Exploring narrative gestures on digital surfaces

A significant amount of research on digital tables has traditionally investigated the use of hands and fingers to control 2D and 3D artifacts, has even investigated people\’s expectations when interacting with these devices. However, people often use their hands and body to communicate and express ideas to others. In this work, we explore narrative gestures on a digital table for the purpose of telling stories. We present the results of an observational study of people illustrating stories on a digital table with virtual figurines, and in both a physical sandbox and water with physical figurines. Our results show that the narrative gestures people use to tell stories with objects are highly varied and, in some cases, fundamentally different from the gestures designers and researchers have suggested for controlling digital content. In contrast to smooth, pre-determined drags for movement and rotation, people use jiggling, repeated lifting, and bimanual actions to express rich, simultaneous, and independent actions by multiple characters in a story. Based on these results, we suggest that future storytelling designs consider the importance of touch actions for narration, in-place manipulations, the (possibly non-linear) path of a drag, allowing expression through manipulations, and two-handed simultaneous manipulation of multiple objects.