Drawing with the Hand in Free Space: Creating 3D Shapes with Gesture in a Semi-Immersive Environment

Abstract

This article presents a new medium in which organic surfaces are drawn in 3D space with the hand. Special interface hardware includes a head-tracked stereoscopic display and sensors that track the body and handheld tools, allowing the artist to share the space of the artwork. Additional tools move and deform the shape. This method provides a fluid, unstructured access to three dimensions, ideal for quick, spontaneous ideation and investigation of complex structures.

The gesture has a spontaneity, a freedom, an unfiltered physicality in its instantaneous choice. There is a depth of communication in this moment—the split second of a photograph, the subtle timing of a comedian. These instants are not planned or contrived but quickly communicated through a developed intuition. Mark-based traditional media, such as drawing and painting, engage this type of moment repeatedly in a form that engages the body. I view drawing, especially sketching, as a way of physically capturing a form of thought [1]. Yet lines and the paper they occupy are two-dimensional; they do not capture the bodily space. My experience with three-dimensional physical media has been arduous: much planning for small moments of interaction and response. Computer modeling programs [2] require an indirect manipulation of form through mathematical quantities, typically with a 2D interface. I find that these laborious processes do not support the spontaneity of creation and the physical, corporeal understanding that I seek. Without this immediacy, I feel the ability to freely explore

Fig. 1.

A thin strip of surface is drawn with the hand. Special viewing hardware makes the stroke appear to float above the table, as depicted in this composite image. (Photo: Vanessa Stump)

and to thoroughly engage 3D space lacking.

Fig. 2.

Basic methods of developing surfaces: (a) The path of the hand in space is captured as a stroke. A sensor on the lower index finger detects the thumb's closing, which initiates the drawing. (b) Sensor-equipped tongs are used to rotate the figure. Two pairs of tongs used together can be used to scale an object. (c) An ergonomic 3D push-button eraser removes a small volume from an object. (d) A magnet held between the fingers attracts a surface. (Photos: Vanessa Stump)

The medium presented here, which I call surface drawing [3], has developed in response to these concerns. This method is an extension of line drawing to 3D space, using the hand in place of a pen. As the hand is moved through space, its path takes form and hovers in the air as a surface. This concept is realized with advanced computer-interface devices and custom software. Each hand motion is sensed by a specially equipped glove, recorded by the computer and displayed as a coherent stroke. An accumulation of these strokes forms an object, in much the same way that 2D lines combine. The action of creating with the hand is somewhat like touching an imaginary object and having it materialize. This method of creation is a recording of gesture, capturing a performative body as object. The relationship between artist and object is two-way, with the object enveloping the artist, affecting the growth of form.

This method does much to provide the relationship to 3D space that I seek. I have pursued this approach both as an artist investigating visual space and as a computer scientist (in collaboration with Peter Schröder and Michael Pruett at California Institute of Technology [Caltech]) developing a system to support this interaction. I am concerned with making a general-purpose system for the artistic community at large. To this end, I first investigated the system's ability to make representational surface drawings. More recently I have created abstract shapes that develop structural relationships unique to this medium. In both of these investigations my focus has been more on geometry than on color. This is due to my emphasis on structure and my anticipation of future work in texture and shading.

The Physical Interface...