Resistance Is Fertile: Gesture and Agency in the Field of Responsive Media

The force that through the green fuse drives the flower
Drives my green age; that blasts the roots of trees
Is my destroyer.
And I am dumb to tell the crooked rose
My youth is bent by the same wintry fever.

The force that drives the water through the rocks
Drives my red blood; that dries the mouthing streams
Turns mine to wax.
And I am dumb to mouth unto my veins
How at the mountain spring the same mouth sucks.

The hand that whirls the water in the pool
Stirs the quicksand; that ropes the blowing wind
Hauls my shroud sail.
And I am dumb to tell the hanging man
How of my clay is made the hangman's lime. ¹

Introduction

Accounts of language, languaging, and the construction of knowledge have turned on logic, on semiotics, on information and cybernetics. Can we appeal instead to notions of embodiment and materiality that do not reduce to these other categories? Scholars such as Katherine Hayles and Mark Hansen have recently emphasized the need to rethink embodiment and materiality in an era saturated with digital media, ² but attempts to reinterpret such notions have tended to slip back into the semiotic or informatic categories they try to exceed. Questions of gesture and agency in the presence of emerging technologies of performance have cast such notions in sharper relief. Given these concerns, it seems reasonable to expect that an account of gesture and agency informed by some direct experience with sensor technology, real-time systems, and computational media as applied to experimental performance could offer more insight to this inquiry. Moreover, since formal accounts of agency have tended to be framed within the categories of the cognitive, if not the linguistic, it seems that experimental projects with music, time-based visual art, fabric art, and theater would constitute fertile ground for phenomenological study.

Motivated by present concerns with embodiment and materiality, I pose the question: How do our notions of gesture and agency mutate in the presence of real-time, dynamically varying computational media? I refract this question through the experiences of building and playing in responsive spaces such as TGarden's TG2001, which was presented as a public experiment at Ars Electronica in Linz, Austria, and at V2 in Rotterdam. ³ What I bring into the conversation is a study of interaction and digital media, and some years of experience with building simulations, visualizations of differential geometric processes, and most recently with responsive media spaces. My claim is that such responsive media spaces, both in their construction and in the experiences that they sustain, call into question linguistic and informatic models of gesture and open new ways to understand gesture and agency as embodied, a-linguistic experience. Part of the strategy has been to materialize the argument in the same responsive spaces that we study, and to reflect upon their design and their performance. Consequent to this approach I suggest a way to understand gesture, agency, and free play, and consider in what manner we can constitute material embodiment.

In this essay I first describe the TGarden environments as the prototypical examples of responsive media spaces, some gestural features of such environments, and the qualities of gesture on which I would like to focus, explaining why one may profitably defer resorting to linguistic categories. I then describe the materialization of gesture in physical movement and computationally mediated response, drawing for concreteness on the particular emergent technologies of computational media: real-time digital video and sound. And finally, based on this richer notion of open and multiply completed topological gesture, I draw a set of implications regarding agency in the presence of responsive media.

TGarden Responsive Media Spaces

A TGarden is a responsive media environment, a room in which people can shape projected sound and video as they move. Upon entering a TGarden space, each visitor—called a player&#8212...