SICIB: An Interactive Music Composition System Using Body Movements

Traditionally, music and dance have been complementary arts. However, their integration has not always been entirely satisfactory. In general, a dancer must conform movements to a predefined piece of music, leaving very little room for improvisational creativity. In this article, a system called SICIB--capable of music composition, improvisation, and performance using body movements--is described. SICIB uses data from sensors attached to dancers and "if-then" rules to couple choreographic gestures with music. The article describes the choreographic elements considered by the system (such as position, velocity, acceleration, curvature, and torsion of movements, jumps, etc.), as well as the musical elements that can be affected by them (e.g., intensity, tone, music sequences, etc.) through two different music composition systems: Escamol and Aura. The choreographic information obtained from the sensors, the musical capabilities of the music composition systems, and a simple rule-based coupling mechanism offers good opportunities for interaction between choreographers and composers.

The architecture of SICIB, which allows real-time performance, is also described. SICIB has been used by three different composers and a choreographer with very encouraging results. In particular, the dancer has been involved in music dialogues with live performance musicians. Our experiences with the development of SICIB and our own insights into the relationship that new technologies offer to choreographers and dancers are also discussed.

Background

In 1965, John Cage, David Tudor, and Merce Cunningham collaborated on Variations V, a multimedia work in which dancers triggered sounds each time they were positioned between one of a dozen photoelectric cells and a light activated each cell. This revolutionary work challenged notions of the traditional relationship between choreographer and composer--a relationship characterized by compromise of one artist in order to fit the work of the other.

The Cage-Cunningham collaboration achieved equality by transferring control from both composer and choreographer directly to the dancer. Recent technologies offer more refined tools for mapping dancer's movements to music. Computer processing of data from motion sensors allows both composer and choreographer to retain control and achieve an integrated work that is not improvisatory. These systems provide any degree of control desired by the composer-choreographer team; they can also provide the dancer with the flexibility of performer nuance and inflection usually associated with musicians rather than dancers.

The motion detection technology can be classified according to the location of the sensors and detectors. These include sensors and detectors attached to the body of the dancer. Examples of such devices include piezo-electric and flex sensors (e.g., Van Raalte, 1998). Another type of technology involves sensors and detectors placed external to the dancer's body (cameras, infrared sensors, etc.) (e.g., Rokeby 1999). A third type of motion detection technology includes sensors attached to the body and detectors placed strategically elsewhere. These devices include electromagnetic sensors, sonar, etc. (e.g., Morales-Manzanares and Morales 1997).

As is often the case with integrated technologies, newfound freedom is often paradoxically accompanied by stifling restrictions. Researchers, musicians, composers, choreographers, and dancers are just beginning to grasp the possibilities these new technologies offer. However, there remains a distinct need for a simpler, easier to understand, and powerful coupling mechanism that mediates between sound and motion.

This paper describes Sistema Interactivo de Composición e Improvisación para Bailarines (SICIB) designed with the above ideas in mind and capable of generating music in real time based on body motion. SICIB receives space coordinates from sensors attached to dancers and obtains choreographic information from them. Combinations of such information are used to satisfy conditions of "if-then" rules whose actions affect musical elements. SICIB can use two different music composition systems: Escamol (Morales-Manzanares 1992) and Aura (Dannenberg and Brandt 1996).

Escamol is a language for creating scores using grammatical rules (Allen 1987). Aura is an object-oriented software foundation for...