Interactive Visual Music: A Personal Perspective

Interactive performance is one of the most innovative ways computers can be used in music, and it leads to new ways of thinking about music composition and performance. Interactive performance also poses many technical challenges, resulting in new languages and special hardware including sensors, synthesis methods, and software techniques. As if there are not already enough problems to tackle, many composers and artists have explored the combination of computer animation and music within interactive performances. In this article, I describe my own work in this area, dating from around 1987, including discussions of artistic and technical challenges as they have evolved. I also describe the Aura system, which I now use to create interactive music, animation, and video.

My main interest in composing and developing systems is to approach music and video as expressions of the same underlying musical "deep structure." Thus, images are not an interpretation or accompaniment to audio but rather an integral part of the music and the listener/viewer experience. From a systems perspective, I have always felt that timing and responsiveness are critical if images and music are to work together. I have focused on software organizations that afford precise timing and flexible interaction, sometimes at the expense of rich imagery or more convenient tools for image-making.

"Interaction" is commonly accepted (usually without much thought) as a desirable property of computer systems, so the term has become overused and vague. A bit of explanation is in order. Interaction is a two-way flow of information between live performer(s) and computer. In my work, I see improvisation as a way to use the talents of the performer to the fullest. Once a performer is given the freedom to improvise, however, it is difficult for a composer to retain much control. I resolve this problem by composing interaction. In other words, rather than writing notes for the performer, I design musical situations by generating sounds and images that encourage improvising performers to make certain musical choices. In this way, I feel that I can achieve compositional control over both large-scale structure and fine-grain musical texture. At the same time, performers are empowered to augment the composed interactive framework with their own ideas, musical personality, and sounds.

Origins

In the earliest years of interactive music performance, a variety of hardware platforms based on minicomputers and microprocessors were available. Most systems were expensive, and almost nothing was available off-the-shelf. However, around 1984, things changed dramatically: one could purchase an IBM PC, a MIDI synthesizer, and a MIDI interface, allowing interactive computer music using affordable and portable equipment. I developed much of the CMU MIDI Toolkit software (Dannenberg 1986) in late 1984, which formed the basis for interactive music pieces by many composers. By 1985, I was working with Cherry Lane Technologies on a computer accompaniment product for the soon-to-be-announced Amiga computer from Commodore. Cherry Lane carried a line of pitch-to-MIDI converters from IVL Technologies, so I was soon in the possession of very portable machines that could take in data from my trumpet as MIDI, process the MIDI data, and control synthesizers.

Unlike the PC, the Amiga had a built-in color graphics system with hardware acceleration. At first, I used the Amiga graphics to visualize the internal state of an interactive music piece that was originally written for the PC. It was a small step to use the graphics routines to create abstract shapes in response to my trumpet playing. My first experiment was a sort of music notation where performed notes were displayed in real-time as growing squares positioned according to pitch and starting time. Because time wrapped around on the horizontal axis, squares in different colors would pile up on top of other squares, or very long notes would fill large portions of the screen, effectively erasing what was there before. This was all very simple, but I became hooked on the idea of generating music and images together. This would be too much for a performer alone, but with a computer helping out, new things became possible.

Lines, Polygons, Screens, and...