- Partial Reflections
This paper describes some of our recent work using virtual physical models as a mediating mechanism between a live instrumentalist and computer-generated sound and video.
This work has resulted in the construction of a number of prototype 'virtual sound sculptures' and the composition and performance of a two-movement electro-acoustic work entitled Partial Reflections. It is the result of collaboration between a composer/musician and a technologist/musician, both with significant professional experience as musicians as well as additional expertise in composition and software development, respectively.
The initial goals of the project were very broad. We wanted to create performance works for trombone and computer where computer-generated sounds and visuals were shaped and directed solely by the sound of the instrument. Thus, we wanted the software component of the work to:
• respond in real-time—that is, respond directly and immediately to live audio input
• provide some kind of audiovisual representation of or response to the music
• be suitable for use in a concert setting (and also potentially in practice and/or teaching)
• encourage musical exploration
• facilitate musical expression.
When using any software which transforms the sound of live audio in real time, the composer  is in a sense composing for 'instrument augmented by software'—a hybrid instrument.
From this perspective one could say that our work involves the design of new musical instruments that are controlled by sound. However, following Perry Cook's advice that instrument designers should, "make a piece, not an instrument or controller" , we do not seek to design new general-purpose instruments suitable for use in a broad range of applications. We aim instead to create software/hybrid instruments and music specifically suited to one another. In line with this, we try to avoid situations where a piece of music is composed for pre-existing software or, conversely where software is constructed as a kind of visualization of pre-composed music.
Of course it may be that more general principles for instrument design emerge from this process, but uncovering these principles is not the focus of the project—at this stage anyway.
To summarise, what we produce are not 'simply' new musical instruments, but composed works for trombone augmented by software which has some instrumental characteristics.
Virtual Sound Sculptures
We began to use virtual physical models early in our work because we were attracted to the idea of giving the musician some kind of 'tangible' control over the audio and video generated by the computer. The use of physical models in audio synthesis is an active research area. In our work we do not focus on the use of physical models to directly generate sound. Instead, we use them as mediating structures between the live audio and computer-generated sound and video [3, 4].
In this approach, the software incorporates a physical model which may be thought of as a kind of virtual sculpture. The software designer builds the sculpture by positioning various objects in virtual space and specifying their physical qualities such as mass . These objects may also be linked together with virtual 'springs' of certain lengths, rigidity, etc. Because this sculpture is programmed to apply the laws of Newtonian physics, it responds in ways that appear natural when forces are applied to it. In our case, the forces are mapped to characteristics of the music that is played. So, for example, if the loudness of the input sound is mapped to the quantity of force exerted on the sculpture, then playing a loud note will cause a large amount of force to be applied to the model and, depending on its structure, it may bounce around the screen, change shape and so on. In our work, these movements also cause the computer to output sounds.
To put it simply, the musician's live sound exerts force on the physical model/virtual sculpture and in response it moves in physically plausible ways. Fig. 1 shows a high-level view of how this works. Note that while it does not necessarily have to be the case, in our work the visual output is a direct representation of the physical model itself. The...