A Multi-Instrument, Force-Feedback Keyboard

When playing a musical instrument, a player perceives not only the sound generated, but also the haptic interaction, arising during the contact between player and instrument. Such haptic interaction, based on the sense of touch, involves several senses in the player: tactile, kinesthetic (i.e., mediated by end organs located in muscles, tendons, and joints and stimulated by bodily movements and tensions), proprioceptive (i.e., of, relating to, or being stimuli arising within the organism), etc. By its nature, the haptic interaction is bidirectional, and this is exploited by musical instrument players, who can better correlate their actions on the instrument to the sound generated. For instance, by paying attention to the interaction force between key and finger, arising during the descent of the key, pianists can detect the re-triggering of the escapement mechanisms and, in turn, can adjust the key motion to obtain the fastest repetition of the note.

Roughly speaking, haptic information allows the player to perceive the "state" of the mechanism being manipulated through the key. By using this knowledge about the state of the mechanism and correlating it with the sound generated, the player learns a strategy to obtain desired tones. This tight correspondence between acoustic response and touch response, however, is lost in many electronic instruments (e.g., in standard commercial synthesizers), in which sound generation is related only to the key attack velocity and pressure. In this type of synthetic instrument, the touch feedback is independent of the instrument being simulated. For instance, the interaction with different instruments like harpsichord, piano, or pipe organ gives the same haptic information to the player. This constitutes a significant limitation for the musician, who loses expressive control of the instrument and, in turn, of the generated sound.

This consideration led to several research activities, aimed at the realization of an active keyboard, in which actuators connected to the keys are driven in such a way that the haptic interaction experienced is the same as if the player were interacting with the keyboard of the real instrument being emulated by the synthesizer (Baker 1988; Cadoz, Lisowski, and Florens 1990; Gillespie 1992; Gillespie and Cutkosky 1992; Cadoz, Luciani, and Florens 1993; Gillespie 1994). Such haptic displays are usually referred to as "virtual mechanisms," because they are designed for the reproduction of the touch feedback that a user would experience when interacting with an actual multi-body mechanism. A very simple example of virtual mechanism is the "virtual spring" shown in Figure 1.

Figure 1a shows the actual mechanism, realized by a spring, anchored to a wall on one side and to a plate on the other. Pushing the plate, a force proportional to the displacement x is percevied. In Figure 1b, the virtual mechanism is shown. Here, the spring has been replaced by a linear motor. By sensing the position of the plate and driving the motor with a current proportional to such displacement, the force perceived by the user is again proportional to the displacement, as if the user were pushing the system with the real spring. Following the same principle, a damping mechanism can be simulated by generating a force proportional to the velocity of the plate, while an inertial term can be added by sending to the motor a current proportional to plate acceleration.

This very simple example can be extended to multi-body mechanisms, composed of several parts, which interact with one another in terms of impacts, constraints, etc. In such a case, the motion of each part of the virtual mechanism must be calculated by a dynamic simulator, which incorporates all the characteristics of the real mechanism and computes the interaction forces among the parts. It is worth noting that, at times, an overly detailed description of the real mechanism leads to a bulky dynamic simulator, not suitable for real-time implementation, as is required in haptic interaction. Moreover, it is usually difficult to tune the parameters of the dynamic simulator, especially when the mechanism to be simulated contains several nonlinear components, such as nonlinear dampers or constraints.

Figure 1.

Virtual spring.

Among all the...