Synthetic Senses

Based on a paper presented at ISEA 2002, 11th International Symposium on Electronic Art, Nagoya, Japan, 2-31 October 2002.

Our experience of the world through the normal complement of human senses is one of great richness and density; yet it is not comprehensive. We are sensitive to a minor fraction of the available spectra. Studies of animals have shown them to possess a considerable range of "exotic" sensory capabilities [1]. Health concerns about various forms of radiation, magnetic and electric fields and certain chemicals indicate that there are additional aspects of the environment that it may be in our interest to apprehend. But beyond utilitarian functionality, it is the potential for perceiving a richer world that drives our research into new sense modalities at the Human Interface Laboratory [2] of the Rensselaer Polytechnic Institute.

Over the last four decades, Paul Bach-y-rita and his colleagues have developed sensory substitution devices that allow the blind to "see" through the surface of the skin [3] by means of electro-tactile transducers. For the blind subject, these devices are essentially a new sense modality. We suggest that, in a similar way, we can create novel sense modalities by using sensor technologies that do not duplicate human senses but respond instead to things that our senses cannot perceive—magnetic fields, for example.

Magnetic fields were chosen for our initial projects because they can occur at the scale of either an object or an immersive environment, from the perspective of the human body. Volitional movements giving rise to the perception of magnetic fields would therefore differ for each of these scales. Perceptual modes arise from the specific sensorimotor contingencies that govern them [4]. In the case of the earth's magnetic field, the contingency is orientation, as the body is small compared with the earth. The relevant movements are locomotive. For object-scale magnetic fields, such movements are primarily exploratory, involving the arm, hand and fingers. Two devices have been fabricated in our laboratory to test these hypotheses.

Fig. 1.

Magnetic field perception devices: compass belt (left) and pen probe (right).

Photo © Ted Krueger

In one project, a magnetic sensor was built into a pen-like probe [5] and its output coupled to a tactile transducer located on a subject's sternum (Fig. 1). A complex grouping of permanent magnets was used to test the ability of the subject to perceive a complex magnetic field as a spatial phenomenon rather than a vibratory sensation on the skin. With sufficient experience, the subject experienced a transformation of the focal awareness, and the magnetic field was perceived in space. This is similar to the way in which the sensation of a tool in the hand becomes transparent and the contact of the tool with a surface enters one's awareness.

In the second project, an automotive compass capable of sensing eight directions was interfaced with pager vibrators contained in an elastic belt [6] (Fig. 1, left). Its purpose was to allow the subject to perceive magnetic north as an aid to navigation. That is, a person wearing the belt would feel a vibration in a position corresponding to north. While effective in this task, the belt also yielded the sensory experience of large-scale magnetic fields surrounding the electric motors of a commuter train and the flux of current in the power lines that fed them. These unanticipated experiences are far more interesting than the initial goal of the project and illustrate the point that there are additional dimensions of the world available for perception if the appropriate technology can be built.

The goal of this research trajectory is to formulate the principles by which arbitrarily chosen sensor technologies might be interfaced to the body to yield a veridical perception of the phenomena in question.