Synesthesia and Intersenses
Intermedia in Electronic Images
The essay focuses on the processes of intermedia in visual media. The author's analysis of the merging of still, moving and computed images reveals that, in intermedia, images tend toward a spatial, rather than temporal, organizing principle. This shift becomes evident in particular in the moving images in such electronic films as Peter Greenaway's Prospero's Books. The limits of intermedia in the electronic medium are unfolded in the concept of the coherent image by Zbigniew Rybczynski. Another concept of compression and convergence is demonstrated by Clea T. Waite in her crossing two- and three-dimensionality in a video installation.
Taking as a starting point the contemporary debates of intermedia describing the phenomena of crossing the borders between traditional media (such as painting and photography) and contemporary media (such as cinema, television, video, computer and other hypermedia), my essay opens the question of how we may consider the shift in the arts that is caused by new technologies. What I am interested in is to show how the interrelationships between analog and digital media result in new forms of media arts. For instance, when different types of images are correlated and merged with each other on the borders of painting, photography, film, video and computer animation, the interrelationships of the distinct elements cause a shift in the notion of the whole image. In particular I discuss the processes of transformation effected through convergence of elements of different media. The characteristics of intermedia may be identified in certain forms of the image, when elements of the static and the moving image are interrelated to create a third form of the image. For example, in the domain of cinema an intermedia state of the image occurs when filmic processes of imaging based upon the interval are used in combination with digital tools (such as the graphic paint box) and elements of hypermedia (multi-dimensional and omni-directional, namely multiple layers). The resultant mixed form of the image may reveal the continuities and similarities and also the discontinuities and changes between analog and digital images, in particular when both are combined in the electronic collage.
Such a shift is made visible, for example, in Peter Greenaway's electronic film Prospero's Books (1991), which merges analog and digital media in a mixed form--the result is a category of media arts known as intermedia, in which elements of differing media images are combined and transformed to create a new form of image. Greenaway produces a new type of image through the simulation of movement when he electronically animates single frames of phase photography. The result is not the moving image generally known in film; rather the re-animation of the static images results in a paradox: the electronic moving image here presents a static and a moving image at the same time. The paradoxical structure means that the interval that separates and connects the different phases of the photography is made visible within a single frame unit--and that is the new quality--while at the same time the continuity of filmic movement is also represented. Mobile and immobile parts of the image are interrelated within one single frame that is digitally processed with the use of a graphic paint box. [End Page 55]
In Prospero's Books Greenaway uses the formal device of a frame-within-a-frame structure to visually express and unfold the contradictory dynamics and directions of a paradoxical image form. These paradoxes in the moving image can only be shown in the electronic simulation of photographic (still) and cinematic (dynamic) images. For example, when the image is divided into different frames, different types of movement are shown within these frames so that each single section has its own specific movement structure. On the surface of an electronically held simulation of freeze frame, the inter-frame movement of any figuration seems to run endlessly in the same location on the film frame without really moving. The video still unfolds the simulation of transfiguration, which can, but does not have to, correspond to natural types of motion. In contrast to the dissected segments in phase photography, the computer-animated images are speeded up to a high degree, whereby speed represents a variable dimension in the composition. The digital re-animation of a freeze frame from phase photography using the paint box cannot be compared to serial photography (e.g. Muybridge, Marey) (Color Plate A No. 1).
The two levels of the mixed image are divided by the use of immobile framing, where the frame-within-a-frame device functions as a digital collage: filmic figuration is represented in the outer frame, and computed simulation of figuration in the inner frame. While the outer frame shows the conventional cinematic moving image that expresses the linear narration of the fiction film, the embedded inner frame is reserved for processed, transformed images that result from the manipulation of previous forms of the image (such as photographic or filmic images). Within these immobile inner frames, the computer animations may produce the effect of continuously moving animals, each within its own image plane, or of a geometric drawing turning around its own axis, thereby engendering three dimensions. The key images of animals or human figures in motion seem to produce continuous movement, but insofar as these animated figures are keyed within a static form represented by the immobile inner frame, the structure of the inner movement collides with the movement represented by the outer framed section--that is to say, the entire frame unit. The effect that occurs is that each animal set in motion through electronic simulation seems to move on its own image plane within the inner frame. At the same time, the outer part of the image maintains the linear continuity of the film.
This paradox of a static and dynamic image results from computer animation insofar as the velocity of the images is varied within one single frame unit. Through the use of digital media, the correlation between static and dynamic images, between photographic and filmic images, can demonstrate how still images can be transformed. This transformation of different types of images, which causes a shift in the organization of the whole image, I will call "intermedia."
It is important to emphasize that intermedia is a formal category that defines an interrelationship between or among distinct media that merge with each other, such as a photographic still image reworked in a film or video. The concept of intermedia, however, differs from mixed media and multimedia (the synchronous occurrence of different art forms within the frame of one integral medium such as theater, opera, film performance), and it is historically linked to the context of intertextuality and intermedia that emerged in print and visual forms of early twentieth-century avant-garde art. In his research on the correlation of word and image in Russian modernism, Aage A. Hansen-Löve discusses the differences between the concepts of intertextuality and intermedia in relation to a system of different art forms, saying that the intertextual indicates a shift within a single art form . Intertextuality addresses a text-text or film-film relationship and should be distinguished from [End Page 56] intermedia, which relies on the interrelationship between two different art forms that have developed separately but through the correlation effects a type of transformation that involves the historically different development of media as well.
First, intermedia differs then from multimedia, which correlates
different media that are presented together synchronously yet remain
distinct. Second, intermedia goes beyond mixed media, which incorporates
elements of one medium in another (e.g. photography in film, painting
in photography.) What is essential to intermedia and intertextuality as
well is the category of transformation. But where intertextuality
expresses a text-text relationship, intermedia means that the reference frame of the entire system of art forms that mediates the intermedial correlation is itself included in the processes of transformation. In relation to visual media, then, this definition of intermedia inherently implies that the processes of transformation are reflected in the form of the images, because it is through the modes of self-reflection that the structural shifts characteristic of new media images are mediated and made visible. The point is that the transformation of elements of at least two (historically) different media creates a new form of image that reveals these differences in a mixed form and mostly reveals the self-reflexivity of the medium in a paradoxical structure.
Film images that are reworked or manipulated with video and computer technologies indicate a visible shift in the cinematic image only insofar as the photographic and cinematic and their related technologies of image design are literally confronted with digitally processed types of moving images. Peter Greenaway has produced just such a striking example of intermedia aesthetics in Prospero's Books, by reworking the interval that relates to photography and film on the digital level to create an image cluster that relates the static and the moving images to each other. The aesthetic function of the interval is important for any comparison of different visual media, i.e. film, video and digital, insofar as it represents a shift within temporal categories. However, whereas the function of the interval has been used in the history of film to express a specific temporal feature of the cinematic image--that is, simultaneously to stress temporal difference and to mediate continuity between two juxtaposed elements--in electronic film the interval may be transformed into a different type of image that is no longer film. When the interval is reworked through electronic processing to create the simulation of a still image that effaces the gap, the resultant type of image is a cluster.
By the cluster, I mean a type of image that is widely used in electronic media through the multiple layerings of different images or image elements, resulting in a spatial density. A cluster, more precisely a visual cluster, implies a simultaneity of different layers within one single image unit; as a term, it can be applied to both film and video, but in particular to electronic images that tend toward the point of absolute density within a frame. What is crucial is the fact that a cluster does not take place as a series but as a spatialization of the moving image. With the cluster and the interval, I introduce two categories of intermedial relations. In this instance, the interval accounts for the temporal notion of the image, whereas the cluster relates to the spatial organization of the image. Thus, the structural elements that are specific to different media, such as the interval and the cluster, are correlated in Greenaway's Prospero's Books.
I introduce the term "electronic film" to describe artworks that basically rely on moving images but also integrate features of painting, photography (static image), film (dynamic image) and the computed simulation of the two (digital image). The correlation and the merging of images that take place in the electronic [End Page 57] simulation, where simulation means an aesthetic operation on the surface of the image, help to create an intermedia design visible on the surface of the image. This is different than in film, where the structure of film may be distorted by the interval such that the linear organization and the image velocity are manipulated. The electronically simulated image effects the cinematographic moving image such that particles of the image are accelerated or slowed down; the gradual succession of images will be stopped altogether in the electronic still image. Temporal features may collapse into spatial, and vice versa. Finally, the spatial features of the electronic film tend to become an element in the merging of film and computer in a new digital cinema where the linear structure of film is disrupted by multiple layers.
Through the dominant use of clusters in electronic film to replace the function of the interval, Greenaway in Prospero's Books aesthetically explains the shift from the temporal to the spatial organization of the image. Self-reflection is found in the visibility of the heterogeneous elements that are put together to build a cluster. Greenaway is more interested in showing the incoherence of the clusters than in producing coherence through the use of matte effects or further layers. Thus, self-reflection is a means to make the twofold function of visual clusters evident: on the one hand, the cluster produces spatial density through coherence and fusion; on the other hand, a cluster may stress and mediate the discontinuity between layers. Greenaway compares the constructing principles of the interval and the cluster at the level of intermedia, in that he simultaneously deals with forms of the static and dynamic image in relationship to the digital (computerized) image. The result of this correlation is a structural comparison of different media that is effected by the collision of different types of movement and velocity within the unit of one single image. This collision of different media images in the cluster I will call a "hyperdynamic image position."
The hyperdynamic image position represents a point of collision in which the structural differences between the two types of images, the static and dynamic as they have emerged in analog media, are confronted with digitally processed images at the level of electronic representation. The paradoxical structure of the collision becomes visible in the mixed form of the image that reinforces the processes of transformation. The temporal features of film as represented through the concept of the interval are thereby turned into image forms of spatial density and simultaneity. The hyperdynamic image position expresses an intermediate step between different visual media, where the moving and the still image collide in the (spatial) density of a cluster. Finally, it declares on the electronic level, which here is the intermedia level of comparison, that the distinction between the media is an essential issue in intermedia. Only that which is distinct and separate can merge.
However there are limitations to this as a conceptual definition of intermedia. In [End Page 58] his HDTV (high-density TV) videofilm Kafka (1992), Zbigniew Rybczynski develops the concept of a coherent image and exposes the shift from analog to digital when he performs processes of multiple layerings that reveal on the representational level that the distinction between media is not relevant. Rybczynksi uses the device of motion simulation to create the effect of a coherent image that encompasses up to 70 layers within one single unit. With the use of a camera motion control system that had been specially developed for this film, Rybczynski makes the possibilities for temporal and spatial camera movements infinite (Fig. 1). The motion control system navigates the simulation of the camera movements to create complex composite shots (Figs 2, 3):
A character in Kafka may appear in several places within the same shot, or may walk away from a character in one room only to encounter himself playing a different character in another room--all within the same takes. . . . Like pieces of a puzzle, the actors and the set's floors, ceilings and walls are photographed separately and then matted together, creating composite images made of up to 70 layers. . . . Because the actors are moving on rotating turntables, they are not bound to the physical limitations of the studio and can go anywhere. The same is true of the camera .
The composite images are built into moving clusters that hide their layers and edit points in order to produce "a seamless, smooth movement into the next location" . Such means of creating layers results in high spatial density and deep focus of an image type that rather reassembles processes of digital images but performed at the analog level. On the edge of digital media, Rybczynski further develops the means of analog media in order to achieve similar effects. The coherent image does not reveal its construction principles. The notion of this image is not self-reflection; on the contrary, it represents the limits of the intermedia image because the differences between the different elements are no longer made visible.
Thus, the coherent image of Rybczynski and the hyperdynamic image position of Greenaway exemplify two different conceptualizations for relating elements of different media to each other. The coherent image makes invisible the techniques of merging and layering. In opposition, Greenaway's incoherent image is categorized as intermedial essentially based on the differences that can be recognized. The differences are expressed self-reflexively in the form and shape of the artwork. Transformation here has a twofold meaning: one concerns the dialogue between distinct elements that merge into each other; the other one is the collision of separate elements.
Collision results from elements of moving images and of static images that are contrasted to each other in the form of another medium, for example, in a computed image. In collision, the different media elements do not entirely lose their attachment to their old media: they are still recognizable as, for example, photography and film brought together. At the representational level, the collision is performed within a structure of exchange that makes clear the combined coherent and incoherent aspects of both elements. Exchange means another structural category, namely the entire setting of interrelationships where the visible interval in film can reveal its photographic quality, or where the concept of a series in painting may evidently relate to film, and vice versa. In exchange, the media-specific types of images would be reworked in other media at the level of form. The transformation is the new visible form that results from collision and exchange, for example the hyperdynamic image position itself. Transformation thus becomes a structural category that expresses the ways that those different elements are connected and merged into each other, thereby creating a new form.
The form of an intermedia artwork is thus defined then not only by collision but also by the exchange and transformation of elements that come from different media, such as painting, photography, film, video and other electronic media. Intermedia therefore is a formal category of exchange. It signifies an aesthetic encompassment of both form and content. In an intermedia work of art, content becomes a formal category that reveals the structure of combination and collision. The related meaning of content is to express such modes of transformation that are effected by the collision of painting and film, of film and electronic media, and so on. The contextual meaning of intermedia is to reveal the media forms themselves. The making visible of elements that are considered media specific can be performed by ways of comparing and transforming elements such as the interval.
Intermedia is a concept of merging based on historically separate developments. In the case of digital media, various types of media are integrated--that is, they merge with each other within the same technical structure. As German computer scientist Wolfgang Coy says, "All written, optical, and electric media with the use of microelectronics and [End Page 59] computer techniques finally will merge into one universal digital medium" . This universal medium is often named "hypermedia," thus signifying a multidimensional structure. The computer is a building block for creating new hypermedia. Hypermedia, the term introduced by Ted Nelson  to describe media that perform "multidimensional ways" of branching, allow one to move in a non-linear way through information. The point is how to access different media; the distinction between media is not the issue in hypermedia. Multidimensional connectivity and interactivity, which are associated with hypermedia, do not rely on the same transformation category that is essentially the concern in intermedia. The non-sequential structure in the first place indicates an option to connect each single digital medium to another one. Networking in hypermedia differs from that in intermedia, in which the connection of different media always involves a transformation.
As said, intermedia differs not only from hypermedia but also from multimedia and mixed media. Both multimedia and mixed media are comparable to hypermedia insofar as they describe the expansion of a single medium in terms of accumulation rather than of transformation. In his essay on intermedia, Dick Higgins describes the different concepts: "Intermedia differ from mixed media; an opera is a mixed medium, inasmuch as we know what is the music, what is the text, and what is the mise-en-scène. In an intermedium, on the other hand, there is a conceptual fusion" . For example, multimedia and mixed media art forms, in particular avant-garde art, can be identified within happenings, Fluxus performances, or related events that combine live art and film. All these connect and combine different art forms on a level that does not necessarily involve changing the structure of each single medium .
The consequence is that intermedia in visual culture is best established by modes of self-reflection. Self-reflection in this instance is a medium-specific strategy that is used to link formal aspects of different visual media, such as painting, film and electronic media. In particular, those forms of an image that have occurred in one single medium undergo a process of remodeling and reshaping when they are transferred into the context of another visual medium. Transferring results in transformation when the structural elements of both media are made evident and visible in a form that reveals their differences. Because self-reflection reveals simultaneously those elements of incoherence and those of coherence, it seems to be an appropriate device to expose the specific interrelationship of two different media. Thus, self-reflection is the most striking device to make clear the twofold structure of transformation, by revealing the ways that different media can be connected in one form. In visual culture, an intermediate state of art occurs when the forms of different media collide in another form effected by transformation.
In the 3D video-installation Kur (1997), Clea T. Waite disassembles the twofold structure of intermedia. The installation consists of four screens, each large enough to represent human movement on a life-size scale. The screens are spatially arranged to form a square. The viewer wears 3D glasses and enters into the middle of the square, where it is impossible to see all the moving images on the four projection screens simultaneously. This is crucial to the concept of the installation, since the four screens present four different versions of the same story about a Sumerian myth. Therefore, the viewer of the installation necessarily has to select which of the different versions he/she wants to see (Fig. 4).
Kur represents parallel time in a three-dimensional simulation of space that reveals the concept of time in space and simulates how different times may exist parallel to each other. The work simultaneously merges different moments in time into one single image and presents the presence of parallel time on four screens. The duality inherent in this concept is expressed by specific image devices: (1) the electronic flicker is used to represent simultaneously two different times; (2) digital feedback makes visible the time trail and produces an aesthetic effect that creates a spatial sculpture that makes visible different moments in time (Fig. 5); (3) particle effects result from the reworking of density and show the distortion and the re-creation of images (Fig. 6). By compressing and decompressing space in time, this process exemplifies a transformation between different forms of images. Kur shows time images that merge into space and spatial images that merge into time. What is important is that the differences and distortions are made visible within the image itself (Fig 4).
Waite's video installation uses digital video to show the emergence of forms of media self-reflection, where the possibilities of a two-dimensional image simulation are expanded onto the level of three-dimensionality. Waite's simulation [End Page 60] image and the one developed by Rybczynski both indicate an expansion in electronic features of the image. They are both concerned with the spatial and temporal dimensions of the image. In particular, they stress the concept of space. The hyperdynamic image position adopted by Greenaway acts as the turning point of time and space that is visible within the paradoxical structure of the image itself. These examples show how media-specific devices change their form and function in an intermedia form of the image so that the temporal concept of interval and the linear structure of montage are opposed by features with reversed functions. In these examples, it is mainly the devices of clusters and multiple layerings that effect a conceptual turn in linear montage when collage elements of spatial density are integrated.
The shift from the temporal to the spatial thus occurs on the level of image design and does not indicate a change of paradigm. It signifies an aesthetic tendency that becomes evident at the structural level when the use of certain aesthetic devices, such as the cluster in electronic images, causes a shift in preference from the temporal to the spatial organization of the image. To avoid any misunderstanding at the level of discourse, I want to make it clear that the phenomena I have discussed as causing a shift from temporalization to spatialization in the organization of visual elements should be discussed and valued in aesthetic terms. The argument that new media images, in particular electronic and computed images, indicate a shift from the concern with temporal features to a preference for spatial features should not be mistaken as a statement claiming a change of paradigm. On the contrary, with regard to Niklas Luhmann's definition that what signifies a change of paradigm requires stating a change in media , we are not dealing with an essential difference in media, but with a categorical difference in aesthetics. What is different here is rather a shift in the notion of the image, mainly effected by the use of electronic devices.
In summary, intermedia is a conceptual term. It should be applied in the first place to a specific type of transformation between different media. Secondly, the term intermedia indicates the structure of a transformation that is effected by a collision of elements taken from different media, and it is an identifiable aesthetic device in the media arts. The term intermedia is useful then on three levels: as a transformation category, as a structural term, and as an aesthetic device.
The discourse on intermedia encompasses an aesthetic practice in media art, the structure of cultural and artistic processes, and a technological metaphor, whereas the distinction from multimedia or hypertext is not so clear. Conceptually the meanings of "inter," "multi" and "hyper" are not coherent, and the incoherence of their naming points to a larger problem that lies within the phenomenon itself. The descriptions of all these phenomena shift, depending on the discourse. Similar phenomena may be described with different terms; different aesthetic practices are subsumed under the rubric of intermedia. Within media studies, for example, intermedia is a conceptual term that stands for processes that integrate structural elements that are specific to different media. But it is important to remember that the meaning of the concept is twofold: on the one hand, it signifies a technical device and, on the other hand, it refers to the technological dimension of both cultural and media processes. What is most basic to the processes of intermedia involves an activity of transformation and not of accumulation, and what is at stake is a new notion of the image that results when analog and digital media are correlated in a mixed form.
I would like to thank Peter Greenaway, Clea T. Waite and Zbigniew Rybczynski for their kind permission to publish illustrations of their works.
Yvonne Spielmann is a teacher of Media Studies at Cornell University. She is the author of numerous books and essays on experimentation and the avant-garde; history and theory of visual media; aesthetic theory in the twentieth century; and media theories, intermedia and visual culture. She is currently writing a book on video.
Yvonne Spielmann (educator), Dept. of Media Studies, Cornell University, Society for the Humanities, 27 East Avenue, Ithaca, NY 14853-1101, U.S.A. E-mail: <email@example.com>.
An earlier version of the essay was published in the journal Iris, No. 25 (1998).
References and Notes
1. Aage A. Hansen-Löve, "Intermedialität und Intertextualität. Problem der korrelation von Wort- und Bildkunst--Am Beispiel der russischen Modern," in Dialog der Texte, W. Schmid and W.-D. Stempel, eds. (Vienna: Wiener Slawistischer Almanach, 1983), Vol. 11.
2. Marsha Moore, "Motion Control. Hi-Def Kafka: Metamorphosis via 'Motion Simulation,'" American Cinematographer 2 (1993) p. 65.
3. Moore .
4. Wolfgang Coy, "Die Turing-Galaxis Computer als Medien," in Weltbilder, Bildwelten, Interface 2, K.P. Dencker, ed. (Hamburg: Hans-Bredow-Institut, 1995) p. 53 (translated from the German).
5. Theodor H. Nelson, "Hypermedia," in Theodor H. Nelson, Computer Lib/Dream Machines (Redmond, WA: Tempus Books of Microsoft Press, 1987) p. 64: "Essentially, today's system for presenting pictures, texts and whatnot can bring you different things automatically depending on what you do. Selection of this type is generally called branching. (I have suggested the generic term hypermedia for presentational media which perform in this (and other) multidimensional ways.)"
6. Dick Higgins, Horizons, the Poetics and Theory of the Intermedia (Carbondale and Edwardsville, IL: Southern Illinois Univ. Press, 1984) p. 16.
7. For example, regarding film as a medium, intermedia aspects can be traced in film's early phase, insofar as film brought together discrete elements from literature, music, dance and theater. But the medium of film is not per se intermedia.
8. Niklas Luhmann, "Paradigmawechsel in der Systemtheorie," in Epochenschwelle und Epochenbewußtsein (Poetik und Hermeneutik, Vol. 12), R. Herzog and R. Koselleck, eds. (Munich: Wilhelm Fink Verlag, 1987) pp. 305-322.
frame--the terms "frame" and "framing" are commonly used in cinema studies to encompass the point of view, the length of the take and the mobility or immobility of the camera. Therefore one can distinguish between immobile framing and mobile framing. For reference see David Bordwell and Kristin Thompson, Film Art. An Introduction, 4th Ed. (New York: McGraw Hill, 1993).
interval--standard technical term in cinema studies defining the distance (temporal and spatial) between single frames: the interval has the function to connect and to separate spatially and temporally. Furthermore, interval is a feature of montage that interrelates different elements into a form that reveals their differences. The interval in the history of film is defined by Dziga Vertov in his theories of the interval montage. The most striking film to reveal the interval montage is Dziga Vertov's The Man with the Moving Camera (1929).
phase photography--to take photographs of single moments of a movement, thereby dividing a continuous movement of a figure into single frames, images, "shots."
self-reflection--term used to describe a phenomenon such as the reflection of a figure onto a surface.
serial photography--a series of photographs taken shortly one after the other and showing only minimal differences between the previous and the next image. Serial photography is a common feature in cartography, where it is used to map a landscape by taking pictures in series from the air.