New Narratives: Stories and Storytelling in the Digital Age

1 From Synesthesia to Multimedia

19 1 From Synesthesia to Multimedia How to Talk about New Media Narrative daniel punday The title of Marku Eskelinen’s controversial essay in the Electronic Book Review, “Cybertext Theory: What an English Professor Should Know before Trying,” tells us a great deal about the problems of talking about new media narrative. Eskelinen offers the gamelike texts that Espen Aarseth calls cybertext as the other to the work that literature professors and narratologists usually discuss. For him, these games are the object of entrenched disciplines’ “colonializing” gaze: cybertext theory must defend “its objects of study from various colonising enterprises from traditional literary institutions, whenever they’d become desperate enough to try” (Eskelinen 2001). According to this description, cybertexts (such as video games) are like some new species recently discovered; any attempt to impose traditional textual categories from other media (literature primarily , but also film) is by definition a misinterpretation of the form. The challenge of interpreting new media texts becomes one of developing native categories and terms that are fair to this new medium. And yet, we all recognize that traditional and new media narratives are not really independent. When Katherine Hayles calls for a “mediumspecific analysis,” she is careful to explain that she “do[es] not mean to advocate that media should be considered in isolation from one another ” (Hayles 2004b, 69). I am afraid, however, that Eskelinen’s celebration of cybertexts encourages precisely this. Marie-Laure Ryan is surely right when she notes that each “narrative medium” will have a “unique combination of features” (2004, 19), among which are the senses that it stimulates , the way that it uses space and time, the technological and material basis for the work, and the cultural context for producing and disseminating the story. The danger of this flexible and powerful characterization, however, is that the word unique can make us forget the word combination , that our search for what is different about a medium distracts us 20 From Synesthesia to Multimedia from how much that medium shares features with others. In fact, media frequently achieve their identities in part by combining elements of other media that have complementary features, such as the use of music in a film, or newsprint in a collage painting. Recently, there has been a growing interest in this mixture under the category of multimodality, which focuses on the way that communicational structures can invoke different senses (hearing, sight, touch) using different semiotic channels (text, image, audio recording, video). Analysis of sensory modalities has its roots in cognitive science and has been applied most extensively in work on interface design and pedagogy (see, for example, Selfe 2007). One limitation of such cognitive approaches to communication, however, is that they generally strip out cultural and material history to construct their models of the fundamental elements of human perception. Maribeth Back frames multimodal text design in such cognitive terms: “Multisensory reading relies on people’s ability to collate and decipher multiple sensory streams simultaneously, much as we interpret the world around us through the use of multiple senses. This is more than a simple struggle between perceptual sensitivities, however; we use cultural cues and personal perceptual history as criteria to interpret sensory data” (2003, 161). Because of my desire to retain the history that cognitive theory can leave behind, I would like to resurrect a term that passed out of vogue some time ago, multimedia, which emphasizes existing cultural models of the various channels through which we encounter information. Early criticism usually described electronic textuality and digital images as multimedia. Some of the first new media texts that everyday consumers encountered were “multimedia” encyclopedias—published on cd-rom and included with many new computer purchases in the 1990s. Although multimedia has remained a consumer buzzword, the term has fallen out of favor with critics who increasingly emphasize the unique qualities of digital works. The term multimedia has, however, two advantages that I want to emphasize in this essay. First, the multimedia work quite clearly combines several forms of already-existing media. In the popular consumer imagination, this usually means that multimedia products add additional layers of content onto a traditional media form. Thus, a book may come with a cd-rom of music, or a dvd may come with production stills and on-screen games. Critics eager to see new media as an independent aesthetic form bristle at the implication that it has been Daniel Punday 21...