Handling Digital Brains: A Laboratory Study of Multimodal Semiotic Interaction in the Age of Computers

2. fMRI Brain Visuals as Fields for Interaction

2 fMRI Brain Visuals as Fields for Interaction fMRI brain visuals are signs in a very straightforward sense: Cognitive neuroscientists observe the human brain and its processes by consulting its fMRI renderings. Alan Gross (2008: 281) has suggested that the character of fMRI brain visuals should be understood in terms of indexical signs. In his proposal, Gross refers to Peirce’s1 famous distinction between icon, index, and symbol,2 articulated with respect to the relationship between the sign and its object (Peirce, C.P.: 4.531). For Peirce, whereas symbol, most closely related to the Saussurian language-like sign, is a conventional sign denoting its object with respect to a rule, iconic and indexical signs are characterized by their materiality and embodiment. Index is the sign that is physically or causally connected to its referent and thus always bound to specific circumstances of its instantiation. Examples are a pointing finger and footprints in the sand. Icon, on the other hand, is a sign that shares characteristics with the object, perceived as having some similarity with it. Usual examples of icons are a realistic painting and a wax statue. The indexical character of fMRI visuals is evident in how they are generated. Just as a photograph has a causal relationship with its subject,3 there is a causal relationship between the brain and its fMRI rendering. However, fMRI visuals are also iconic. The claim for the iconic character of brain visuals, though, should not be equated with a naïve idea of similarity: fMRI visuals are not iconic signs because they look like the brain and its processes. Rather, fMRI visuals are iconic as they are understood through an active visual inspection and embodied engagement . In other words, the iconicity of fMRI visuals comes to the fore when they are considered from the perspective of real-time, practical engagement. 24 Chapter 2 This chapter looks at a published fMRI figure to show that fMRI visuals are not iconic signs in terms of the naïve idea of similarity, but that they generate meaning by relying on a variety of semiotic structures that function as their “infrastructure for seeing.” The published fMRI figure does not directly “reveal” to a passive eye the brain and its processes; instead, it relies on a variety of signs that indicate what the figure shows as they call upon the viewers’ cultural knowledge and experiential engagement . One should, however, ask what functions as the infrastructure for seeing when scientists engage fMRI visuals during their everyday laboratory work. The question of iconicity and semiotic infrastructure is important for at least two reasons. First, it problematizes the productivity of the dichotomy between the visual and the digital. Second, it calls for a reconceptualization of scientific visuals and their boundaries. fMRI scans, when considered from the perspective of their everyday, real-time engagement, are neither only visual nor only digital; they are at the same time visual and digital. This engagement with brain visuals can be tackled in terms of written as well as gesturally enacted signs. Because they do not generate meaning in the absence of their infrastructure for seeing, such infrastructure is their constitutive element. Digital scientific visuals are, thus, fields for interaction as they have to be understood with respect to how they are worked with and experienced. In other words, their character is not necessarily representational, but it concerns the participation of their readers/ writers. In making this argument, the chapter relies on the interpretative semiotics of Peirce and his follower—Umberto Eco. Whereas Peirce speaks to social studies of science and technology through his own writing in the philosophy of science (e.g., Rescher, 1978), I want to highlight some of the features of Peirce’s semeiotic and pragmati(ci)sm not originally aimed at studies of science and less commonly referred to in STS. This overview, however, is not intended as an exposition of the theory that underpins the practice (expounded in the chapters that follow). Rather, it is to clarify some of the concerns that sustain the practice-oriented analysis that constitutes the core of this book. At the same time, in providing empirical examples and analyzing videotaped material of laboratory work and interaction , the goal is to generate a sense of how the next step in engaging Peirce’s semiotics in STS can be taken. fMRI Brain Visuals as Fields for Interaction 25 Iconicity and fMRI Brain...