Handling Digital Brains: A Laboratory Study of Multimodal Semiotic Interaction in the Age of Computers

1. In the fMRI Laboratory

1 In the fMRI Laboratory It is 2002, and we are in a cognitive neuroscience laboratory at the University of California, San Diego. There, we encounter two researchers seated in front of a computer screen (figure 1.1). One of them, the laboratory director, Paul (a professor with a distinguished record of publishing and teaching in the field of cognitive neuroscience), is talking with a graduate student named Jane (a promising Ph.D. candidate in cognitive science) seated next to him (featured on the right in figure 1.1). The two researchers are engaged in the practice of functional magnetic resonance imaging (fMRI). fMRI, together with its forerunner, MRI, is a key modern digital imaging technology used for medical and scientific purposes. The goal of MRI is to provide detailed static renderings of the anatomic structure of internal body parts, such as the brain. This technique uses radiofrequency, magnetic fields, and computers to create visual renderings (“visuals”) based on the varying local environments of water molecules in the body. To obtain such visuals, a person (or, in fMRI practitioners’ jargon, an experimental subject or a subject) is scanned. During a brain MRI scanning session, hydrogen protons in brain tissues are magnetically induced to emit a signal that is detected by the computer. Such signals, represented as numerical data, are then converted into visuals of the brain as the brain anatomy of the experimental subject is imaged. The mapping of human brain function by use of fMRI represents a new dimension in the acquisition of physiologic and biochemical information with MRI. The technique is used to observe dynamic processes in the brain that are demonstrated by visualization of the local changes in magnetic field properties occurring in the brain as a result of changes in blood oxygenation . The role of fMRI visuals is, thus, to display the degree of activity 2 Chapter 1 in various areas of the brain; if the experimental data are obtained while a subject is engaged in a particular cognitive task, the visual can indicate which parts of the brain are most active during that task. To show these results, however, fMRI visuals require extensive analysis in the laboratory. During analysis sessions, such as the one in which Jane and Paul are involved, fMRI practitioners use computers to engage their data, shaping the appearance of fMRI visuals. The engagement with digital material allows the practitioners to enhance their understanding of the imaged biological matter. This means that observation by an fMRI practitioner is multimodal and cyborg-like; it is accomplished by coordinating the eyes with digital technology, an array of instruments, graphical inscriptions , and actions of the hands.1 Rather than passively gazing at static visual images, fMRI researchers interact with each other and the technology , engaging their experiential and semiotic bodies. They manipulate, listen to, and touch computers and other instruments, and they also talk, gesture, interactionally engage their heads, necks, and torsos as they attend to each other in the work of cognitive neuroscience. One of the ways practitioners can engage digital material to enhance their understanding is to flatten computationally the imaged cerebral cortex to see not only what is shown on its ridges (or gyri) but also what otherwise would be hidden in the fissures (or sulci) that surround such ridges. This is exactly what Jane and Paul are doing when we join their data analysis session. Because Jane still needs to acquire skills in data analysis, Paul, while talking about the brain visuals displayed on the (a) (b) Figure 1.1 fMRI researchers working on a laboratory computer. In the fMRI Laboratory 3 computer screen, shows Jane how to use fMRI software to generate visuals that appear flattened. When the computer screen displays a set of data as an “inflated surface,” Paul identifies portions of the imaged visual cortex: “That’s the center of the gaze and that’s that other thing that I said don’t look at it. That’s up, right there.” As he points out a section that does not look like he thinks it should, Paul chuckles while jokingly warning Jane not to pay attention to it. During the entire sequence, as Paul holds the computer mouse in his right hand and skillfully directs the changes of the displayed visuals, he indicates specific brain areas by using the cursor and pointing with his left hand. While attentively listening, Jane takes notes...