Typology of Emotion Assemblages in Art and Science

Abstract

This article analyzes how artists and scientists use emotion as a methodological tool to study participants and cocreate projects. It uses a science and technology studies approach to identify a typology of five emotion assemblages and the material, social, and cultural components that make them operate. These assemblages include or eliminate cultural and environmental context, the role of participants, and artist/researcher reflexivity. The paper argues that there is an imperative for artists, designers, and researchers to make deliberate choices as to who or what to include and exclude when working with emotion.

Art and emotion

Art has historically been seen as a space to express emotions in artworks and invoke emotions in the audience [1]. Yet more recently, there has been a shift toward artists using emotions as a methodological tool to understand themselves and the participants within their projects and to cocreate artworks. When emotions are used as a method, they are the central object of analysis, or the means to carry out research, rather than a mere byproduct. In the arts, these inquiries have often involved shifting sensor technologies and conceptual approaches from the natural and social science disciplines into the arts. Yet this adoption has often taken place in a haphazard way, without critical reflection on the epistemological, ontological, and political assumptions of how to work with emotion. In this article I address this gap by asking the normative question: How should artists use emotion as method? While I primarily address art and design practitioners, I aim to make this approach equally valid for natural and social science researchers, and throughout the article I will be addressing a conjoined artist/researcher. Theoretically I use an approach from science and technology studies and new materialism, which sees emotion as co-constructed with the living and material world [2], to foreground material components such as physiological sensors, research settings, and visualizations that enable the use of emotion as method. This mix of physical, social, and conceptual elements is what I refer to in the paper as an "emotion assemblage."

Methodology

The article is based on my observations of the Geography of Feelings project and discussions with the team of Photini Papahatzi and Myrto Zarakosta [3], which highlighted a lack of discourse within art and design about how to deploy emotion as a method. I attempt to address this gap by identifying a range of emotion assemblages. Creating the typology involved a limited literature review of academic texts on Google Scholar for "art + emotion" and "emotion + method," as well as targeted searches for "emotion" within art and design journals to identify projects. These were clustered to identify five assemblages that are distinguished by how they situate emotion: in the laboratory, as public data, or in galleries, other cultural settings, and communities. I then visualized the assemblages via a diagramming approach inspired by Klooster [4] to highlight salient aspects such as how the assemblages conceptualize participants and position the artist/researcher. The resulting typology is not intended to be exhaustive but to start to address the practical needs of artists and designers to differentiate and classify ways of working with emotion.

A brief review of the study of emotion

Emotion is talked about in the mainstream media in addition to being a topic of academic research. The study of emotion has a history going back hundreds of years and has been discussed in many disciplines, including philosophy, psychology, neuroscience, and linguistics. Yet there are disciplinary differences as to what constitutes emotion and how to study it. Psychology, for example, defines emotion as "experiential, behavioral, and physiological elements, by which an individual attempts to deal with a personally significant matter or event" [5]. Mauss and Robinson provide a useful overview of different methodological approaches to emotion [6], listing a wide range of methods for capturing emotion, such as self-reporting, physiological measures, brain state measures such as EEG, neuroimaging, and behavioral aspects such as vocal and facial characteristics. A key reference point is the notion of affective computing [7], which synthesized the history of physiological research and integrated it with computation methods to create sensors that are accessible to artists and designers. Regardless, we find no consensus around emotion capture approaches but rather many disciplinary discussions about the "right way to study emotion." The discussions are broadly divided between framing emotion as an unconscious physiological artifact of the body or as something that can be self-reported by research subjects [8]. There have been proposals to integrate these methods via a "biosocial" approach that combines physiological measurement with verbalized emotions via interviews [9].

Here I use a theoretical approach from science and technology studies and new materialism that conceptualizes emotion as an "affective" assemblage of social relations and material elements rather than occurring within the body of an individual [10]. This approach has been used to analyze what emotions do, rather than what they are, in order to understand how emotions are deployed for strategic and political purposes at both micro and macro levels [11]. An emotion assemblage, in these terms, is a structural arrangement that stages material elements such as research settings and sensors as well as cultural contexts and social relations between researcher and participants. When deployed in different situations, these assemblages have some flexibility to adapt but maintain a series of distinct structural relationships. I use the term "staging," taken from the philosopher Annemarie Mol's work, to highlight the performative effect of these arrangements, which don't passively record emotion but actively transform their setting and participants [12]. The suggestion is that different stagings of participants, sensor technologies, and artists/researchers shape the world in radically different ways. The philosopher Bruno Latour, in his paper "How to Talk About the Body" [13], argues that sensation is distributed beyond the body and requires material devices to become articulate. He gives the example of a perfume kit that contains a variety of scents, which allows the perfumier to become sensitive to olfactory difference and thus develop a "nose." Latour's argument is that senses (and emotions) are not located in the human body but are coarticulated with the material world. What matters is how sensitive stagings are to difference and whether they generate findings that are engaging, transformative, and available to be publicly disputed. In effect, Latour offers a normative framework for analyzing a range of emotion assemblages across the natural and social sciences, as well as art and design.

Five emotion assemblages

The assemblages are labeled by locations that define and constrain their way of staging emotion.

Assemblage 1: Laboratory Emotion

This assemblage (Fig. 1) is present in natural sciences such as neuroscience and psychology. It is characterized by the researcher setting up a controlled test stimulus and capturing

Fig 1.

Diagram of "Laboratory Emotion," with statistical analysis as its output. (© Christian Nold)

the emotional response from participants, who are treated as universal representatives of human behavior. The methods for emotion capture vary wildly, from vision capture of facial expression to neuroimaging or physiological metrics [14]. What is significant about this approach is that participants are treated as subjects and not individually involved in the staging of the process, such as defining the stimulus or parameters of the experiment. The outputs of this research are statistical data. While visualizations are often produced, researchers tend to use these as byproducts to communicate the research.

Assemblage 2: Public Emotion

This assemblage (Fig. 2) is often seen in computer science and design, and sometimes in art contexts. Methodologically this assemblage involves capturing public emotion by extracting social media data and identifying semantic patterns that have emotional valance, such as "I love," seen in projects such as We Feel Fine [15]. Other methods involve creating custom smartphone apps such as Mappiness [16] for participants to download and use to generate emotion data, or physiological sensors to record participant arousal, as in the People as Sensors concept [17]. What is significant about this assemblage is that while it uses natural science methods and epistemology, it is often focused on producing visualizations as its output, where emotion is treated as an aggregate in order to talk about "public emotion." Typically, participants are not aware of their emotion data being used nor are they involved in generating the final visualizations.

Fig 2.

Diagram of "Public Emotion" using aggregative analysis that results in visualization as its output. (© Christian Nold)

Assemblage 3: Gallery Emotion

This assemblage (Fig. 3) is common within the arts and interaction design. It is characterized by multimedia installations that involve near-real-time emotion capture from participants that is then fed back to the participants visually or via audio. The artworks tend to use galvanic skin response [18], heart rate [19], or body gesture [20] to capture emotion data and directly translate this into an artwork. The input of the projects is usually the behavior of the audience or artist and the immediate context of the artwork, such as the gallery or museum. What is distinctive about this assemblage is that the participants are interactively involved in the generation of the artwork, often taking the role of players or performers. The focus is on interaction, and there is often no final visualization output.

Assemblage 4: Cultural Emotion

This assemblage (Fig. 4) is common in the social sciences, which use qualitative methods of participant observation and ethnographic methods. One of its key techniques is observing how participants interact with each other within their social and cultural contexts. The premise is that emotions are verbalized by participants themselves or can be

Fig 3.

Diagram of "Gallery Emotion" employing emotion feedback between the artwork and participants. (© Christian Nold)

inferred by the researcher through observation. The aim is not to "capture" emotion but to note affective episodes of disconcertment [21] to understand broader epistemic conflicts. In this assemblage, the role of the researcher is complex, since they observe emotion in participants but also feel emotions themselves. Hovland [22] argues that anthropology has a difficult relationship with the researcher's own emotions, which have traditionally been seen as illegitimate and have thus been excluded from anthropological narratives. One response is to use emotions as an explicit method, where "the researcher's emotional empathy is a relevant methodological stance to uncover the role and centrality of emotions in the behavior of the public servants we study. This positioning allows the ethnographer to feel like a bureaucrat and develop cognitive empathy" [23]. It is only by involving their own emotions that the researcher learns to "feel" and understand the situation of their participants. Guest and Seoighe extend this to suggest that researchers have an ethical and political imperative to narrate their own emotions in relation to contested research situations such as prison environments [24]. What is distinctive about this assemblage is that it highlights the role of the researcher as an active participant in the research who is analyzing the emotions of participants as well as their own. The result is that the researcher is actively present and sometimes even an "obstructive" element [25] that develops self-reflexivity about their own positionality in relation to the participants and the research setting. Participants are often framed as cultural representatives but not normally involved in defining the research process.

Fig 4.

Diagram of "Cultural Emotion," where participants are treated as existing within a cultural and situational context. The presence of the researcher (grey) is highlighted for observing participants' emotion as well as reflecting on their own experience. (© Christian Nold)

Assemblage 5: Community Emotion

This assemblage (Fig. 5) is found in socially engaged and community art as well as social science research. It is characterized by artists/researchers working directly in spatially delimited settings with local participants to focus on their articulation of emotion. In the Geographies of Feelings project [26], the participants explore the emotional context of their cultural and family history of forced migration by bringing in old family photos that the artistic team transforms into photographic sculptures. These objects are then used by the participants to create public performances that narrate their family histories, which are shown to an audience of other locals. In the Bio Mapping [27] project, participants take part in an emotion mapping of their surroundings using a physiological arousal sensor to produce visualizations that they narrate to a group of local participants. What is significant about this assemblage is that the participants take a substantive role in the process of choosing how to perform their own experiences and influence the final representation of the artwork as a form of cocreation. The artist/researcher is a visible and active part of staging the whole process and personally involved in building relationships with participants to invite them to take part. The artist/researcher also includes themselves and their positionality as part of the social process. In Papahatzi's work, the artist shares the same background of family dispossession as her participants, enabling personal empathy and insights into the emotion and trauma of lost ancestry and identity. This presence of the artist/researcher enables a critical reflexivity as to how emotion is deployed politically within the cultural narratives of conflict zones. In this assemblage, emotion is framed as expanding beyond the human body to include the sociomaterial, cultural, and environmental situation and is mediated with artifacts and in relation to a community and its history.

Discussion

How should artists use emotion as method? In this article I have shown that there are significant differences between assemblages that situate emotion in different sites: within the laboratory, within the gallery, as felt by the public, as cultural, or as the collective dynamic of communities. These different ways of locating emotions structurally include and exclude particular elements, such as environment, communities, and histories. They also have the effect of placing participants into a range of roles, from subjects to cocreators. Taking up Latour's challenge to make normative choices about how emotion should be staged, there are three points that distinguish the assemblages:

The Cause of Emotion

How do emotion assemblages enable or constrain what can be felt by participants? Natural science experiments on emotion (Assemblage 1) use controlled stimuli to simulate a variety of contexts. Emotion is treated as interior to the body and something that can be triggered and measured within the laboratory. There are similarities to real-time feedback artworks (Assemblage 3) where the interactive feedback is generated within the confines of a white box gallery or museum. In contrast, public (Assemblage 2), cultural (Assemblage 4), and community (Assemblage 5) approaches present emotions as responses and interactions with multiple real-world factors. The point is that the assemblages make different ontological claims about the causation of emotion. Some conceptualize it as confined to the human body, while others treat it as entangled with the messy multitude of place, culture, history, and environment.

The Position of Participants

In natural science experiments (Assemblage 1), as well as crowdsourced emotion (Assemblage 2), the participants act as an input source for emotion data and are not involved in the staging of the research context. In contrast, the other three assemblages offer participants more substantive roles: as real-time performers, cultural representatives, or coproducers of the artwork. This increased agency can allow participants

Fig 5.

Diagram of "Community Emotion," which treats participants as situated within historic and environmental contexts. This assemblage emphasizes the presence of the artist (grey) as staging the overall process. (© Christian Nold)

to contest the way that the research or artwork is created and propose alternatives.

The Reflexivity of the Researcher/Artist

The natural science epistemology (assemblages 1 and 2) is premised on an invisible and non-interventive researcher. This is paralleled by the role of the artist in gallery exhibitions (Assemblage 3), where the participants have emotional interactions with the artwork, while the artist themself is absent. This is countered by the cultural (Assemblage 4) and community (Assemblage 5) approaches, where the artist/researcher is an active component of the project and directly staging the process. The physical proximity and personal involvement of the artist/researcher enables reflection on their role in the project and enables questions as to whose emotion is being narrated and whose is excluded.

My central observation in this article is that assemblages for using emotion as method have their own agendas and structural agency. Emotion assemblages are both material and ideological; they consist of physical sensors and social arrangements of participants and researchers that have structuring force even if they are shifted into an arts context. Even humble technology such as a tape recorder used to record an interview involves a staging of emotional experience that actively transforms the world [28]. While natural scientists have specific epistemological reasons for staging emotion in highly controlled settings, this is not the case for artists and other social science researchers who welcome the messy complexity and multiplicity of the real world. For too long, the arts have uncritically borrowed emotion-sensing methods and technologies from the natural sciences that contain assumptions about the nature of emotion and that perpetuate the staging of emotion in controlled and hierarchical settings. I argue that artists need to move away from haphazard approaches toward the techniques of emotion capture and make deliberate and critical choices for how to stage emotions with participants. I intend the typology in this paper to help with these choices and to highlight new directions for future emotion assemblages. I believe artists/researchers need to invent their own customized assemblages that can stage emotion in an equitable way, expand beyond the indi vidual body, and support communities situated in distinct cultural and historic contexts. Ultimately, all artists, designers, and researchers face the imperative to make deliberate choices as to who and what to include and exclude when it comes to emotion.