In lieu of an abstract, here is a brief excerpt of the content:

20 The Research on Audiovisual Perception of Temporal Order and the Processing of Musical Temporal Patterns: Associations, Pitfalls, and Future Directions Argiro Vatakis and Georgios Papadelis 20.1 Introduction Almost all everyday acts are governed by time. Acts such as walking and speaking are often taken for granted and are considered easy to execute and understand. These acts, however, are not nearly as easy as they appear to be. They are in fact highly complex acts that unfold over time and require attending and perceiving the temporal order of the individual units of each single act. Temporal perception (in terms of temporal synchrony) is also essential (along with space and semantic congruency) in multisensory integration in determining whether the observer will experience a unified audiovisual event or two separate auditory and visual events (e.g., Calvert, Spence, & Stein, 2004; de Gelder & Bertelson, 2003; Doehrmann & Naumer, 2008; Driver & Spence, 2000; Kallinen & Ravaja, 2007; Sekuler, Sekuler, & Lau, 1997; Slutsky & Recanzone, 2001). Multisensory integration is indeed enhanced when multiple sensory signals are close in time (e.g., see Calvert et al., 2004; de Gelder & Bertelson, 2003), however precise temporal coincidence is not mandatory for the human perceptual system to create a unified perceptual representation of a multisensory event (e.g., Dixon & Spitz, 1980; Engel & Doherty, 1971; Grant, van Wassenhove, & Poeppel, 2004; Kopinska & Harris, 2004; Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Navarra et al., 2005; Rihs, 1995; Soto-Faraco & Alsius, 2007, 2009; Sugita & Suzuki, 2003; Vatakis & Spence, 2010). Thus, even though information received by our different senses may require different processing times, we still experience the illusion of a synchronous event (Spence & Squire, 2003; Zeki, 1993). In order to estimate sensory latencies and their effects on the integration of a multisensory event, many psychophysical methods have been applied in the laboratory. This chapter will mainly focus on one such method, the temporal-order judgment (TOJ) task (other methods include reaction time, simultaneity judgment, and ternary response tasks; Vatakis & Spence, 2010). The TOJ task has been central in experimentation on temporal perception for more that a century now (e.g., Exner, 1875; Hamlin, 1893; Jaśkowski, Jaroszyk, & Hojan-Jezierska, 1990; Rutschmann & Link, 1964). The idea of this method is that if a given stimulus is 410 Argiro Vatakis and Georgios Papadelis processed faster, then it should also produce a faster reportable sensation in comparison to another stimulus. Consequently, a participant’s judgment in regard to the point in time they experienced a sensation should represent the processing time of the stimulus responsible for that sensation (Neumann & Niepel 2004). In a typical TOJ task, therefore, the participants are presented with a pair of stimuli (e.g., an auditory and a visual one) at various stimulus-onset asynchronies and asked to make a judgment about the order of stimulus presentation (i.e., “Which stimulus was presented first?”; e.g., Bald et al., 1942; Hirsh & Sherrick, 1961; Spence, Shore, & Klein, 2001; Sternberg, Knoll, & Gates, 1971; Vatakis & Spence, 2010; or “Which was presented second?”; e.g., Parise & Spence, 2009). The data obtained from a TOJ task allows for the calculation of two measures, the just-noticeable difference (JND) and the point of subjective simultaneity (PSS; see figure 20.1). The JND provides a standardized measure of the sensitivity with which participants can judge the temporal order of the two stimuli that have been presented at a given performance threshold (typically 75 percent correct; see figure 20.1A). The PSS provides an estimate of the time interval by which the stimulus in one sensory modality has to lead or lag the stimulus in the other modality in order for the two to be perceived as having been presented synchronously (see figure 20.1B). These two measures are used for the calculation of the temporal window of multisensory integration using the formula PSS±JND. (Note that the term temporal window of multisensory integration does not imply an active process, but rather refers to the interval in which no signal discrepancy is perceived; anything beyond this interval will normally be perceived as being desynchronized.) Just Noticeable Difference (JND) Point of Subjective Simultaneity (PSS) SOA (ms) 0.01 0.08 –250 –150 –50 50 150 250 0.5 0.75 0.99 Proportion of “Vision first” Responses Vision first Sound first SOA (ms) –250 –150 –50 50 150 250 Vision first Sound first 0.92 0.25 A Proportion of “Vision first” Responses B Temporal order uncertainty 0.0 0.2 0.4 0.6 0.8 1.0 PSS Figure 20.1 Measures of (A) the just-noticeable difference...

Share