3D Notations and the Immersive Score

Abstract

The author discusses his use of generative three-dimensional notations for representing musical forms. Several key works, programmed in the Max/OpenGL platform, are described in detail, and the author discusses current development with Microsoft's HoloLens. The author argues that such immersive technology promotes a physical engagement with the score in which the work is an emergent property of an open-ended play.

For over 15 years I have been exploring various nonlinear open-form musical structures afforded by generative real-time scores. Such scores often integrate complex nonlinear processes within generative techniques ranging from the use of Markov chains or other stochastic processes to determine the temporal ordering of events through to the use of data derived from timbral analysis to drive low-level structural transformations such as pitch distributions. The procedural generation of musical forms has been explored by a large number of composers, but only over the past ten years have composers begun to explore procedurally generated real-time performance scores [1]. Of particular interest in my creative and research practice are the use of three-dimensional scores in dynamic visualizations, which present performers with representations of musical form, the potentialities of which are explored through guided play [2].

Three-dimensional scores fundamentally present an effort to transcend the materiality of the printed page. While visual artists have grappled with the affordances of perspective since the fourteenth century, the applications of perspective and three-dimensional structures in musical notations have been of only relatively recent interest. The use of depth as a structural determinant is suggested in works such as Fontana Mix (1958), Cartridge Music (1960) or Variations III (1962) by John Cage [3–5] or in Toshi Ichiyanagi's Music for Piano No. 7 (1961) [6], where printed transparencies are overlaid to create musical structures, and Kenneth Gaburo explores the use of superimposed text in his Lingua II: Maledetto II (1967–1968) [7]. In each of these works, while form emerges from material depth, the two-dimensional surface upon which the works are denoted is a fundamental constraint [8].

While I alluded to three-dimensional notations in my Valses and Etudes (2005, rev. 2011), for piano and computer, where pages of musical notation fly in and out of a musical display, it was not until my point studies no. 2 (2013), for any two pitched instruments and computer, that I started to explore the musical affordances of three-dimensional notated structures with greater focus. The techniques I developed in point studies no. 2, including the use of stochastic processes to instantiate pitch and rhythmic structures and transformational grammars to drive temporal development, have motivated much of my subsequent research in this area. In point studies no. 2, a three-dimensional construct of stochastically distributed colored nodes connected by thin white lines is presented to the two performers through the OpenGL platform on a video display (Fig. 1). Node colors denote set pitches, while the length of lines connecting nodes denote the potential duration of those pitches. The number and distribution of nodes is uniquely determined for each performance, providing a variety of possible pathways through the score that performers may trace. The performers are presented with a constantly shift ing view of the node distributions as the OpenGL camera moves along predetermined trajectories around the nodal construct. This facilitates the ability of the performers to discern pathways through the score as it is displayed on a two-dimensional surface.

Fig. 1.

Sample instantiation of the score for point studies no. 2 (2013). (© David Kim-Boyle)

Fig. 2.

Screen capture from the score for 64×4×4 (2017). (© David Kim-Boyle)

Fig. 3.

Screen capture from the score for 5×3×3 (2019). (© David Kim-Boyle)

The techniques developed in point studies no. 2 are extended in 64×4×4 (2017) for string quartet and 5×3×3 (2018) for any three wind instruments. In 64×4×4, the virtual size of the score is extended to a maximum potential density of 1024 nodes in comparison to point studies no. 2's 512 nodes. In 64×4×4 node colors denote string harmonics rather than pitch, while line colors (red, green, blue, white) denote different strings on which those harmonics are performed. The increase in nodal density creates a greater number of potential pathways through the score for the performers to explore but also allows a broader scope of movements for the OpenGL camera's trajectories through the score. Like point studies no. 2, 64×4×4 also features the use of transformational grammars where the spatial distribution of nodes along x, y and z planes is dynamically scaled during performance (Fig. 2).

In 5×3×3 (2019), colored nodes, which in this instance represent different types of articulations, are connected by colored Bézier curves, the curvature and length of which denote particular timbral transformations and event durations. Line curvatures are driven by FFT analyses of the three instrumental sounds (Fig. 3). Each performer is presented with a different perspective on the nodal construct, which shifts as they physically navigate the performance space. As in 64×4×4, transformational grammars are used to dynamically transform the node's x, y and z spatial positions affecting the curvature and length of connecting lines.

In point studies no. 2, 64×4×4, and 5×3×3 the z-axis of the displayed score is not mapped to any unique musical parameter. Rather, information is displayed in three dimensions to facilitate more efficient representations of complex nonlinear structures and to allow more complex nonlinear processes to be integrated within temporal development. Despite these aesthetic goals, the score for each of these works is still fundamentally constrained by the two-dimensional surface upon which it is displayed. While I have considered the use of 3D printing as a means of transcending this limitation, I have ruled this out as the generative and dynamic temporal processes on which my scores are grounded cannot be preserved, and the ensuing impracticalities regarding performance are of substantial concern. Of greater value is the use of immersive, mixed reality displays such as Microsoft's HoloLens; to that end, I have begun porting work to Unity, the development platform with which HoloLens applications are built, for display on this device.

While the HoloLens has received growing investigative interest in fields ranging from medicine [9] to architecture [10], its use in the creative and performing arts has to date been more limited. As a device for facilitating the display of compelling 3D visualizations, the HoloLens affords a ready solution to the challenges involved in displaying three-dimensional notational schemas, including reducing the occlusion that occurs when a three-dimensional structure is presented on a two-dimensional surface. At the same time, the device opens new aesthetic affordances through foregrounding the physical relationship between performers and notational schema. In 64×4×4, for example, the first of my works to be ported to the HoloLens, the physical engagement with the score becomes an essential means of uncovering its various potentialities [11]. The pathways through the score, uniquely instantiated for each performance, may only be discovered when the performers physically navigate the space in which the score is displayed. In this respect, performance approaches a type of dérive [12], where the work's form emerges as a result of ludic engagement with its latent possibilities. The aesthetic foregrounding of play not only situates 64×4×4 within a rich artistic heritage of psychogeography [13] but fundamentally underscores the aesthetic value of play [14] and its political agency [15].

While relationships between spatial geometry and musical structure have been explored in a vast body of creative work by composers such as Xenakis [16] and theorists such as Deleuze and Guattari [17] and Lefebvre [18], immersive technologies such as the HoloLens offer new insights into this relationship through the resituation of play within an aesthetic space. On a more pragmatic level, such technologies, while not without technical constraints, offer a viable means for the creation of dynamic, three-dimensional performance scores musically articulated through playful exploration that in my own creative work continue to be a powerful source of stimulation.

Supplementary Material

• [ Click to download ] Excerpt from a performance of point studies no. 2 (2013) for any two wind instruments and computer.
Copyright credit: David Kim-Boyle