Language, Vision and Music

Readers can surely be forgiven for bloated expectations upon seeing the title of this volume. They can also be forgiven, surely, for falling into a funk upon opening the book and discovering a heterogeneous collection of 33 proceedings papers under three diffuse headings with almost no common vocabulary from one paper to the next. The actual utility of the book, readers will eventually discover (or at least this reader did), lies somewhere between the bloated and the funky.

The conference from which these papers are mainly drawn appears to have been a smashing success. There is a kind of jauntiness in the Introduction (pp. 1–8), written by all three editors, that suggests that the Galway occasion was suffused with uisgebeatha, Irish for high spirits. The Introduction, together with the section introductions by each of the editors ("Language and vision" by McKevitt, pp. 9–14; "Language and music" by Ó Nualláin, pp. 189–190; "Creativity" by Mulvihill, pp. 339–340) help give shape to the volume, as far as possible, and are unfailingly clear and informative.

The Introduction reprints the call for papers that gave birth to the conference (p. 2), and its terms of reference are couched in a set of large questions that could grow into a momentous interdisciplinary research program:

Language, vision and music: What common cognitive patterns underlie our competence in these disparate modes of thought? Language (natural and formal), vision and music seem to share at least the following attributes: a hierarchical organization of constituents, recursivity, metaphor, the possibility of self-reference, ambiguity, and systematicity. Can we propose the existence of a general symbol system with instantiations in these three modes or is the only commonality to be found at the level of such entities as cerebral columnar automata?

The third section, on creativity, though not anticipated in the title of either the conference or the book, was also part of the original call for papers. The organizers noted that "In AI we have failed to get much handle on creativity", and to that end they instituted a special session on it with similarly overarching terms of reference (p. 3): "What are the links between language, vision and music? Is creativity the same for each? and by the way, What is creativity?"

Out of this ambitious call came a conference with four invited speakers, five posters and 30 oral presentations under nine general headings. Speakers represented "a balanced set ... from both Humanities and Engineering" (p. 2). Sheldon Klein's invited talk on creativity, we are told, coincided with a solar eclipse (p. 5). One of the workshops opened with sprinkles of Irish whiskey as a "pre-christian blessing".

The editors "hope that the book conveys some of the excitement of the event" (p. 5). Some of it comes through, indeed, but it is too much to hope that the printed page might capture the thrill of a Jameson mist on your forehead. More gravely, much of the excitement is overwhelmed in the print version by sheer bulk, and by the need to re-orient the mindset from one paper to the next. Members of the program committee who were readily familiar to linguists (Charles Fillmore, Shalom Lappin among them) apparently did not contribute papers to the proceedings. Some papers in the book were not conference papers at all (for instance, a 1999 publication by Ipke Wachsmuth, "Communicative rhythm in gesture and speech", pp. 117–132); good as they might be, their selection does not seem in the service of providing links between conference papers but appears to proliferate sub-themes.

The best of these papers are intellectual excursions in search of a theory. Rather than discuss the chapters one by one as is sometimes done in reviews of anthologies, I intend to look at a few themes as they are dealt with by several authors. The themes I discuss are arbitrary, determined solely by things that caught my fancy (and might not on the next reading). Enumerations of chapters are almost always tedious and in this case would hardly provide an overview. Nevertheless, mentioning some of the more evocative chapter titles might convey the range of interests housed herein: Isabella Poggi and Catherine Pelachaud, "Signals and meanings of gaze in animated faces" (pp. 133– 144); Elisabeth Ahlsén, "Speech, vision and aphasic communication" (pp. 145–156); Kai Karma, "Auditory structuring in explaining dyslexia" (pp. 221–230); Isabella Poggi, "The lexicon of the conductor's face" (pp. 271–284); A. Bonardi and F. Rousseaux, "How do interactive virtual operas shift relationships between music, text and image?" (pp. 285– 294); T. Rickards, "Creativity in humans, computers and the rest of God's creatures" (pp. 373–384).

Sheldon Klein, in "Analogical foundations of creativity in language, culture and the arts: the Upper Paleolithic to 2100CE", the talk that was interrupted by the eclipse, boldly begins by placing two constraints on theories of human cognition which he apparently considers necessary constraints. Such theories, he says, are adequate only if they:

1. 1. can account for the exponential growth in human symbolic behaviour that began about 40,000 years ago "when anatomically modern humans, after more than 150,000 years of behavioral banality, began to do rather interesting things" (p. 359), and

2. 2. can account for human computational superiority over machines even though the machines compute a million times faster (pp. 347–348).

He then launches into a formalistic, interdisciplinary exposition of multi-valued logical operations augmented by analogical transformation operators "that can be used to derive new analogies on the basis of prior examples" (p. 349). Presumably Klein believes that growth in human symbolic behaviour has its roots in this development. But the real crux of his claim rests on the cognitive development of global classification schemes that allowed people who shared them to comprehend analogical extensions to novel domains. "The essential requirement for complex social organization is that the participants can predict the behavior of others", he says (p. 363), and with that no one would argue. How the classification schemes evolved he does not say, but once in place they could plausibly provide the social basis for analogical reasoning that requires "at least one hidden layer" (p. 348) beyond machine-readable processing. Presumably, again, Klein believes that the hidden layer allows humans computational capabilities that are out of bounds for machines.

Middle Paleolithic humans capable of analogical reasoning from global classification schemes, Klein claims (p. 369), would not require "the genetically determined innate capacities in either Chomsky's or Pinker's version of such theories ... . Observed linguistic universals appear because the architecture of the brain makes some language structures computationally inefficient." We would like to believe that; presumably even Chomsky and Pinker would like to believe it. But in the end, Klein's formalisms make much of high-powered processing and leave the mystery of classificatory cognition ineffable. It is, however, in focus as problematic. Klein is erudite, clever and provocative, and his chapter probably has the most depth in the volume.

Three papers in the volume deal with synaesthesia, the confounding of sensations, usually experienced as visual associations with sounds but sometimes also, apparently, with the hearing of colours and tasting of shapes (according to John G. Gammack, "Synaesthesia and knowing", pp. 157–170). Theoretically, it can apparently occur for all 20 paired combinations of the five senses (according to Sean A. Day, "What synaesthesia is (and is not)", pp. 171–180) though most of them are unattested. Surprisingly, all three papers cover very similar ground, though the authors are from Australia (Gammack), Taiwan (Day) and Russia (Bulat Galeyev, "Synaesthesia is not a psychic anomaly but a form of non-verbal thinking", pp. 181–187). All three compare the composer Scriabin, who experienced the key of C major as red, to Rimsky Korsakov, for whom it was white, and two of the three consider Scriabin "genuine" and Rimsky Korsakov fraudulent for reasons that are not specified. All of them cite Aristotle and an assortment of other classical and Enlightenment authors on synaesthesia. All invoke homely metaphors like associations of warmth at the red end of the spectrum and cool at the blue end, or brightness of high pitches and darkness of low ones. They are all fairly entertaining (if repetitive). But if synaesthesia is truly interesting cognitively it must be uncontrollable, not metaphorical, and irrepressible, not artsy. None of these authors report cases of synaesthetic madness involving, say, people hallucinating over fire sirens or conversing with otherworldly voices emanating from Vanity Fair ads. These authors are philosophical rather than clinical, though not philosophical enough, evidently, to wonder aloud how they got to be participants at a conference on language, vision, and music.

Niall Griffith poses some hard, topical questions in "Music and language: metaphor and causation" (pp. 191–203). What is the developmental relationship between linguistic prosody and music? What is the evidence for the evolution of music from language? Answers to these questions are, predictably, hard to come by. Griffith begins with a simple contrast based on Popper's four functional levels of language, claiming that music shares Expressive and Signaling functions with language but lacks Descriptive and Argumentative functions. Even here, it seems questionable that Signaling refers to the same activity in music as in language, where signaling is defined in terms of communicative acts eliciting a "general" (presumably common) response. Some of Griffith's ruminations may be interpreted as an attempt to reconcile signaling in the two modes, though he does not put it that way. Later on he concedes, for example, that "music, while involving categorical perceptions—we hear pitches and phrases—does not attach meaning to the arbitrary signs it identifies" (p. 200), and it would have been a short leap for him to infer that that difference must yield signaling discrepancies.

Although Griffith cites several linguists (Chomsky, Fodor, Jackendoff, Lakoff, Pinker, Winograd), readers who know some linguistics are likely to question him on many points. "Prosody is essentially musical", he says (p. 194), but it seems much more likely to be accidentally musical and essentially linguistic. "Music originates in ... physical activity" (p. 200), but it seems originally cognitive and only manifested physically. "While we may be able to describe a piece of music with a grammar", Griffith says (p. 193), "this is not the same as saying that the music is caused by a process that is described by the grammar" (p. 193), but that seems equally true of language and grammar. The importance of Griffith's good questions is not that they find good answers but that they might open up productive debates.

Are there, as the call for papers asks, "common cognitive patterns [that] underlie our competence" in language, vision and music? Probably. Are they identified or hypothesized or suggested in this volume? Not obviously and perhaps not implicitly. Should we be surprised or disappointed? Probably not. The reader's bloated expectations may not have been planted there intentionally by whoever gave the book its all-encompassing title. The editors, after sitting through the conference presentations for four days and then assembling its proceedings for almost three years, surely knew that their volume would not be all-encompassing. Their decision to try to beef up the contents with previously published papers suggests that they made a sober appraisal of its shortcomings. Basically, what they accomplished was to bring together numerous academics from several disciplines to tell us about some thoughts they are entertaining on a host of individually defined topics. It can be enjoyable for readers who expect no more.