-
6. Getting Real about Systematicity
- The MIT Press
- Chapter
- Additional Information
1 Introduction 1.1 Systematicity and Reality In the twenty-five years since its inception, the systematicity debate has suffered from remarkably weak empirical grounding. For a large part, the debate has relied on purely theoretical arguments, mostly from the classicists ’ side (e.g., Aizawa 1997a; Fodor and Pylyshyn 1988; Phillips 2000), but occasionally from the connectionist camp as well (Bechtel 1993; Van Gelder 1990). And although there have been many attempts to empirically demonstrate (lack of) systematicity in connectionist models, it remains doubtful how these demonstrations bear upon reality, considering that they are always restricted to hand-crafted, miniature domains. This is the case irrespective of whether they are presented by supporters of connectionist systematicity1 (Bodén 2004; Brakel and Frank 2009; Chang 2002; Christiansen and Chater 1994; Elman 1991; Farkaš and Crocker 2008; Fitz and Chang 2009; Frank 2006a,b; Frank and Čerňanský 2008; Frank, Haselager , and van Rooij 2009; Hadley, Rotaru-Varga, Arnold, and Cardei 2001; Jansen and Watter 2012; McClelland, St. John, and Taraban 1989; Miikkulainen 1996; Monner and Reggia 2011; Niklasson and Van Gelder 1994; Voegtlin and Dominey 2005; Wong and Wang, 2007) or by those who are more skeptical (Marcus 2001; Phillips 1998; Van der Velde, Van der Voort van der Kleij, and De Kamps 2004). My goal in this chapter is to approach the systematicity problem in a fully empirical manner, by directly comparing a connectionist and a symbolic sentence-processing model in a (more or less) realistic setting.2 As far as this chapter is concerned, getting real about systematicity means three things. First, connectionists can no longer get away with presenting models that function only within some unrealistic toy domain. To the extent that the systematicity issue is relevant to real-life cognitive systems, 6 Getting Real about Systematicity Stefan L. Frank 148 Stefan L. Frank connectionists should be able to demonstrate that (alleged) instances of systematicity do not depend crucially on the artificial nature of the simulation. Second, I also aim to raise the bar for classicists, who need to back up their claim empirically that symbol systems are necessarily systematic. Aizawa (1997b) argues that compositionality is not a sufficient condition for systematicity, and, indeed, to the best of my knowledge it has never been empirically demonstrated that symbol systems are any more systematic than neural networks. Nevertheless, even many connectionists accept the premise that symbol systems explain systematicity. Third, rather than defining particular levels of systematic behavior based on the specifics of training input and novel examples (as in, e.g., Hadley 1994a,b), the question of how systematic cognition really is will be avoided altogether. People learn language from what is “out there” and, subsequently, comprehend and produce more language “out there.” Hence, the generalization abilities of the models presented here are investigated by training and testing both models on a large sample of sentences from natural sources. There is no invented, miniature language, and no assumptions are made about which specific syntactic construction in the training data should result in which specific systematic generalizations. 1.2 Statistical Modeling of Language While the systematicity debate in philosophy and cognitive science revolved around theoretical arguments and unrealistic examples, actual progress was being made in the field of computational linguistics. The development of statistical methods for learning and processing natural language resulted in many successful algorithms for tasks such as sentence parsing, translation, and information retrieval. Recently, there has been a growing interest in applying such models to explain psychological phenomena in human language comprehension (e.g., Boston, Hale, Patil, Kliegl, and Vasishth 2008; Brouwer, Fitz, and Hoeks 2010; Levy 2008), production (e.g., Levy and Jaeger 2007), and acquisition (e.g., Bod and Smets 2012; Borensztajn, Zuidema, and Bod 2009b). The systematicity controversy tends not to arise here; for one because computational linguists are often concerned more with practical than with theoretical issues. Also, and perhaps more importantly, these models are typically symbolic and thereby dodge the systematicity critique. What recent models from computational linguistics share with connectionist ones is their statistical nature: they are concerned with the problem of extracting useful statistics from training data in order to yield [44.202.128.177] Project MUSE (2024-03-28 16:00 GMT) Getting Real about Systematicity 149 optimal performance on novel input. As Hadley (1994a) pointed out, this issue of correct generalization to previously unseen examples is exactly what systematicity is all about. Hence, one might expect statistical computational linguistics to be vulnerable to the same systematicity critique...