Misapplying working-memory tests: A reductio ad absurdum

The preceding discussion note by Sprouse, Wagers, and Phillips (2012b; SWP2) sidesteps the objections we raised in our critique (Hofmeister et al. 2012; HSS) of their 2012a article (SWP1). In the process, they attempt to deflect attention away from the fundamental issue at stake: is the experimental method used by SWP1 an effective diagnostic of the role of processing factors in acceptability contrasts? The answer to this question, as far as anyone knows, is 'no'. Hence, as we argue in HSS, the absence of a correlation between memory measures and acceptability contrasts is unsurprising and in no way undermines processing-based accounts of island effects.

Processing-based accounts of islands centered around working-memory (WM) limitations, like those of Kluender & Kutas 1993 and Hofmeister & Sag 2010, DO predict a relationship between WM measures and judgments for islands IF AND ONLY IF: (i) the relevant WM measures tap the resources used in processing sentences with island-violating dependencies, and (ii) the relevant examples allow for individual variation to emerge.¹ But neither concern has been satisfied by SWP. They do not establish that their chosen memory measures (or any other) systematically relate to judgments for ANY hard-to-process but grammatical sentences, nor that this relationship applies universally across the spectrum of sentence processing difficulty.

SWP2 fail to even address point (ii), while they attempt to defuse point (i) by a purely 'in-principle' argument. In essence, they try to deflect concern about (i) by speculating that other WM tasks would have produced similar results, implying that there should be little concern about their choice of memory measures. Putting aside serious concerns about whether memory tasks are so interchangeable (and the unaddressed fact that the n-back and serial-recall tasks are not WM tasks; see Kane et al. 2004), this still leaves unanswered the question of how the magnitude of acceptability contrasts tied to resource limitations relates to memory measures. In other words, it is premature to argue about whether all memory tasks relate in the same way to judgments for hard-to-process structures if we do not know that ANY memory measure systematically predicts acceptability judgments for such structures! The relevance of (ii) to this issue is not merely a theoretical concern, either. In practice, both HSS and SWP have independently validated objection (ii). In HSS, we showed that perhaps the most widely used WM measure, the reading span, fails to predict variation in judgments for extremely hard-to-process sentences (see also Waters & Caplan 1996), even though they predict variation for 'moderately' difficult items. Indeed, both SWP1 and SWP2 fail to mention their finding that judgments for the poster child of grammatical, hard-to-process structures—multiple center-embeddings—were, in fact, higher for individuals with LOWER MEMORY SCORES on their tests (Sprouse 2009), contrary to the hypothesis that higher scores should equate to higher acceptability judgments for sentences that stress WM resources. This evidence belies SWP2's claim that 'the naturalness or discourse felicity of our materials have little to do with the debate' (p. 402), since in any investigation of individual differences, the nature of the test materials matters. For example, if the materials universally cause computational breakdown, then individual differences will be inconsequential.

What we can say for certain on the basis of this evidence is that there is no known UNIVERSALLY APPLICABLE way in which memory measures relate to acceptability judgments. Beyond this, we are faced with a stark example of the conclusions we would be forced to draw if we accepted SWP's logic: if a failure to find a correlation between WM measures and acceptability decrements is evidence that grammatical constraints—and not processing constraints—underlie a contrast, then both HSS and SWP have 'evidence' that multiple center-embeddings are ruled out by grammar and not by processing.

In light of this reductio, we conclude that there is simply nothing new to be learned from either SWP1 or SWP2 about how grammatical constraints and processing factors like WM limitations interact in the processing of island structures.² Their conclusions that processing...