Change or Not to Change -- Is There a Question? A Response to Pike

Gary Pike is certainly one of the most astute methodologists working in our field. We congratulate him on another important and timely contribution. We think, however, that what we demonstrated in our paper, "Explaining Student Growth in College When You Don't Think You Are," may be different than what Pike believes we did.

Let us briefly reiterate the point of our paper. In correlational and quasi-experimental research, the type of inquiry that overwhelmingly dominates the college impact literature, the most internally valid designs are based on longitudinal data with a pretest measure of the outcome variable (or posttest). Because they are typically characterized by student self-selection, such designs do not provide the same level of internal validity or causal inference as randomized experiments. Indeed, as Lord (1958, p. 450) referring to gain scores pointed out, and Pike (1992, p. 78) quoted in a previous paper: "Unless the two students started at the same point in the score scales, however, it cannot be concluded that the first student really learned more than the second, except in some very arbitrary sense." However, longitudinal pretest-posttest designs are much preferable to cross-sectional designs or longitudinal designs that do not permit one to take students' precollege scores on the outcome variable into account. The reason for this is straightforward. Whether one is using posttest scores or gain scores (posttest-pretest) as the dependent variable, part of the total variance in each will be due to college experiences, but another part of the total variance in each will also be attributable to students' pretest scores. For example, the pretest usually has a high, positive correlation with the posttest (i.e., the student who starts high will usually finish high and vice versa). Because of regression to the mean, however, the pretest score tends to have a negative correlation with pretest-to-posttest gain (i.e., students starting low tend to exhibit larger gains than those starting high). Consequently, unless the influence of the pretest is taken into account in correlational or quasi-experimental designs, it is difficult, if not impossible, to determine which part of the total variance in posttest or gain scores (post-pre) is uniquely associated with the college experience variables. (This is potentially a major problem in interpreting the regression coefficients and significance tests in Model 3 of Table 1 in Pike's paper.) Unless one has conducted a randomized experiment, in which comparison groups differ in only chance ways on a pretest, any analysis of posttest scores or pre-to-post gain scores that does not take the confounding effects of the pretest into account is quite problematic. One possible exception to this is when one is analyzing gain scores with a variable that has no measurement error and, therefore, no artifactual regression to the mean. (An example would be body weight in Pike's paper.) However, such perfect measurement of constructs is nearly nonexistent in research that attempts to estimate student growth during college.

In our paper, we demonstrated that the two following equations (i.e., Equations 1 and 2) yield the same metric regression coefficients and significance tests (i.e., t ratios) for variables x, y, and z.

Equation 1: Posttest = b(Pretest) + b(x) + b(y) + b(z)

Equation 2: Posttest - Pretest = b(Pretest) + b(x) + b(y) + b(z)

This means that when one is predicting a posttest score with a regression specification that includes the pretest (Equation 1 above), the estimated metric effects and statistical significance of all other independent (predictor) variables in the equation will be the same as if one were predicting change or growth on that variable with the same equation (i.e., Equation 2 above). For example, assume that x represents working versus not working during college. The metric regression coefficient and t test for work in the prediction of posttest scores (Equation 1) will be exactly the same as the metric regression coefficient and t test for work in the prediction of change or growth (Equation 2). The "take-home message": One...