Identifying Meaningful Individual-Level Change in Educational Experiences: Adding to Our Methodological Toolkit

In recent years, improving the quantitative methods used to assess the effect of college, and particular college experiences, on student outcomes has received increased attention (e.g., Mayhew et al., 2016). In How College Affects Students, Mayhew et al. (2016) highlighted the importance of issues of practical vs. statistical significance, self-selection into college (and by extension, self-selection into particular experiences), and direct and indirect effects, among other methodological challenges in identifying the relationships between college experiences and student learning and success. One particularly difficult challenge is identifying the conditional effects of experiences on student outcomes. Who benefits, or who does not, from particular experiences? There is growing evidence that the effects of educational experiences may differ among students, and in some cases, effects that may be positive for some students are negative for others (e.g., Mayhew et al., 2016; Seifert, Gillig, Hanson, Pascarella, & Blaich, 2014).

The most common methods of assessing conditional effects rely on group-level analyses (e.g., introducing interaction terms or conducting subgroup analyses). Yet, these methods do not provide a way to determine whether an experience has had a positive, negative, or neutral effect on an individual student. Examining individual-level change can help researchers and practitioners further understand the complexities of how educational experiences affect students. With this article we aim to build on the work on conditional effects in higher education (e.g., Seifert et al., 2014) to provide a way to assess meaningful individual-level change. We provide a theoretical framework for understanding why educational experiences might lead to positive or negative outcomes; discuss the challenges in assessing individual-level change; describe one method of assessing individual-level change; provide an example of how researchers might use this method to consider positive and negative outcomes for individual students; and discuss how this consideration might change the way we view college experiences.

THEORETICAL FOUNDATIONS

Experiential learning theories are the basis of many educational experiences in higher education; however, Dewey (1938) cautioned that experiences are not automatically educational and can even be "mis-educative." Research indicates that educational experiences do not always lead to positive outcomes (e.g., Kilgo, Sheets, & Pascarella, 2015) and may have the potential for negative outcomes. For example, Becker and Paul (2015) argued that service-learning experiences may reproduce color-blind racism by reinforcing racist stereotypes. Ogden (2006) highlighted that study abroad experiences can similarly reinforce stereotypes of another culture, even when the students perceive that they had a transformative, positive experience.

Despite the potential for experiences to be mis-educative, most research on educational experiences in college reports positive benefits. In a review of 35 studies on out-of-class activities, Simmons, Creamer, and Yu (2017) found that only 6 referred to negative or neutral outcomes. Even scholars who question the benefits of particular experiences often fall short of pointing out the potential for negative outcomes. For example, Kilgo et al. (2015) found that capstone courses and service-learning both had a negative effect on certain educational outcomes. The authors, however, concluded that these "surprising" findings may not mean that these experiences always lead to negative outcomes, but encouraged further study on their effectiveness (p. 522).

The scarcity of literature on negative or neutral effects of educational experiences might be due to a bias against publishing negative results or to a lack of longitudinal research in recent years (Mayhew et al., 2016) or to the fact that many experiences may actually lead to positive gains on average (e.g., Varela, 2017). Averages, however, obscure individual differences and thus do not provide the full picture of the learning that results from an educational experience. For example, in a large-scale study, Vande Berg, Connor-Linton, and Paige (2009) found an overall positive effect of studying abroad on intercultural competence, despite the fact that over a third of the women in the study "showed statistically insignificant intercultural gains or actual decline" (p. 25). Although this is one of the few studies that highlights neutral or negative effects, the researchers conflated "statistically insignificant … gains" with "actual decline...