publisher colophon

Few scholars have comprehensively examined benefits of undergraduate research (UGR) participation for students at an institution campus-wide. In this study we examined benefits of UGR participation at a Hispanic-majority institution using National Survey of Student Engagement data. Generalized estimating equations were used to examine the influence of UGR participation on 5 student outcomes: gains in knowledge and skills, institutional support, overall satisfaction, grade point average, and student–faculty interaction. Results indicate that UGR participation is a robust positive predictor of all 5 outcomes. We provide insights into strategies for enhancing the beneficial impacts of UGR participation, especially for students from underrepresented groups.

Racial and ethnic minorities are underrepresented among bachelor's degree recipients, a disparity that is magnified in graduate degree completion (Kena et al., 2015). This racial/ethnic educational attainment gap is driven in part by a preparation gap that adversely impacts the persistence of students from underrepresented minority groups in higher education (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine [NAS/NAE/IOE], 2010). These unequal benefits of higher education serve to reinforce other social and economic disadvantages experienced by some racial/ethnic minority groups in the United States, which suggests that those disparities will grow in the future without the provision of targeted educational resources (Johnson & Bozeman, 2012). Numerous undergraduate educational intervention programs have been funded in the United States to increase the success of underrepresented minority students in higher education and eventually in the high-skilled labor market.

An approach common to these interventions has been to provide faculty-mentored research experiences for undergraduate students. Few scholars, however, have examined the relationship between undergraduate research (UGR) participation and educational outcomes with large samples, accounting for important individual characteristics, and none have done [End Page 583] so at a minority-serving institution (MSI) or across all undergraduate degree programs at an institution. In this study we examine effects of UGR participation and strategies for increasing impacts of UGR for students from all degree programs at a Hispanic-majority institution (HMI), which is of practical importance. Given the national deficit in high-skilled workers, the global competitiveness of the United States now depends on improving the training of increasing numbers of students from underrepresented minority backgrounds who are flowing into American colleges and universities (Hurtado, Cabrera, Lin, Arellano, & Espinosa, 2009; NAS/NAE/IOE, 2010). Consideration of students across all degree programs in our analysis enables generalization of the effects of UGR participation beyond one discipline or UGR program, while our focus on a MSI enables clarification of the impact of UGR participation on educational outcomes within a context of particular importance to the goal of bolstering America's high-skilled workforce.

LITERATURE REVIEW

Empirical studies of the student developmental benefits of UGR have increased in recent decades, largely due to emerging funding opportunities for UGR. Findings from numerous studies suggest that UGR promotes student success through improved learning (e.g., technical, analytic, critical thinking, and communication skills), retention, persistence to degree completion in their chosen field, professional self-confidence, and preparedness for and plans to pursue graduate study (Finley & McNair, 2013; Ishiyama, 2002; Kardash, 2000; Laursen, Hunter, Seymour, Thiry, & Melton, 2010; Lopatto, 2004, 2007, 2009; Russell, Hancock, & McCullough, 2007; Seymour, Hunter, Laursen, & Deantoni, 2004). Research experiences involving engagement with faculty mentors are conducive to the psychosocial processes by which students develop identities as scholars and persist in high-skilled careers (Hunter, Laursen, & Seymour, 2007; Merolla & Serpe, 2013). Working closely with faculty mentors has been linked to graduate school matriculation, and mentors are critical influences on student success (Tsui, 2007).

The success of underrepresented minority (URM) students has been shown to be influenced by participation in faculty-mentored UGR (Eagan et al., 2013; Jones, Barlow, & Villarejo, 2010; Schultz et al., 2011; Villarejo, Barlow, Kogan, Veazey, & Sweeney, 2008). Research mentoring may be particularly important to promoting URM student success in the many White-male-dominated fields of higher education, where students from traditionally underrepresented backgrounds may feel less like they belong (Fadigan & Hammrich, 2004). Studies of programs targeted toward URMs suggest that UGR increases the academic and social integration of these students through engagement activities, including intensive faculty interactions (Clewell, Cohen, Deterding, & Tsui, 2005). For example, Hurtado et al. (2009) found that URM students engaging in UGR experienced developmental benefits that extended beyond the research setting (e.g., into the classroom) through enhanced self-efficacy; access to key resources (e.g., professional development activities, support from staff); collaborative relationships with peers, graduate students, and faculty; and immersion in competitive social networks that fostered motivation to work harder and perform better.

Since the environments of predominantly White institutions (PWIs) differ from those of MSIs, undergraduate students' research experiences (and outcomes) may vary between those institutional contexts. Due to their marginal inclusion within many PWIs, URM students in those contexts may be more sensitive [End Page 584] than nonminorities to faculty engagement, involvement in campus activities, and access to role models (Anaya & Cole, 2001; Cole & Espinoza, 2008; Fischer, 2007), all of which are components of UGR experiences. While findings indicate that UGR may function especially well in promoting URM student success at PWIs (Jones et al., 2010; Russell et al., 2007), they may not be applicable to MSIs.

Few studies have focused on UGR experiences of URM students at MSIs, and none have focused on examining differential effects of UGR participation by racial/ethnic status on student outcomes at MSIs, controlling for selection bias and other relevant variables. Given that findings regarding enhanced benefits of UGR participation for URM students from existing quantitative studies are applicable to PWIs, there is a gap in knowledge regarding MSIs. Hurtado et al.'s (2009) phenomenological study of how URM students experience UGR across four institutions—including one PWI, two Hispanic-serving institutions (HSIs), and one historically Black university—offers some insight into distinctions in URM students' UGR experiences at PWIs versus MSIs. In terms of factors with potentially negative effects on their academic development, at the PWI, URM students experienced racial stigma via skepticism from peers and faculty regarding their intellectual capabilities, leading to feelings of academic intimidation that placed them at risk to stereotype threat. In contrast, among the MSI-based URM students, none experienced racial stigmatization directly through their engagement in UGR and some benefitted through cultural congruity (e.g., by working with supportive peers and faculty from their own backgrounds), which enhanced their confidence and motivation to excel. Thus, Hurtado et al. (2009) show that URM students engaging in UGR at PWIs and MSIs have different experiences, and that the stigma experienced at PWIs may more directly inhibit their academic development.

In terms of limitations of the literature we address here, while extant studies document how UGR functions to promote student development and how students experience UGR, there is a general need for more comprehensive quantitative research on relationships between UGR and multiple student developmental outcomes, as well as research focused on URM student outcomes at MSIs in particular. Most studies of UGR and student outcomes have limitations associated with small sample size, selection bias, omitted variables, and narrow disciplinary focus (except Eagan et al., 2013; Jones et al., 2010). These limitations create substantial uncertainty in our current knowledge of the effects of UGR. Without rigorous tests using larger data sets to examine the association of UGR participation with student outcomes, adjusting for important individual-level input (i.e., precollege) and college environment characteristics, it is impossible to determine the extent of this relationship. Most prior studies have focused on the impacts of UGR participation for students in specific training programs or majors, and only one to our knowledge has focused on impacts for students campus-wide (Craney et al., 2011). Only a few prior studies have examined racial/ethnic differences in the effects of UGR participation on student outcomes, and none have examined the association between UGR participation (vs. nonparticipation) and student outcomes at MSIs.

CONCEPTUAL MODEL, RESEARCH QUESTIONS, AND HYPOTHESES

We employed the input–environment–output (I-E-O) model to orient the study (Astin & Antonio, 2012). A methodological limitation of all studies focused on the impact of UGR [End Page 585] experiences is that the students are not assigned at random to UGR participant and control groups. The purpose of the I-E-O model is to inform multivariate analyses that control for the effects of differences in initial student attributes and in student exposure to educational environments while in college to minimize the chances of arriving at invalid inferences regarding the influence of particular educational interventions on student developmental outcomes. In other words, the aim is to adjust for potentially biasing variables to accomplish by statistical means what random assignment enables in controlled experiments.

Table 1. Input, Environment and Outcome Measures, Based on the I-E-O Modela
Click for larger view
View full resolution
Table 1.

Input, Environment and Outcome Measures, Based on the I-E-O Modela

In the I-E-O model, outcomes refer to what an institution influences or attempts to influence in terms of student development through its educational programs. From a statistical perspective, outcomes are synonymous with dependent variables. Astin and Antonio (2012) propose examination of outcomes in the following domains: (a) cognitive outcomes–psychological data, (b) affective outcomes–psychological data, (c) cognitive outcomes–behavioral data, and (d) affective outcomes–behavioral data. Whereas cognitive outcome measures are designed to gauge student knowledge and command of higher order mental processes, affective outcome measures gauge students' subjective assessments of their experiences, in terms of their feelings, attitudes, self-concept, aspirations, and social and interpersonal relationships. Psychological data reflect the internal states or traits of students, while behavioral data apply to students' observable activities. Table 1 lists the four domains and the variables we use in each area (Astin & Antonio, 2012).

Inputs denote attributes that students bring initially into the institution's educational programs (e.g., gender, race/ethnicity, academic preparation); these measures typically function as control variables in analyses. The environment refers to students' experiences during their time at the institution; it is operationalized with independent variables (e.g., educational experiences or interventions) that represent influences on student development (Astin & Antonio, 2012). Table 1 lists the input and environment measures we analyze. We selected these input and environment measures because prior research indicates that they influence a wide range of college-level student outcomes such as engagement, satisfaction, grade point average, persistence, and graduation (Bean & Metzner, 1985; Kim & Sax, 2009; Packard, Gagnon, LaBelle, Jeffers, & Lynn, 2011; Pike & Kuh, 2005; Reyes, 2011).

Our analyses address two specific research [End Page 586] questions regarding the role of UGR participation in student outcomes:

  1. 1. Is participation in faculty-mentored UGR positively associated with improved outcomes, adjusting for relevant input and environment variables?

  2. 2. If so, do the associations differ for Hispanic students as compared to non-Hispanic White students?

Based on our literature review, we formulated hypotheses regarding the influence of UGR participation and other relevant influences on student outcomes. The hypothesized influences are operationalized as independent variables in the statistical models, which are described below. In terms of direct effects (Research Question 1), previous studies support the hypothesis that research participation is associated with more successful student outcomes. In addition, they support the hypotheses that better precollege academic achievement, female gender, and full-time enrollment status are associated with more successful outcomes, and, in contrast, that Hispanic status, Black / African American status, working off campus, caring for dependents, reentry student status, first-generation student status, and transfer student status are related to less successful outcomes. In terms of Hispanic ethnicity as an effect modifier of relationships between UGR participation and student outcomes (Research Question 2), we hypothesize that UGR participation has a significantly stronger relationship with more successful outcomes for Hispanic students as compared with non-Hispanic White students. Due to small cell sizes, Black / African American students and those from the other URM groups were not included in interaction effect models.

METHOD

Data

Our university provided National Survey of Student Engagement (NSSE) and student records data. While the NSSE has been widely employed in research on educational outcomes, no prior study has used the NSSE to analyze the benefits of UGR. Our analysis highlights the potential value of the NSSE to research on UGR. The NSSE data set includes 1,739 university seniors who completed the survey as part of five annual samples from 2008 to 2012. For the 2008 and 2009 NSSE administrations, students received a paper version of the survey, with an option of completing a Web-based version. To reduce the potential for nonresponse bias, for the 2010 to 2012 administrations, students were initially invited to participate in a Web-based version via e-mail, and a subgroup of nonrespondents was systematically selected to complete a paper version. Response rates, sampling error, and number of respondents for the samples are as follows: 2008 = 37% (response rate), 7.9% (sampling error), and 158 (number of respondents); 2009 = 39%, 6.8%, 194; 2010 = 32%, 4.2%, 511; 2011 = 27%, 4.0%, 521; and 2012 = 20%, 5.4%, 355. Research indicates that similar survey response rates can yield representative samples (Keeter, Kennedy, Dimock, Best, & Craighill, 2006). Furthermore, the NSSE data collection protocol implemented at our university has been adapted to reduce nonresponse bias.

The resulting samples are generally representative of the university's undergraduate student population. For example, in terms of race/ethnicity, our combined NSSE samples are 84% Hispanic, 10% non-Hispanic White, and 3% non-Hispanic Black / African American. Institutional data reveal comparable figures for the total undergraduate student population, which is 87% Hispanic, 9% non-Hispanic White, and 3% non-Hispanic Black. The individual-level NSSE data were paired with student records data (on precollege educational attainment, race/ethnicity, gender, age, college of major, part-time vs. fulltime [End Page 587] enrollment status, and GPA). Data for all years were pooled together into one sample to provide more statistical power for the analyses. Table 2 provides descriptive statistics for all independent and dependent variables, including means or proportions for categorical variables and standard deviations for scale variables.

Independent Variables: Input

Race/Ethnicity

Using institutional data, the following categorical race/ethnicity measure was constructed: 1 for Hispanic and 0 for non-Hispanic; 1 for Black (non-Hispanic) and 0 for non-Black; 1 for Native American (non-Hispanic) and 0 for non–Native American; and 1 for Asian / Pacific Islander (non-Hispanic) and 0 for non–Asian / Pacific Islander. When employing categories in analyses, we used the reference group of White only (non-Hispanic). Of the sample analyzed, Hispanics compose 84.5%, Blacks 2.9%, Native Americans 0.3%, Asians / Pacific Islanders 1.2%, and Whites 9.7%.

Gender

Institutional data were obtained on student gender. Females were coded as 1, males 0. Higher percentages of females than males enroll and graduate from our university, which is reflected in the sample distribution of females (63%) and males (37%).

First-Generation Student

First-generation status was defined using NSSE guidelines such that a student coded as first generation reported neither a mother nor father having any postsecondary experience (i.e., the highest parental level of education is the completion of high school; Item 27). Students meeting those parental educational attainment criteria were coded 1 as first-generation students and all others were coded 0 as non-first-generation students. Among students in our sample, 39% were first-generation college students.

Reentry Student

Reentry status was calculated by subtracting the institutionally reported birth year from the year of NSSE administration. Individuals 25 years and older were recoded as 1 for reentry, and individuals 24 years and younger were recoded as 0 for traditional. Over half (57%) of students in our sample were reentry students based on this definition.

Transfer Student

This variable was constructed using responses to the NSSE question that asked if the student began his or her education at our university or elsewhere (Item 20). Response options were "Started here" (coded 0 for non–transfer students) or "Started elsewhere" (coded 1 for transfer students); transfer students compose 52% of students in our sample.

High School Percentile

Institutional data on high school percentile, which are employed here as a measure of precollege academic achievement, were obtained. High school percentile is calculated by dividing the student's high school class rank by the class size and then subtracting that term from 1. For example, the high school percentile of a student who ranked 46 in a class of 598 would be 92%. The average high school percentile of seniors in our sample was 75%.

Independent Variables: Environment

UGR Participation

Through the NSSE, students were asked whether they had participated or planned to participate in research with a faculty member outside of a course or other requirements (Item 7d). Likert-type response options were done or in progress, plan to do, do not plan to do, and have not decided. The NSSE has not been employed in prior research testing the effects of UGR participation on educational outcomes. While scholars have not previously systematically examined the validity of this NSSE item in comparison to other measures of UGR participation, researchers at the University of California, Riverside found general correspondence between rates of UGR [End Page 588]

Table 2. Descriptive Statistics for Analysis Variables
Click for larger view
View full resolution
Table 2.

Descriptive Statistics for Analysis Variables

[End Page 589] participation based on measures derived from student self-reports versus institutional records (Rorive, Coyne, & Victorino, 2014). For the UGR participation variable used in our analyses, we recoded UGR participation (done or in progress) as 1 and nonparticipation (which also includes plan to do and have not decided, since our sample included only seniors) as 0. Among seniors in our sample, 21% had participated in UGR.

Working Off Campus

This variable was constructed using responses to the NSSE item related to hours per week worked off campus (Item 9c); see the copyrighted NSSE survey for exact item wording. Response options included 0 hours, 1–5 hours, 6–10 hours, 11–15 hours, 16–20 hours, 21–25 hours, 26–30 hours, and more than 30 hours. Responses were recoded according to NSSE guidelines: The midpoint for each response category was used (e.g., a response of 1–5 hours was recoded as 3), and a response of more than 30 hours was recoded to 31. Students in our sample spent on average nearly 15 hours per week working off campus.

Caring for Dependents

This variable was constructed using responses to the NSSE item that measured hours per week student respondents spent caring for dependents living with themselves (parents, children, spouse, etc.; Item 9f). Response options for hours caring for dependents were identical to working off campus, as was our recoding procedure. Students in our sample spent an average of nearly 20 hours a week caring for dependents.

Full-Time Student

Data on enrollment status were obtained from student records. In correspondence with institutional policy, full-time students (coded as 1) were enrolled in 12 or more course credit hours and those with less than 12 course credit hours were coded as 0. Among individuals in our sample, 67% were full-time students.

Dependent Variables: Outcome

To examine the multidimensional impacts of UGR participation on undergraduate educational outcomes, we analyze five outcomes. These include self-reported gains in knowledge and skills, perceived institutional support, overall satisfaction with the undergraduate experience, grade point average, and student–faculty interaction.

Outcome Domain 1: Gains in Knowledge and Skills

We examine a self-report-based measure of student gains in knowledge and skills because student growth is highly relevant to UGR experiences. There are limitations to self-reported measures of knowledge acquisition documented by recent studies, which highlight the inconsistent correspondence between standardized test measures of college student learning versus student self-reported measures of knowledge/skills gains (Bowman, 2010; Gordon, Ludlum, & Hoey, 2008). Those studies suggest that measures of self-reported gains should not be employed as simple proxies for actual student learning. However, it is important to examine self-reported gains in the context of UGR participation, since subjective aspects of student development that are enhanced through UGR experiences may predict future student success in objective terms. Recent research has documented the importance of UGR to the development of science identity, confidence, motivation, and, ultimately, persistence, especially among URM students (Graham, Frederick, Byars-Winston, Hunter, & Handelsman, 2013). Those subjective gains that accrue through UGR experiences, which enhance persistence in objective terms, are not gauged by the standardized-test measures of learning gains used for comparison purposes in the studies noted above, but are likely reflected in measures of students' self-assessed gains. [End Page 590]

To measure the extent to which students perceive their educational experiences at our university have contributed to their knowledge, skills, and personal development, we selected nine items from the NSSE to provide the basis of a composite variable. Students were asked about the extent to which their experience at the university contributed to their development in the following areas: (a) critical thinking (Item 11e), (b) computing and technology (Item 11g), (c) learning on your own (Item 11j), (d) working with others (Item 11h), (e) solving problems (Item 11m), (f) analyzing quantitative issues (Item 11f), (g) speaking skills (Item 11d), (h) job skills and knowledge (Item 11b), and (i) writing skills (Item 11c). Response options were very little, some, quite a bit, and very much and were recoded as per NSSE (0, 33.33, 66.67, and 100, respectively). A composite of gains in knowledge and skills was calculated (Cronbach's α = .902), which combines responses to the nine items, such that higher scores indicate greater knowledge and development in the student experience.

Outcome Domain 2: Institutional Support

To gauge students' perceptions of the academic and nonacademic support they had received at our university, we selected two items from the NSSE (providing you with the support you need to succeed academically [Item 10b] and helping you cope with your nonacademic duties (work, family, etc.; Item 10d). Response options were very little, some, quite a bit, and very much and were recoded as per NSSE (0, 33.33, 66.67, and 100, respectively). A composite of institutional support was constructed (Cronbach's α = .656), which combines recoded responses to the two items, such that higher scores indicate greater support.

Outcome Domain 2: Overall Satisfaction

We selected one item from the NSSE that gauges students' overall satisfaction with their entire educational experience at our university (Item 13). Response options were poor, fair, good, and excellent and were recoded as per NSSE (0, 33.33, 66.67, and 100, respectively). Higher scores indicate greater overall satisfaction.

Outcome Domain 3: Grade Point Average (GPA)

As a measure of academic performance, we employed each NSSE senior respondent's GPA. GPA is reported on a 4.0 scale. Institutionally reported cumulative GPA data were provided to us with no missing data for any students in the sample. See Table 2 for descriptive statistics for all analysis measures.

Outcome Domain 4: Student–Faculty Interaction

We selected four items (from the NSSE) to gauge the extent to which students interacted with faculty members (i.e., received prompt feedback from faculty [Item 1q]; discussed ideas with faculty members outside of class [Item 1p]; worked on activities not related to coursework with a faculty member outside of class [Item 1s]; and talked about career plans with a faculty member [Item 1o]). Likert-type response options were never, sometimes, often, and very often and were recoded as per NSSE (0, 33.33, 66.67, and 100, respectively). A composite of student–faculty interaction was created (Cronbach's α = .733), which combines recoded responses to the four items, such that higher scores indicate more student engagement with faculty.

Analyses

To reduce nonresponse bias, the missing values of all analysis variables were multiply imputed. Multiple imputation (MI) involves creating multiple sets of values for missing observations using a regression-based approach and is currently considered a best practice for addressing missing data in statistical analysis (Enders, 2010). Using IBM® SPSS (version 20) statistical software, 20 imputed data sets were specified to increase power and 200 between-imputation iterations were used (Enders, 2010). The percentage missing for the variables ranged from 0.0% (e.g., grade point average) [End Page 591] to 48.6% (high school percentile; highest percentage missing; see Table 2). There is a relatively high proportion of missing data for high school academic achievement because our institution is by mission open-access, where admission/enrollment requirements are limited to high school graduation or equivalency; thus, the institution lacks data on precollege academic achievement for many students.

Multiply imputed data were used to calculate bivariate correlations to clarify basic relationships between the explanatory and dependent variables. Then, multiply imputed data analyses were performed using a series of generalized estimating equations (GEEs) with robust covariance estimates, which model the independent variables as predictors of undergraduate student educational outcomes. GEEs extend the generalized linear model (Nelder & Wedderburn, 1972) to provide a general method for the analyses of clustered response variables, and relax several assumptions of traditional regression models (Diggle, Heagerty, Liang, & Zeger, 2002). For our purposes, GEEs are preferable to other modeling approaches that account for nonindependence of data (e.g., hierarchical linear models). This is because GEEs estimate unbiased regression coefficients, even with misspecification of the correlation structure when using a robust variance estimator (Liang & Zeger, 1986). In addition, because our focus is on predictors of educational outcomes at the individual level, not on higher level (e.g., institutional) effects, GEEs are most appropriate because the intracluster correlation estimates are adjusted for as a nuisance and not modeled (as in hierarchical linear models).

The GEEs assume that observations from within a cluster are correlated, while observations from different clusters are independent. GEEs with clusters defined based on the college of major of the student were used in the final models presented here (Table 2). Accounting for this level of clustering was necessary because funded UGR programming varies dramatically by college, as do rates of student participation in UGR. For example, while 41% of science students participate in UGR, only 7% of business administration students do.

GEEs imply no strict distribution assumptions for independent variables and are appropriate for use with non-normally distributed outcome variables, which is the case here. Gamma distributions with a logarithmic or an identity link (as applicable) function were specified for each GEE, because the dependent variables are composed of positive scale values skewed toward larger positive values. To select the best fitting model, we ran all GEEs with both logarithmic and identity link functions, using quasi-likelihood under independence criterion (QIC) goodness-of-fit coefficients to select the link function that gave the best model fit (Garson, 2013). While the QIC allows the analyst to select the best fitting model, it does not indicate the proportion of variance explained. For our analyses, the exchangeable correlation structure specification was selected as it assumes constant intracluster dependency, and when there is no logical ordering to the observations, the exchangeable matrix is recommended (Diggle et al., 2002). Specifications used for each GEE are summarized in the "Model Spec." row of Table 3.

We estimated GEEs to analyze the main effects of the predictor variables on educational outcomes (i.e., gains in knowledge and skills, institutional support, overall satisfaction, GPA, and student–faculty interaction) to address Research Question 1. We also estimated GEEs by adding interaction terms for Hispanic status and UGR participation to address Research Question 2. According to variance inflation factor, tolerance, and condition index criteria, inferences from the GEEs are not influenced by multicollinearity problems. [End Page 592]

Table 3. Generalized Estimating Equations Predicting Student Outcomes (N = 1,739)
Click for larger view
View full resolution
Table 3.

Generalized Estimating Equations Predicting Student Outcomes (N = 1,739)

[End Page 593]

Sensitivity Analyses

We conducted four sensitivity analyses of the effects of UGR participation on the suite of educational outcomes by estimating GEEs with (a) the high school percentile variable excluded from the analysis of the multiply imputed data sets, since it contains a relatively high proportion of missing data; (b) students nested within their majors instead of colleges, to examine whether results are sensitive to more specific definitions of disciplinary clustering; (c) students nested in their year of NSSE administration (2008–2012) and their colleges, to examine whether results are sensitive to annual cohort effects; and (d) students with any missing data excluded, to examine whether results are sensitive to the MI approach.

RESULTS

Table 3 provides GEE estimates of relationships between UGR participation and each of the educational outcomes for the entire sample, adjusting for the effects of relevant input and environment variables. UGR participation exhibits significant and positive associations with all outcomes based on the GEE estimates, such that seniors who had engaged in UGR self-reported greater frequency of meaningful interactions with faculty members, greater academic and nonacademic campus support, greater gains in knowledge and skills from their undergraduate experiences, and higher levels of satisfaction with their overall educational experiences. In addition, UGR participation predicted higher GPA, such that students who had engaged in UGR had accumulated higher grades throughout their undergraduate careers.

Beta coefficients (under the B columns in Table 3) indicate the relative strength of the predictors for each of the five GEEs. Using the result for research participation in the GEE predicting student–faculty interaction as an example, the coefficient of 0.35 is larger than the coefficients for other predictors that exhibit significant associations with student–faculty interaction, indicating that research participation is a relatively stronger predictor than other independent variables in that model. Note that the effect sizes of the coefficients for the independent variables are comparable within an individual GEE but not between GEEs.

It is also important to consider the size of the effect of UGR participation on the five student outcomes. The only model reported in Table 3 that uses an identity link function to predict the dependent variable linearly is institutional support, since the identify link function yielded the best fitting GEE in that case. Note the coefficient for research participation is 7.79, which indicates the following: if a student had (vs. had not) participated in faculty-mentored research, the difference in the institutional support score would be expected to increase by 7.79 units (on a 100-point scale), while holding other variables constant. The effect sizes of the coefficients in the other four GEEs, which fit best when specified with logarithmic link functions, are interpretable only relative to the natural log of the dependent variables. If those GEEs are instead predicted linearly, then research participation is associated with point increases of 15.78, 6.12, and 4.15 respectively (on 100.00-point scales), in student–faculty interaction, gains in knowledge and skills, and overall satisfaction, and with a 0.32-point increase in GPA (on a 4.00-point scale). Those estimates indicate that UGR participation has statistically and practically significant effects across the five educational outcomes.

In terms of input variables, results indicate that students with higher high school percentile scores had significantly fewer interactions with faculty, but had significantly higher [End Page 594] GPAs, net of the effects of other variables. Hispanic students had significantly greater academic and nonacademic campus support and significantly greater gains in knowledge and skills, but maintained significantly lower GPAs. Black students maintained significantly lower GPAs. First-generation college students had higher overall satisfaction with their undergraduate educational experiences. Reentry students had significantly lower GPAs. Students who transferred from other institutions had significantly lower gains in knowledge/skills and lower GPAs. In terms of environment variables, students who spent more time working off campus reported lower overall satisfaction with their educational experiences and had lower GPAs. Students who spent more time caring for dependents reported significantly greater interactions with faculty, more institutional support, greater gains in knowledge/skills, and higher overall satisfaction with their educational experiences. In addition, full-time students reported greater interactions with faculty members.

In a second GEE series (results not shown), we added interaction terms for Hispanic status by UGR participation to each of the five models. The interaction term was significant in the model predicting gains in knowledge and skills (p = .028), and approached significance in the model predicting institutional support (p = .057). This indicates that UGR is associated with significantly lower self-reported gains in knowledge and skills and nearly significantly less perceived institutional support among Hispanic students as compared to non-Hispanic White students. For Hispanic students, UGR participation is positively associated with gains in knowledge and skills and institutional support; however, the positive effects of UGR participation on those outcomes are significantly greater among non-Hispanic White students.

Results of the four sensitivity analyses (results not shown) of the effects of UGR participation are as follows: In GEE models that (a) exclude the high school percentile variable, (b) nest students within their majors instead of colleges, (c) nest students within their year of survey administration and college, and (d) exclude all students with any missing data, UGR participation is found to predict significantly more positive student outcomes across all dependent variables. Thus, the effect of UGR participation on positive student outcomes is robust across a range of model specifications.

DISCUSSION

This study is one of a few to examine benefits of UGR participation for students at a MSI, and the only one that has done so using an approach that enables examination of associations between UGR participation and student outcomes, accounting for salient input and environment variables. In response to Research Question 1, results indicate that UGR participation was a statistically significant and positive predictor of the five outcomes; sensitivity analyses results provide evidence for the robustness of that relationship. None of the input or other environment variables significantly predicted as many of the outcomes as did UGR participation. In sum, UGR participation was the most consistent predictor of improved student outcomes across the models.

There are two hypotheses derived from the literature directly relevant to Research Question 2, both of which point toward UGR experiences imparting greater benefits to URM versus non-Hispanic White students. First, studies indicate that URM students may benefit more from UGR due to the increased barriers they confront to academic and social integration within institutions of higher education and, concomitantly, to their increased sensitivity to engagement with [End Page 595] faculty mentors/role models and campus life more generally via UGR experiences (Jones et al., 2010; Lopatto, 2004; Russell et al., 2007). Note that this hypothesis is based largely on the experiences of URM students at PWIs. Second, the results of Hurtado et al. (2009) suggest that URM students at MSIs as opposed to PWIs stand to receive greater developmental benefits via UGR engagement due to experiences occurring within their UGR training environments, wherein they have a greater likelihood of experiencing cultural congruity at MSIs vs. racial stigmatization (and academic intimidation) at PWIs. Note, however, that Hurtado et al. (2009) focused only on the experiences of URM student participants in UGR programs, and their findings provide little insight into the developmental deficits experienced by URM and nonminority students alike who do not engage in faculty-mentored UGR (relative to their counterparts who do participate in UGR) in the contexts of PWIs or MSIs.

Our GEE interaction results revealed that associations between participation in UGR and two of five educational outcomes differed for Hispanic as compared to non-Hispanic White students. Contrary to expectations, those interaction results suggest that UGR participation was less beneficial to Hispanic students compared to non-Hispanic White students at this HMI, in terms of enhancing gains in knowledge/skills and perceptions of institutional support. Statistically nonsignificant interaction term results in models for the other three educational outcomes suggest that Hispanic and non-Hispanic White students experienced commensurate benefits from their UGR experiences. Lopatto (2004) found something similar: no racial/ethnic differences in the effects of UGR on satisfaction with the overall UGR experience or on plans to pursue postgraduate education. Thus, in terms of any differences in the effects of UGR on student outcomes based on racial/ethnic status, counterintuitively, participation in UGR appears to have conferred some enhanced educational benefits to non-Hispanic White rather than Hispanic students this HMI.

This finding contradicts the hypotheses in the literature, which has not examined MSIs. There are a few plausible explanations for this counterintuitive finding, which suggest avenues for future research. First, MSIs tend to be highly successful in promoting academic success for URM students, irrespective of their participation in UGR (Flowers, 2002). While there are fewer studies of the role that HSIs (or HMIs) play in promoting educational outcomes for Hispanic students (compared to studies of HBCUs and Black students; Allen, 1992; Outcalt & Skewes-Cox, 2002), like HBCUs, HSIs tend to promote more inclusive campus environments, high levels of student engagement, and a sense of belonging, which serve to increase the cultural congruity between Hispanic students and the institution (Abraham, Lujan, López, & Walker, 2002). Thus, Hispanic students at HSIs, and especially HMIs, face reduced barriers to academic and social integration as compared to Hispanic students at PWIs. This suggests that the enhanced benefits of UGR participation for Hispanic students at PWIs are less likely to be observed at HSIs or HMIs, where academic and social integration may be easier for Hispanic students, including those who do not participate in UGR. Indeed, our results for the main effects of Hispanic status on the student outcomes suggest that Hispanic students experience greater student–faculty interaction, institutional support, gains in knowledge and skills, and overall satisfaction at this HMI than White students, controlling for UGR participation (Table 3). In contrast, at PWIs, UGR experiences may be more critical to academic and social integration [End Page 596] for URM students.

Second, Hurtado et al. (2009) suggest that the racial stigmatization and academic intimidation experienced by URMs at PWIs—both outside and within faculty-mentored UGR training environments—create discriminating pressures that exclude all but the grittiest and most highly motivated URM students from UGR participation. It is plausible that URM students selected for UGR participation at PWIs are those who are primed to experience the greatest growth relative to all other student groups. More research is needed to enhance understanding of the differential effects of UGR on developmental outcomes by race/ethnicity for students across varied institutional contexts (i.e., MSIs and PWIs).

In terms of practical relevance, given the strong association between UGR participation and improved student developmental outcomes we found across one MSI, expanding access for minority students to UGR experiences nationwide may be an effective way of helping close the racial/ethnic higher educational achievement gap in the United States. The National Institutes of Health is making a large investment in enhancing access to UGR opportunities for students from traditionally underrepresented groups through the Building Undergraduate Infrastructure Leading to Diversity (BUILD) initiative. MSIs hold potential for scaling up UGR opportunities while enhancing participation rates among URMs. However, there are challenges to achieving the goals of such initiatives at MSIs. Not all students have equal opportunities to take part in faculty-mentored research (Finley & McNair, 2013). At a national level, URM students participate substantially less in UGR than non-Hispanic Whites. While Hispanic, Black, and Native American undergraduate students have UGR participation rates of 19%, 17%, and 21%, respectively, the UGR participation rate for White students is 24% (NSSE, 2013). Unless programs are well-targeted to meet the needs of URM students, particularly those at resource-strapped MSIs, the existing racial/ethnic higher educational attainment gap will likely expand in the future, regardless of the size of the investment in UGR programs.

There are several approaches that institutions of higher education, and MSIs in particular, may implement to increase participation in UGR. First, URM students face specific barriers to engaging in UGR because they are more likely to be economically disadvantaged, have off campus work demands and caretaking responsibilities, and be first-generation college students (Finley & McNair, 2013). Targeted resources and interventions—including student stipends, tuition/fees scholarships, daycare support, UGR bridge programs to support transfer students, and engagement activities with parents/family members to promote UGR—must be applied to help students surmount these specific barriers (Johnson & Bozeman, 2012). Second, UGR opportunities in US universities are typically offered to upper-division students, yet there is evidence that students (especially URMs) benefit most from UGR during their freshman and sophomore years (Graham et al., 2013). Thus, funding organizations must be called on to provide resources to an array of universities—and MSIs should be prioritized—to expand UGR engagement via the development of course-based research training opportunities at the early undergraduate level. Third, UGR opportunities in U.S. universities are concentrated in the natural sciences (NSSE, 2013), and there is a relative scarcity of funded UGR opportunities in the sociobehavioral sciences and education. This is despite the fact that students across all degree programs stand to benefit from UGR opportunities, as our findings indicate. To broaden the impacts of UGR training, funded UGR opportunities need to be expanded to include a broader range of academic fields. [End Page 597]

CONCLUSION

This is the first comprehensive quantitative study of the association between UGR and student success across all degree programs at an institution, adjusting for input and environment variables known to influence educational outcomes. It is also one of a few studies of UGR benefits to address selection bias by including an institutionally representative sample of students across all fields, some of whom had participated in UGR compared to others who had not. While students were not randomly assigned to UGR participant (treatment) and non-UGR participant (control) groups, our statistical models included a suite of variables that are known to influence student participation in UGR and educational outcomes. This enabled us to address selection biases in a manner achieved by few studies of the role of UGR participation in college student development.

This study has limitations that should be addressed in future research. First, we examined one MSI with open-access admissions criteria. While the institutional context differs from others, through our analysis approach we effectively isolated the effects of UGR on student outcomes by controlling for input and environment variables. Thus, our findings for the positive effects of UGR on a range of student outcomes are robust and, we hypothesize, generalizable to other institutional contexts. Second, there is an unknown degree of inaccuracy in our self-reported measure of UGR participation; moreover, this measure treats a multifaceted experience as a dichotomous phenomenon, which bypasses issues of timing, duration, and quality as well as more specific features that characterize UGR experiences. We lack access to comparable data on such features of UGR experiences campus-wide. As Linn, Palmer, Baranger, Gerard, and Stone (2015) observed, "The field needs agreed-upon criteria for undergraduate research experiences and validated, generalizable assessments for these criteria" (p. 5). The development of improved, validated measures of UGR experiences in tandem with future data collection efforts could enhance understanding of the role of UGR experiences in students' educational and career outcomes. [End Page 598]

Timothy W. Collins

Timothy W. Collins is Professor of Geography

Sara E. Grineski

Sara E. Grineski is Professor of Sociology at the University of Texas at El Paso.

Jessica Shenberger

Jessica Shenberger is a doctoral candidate of Psychology at the University of Texas at El Paso.

Xiaodan Morales

Xiaodan Morales is a postdoctoral research fellow in the BUILDing SCHOLARS Center at the University of Texas at El Paso.

Osvaldo F. Morera

Osvaldo F. Morera is Professor of Psychology at the University of Texas at El Paso.

Lourdes E. Echegoyen

Lourdes E. Echegoyen is Director of the Campus Office of Undergraduate Research Initiatives and of the BUILDing SCHOLARS Center at the University of Texas at El Paso. Research supported by the National Institute on Minority Health and Health Disparities of the National Institutes of Health (P20MD008700) as well as the National Institute of General Medical Sciences of the National Institutes of Health (RL5GM118969, UL1GM118970, and TL4GM118971). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors gratefully acknowledge Roy Mathew, Cathe Lester, and Denise Carrejo from UTEP's Center for Institutional Evaluation, Research and Planning, who provided the NSSE and student records data.

Correspondence concerning this article should be addressed to Timothy W. Collins, Department of Sociology & Anthropology, University of Texas at El Paso, 500 W. University Ave., El Paso, TX 79902; twcollins@utep.edu

REFERENCES

Abraham, J., Lujan, S., López, E., & Walker, M. (2002). Graduating students' perceptions of outcomes of college experiences at a predominantly Hispanic university. Journal of Hispanic Higher Education, 1, 267-276.
Allen, W. (1992). The color of success: African American college student outcomes at predominantly White and historically Black public colleges and universities. Harvard Education Review, 62, 26-44.
Anaya, G., & Cole, D. (2001). Latina/o student achievement: Exploring the influence of student–faculty interaction on college grades. Journal of College Student Development, 42, 3-14.
Astin, A., & Antonio, A. (2012). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. Lanham, MD: Rowman & Littlefield.
Bean, J., & Metzner, B. (1985). A conceptual model of nontraditional undergraduate student attrition. Review of Educational Research, 55, 485-540.
Bowman, N. (2010). Can 1st-year college students accurately report their learning and development? American Educational Research Journal, 47, 466-496.
Clewell, B., Cohen, C., Deterding, N., & Tsui, L. (2005). Final report on the evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation program: Full technical report and appendices. Retrieved from http://www.urban.org/publications/411301.html
Cole, D., & Espinoza, A. (2008). Examining the academic success of Latino students in science, technology, engineering, and mathematics (STEM) majors. Journal of College Student Development, 49, 285-300.
Craney, C., McKay, T., Mazzeo, A., Morris, J., Prigodich, C., & de Groot, R. (2011). Cross-discipline perceptions of the undergraduate research experience. Journal of Higher Education, 82, 92-113.
Diggle, P., Heagerty, P., Liang, K., & Zeger, S. (2002). Analysis of longitudinal data (2nd ed.). Oxford, England: Oxford University Press.
Eagan, M., Hurtado, S., Chang, M., Garcia, G., Herrera, F., & Garibay, J. (2013). Making a difference in science education for underrepresented students: The impact of undergraduate research programs. American Educational Research Journal, 50, 683-713.
Enders, C. (2010). Applied missing data analysis. New York, NY: Guilford.
Fadigan, K., & Hammrich, P. (2004). A longitudinal study of the educational and career trajectories of female participants of an urban informal science education program. Journal of Research in Science Teaching, 41, 835-860.
Finley, A., & McNair, T. (2013). Assessing underserved students' engagement in high-impact practices. Washington, DC: Association of College and Universities.
Fischer, M. (2007). Settling into campus life: Differences by race/ethnicity in college involvement and outcomes. Journal of Higher Education, 78, 125-161.
Flowers, L. (2002). The impact of college racial composition on African American students' academic and social gains: Additional evidence. Journal of College Student Development, 43, 403-410.
Garson, G. (2013). Generalized linear models and generalized estimating equations. Asheboro, NC: Statistical Associates.
Gordon, J., Ludlum, J., & Hoey, J. (2008). Validating NSSE against student outcomes: Are they related? Research in Higher Education, 49, 19-39.
Graham, M., Frederick, J., Byars-Winston, A., Hunter, A., & Handelsman, J. (2013, September 27). Science education: Increasing persistence of college students in STEM. Science, 341(6153), 1455-1456.
Hunter, A., Laursen, S., & Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students' cognitive, personal, and professional development. Science Education, 91, 36-74.
Hurtado, S., Cabrera, N., Lin, M., Arellano, L., & Espinosa, L. (2009). Diversifying science: Underrepresented student experiences in structured research programs. Research in Higher Education, 50, 189-214.
Ishiyama, J. (2002). Does early participation in undergraduate research benefit social science and humanities majors? Journal of College Student Development, 36, 380-386.
Johnson, J., & Bozeman, B. (2012). Perspective: Adopting an asset bundle model to support and advance minority students' careers in academic medicine and the scientific pipeline. Academic Medicine, 87, 1488-1495.
Jones, M., Barlow, A., & Villarejo, M. (2010). Importance of undergraduate research for minority persistence and achievement in biology. Journal of Higher Education, 81, 82-115.
Kardash, C. (2000). Evaluation of an undergraduate research experience: Perceptions of undergraduate interns and their faculty mentors. Journal of Educational Psychology, 92, 191-201.
Keeter, S., Kennedy, C., Dimock, M., Best, J., & Craighill, P. (2006). Gauging the impact of growing nonresponse on estimates from a national RDD telephone survey. Public Opinion Quarterly, 70, 759-779.
Kena, G., Musu-Gillette, L., Robinson, J., Wang, X., Rathbun, A., Zhang, J., . . . Dunlop Velez, E. (2015). The condition of education 2015. Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Kim, Y., & Sax, L. (2009). Student–faculty interaction in research universities: Differences by student gender, race, social class, and first-generation status. Research in Higher Education, 50, 437-459.
Laursen, S., Hunter, A., Seymour, E., Thiry, H., & Melton, G. (2010). Undergraduate research in the sciences: Engaging students in real science. San Francisco, CA: Jossey-Bass.
Liang, K., & Zeger, S. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73, 13-22.
Linn, M., Palmer, E., Baranger, A., Gerard, E., & Stone, E. (2015, February 6). Undergraduate research experiences: Impacts and opportunities. Science, 347(6222). doi:10.1126/science.1261757
Lopatto, D. (2004). Survey of Undergraduate Research Experiences (SURE): First findings. CBE—Life Sciences Education, 3, 270-277.
Lopatto, D. (2007). Undergraduate research experiences support science career decisions and active learning. CBE—Life Sciences Education, 6, 297-306.
Lopatto, D. (2009). Science in solution: The impact of undergraduate research on student learning. Tucson, AZ: Research Corporation for Science Advancement. [End Page 599]
Merolla, D., & Serpe, R. (2013). STEM enrichment programs and graduate school matriculation: The role of science identity salience. Social Psychology of Education, 16, 575-597.
National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. (2010). Rising above the gathering storm, revisited: Rapidly approaching category 5. Washington, DC: National Academies Press.
National Survey of Student Engagement. (2013). A fresh look at student engagement—Annual results 2013. Bloomington: Indiana University Center for Postsecondary Research. Retrieved from http://nsse.iub.edu/NSSE_2013_Results/pdf/NSSE_2013_Annual_Results.pdf
Nelder, J., & Wedderburn, R. (1972). Generalized linear models. Journal of the Royal Statistical Society, Series A, 135, 370-384.
Outcalt, C., & Skewes-Cox, T. (2002). Involvement and satisfaction: The human environment at HBCUs. Review of Higher Education, 25, 331-347.
Packard, B., Gagnon, J., LaBelle, O., Jeffers, K., & Lynn, E. (2011). Women's experiences in the STEM community college transfer pathway. Journal of Women and Minorities in Science and Engineering, 17, 129-147.
Pike, G., & Kuh, G. (2005). First-and second-generation college students: A comparison of their engagement and intellectual development. Journal of Higher Education, 76, 276-300.
Reyes, M. (2011). Unique challenges for women of color in STEM transferring from community colleges to universities. Harvard Educational Review, 81, 241-263.
Rorive, V., Coyne, G., & Victorino, C. (2014). University of California, Riverside: Undergraduate research tracking report. Retrieved from http://ueeval.ucr.edu/undergraduateresearchtrackingreport_2014.pdf
Russell, S., Hancock, M., & McCullough, J. (2007, April 27). Benefits of undergraduate research experiences. Science, 316(5824), 548-549. doi:10.1126/science.1140384
Schultz, P., Hernandez, P., Woodcock, A., Estrada, M., Chance, R., Aguilar, M., & Serpe, R. (2011). Patching the pipeline: Reducing educational disparities in the sciences through minority training programs. Educational Evaluation and Policy Analysis, 33, 95-114.
Seymour, E., Hunter, A., Laursen, S., & Deantoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education, 88, 493-534.
Tsui, L. (2007). Effective strategies to increase diversity in STEM fields: A review of the research literature. Journal of Negro Education, 76, 555-581.
Villarejo, M., Barlow, A., Kogan, D., Veazey, B., & Sweeney, J. (2008). Encouraging minority undergraduates to choose science careers: Career paths survey results. CBE—Life Sciences Education, 7, 394-409. [End Page 600]

Additional Information

ISSN
1543-3382
Print ISSN
0897-5264
Pages
583-600
Launched on MUSE
2017-06-26
Open Access
No
Back To Top

This website uses cookies to ensure you get the best experience on our website. Without cookies your experience may not be seamless.