Russell Sage Foundation

Hiring is an opportunity for school districts to find educators with values and beliefs that align with district goals. Yet beliefs are difficult to measure. We use administrative data from more than ten thousand applications to certificated positions in an urban California school district in which applicants submitted essays about closing achievement gaps. Using structural topic modeling (STM) to code these essays, we examine whether applicants systematically differ in their use of these themes and whether themes predict hiring outcomes. Relative to white applicants, Hispanic and African American applicants are more likely to identify structural causes of inequities and discuss educators' responsibilities for addressing inequality. Similar differences in themes emerge between applicants to schools with different student populations. Techniques like STM can decipher hard-to-measure beliefs from administrative data, providing valuable information for hiring and decision making.

Keywords

teacher hiring, structural topic modeling, educational equity

Schools across the country serve an increasingly diverse student body, and California's schools lead the nation in diversity. Currently some 75 percent of public school students in California identify as a race-ethnicity other than white and nearly 43 percent speak a language other than English at home (California Department of Education 2016a, 2016b). Yet these students have not had equal educational opportunities or successes. Despite repeated [End Page 103] efforts to equalize educational experiences and outcomes for students, school segregation and racial achievement gaps persist and economic achievement gaps have widened (Harris and Herrington 2006; Reardon 2011; Reardon and Bischoff 2011; Reardon and Owens 2014).

Hiring presents school districts with an opportunity to identify educators who have not only relevant experience, but also the beliefs, attitudes, and pedagogical skills that enable them to promote engagement and achievement among students with varied experiences and backgrounds. However, districts often struggle to identify candidates with the characteristics they most desire, and identifying candidates whose beliefs align best with district policy strategies and goals is challenging. Although recent evidence suggests that well-developed applications and interview processes can provide valuable insights into applicant quality, such hiring methods can be cost prohibitive for many districts (Jacob et al. 2016; Rockoff et al. 2011).

This article examines how educator applicants address issues of equity and diversity in their application materials and considers how such beliefs and attitudes influence application decisions and hiring outcomes. We use seven years of unique essay data from more than ten thousand applications to our partner district in California to examine how educator applicants express their beliefs and how the district evaluates them in its hiring process.1 Because equity is central to the district's mission, it explicitly asks all applicants to write a short essay discussing how they would address the achievement gap in their classrooms if hired. We use structural topic modeling (STM) to code the ten most common essay themes from applicant responses to this prompt. We describe variation in the prevalence of each essay theme across educators and contrast response themes by educator race. We then use regression analyses to test whether applicants that articulate the most common beliefs, values, and strategies about how to address achievement inequities apply to schools with more disadvantaged students, rate more highly in application review, and are ultimately more likely to be hired by the district.

Educator short-essay responses cover a wide range of topics, some expressing more general sentiments about the achievement gap and others describing specific strategies and beliefs about how they would address it. Essay content varies by race-ethnicity: relative to white applicants, African American and Hispanic applicants focus more on naming structural causes of inequities facing students, advocating solutions rooted in educators' responsibilities to address inequity and their own biases and prioritizing cultural and linguistic diversity; Asian applicants focus more on building supportive classrooms and family and community engagement. Several essay themes differentially predict applicants' rubric scores on evaluations of their applications and the likelihood that the district hires them. Relative to differentiated instruction, a common theme, applicant essays that describe experience with cultural and linguistic diversity and educators' responsibilities to address inequities receive the highest rubric scores and are associated with the highest likelihood of being hired in the district. Further, several themes are positively related to applying to and being hired in schools with the largest populations of traditionally underserved students—particularly Hispanic students and English-language learners. These results are robust to a variety of model specifications, including those that control for the writing quality of the essays, essay sentiment, educator demographics, [End Page 104] experience, and credentials, and fixed effects for the specific jobs to which applicants applied.

Structural topic modeling techniques are a promising tool for examining administrative data from school districts as well as other settings, particularly qualitative and long-form data that are typically very resource-intensive to analyze. In this application and elsewhere, STM can help examine hard-to-measure practices and beliefs, providing more information at the point of hire and informing decision making in other areas.

background and motivation

Although somewhat diminished in recent decades, racial achievement gaps remain an enduring feature of schools nationwide (Gamoran 2001; Murphey 2014; Reardon 2011). Economic achievement gaps, meanwhile, are expanding markedly, especially between those in the middle of the distribution and their highest income peers (Reardon 2011). School segregation remains entrenched and perpetuates unequal learning opportunities along racial-ethnic and economic lines (Reardon and Owens 2014; Reardon and Bischoff 2011). Our society expects educators to address these persistent challenges and to create classrooms that provide opportunities for all students. However, because we struggle to agree on the goals of schooling, we have no singular blueprint for how to do so (Labaree 1997).

Instead, teachers have a variety of goals for their teaching, and some may not prioritize reducing inequality (Rimm-Kaufman et al. 2006). If educators in some schools think addressing inequality is a key priority of their work but their counterparts in other schools do not, they might create very different learning environments for their students, further exacerbating inequality. Educators also have differing levels of comfort and experience in planning and enacting solutions to address inequality in their work. Understanding this variation is important because teachers' beliefs are an important indicator of what they are likely to do in their classrooms (Buehl and Beck 2015; Pajares 1992; Wilkins 2008; Opfer and Pedder 2011). Despite the central role we expect educators to play in addressing inequality, we know little about what educators think about inequality and how they should work to address it.

We need a better understanding of what educators think about inequality because educators' identities and beliefs are consequential for learning environments, opportunities, and student outcomes, particularly for marginalized students. We know that teachers' identities shape students' schooling experiences and outcomes. For example, teacher-student race and gender congruence positively affects student achievement, grades, their academic perceptions and attitudes, and discipline outcomes (Dee 2004; Egalite and Kisida 2018; Egalite, Kisida, and Winters 2015; Fox 2016; Gershenson et al. 2017; Lindsay and Hart 2017). These effects are often particularly large for black students. The mechanisms driving these effects—typically presumed rather than empirically tested—are that same-race or same-gender educators not only offer representation but also convey values and attitudes that recognize and affirm the unequal experiences of students from marginalized backgrounds.

Underneath demographic similarities, educators have a wide variety of experiences and knowledge that ultimately shape their beliefs about their work and their students. Teacher beliefs are an integral part of teacher practice, learning, development, and identity (Hollingsworth 1989; Opfer and Pedder 2011; Smagorinsky et al. 2004). Teachers' perceptions affect how they treat their students, the types of help and support they provide, and teachers' empathy toward students (Calarco 2011; Ferguson 2003; Okonofua, Paunesku, and Walton 2016). Teacher perceptions, expectations, and stereotypes can influence disciplinary practices and students' later course trajectories and achievement, contributing to the growth of achievement gaps (Ferguson 2003; Okonofua and Eberhardt 2015; Baker et al. 2015; Alvidrez and Weinstein 1999). Teachers' beliefs about overcoming students' social disadvantage influence student achievement (Rochmes 2018). Moreover, teachers' expectations for student ability and achievement matter most for the outcomes of students from disadvantaged groups (Dee 2005; Downey and Pribesh 2004; Ferguson 2003; McGrady and Reynolds 2013; Rist 1970).

Understanding what educators think about [End Page 105] how to address inequality is important not only because of how beliefs influence educator practice, but also because if educators' goals are misaligned with school or district goals, policy solutions to reduce inequality are unlikely to work. Teachers can promote district goals but can also impede and even derail them if they are not well aligned with the goals of individual teachers or the collective goals of their instructional teams (Coburn 2001, 2004; Coburn, Hill, and Spillane 2016; Golann 2018; Spillane 1999).

Because educators play such an important role in carrying out district priorities, the alignment between district goals and educators' goals is an important consideration for educator hiring. Once we know what educators think about how to address inequality, we can also examine how well aligned these beliefs are with district priorities. Districts might use such data in a range of human resource decisions, including hiring. For districts that are particularly concerned with reducing opportunity and achievement gaps for diverse students, hiring presents an important opportunity to identify educators equipped to meet the needs of marginalized students.

In response to national priorities and district interests, many districts prioritize equity-focused beliefs in their recruitment, demonstrating this commitment by prioritizing the diversification of their teaching force (Bireda and Chait 2011; Brown and Boser 2017; Villegas and Irvine 2010). Despite concerted efforts to increase educator diversity through recruitment, training, and retention efforts, the pace of diversification is slow; in 2016, only 18 percent of the nation's teachers identified as teachers of color, an increase of only 1 percent from 2004 (King, McIntosh, and Bell-Ellwanger 2016; National Center for Education Statistics 2004).

Other avenues are possible for making learning opportunities and outcomes more equitable by carefully considering the beliefs and values of all prospective educators, regardless of demographic background. Districts interested in prioritizing candidates with particular beliefs and values can use interview and application essays to collect this information. Earlier research indicated that applications and interviews could not help districts make effective judgments to fulfill hiring priorities; more recent evidence, however, suggests that thoughtful, well-targeted recruitment efforts can provide meaningful information about applicants (Balter and Duncombe 2006; DeArmond, Gross, and Goldhaber 2010; Jacob et al. 2016; Rockoff et al. 2011). In particular, careful consideration of applicant characteristics through rigorous screening methods and interview processes can identify teachers with desired characteristics and behaviors who have a positive impact on student achievement (Goldhaber, Grout, and Huntington-Klein 2016; Jacob et al. 2016).

If districts want to develop a teaching force that has competencies, beliefs, and mindsets that align with district goals for addressing inequality, then they need to create a recruitment process that can identify these qualities among its applicants. This article draws on data from a partner district in California that uses its application to examine educators' approaches to equity goals during its recruiting process. In particular, it asks all applicants to respond to a short-answer essay prompt asking about the approaches they will use to address achievement gaps in the district. The district's human resources department scores these essays using a rubric during an initial screening process, reviewing for general evidence and understanding of equity commitments rather than specific beliefs, attitudes, or teaching strategies. This limited consideration is understandable given that application review is a time-intensive process, particularly with a large volume of applications.

Given resource constraints, it is difficult to measure educators' beliefs for a large number of individuals in other ways. It is time consuming and expensive to conduct multistage interviews or surveys. Such methods may also provide only a narrow slice of what educators think and can be prone to response bias or adverse reactions to particular types of interviewers. For districts wanting to learn more about applicants' beliefs, such methods can be time and cost prohibitive.

Recent advances in text mining paired with improvements in district administrative data collected from application materials provide new avenues for learning about applicant beliefs outside of extensive interviews. Social science [End Page 106] research is just beginning to use computational approaches to analyze text and dialogue to reveal additional features about individuals, their conversations, and verbal and written statements (Bettinger, Liu, and Loeb 2016; Gentzkow, Kelly, and Taddy 2017; Kelly et al. 2018; Liu 2018; McFarland, Jurafsky, and Rawlings 2013). The economist Jens Ludwig highlights the utility of data-mining techniques to help solve complex policy problems (2018). The sociologists Roberto Fernandez and Brian Rubineau identify important insights that applicant pool and referral data can provide about hiring dynamics (2019). Given important insights provided by hiring data (Fernandez and Rubineau 2019), we argue that hiring data provide a particularly exciting opportunity for using these techniques.

This study provides an important first case of applying machine-learning techniques to text-rich application data to examine the role of applicant beliefs in hiring processes and outcomes. It examines these relationships for more than ten thousand applicants over seven years in a large, urban district that has often struggled to attract and retain educators, particularly in its most disadvantaged schools. In addition to these methodological and data advances, this study has the novel purpose of focusing on educator attitudes toward and solutions for addressing the achievement gap, which is difficult to measure in a large sample. This study tests whether a district that expresses a deep commitment to equity and justice actually selects applicants that articulate attitudes and solutions that cohere with the district's priorities. Together, these results shed light on two often-neglected pieces of the inequality puzzle—what educators think and whether districts act in line with their values.

Our investigation of the role that educator beliefs about inequality play in hiring examines the following research questions:

What themes predominate in applicants' attitudes about achievement gaps?

Do the themes discussed vary by applicant race?

Are individuals who write about particular themes more likely to apply to schools with particular characteristics?

Do the essay themes and scores predict hiring outcomes, overall and at particular schools?

data and methods

This article uses school-district administrative data to describe educator attitudes and examine how these attitudes affect hiring outcomes. Our source for these attitudes is short-answer responses on job applications. These data are conventional in the sense that many districts screen applicants with some type of writing exercise (Jones 2012). Our use of them, however, is novel because we code them for applicants' attitudes and dispositions using structural topic modeling. These data provide the first use of machine-learning coded applicant essays to learn more about the role of applicants' beliefs in hiring. This district's screening question is particularly useful for examining educators' attitudes because it asks applicants to wrestle with a problem that will directly affect their daily teaching experiences. As a result, it provides a rich lens to uncover educators' feelings about this challenge.

Our partner district is an urban, public school district in California that employs more than 3,500 educators and administrators to serve a diverse student body of more than fifty thousand students (California Department of Education 2015). It is also a district with highly unequal outcomes for its students. In recent years, among California's large urban districts, it had both the highest average achievement and the widest gap between that average and the district's lowest-performing students, as well as large racial-ethnic achievement gaps. Although the district has made progress, gaps in suspensions, graduation rates, and achievement are ongoing challenges.2

To examine these relationships, we use applicant data from our partner district's human resources department. Applicants to positions in the district applied through a proprietary online interface. Data from this interface were then linked to other administrative records [End Page 107] from human resources by the research team. Current school district and state databases increasingly include detailed information about employees, but it is far less common that data from applications to district positions are linked to other district data sources (but, for notable exceptions, see Goldhaber, Grout, and Huntington-Klein 2016; Jacob et al. 2016; Saavedra et al. 2017). This type of connection allows for new research approaches that link the applicant pool to the pool of employed educators and other employees within a district. This approach has provided novel insights about hiring pathways, sorting, and discrimination in other sectors (Fernandez and Friedrich 2011; Petersen and Togstad 2006). However, research has not used applicant data that includes long-form writing and would typically be considered qualitative. By applying structural topic modeling to applicant essays, this study is the first to link attitudes within the applicant pool to hiring outcomes.

Figure 1. Number of Job Postings, Applications, and Applicants Hired Source: Authors' compilation based on partner district data. Note: Number of hires = 2,883.
Click for larger view
View full resolution
Figure 1.

Number of Job Postings, Applications, and Applicants Hired

Source: Authors' compilation based on partner district data.

Note: Number of hires = 2,883.

We examine data for all applicants to certificated positions from March 2009 to October 2015. Figure 1 shows the number of job postings, applications (in thousands), and applicants hired by year.

After an initial acceleration as the system came online, postings, applications, and number of applicants hired stayed fairly consistent across the seven years of our panel, save for an uptick toward the end of the financial crisis in 2011. During this window, this district posted 11,599 unique jobs, of which we use the 6,706 positions for which one or more applications were submitted. Postings received eighty-seven applications on average (from 1 to 544). The district received 218,196 applications for all certificated positions from 14,421 unique individuals. Our analytic sample consists of the 10,188 applicants (13,016 applicant-year observations) that completed essays (164,367 applications). Applicants submitted an average of sixteen applications each over this period (an average of 22.97 at the applicant-year level). The district ultimately hired 2,883 individuals from this pool.3 Table 1 presents descriptive characteristics, experience, and qualifications for the district applicant pool to all certificated positions between 2009 and 2015, and compares these [End Page 108]

Table 1. Descriptive Statistics of Applicant Sample by Hired Status
Click for larger view
View full resolution
Table 1.

Descriptive Statistics of Applicant Sample by Hired Status

[End Page 109]

characteristics for successful and unsuccessful applicants.

Hired applicants differ on many dimensions from those who were not hired. More of the hired applicants were Asian or Latino and fewer were white.4 The hired applicants had fewer years of experience, but were more likely to have worked in the district before applying for the job for which we observe them being hired. More hired teachers had single-subject credentials (eligible to teach in a particular content area in grades six through twelve) and fewer had multiple-subject credentials (eligible to teach all subjects in grades pre–K through eight) relative to the nonhired teachers.

Coding Applicant Essays to Identify Most Common Themes

The district's application also asks for responses to three short-answer essays detailing how applicants would address particular social issues and problems of practice that are relevant for teaching in the district context. Because of the district's equity emphasis, we focus solely on the following short-essay question:

Superintendent [Name] has stated that the achievement gap is the greatest civil rights issue facing our district today and closing that gap is the foundation and vision for the critical work of our teachers, staff, and administrators every day. As an educator, what is your role in working towards closing the achievement gap in [district]? In considering the demographics of our student population, what experiences or skills make you well-positioned to close the achievement gap in the context of a diverse district such as [district]?

This study uses responses to this question to indicate job candidates' attitudes toward equity and approaches to addressing inequality. Notably, the district leaves the specific meaning of the achievement gap to the interpretation of the applicant, not defining priorities about racial achievement gaps, income achievement gaps, or simply gaps between high and low achievers.

We use structural topic modeling, an unsupervised machine-learning technique, to detect the most common topics applicants discuss and their distributions across all of the essays (see, for example, Roberts, Stewart, and Tingley 2014). STM assumes that each document is a mixture of topics and that each topic is a mixture of a set of representative words. STM is particularly useful for evaluating massive textual data, as in our case, and the output of STM can provide metrics about the text content to be used in quantitative analysis.

Before we estimated the STM, we conducted standard preprocessing to prepare the texts for analysis. We first removed all stop words and punctuation.5 We then transformed all the words to lower case and reduced words to their root form, a procedure called stemming. As an unsupervised method, STM requires us to set the number of topics before we run the model. The optimal number of topics needs to balance model fit and substantive interpretation. We estimated the model multiple times to identify the ten, fifteen, twenty, and thirty most common, unique topics. We also controlled for whether an applicant previously worked in the district and their total years of K–12 teaching experience in our topic estimation models.

Our research team then examined the output from each model to classify the topics into essay themes. Four human coders examined model output, including top words, the most common words for each theme; FREX, the frequency and the exclusivity of each word to each theme; lift, which weights words by their frequency in a specific topic and other topics; and score, the top words from an index commonly used in linear discriminant analysis. Coders independently reviewed example essays that were the most emblematic of each topic to assign a topic label. From this, we discussed disagreements [End Page 110] as a team and developed harmonized theme labels for the various topics within each coding set. Many model themes were consistent regardless of the number of distinct topics. In models with fewer topics, several of the themes identified in the more numerous coding schemes combined into a single theme. The coders agreed that the most concise variant with only ten distinct topics captured the majority of the themes from across the variants with a larger number of topics. Diagnostic measures of essay topic classification (not shown) indicated that we lose little information by using ten topics rather than fifteen, twenty, or thirty. In addition, our interrater reliability of theme naming was highest with the smallest set of topics (80 percent exact agreement on labels with ten topics as opposed to 50 percent with twenty topics). We use this model to describe the prevalence of each theme across the pool of applicant essays and use the themes as key predictors in our analyses.

In addition, the STM algorithm codes each essay for the proportion of the text that addresses each theme and gives a rating of the density with which the essay covers each theme. We standardize the proportion of their essays that address each theme. We then use these values in regression models examining which applicants write most about which themes and which themes predict hiring outcomes most strongly. Theme ratings are not mutually exclusive and a given essay can have elements that correspond to multiple themes simultaneously.

Because the district's evaluation of an essay might be influenced by other features of the writing, we measure several characteristics of writing quality, including sentiment, readability, lexical density, number of misspelled words, and text length. We evaluated essays based on whether their overall sentiment or tone is more positive or negative using three off-the-shelf dictionaries to score the sentiment. Correlations between the sentiment measures were fairly high (r = 0.73 to 0.78), and thus we include only the Syushet sentiment scores (Jockers 2017). Readability, or understandability, refers to the level of education one needs to be able to read a piece of text easily. Readability scores are calculated using a grade-level scale, which roughly corresponds to the number of years of education one needs to read a given text. We tested the readability of the essays using the four most popular readability measures. Correlations between the readability measures were very high (r = 0.86 to 0.97), and thus we control for only the automated readability index in our analysis (Senter and Smith 1967). Lexical density refers to the number of lexical, or content, words in a sentence divided by the total number of words. Lexical words are those that give a sentence its meaning and include the nouns, verbs, adjectives, and adverbs. A high lexical density indicates a large number of information-carrying words, which is generally more difficult to read. Lexical density is negatively correlated with the length of a text. We use a type-token ratio to measure the lexical density of the applicant essays (Templin 1957). Finally, because essay screeners may be influenced by overall essay length and obvious errors, we control for the total number of words applicants used in their essays and the percentage of those words that are misspelled.

Examining Variation in Essay Theme Content

After identifying the most common themes discussed in applicants' essays, we test whether theme coverage varies by race-ethnicity. Because we are interested in whether applicants of color and white applicants share similar beliefs about solutions for inequality and achievement gaps, we focus our discussion of the results on this dimension.6 To examine how essay themes vary by race, we estimate ordinary least squares regression models predicting the degree to which an essay includes content about each of the ten themes. These models take the form

inline graphic

where Tint is an indicator for the proportion of applicant i's essay that covered a specific theme n, in year t which is a function of their race-ethnicity βit, a vector of controls Xit for other [End Page 111] applicant demographics, qualifications, and experience, year fixed effects γt, and an error term. We estimate separate models predicting variation in the coverage of each of the ten themes.

Application and Hiring Outcomes

To be hired in the district, an applicant must first upload their entire application content, including the essays, and then apply for a particular position. Applicants may submit applications for multiple positions. More than 83 percent of applicants applied for two or more jobs, but they used the same application for all district positions for which they apply.

The essay responses are likely indicative of some type of social desirability bias on the part of the applicant, given that they are seeking employment. This is likely true of all data collected about applicants because of the very nature of seeking a job. This might lead them to write responses that they believe will elicit specific reactions on the part of principals in the district. However, because applicants cannot change their essay, they are unlikely to write content tailored to different positions. Moreover, job postings for specific schools do not reference attitudes or values that principals are seeking; typically, an applicant applying to multiple positions would simply see variation in the job description or subject area needed for each posting to which they submit an application.

We do not have a good understanding of what motivates educators to apply for some positions and not others. Research indicates that teachers move to higher-achieving, whiter, more-advantaged schools over time, but it is also possible that their beliefs and attitudes influence the initial school placements they seek in a district (Boyd et al. 2005). To test this, we examine the degree to which writing about particular essay themes predicts whether applicants apply to schools with student populations that are above the annual district average for different characteristics. We code each school as having student populations that are above district averages for three characteristics based on the district's public reports of school composition: race-ethnicity, English-language learners (ELLs), and free or reduced-price lunch enrollment. To examine how applicant essays predict whether applicants apply to schools with different student populations, we estimate a series of linear probability models that take the following basic form:

inline graphic

where Yit is an indicator for whether applicant i applied to a school with a student population that was above the district average in that year (for example, above-average percentage ELL) in year t which is a function of several essay elements βit including essay themes, indicators of writing quality, and essay scores, depending on the model, a vector of controls Xit for applicant demographics, qualifications, and experience, year fixed effects, γt, and an error term. In this analysis, we exclude applications that individuals submit to positions not located at a specific school site.7

After applicants submit their application for a specific position in the district through the online system, human resources staff conduct an initial screening of the application. This consists of reviewing the materials for completeness, confirming that the appropriate certification documents and transcripts have been submitted with the application, and reviewing and scoring the application essays. This screening is done with a rubric (not shared to preserve district confidentiality) that evaluates each of the three essays for specific criteria and competencies, awarding zero to three points for no evidence, mixed or limited evidence, satisfactory evidence, or strong evidence of each competency. These scores are then aggregated across essays. Applicants with a score of four or lower are removed from the pool; those with scores of five or higher are passed on to an internal database for principals and central office administrators to review. [End Page 112]

We use the rubric scores on each of the three application essays as an indicator of applicant quality. We create a global score for the three essays; we use the score on the achievement gap essay as a stand-alone; and we aggregate the two non-achievement gap essay scores for control variables in some models. Mean scores for these essays for the full sample and separately for hired and nonhired applicants are shown in table 1.

To examine whether discussing a specific essay theme predicts rubric scores or being hired by the district, we estimate analogous models to equation (2), where rubric score or hired is the dependent variable. The same set of predictors is included in these models. We fit the same models as equation (2) to predict whether applicants are hired by schools with student populations that are above reported district averages across several demographic categories.

results

Educators provide a great variety of responses to the essay question, suggesting that they have meaningfully different values and attitudes about the social and achievement inequities facing district students regardless of the image they attempt to project toward their prospective employer. The content covered and omitted, as well as the general tenor, argumentation, and structure help classify applicants based on their perspectives on the underlying causes and challenges involved in addressing achievement gaps. Clear distinctions appear in the approaches to addressing this challenge and in the attitudes and biases of these applicants.

We summarize this rich detail into the ten most common themes that emerge from our structural topic model coding of applicants' essays. Table 2 lists these themes. Each label summarizes the approach that educators espouse to complete the phrase, "use _______ to reduce inequality." For example, some applicants would "use special services to reduce inequality and close achievement gaps." For example texts most emblematic of each of the ten themes, see the appendix.8

Several themes describe pedagogical approaches and experiences that applicants believe equip them to address achievement gaps. Two themes speak to the ways in which applicants adapt and customize their instruction to the needs of diverse learners by using differentiated instruction and cross-subject strategies, integrating content across subject areas and adapting it for different skill-levels to reduce inequality. One theme draws heavily upon recent school reform approaches and advocates using standards and assessment to improve and monitor instruction thereby reducing inequality. Another argues that educators are best equipped to reduce inequality by drawing upon their personal and instructional experiences with cultural and linguistic diversity.

Other applicants focus more on creating a supportive, inspiring classroom and engaging school, family, and community resources in their efforts to reduce inequality. Two of the themes describe resources outside the classroom that the applicant would access. Applicants who indicate that they would access special services write about working with school services such as therapy and speech pathologists to support diverse learners. Those who discuss family and community engagement write about working with the broader community and families and actively inviting them into the school. Essays also describe a more general idea of having a supportive classroom rather than referring to a specific strategy. Another theme focuses on being supportive, but relies heavily on general ideas about inspiring and believing in students to overcome challenges without clearly connecting these ideas to specific behaviors.

Finally, two themes highlight the societal power structures that create and maintain unequal conditions and achievement gaps in the district. Essays naming structural causes detail the structural challenges facing marginalized students and the ways societal inequities contribute to achievement and opportunity gaps, arguing that these inequities must be interrogated and addressed to make progress in reducing inequality. In some cases, this theme also includes language typically associated with deficit orientations, emphasizing group and family [End Page 113] conditions that reinforce societal inequities. A similar theme identifies societal causes for achievement gaps but discusses educators' responsibilities to work through these challenges to reduce inequality. Educators discussing this social justice–oriented theme decry opportunity gaps, want to challenge biases, and discuss drawing on students' funds of knowledge and using critical and culturally relevant pedagogies to promote equity.

Table 1 includes the average fraction of the essay text that discusses each topic and comparisons of the degree to which hired and nonhired applicants' essays discuss each of the ten topics. Tests of differences in the use of each topic by group are also shown in table 1, suggesting significant differences in topic use between hired and nonhired applicants on nearly every essay topic. Table 1 also includes descriptive comparisons of the other essay features, including readability, lexical complexity, and number of words for the full sample and between hired and nonhired applicants.

On average, the degree to which essays address each theme may seem low, but this is in part because each essay also includes text that is either not specific enough to correspond to a particular theme or addresses another topic that was not part of these ten most common themes. The most prevalent themes are supportive classroom (covered in 17 percent of essays) and family and community engagement (covered in 14 percent of essays). The least prevalent theme is cross-subject strategies, which is covered in only 5 percent of essays.

In addition to variation in percentage of applicants who cover each theme across the body of essays, applicants also vary in the degree of their essay text that is devoted to covering a specific theme. This ranges from educators' responsibilities, about which an essay has as much as 40 percent of its content covering this theme, to standards and assessment, about which an essay has as much as 68 percent of its content about this theme. As noted elsewhere, applicants often have some coverage of multiple themes in their essays, one theme frequently dominating one or more others.

Figure 2 presents a graphical representation of theme density within the entire corpus of essays, showing how two themes overlap, and displays portions of example texts that are among the most emblematic of standards and assessment and educators' responsibilities. The essay emblematic of the standards and assessment theme indicates the applicant's strong belief in standards-aligned curriculum and lesson plans, frequent assessment of student progress, and accountability to help address the achievement gap. The example educators' responsibilities essay highlights the need to use culturally relevant teaching and learning and to validate student cultures and experiences, and advocates for systemic, justice-oriented reform. These applicants clearly have different values and beliefs about educators' roles in addressing inequality and preferred strategies for addressing achievement gaps.

Essay themes vary by the race-ethnicity of the applicants. Table 2 displays results from models predicting theme content based on applicants' race-ethnicity, controlling for other demographic characteristics, experience, and credentials. A number of differences emerge. Applicants from different racial-ethnic groups cohere around particular sets of themes, relative to white applicants. Black applicants focus more on naming structural causes of inequality, such that their essays include 0.23 standard deviations more text about structural inequities than white applicants' essays do. They also call for greater educator responsibility than white applicants do. Asian applicants tend to focus more than white applicants on themes that engage community, family, and special services and build supportive classrooms. Compared with white applicants, Hispanic applicants focus more on creating supportive classrooms, invoke their experiences with cultural and linguistic diversity, and frame their approaches to the achievement gap by naming the structural causes of inequality and identifying educators' responsibilities to address them.9

Application and Hiring Outcomes

Applicant essays are predictive of candidates' application decisions. Table 3 presents results from models predicting the characteristics of [End Page 114] schools to which applicants apply. These models include controls for the applicants' demographic characteristics, experience, credentials, and measures of writing quality. In each case, the outcome is whether the applicant applied to a position in a school that was above the district-average demographic composition for that student group in that year. Given the specificity and ubiquity of differentiated instruction in teacher preparation programs and widespread use in classrooms, and because it is the theme that most closely resembles a solution for gaps between high and low achievers rather than race or income gaps, it is the reference category theme throughout.

Figure 2. Theme Coverage Comparison: Standards and Assessment and Educators' Responsibilities Themes
Click for larger view
View full resolution
Figure 2.

Theme Coverage Comparison: Standards and Assessment and Educators' Responsibilities Themes

Strong educators' responsibilities essay: "I not only have an obligation to provide my students with an equitable education, but also to be an advocate for change. We need to move toward a curriculum and pedagogy that is culturally responsive and relevant to our students. . . . Closing the achievement gap, creating schools that are equitable and just, changing the way we view education, and ensuring that every educator reflects on their own experiences and biases, will take time and is a journey that I am completely dedicated and committed to. I strive to teach for social justice."

Table 3 shows notable differences in the essay topics covered by applicants to schools with different demographic compositions. In particular, applicants who apply to schools that have above-average Hispanic populations write least about differentiated instruction and family and community engagement. In contrast, in above-average Asian schools, applicants write less about nearly every theme than differentiated instruction, although the coefficients are only significantly different for standards and assessment. Applicants to above-average ELL schools focus most on experiences with cultural and linguistic diversity and standards and assessment, which is consistent both with the [End Page 115]

Table 2. Predictors of Themes
Click for larger view
View full resolution
Table 2.

Predictors of Themes

[End Page 116]

needs these schools have and the recent policy prescriptions for schools whose students struggle with English proficiency. Applicants who apply to above-average free or reduced-price lunch schools discuss themes somewhere between those who apply to above-average Hispanic schools and above-average ELL schools. Only one theme, naming structural causes, significantly predicts applying to above-average black schools.

Table 3. Do Essay Themes Predict Applying to Schools with Particular Characteristics?
Click for larger view
View full resolution
Table 3.

Do Essay Themes Predict Applying to Schools with Particular Characteristics?

After an applicant submits their application, screeners in the district's human resources department read it and score each of the three essays using an internal rubric. In models (not shown), we examine the relationship between overall scores and specific essay scores. Higher essay scores positively predict being hired, but the achievement gap essay score matters much more for being hired than scores on the other two essays. An additional point on the achievement gap essay increases an applicant's likelihood of being hired by 6.6 percentage points but the other two essays combined by only 3.7 percentage points (22 percent of applicants were eventually hired). This suggests that information in the achievement gap essay beyond simple evaluation scores benefits candidates. Given the importance of the essay scores and themes, [End Page 117]

Table 4. How Essay Themes Predict Achievement Gaps Essay Scores and Being Hired
Click for larger view
View full resolution
Table 4.

How Essay Themes Predict Achievement Gaps Essay Scores and Being Hired

table 4 displays results from models examining the relationship between essay themes, essay scores given by the initial application screening, and being hired by the district.

Discussing almost every one of the themes is associated with higher essay scores than discussing differentiated instruction is (except family and community engagement and believing in students to overcome). This suggests that the district prioritizes many other beliefs and responses to achievement gaps over something that is part of typical educator training and practice, such as differentiated instruction. The themes scored most highly are educators' responsibilities and experience with cultural or linguistic diversity, but standards and assessment and supportive classroom also scored fairly high. This pattern holds even with controls for racial-ethnic identity and is thus not an artifact of the district simply preferring candidates of color, who are more likely to discuss these themes in their essays.

Many of these themes are also associated with being hired in the district. In general, the themes that received higher scores in the initial screening process are associated with the greatest likelihood of being hired. Although the coefficients might seem small, it is important to consider them relative to the means and standard deviations for the sample. For example, the average achievement gap essay score was 1.9 (on a 0 to 3 scale) with a standard deviation of 0.8. In discussing educators' responsibilities more than differentiated instruction, an applicant [End Page 118] increases their achievement gap essay score by 0.1 points, which is 13 percent of a standard deviation. Moreover, that same content increases the likelihood that they are hired by 2.5 percentage points, regardless of the position sought, which is roughly a 10 percent increase. Additional models (not shown) omit different themes to serve as the reference category to test the relative importance of each theme. Although writing about almost any of the themes improves the likelihood of getting hired more than discussing differentiated instruction, two themes relate to the highest likelihood of being hired relative to all others: experience with cultural-linguistic diversity and educators' responsibilities. The two themes are not significantly different from one another, although the coefficients for educators' responsibilities are positive relative to those of cultural-linguistic diversity.

An additional concern is the distribution of educators across schools. In particular, if districts think that having educators with particular pedagogical orientations and experiences might be better at promoting equity, they might also want to concentrate such individuals in traditionally underserved schools. However, such candidates might be attractive to all types of schools and, by virtue of a decentralized hiring process, may end up with placements at schools with relatively more advantaged students. Table 5 presents results from models investigating whether applicant essay themes are differentially associated with being hired in schools with different student populations, conditional on having applied. As in table 3, the outcomes in this table are whether the school in which the educator was hired has above district-average concentrations of students from particular demographic groups.

The results in table 5 have some patterns in common with table 3, but a few notable differences and smaller coefficients as well. Two themes positively predict being hired in schools with large concentrations of many types of students: special services and educators' responsibilities. Only one other theme, experience with cultural-linguistic diversity, predicts a higher probability of being hired in schools with above-average Asian populations, and supportive classroom predicts higher probability of being hired in schools with above-average numbers of students enrolled in free or reduced-price lunch programs. As with the likelihood of applying, writing about nearly every essay theme increases an applicant's likelihood of being hired in schools with above-average populations of Hispanic students over writing about differentiated instruction, excepting family and community engagement. Although structural causes is the only theme to significantly predict applying to above-average black schools, successful applicants to the same schools discuss a blend of classroom climate themes (family and community engagement, supportive classroom, and special services) and educator strategy themes (standards and assessment and educators' responsibilities).10

discussion

This study uses school-district administrative data to examine how educators' beliefs and attitudes about inequality discussed in application essays impact application behavior and hiring outcomes. It uses machine-learning techniques to identify the most common themes across over ten thousand essays and tests whether applicants who write about particular themes are more likely to apply to schools with certain student populations, are rated more highly by the district, and are more likely to get hired. This multifaceted investigation makes three contributions. First, it extends work examining current and preservice educators' attitudes about inequality by investigating equity attitudes among a large sample of educator applicants. Second, it contributes to literature evaluating educator-hiring processes and hiring in other settings to consider the ways in which attitudes that applicants express in their [End Page 119]

Table 5. Do Essay Themes Predict Teacher Hire in Schools with Particular Characteristics?
Click for larger view
View full resolution
Table 5.

Do Essay Themes Predict Teacher Hire in Schools with Particular Characteristics?

applications affect hiring outcomes. Third, it harnesses existing administrative data and uses text-as-data empirical techniques to glean new knowledge about beliefs and attitudes, which are difficult to measure and evaluate systematically.

Although one might worry that the essays do not reveal applicants' true beliefs, attitudes, and values about equity and achievement gaps, the application essays do provide insight into applicants' perceptions of what they think an employer would like to hear about such topics. Most applicants write thoughtful statements about how they might address a very real problem facing the district. The wide variety of responses suggests that educators have differing perspectives about how to address inequality, even when trying to impress an employer.

The essays show pronounced differences in themes that are differentially related to hiring outcomes. Distinct profiles emerge that vary by applicant race-ethnicity as well as the student population they aim to teach. Black and Hispanic applicants—groups traditionally marginalized in educational settings—write more about issues around structural inequities facing students, advocate for solutions that stress educators' responsibilities for addressing inequities, [End Page 120] and draw on experiences with cultural and linguistic diversity. Many of these themes are positively related to essay scores and hiring outcomes, regardless of the race of the applicant, relative to differentiated instruction, perhaps because differentiated instruction among marginalized populations may be perceived as a way to separate, track, and withhold educational opportunities. Some of the themes also predict an applicant's interest in teaching in schools with marginalized student populations, but a somewhat different set are associated with actually being hired in such schools.

In contrast, Asian and white applicants—groups traditionally advantaged in this district and schools more broadly—focus more on family and community engagement and differentiating instruction. Applicants seeking to work in schools with more-advantaged students talk less about structural inequity and instead focus on meeting individual student needs through differentiated instruction. However, almost all of the other themes are more strongly related to being hired than differentiated instruction is. In other words, educators' approaches to this issue, and their underlying values, attitudes, and beliefs that inform their responses matter in district hiring decisions.

The degree to which educators cultivate distinct equity-oriented personas or profiles in response to this question is striking. It parallels work by the sociologist Lauren Rivera that examines how students in elite colleges work to cultivate the cultural capital needed to successfully match to high-status private-sector jobs (2016). In our setting, educator applicants work not to brandish their elite cultural bona fides but instead to demonstrate that their dispositions position them to work as change-agents in a district seeking to combat systemic inequality (Bourdieu 2000). Yet, although multiple profiles appear to be desired by the district, not all applicants are equally successful at projecting this particular image. Educators from underrepresented racial backgrounds are particularly good at characterizing themselves in a manner consistent with the district's social-justice aims, but many white applicants also reflect thoughtfully on the ways they would address inequality, and the district appears to value this change-agent cultural capital regard less of the demographic background of the applicant who displays it.

The limitations of this study should be kept in mind when interpreting the results. In particular, we do not observe all stages in the hiring process. Between the district's initial screening and the time that applicants are hired for a position, hiring procedures are largely left to the discretion of individual principals. This makes it difficult to determine what type of interview process occurred and how much additional information principals had beyond the application materials with which to make their hiring decisions. This opacity in turn makes it difficult to determine what information principals used to select one candidate over another. Another limitation is that although structural topic modeling does allow for a careful screening of a large volume of text, the algorithms may overlook important insights that human readers would identify. Human coders might see meaning from particular phrases or topics that machine-learning techniques cannot.

Finally, this study takes place in a district that is extremely cognizant of structural inequality and particularly mindful of its role in creating and combating these inequities. Not all districts have this focus or awareness. Thus, the types of candidates attracted to work in this setting and the types of responses applicants provide to this prompt are likely different than they would be if many other districts asked for the same information. Although the generalizability of these results might be limited, they do provide helpful insights about what individuals think (and the variability in what individuals think) about the achievement gap and equity in a setting that brands itself as being actively engaged in the work of trying to combat it. Many U.S. school districts aim to develop a workforce that puts equity at the center of their work; these results indicate that applicants do have different ideas about how best to do so. Incorporating some type of application essay on this topic can help discern important differences and improve selection processes. Moreover, essay prompts discussing other educational challenges could be implemented to similar effect.

This study identifies important variation in applicants' beliefs about how to address a major [End Page 121] educational challenge: the achievement gap. It also reveals that such responses influence both application and hiring behaviors in this district. As in this study, data-mining techniques can unlock novel insights from text-rich administrative data and can inform conversations about educator hiring, diversity, retention, and many other policy-relevant issues. This method summarized complexities about educators' beliefs, values, and preferences that are otherwise costly and challenging to collect across a large number of individuals. Such data could be further linked to student and educator records to examine relationships between applicant attitudes, student outcomes, and educator job performance and tenure—relationships we will address in our setting in future work. Similar data in other settings could be matched to other employee outcomes including performance metrics, satisfaction, and retention. In drawing insights from existing administrative data, this relatively new combination of data and methods has the potential to make novel contributions to general knowledge about human thought and behavior as well as innovative improvements in policy and practice, in education and elsewhere. [End Page 122]

Emily K. Penner

Emily K. Penner is assistant professor of education at the University of California, Irvine.

Jane Rochmes

Jane Rochmes is assistant professor of sociology at Christopher Newport University.

Jing Liu

Jing Liu is a postdoctoral research associate in the Annenberg Institute at Brown University.

Sabrina M. Solanki

Sabrina M. Solanki is a doctoral candidate in the School of Education at the University of California, Irvine.

Susanna Loeb

Susanna Loeb is director of the Annenberg Institute and professor of education and international and public affairs at Brown University.

Emily K. Penner at emily.penner@uci.edu, 3200 Education, University of California, Irvine, Irvine, CA 92697
Jane Rochmes at jane.rochmes@cnu.edu, Department of Sociology, Social Work, and Anthropology, Christopher Newport University, 1 Avenue of the Arts, Newport News, VA 23606
Jing Liu at jing_liu3@brown.edu, Annenberg Institute for School Reform, Brown University, 383 Benefit St., Providence, RI 02903
Sabrina M. Solanki at ssolanki@uci.edu, 3200 Education, University of California, Irvine, Irvine, CA 92697
Susanna Loeb at loeb@brown.edu, Annenberg Institute for School Reform, Brown University, 383 Benefit St., Providence, RI 02903.

Support for this project was provided by grants from the National Science Foundation (IBSS-L1620419), Institute for Education Sciences (R305B130017), and the Stanford Graduate School of Education Incentive Fund.

appendix

The text that follows includes excerpts from applicant essays that exemplify each of the ten most common essay themes.

Special Services

In order to engage all students in their learning process, it is my role as an educator to 1) establish a consistent, comfortable, allinclusive, stimulating and trust-enhancing environment inside the classroom; 2) to maintain open and trusting ongoing communication with students, their families and caretakers in the classroom and an open-door policy; 3) maintain a school-wide support system with other staff members, all students, and their communities; 4) modify curricula based upon the IEPs of students with special needs and ongoing data based on students' experiences, cultural and language background, interests, progress and behavior; 5) develop appropriate, ongoing, and viable rewards systems for individual students as well as student teams; 6) use caution in applying research-based interventions to extinguish undesirable behavior; 6) work closely with teaching team for optimal teaching and school-wide environment.

Family and Community Engagement

I believe that it is essential to get the whole community involved in the education of our children. This is one of the main reasons I want to work in [district], because I live here. It is important to reach out to parents, do home visits, get them involved in school activities. If the entire family feels a part of the school setting then it will be easier for all of us to work together to close the achievement gap.

Believe to Overcome

Students will not here anything you are saying unless they believe you actually care about them. This comes from having a heart and spirit of humbleness and sincerity. You cannot fool these kids into believing what you are saying is truth unless they know you care about their well-being. Once you as a teacher have established in your heart that you are in this profession for the kids and not for any other reason then as a teacher your presence will convey and show this to the kids. Next you can present the content matter that you have a passion for. I believe it is the teachers job and goal to inspire these kids. There have been many methods and programs in our schools to try and do this, but one cannot inspired unless one is inspired. This means as a education you must be full of courage and strength exhibiting genuine love to these kids no matter how hard they may be on you as a teacher or how negative their personal situation is at home. As a teacher you must believe in yourself and the kids. Energy and love must flow from your being to help these kids because a lot of them do not have many people who really care about them.

Experience with Cultural-Linguistic Diversity

Being multi-lingual is essential to closing the achievement gap. Many [district] students speak one language at home and another language at school: English. I have spoken Spanish with parents, translators, students, and with my paraprofessionals. I have even learned some Chinese and Arabic to speak with my students with Speech needs. I think the District should provide condensed History lessons for all teachers of all the various cultures of our students: African-American, Cambodian, Vietnamese, Chinese, Latino, European and Southeast Asian cultures. Since I have travelled throughout Mexico, I have some understanding of the culture. Having studied American History, I have some idea of the African-American experience.

Standards and Assessment

It takes an outstanding educator to meet the needs of students and work toward closing the achievement gap. An outstanding educator is constantly and consistently assessing student learning as well as involving students in assessing their own progress. Progressive educators use the information from ongoing formal and informal assessments to guide the content and methodology of instruction to meet the needs of individual students. Documenting and communicating progress to students, parents, and those who work with the student not only conveys to the student the importance of being a self-motivated learner, but also improves professional practice.

Educators' Responsibilities

Additionally, we need to critically examine the practices that are in place within the educational system, despite our best intentions, the very practices that we use daily work against students from minority ethnicities and low SES backgrounds. Utilizing culturally responsive teaching is at the epicenter of closing the achievement gap. Acknowledging our students' diverse backgrounds gives legitimacy to each student's culture, which in turn creates a clear connection between home and school, which is crucial in promoting student achievement. When this approach to instruction is used, learning becomes more appropriate and effective, because instruction stems from, and responds to the students' strengths and existing knowledge. Due to the fact that non-dominant discourses are often ignored or shamed inside classrooms, it is crucial that the voices of students with non-dominant discourses are given focus and emphasis. The very nature and spotlight of spoken academic language in the classroom, needs to be shifted to fit the voices of the students.

Supportive Classroom

It is truly critical for the students themselves to value education and be self motivated. However, the gap will not be closed by the students or the teachers alone; the gap will ultimately be closed by the joining of the school, the families, and the [district] community. By making education a community effort, students will feel supported and understand the importance of their education. Thus my main role, as an educator, is to facilitate the learning of the students. I truly believe in a positive classroom environment in which the students are free to express themselves and learn with a lowered affective filter. I will do this by promoting an encouraging learning environment and showing students the true life value of education.

Naming Structural Causes

In his book Savage Inequalities the education writer Jonathan Kozol describes how urban public school systems in the 1990s struggled to serve the needs of the poorest families in the country, especially in predominantly African-American and immigrant communities. He cites research indicating that the factors most closely correlated with high achievement in school are family income and education level of parents. Thus inner city schools working to improve achievement face the greatest challenge in the nation. Kozol's own research revealed the stark reality that those same schools are funded by the lowest expenditures per student and equipped with the poorest physical resources. He reports on both the deplorable conditions in many impoverished school districts and the contrasting profusion of resources available in neighboring wealthy districts.

Cross-Subject Strategies

I believe that it is very important for the educator to come up with creative strategies that are specifically tailored to each student's needs. . . . I designed and taught nine week project classes with integrated curriculum. The 6th, 7th, and 8th grade students signed up for the project class that they wanted to be in. (Student choice is very important in motivating the student to achieve.) One example of a project was Mystery Play. For Language Arts, the students read and wrote mystery stories. In Math, they did logic and secret codes. In Social Studies, they studied actual cases. In science, they studied forensics. Each project had a final challenge and in this project the final challenge involved writing, producing, and performing a whodunit mystery play using the elements that they had learned.

Differentiated Instruction

I believe that the number one key component in closing the achievement gap in schools today is adapting instruction to tailor to the specific needs of individual students. . . . I try to create interesting lessons and differentiate instruction so I can reach each of my students. . . . It is a well known fact that not all students learn in the same way. Other than basic learning style differences, some students have learning disabilities or impairments which can significantly alter the way they receive and process information. I make a conscious effort to be very aware of things that may cause students to learn differently, and I use that information to help differentiate my instruction and future lessons. One way that I differentiate my instruction is through the use of centers. I like to set up various learning centers during a lesson to provide a variety of learning experiences that will allow students to learn according to their individual learning styles. The centers can include a variety of direct instruction of material, hands on learning activities, historical documents that students can analyze, and a center where students take the knowledge they have gained and create something.

Footnotes

1. We include applicants to all certificated positions, including classroom teachers, principals, therapists, and others, because some applicants apply to both teaching positions as well as some other type of certificated position for which they believe they are qualified, making the search processes for both positions contingent on one another. For parsimony, throughout the article we refer to all applicants as educators or teachers, reflecting the fact that the majority of the existing literature on educator hiring focuses on teachers. Results discussed are largely consistent whether we use the full sample of educator applicants or only the subsample of applicants to teaching positions (81 percent of the full applicant pool and 85 percent of the hired pool). Nearly all coefficients are similar in sign, significance, and magnitude. A small subset lose or gain significance with the teacher-only sample.

2. This information is from the district website and other local sources published between 2008 and 2017, withheld here to preserve the anonymity of the district. For more information, please contact the authors directly.

3. Some applicants (27 percent) applied in multiple years, and some applicants applied and were hired in multiple years. These applicants have distinct application data from each job search and are thus treated separately in all analyses. Our sample of 2,883 hired individuals includes 2,442 unique applicants, some hired multiple times.

4. The applicant pool is less diverse than the current teaching force in the district where approximately 50 percent of teachers identify as nonwhite.

5. Stop words are considered the most common words in a language, such as the, about, and own. In addition to removing these words, we remove many of the words used in the question itself, particularly the name of the superintendent.

6. We also test for differences based on variation among other demographic, experience, and quality dimensions (not shown).

7. Supplemental models examine all applications per applicant and include fixed effects for the general job category (such as social studies teacher, counselor, bilingual Chinese teacher), or fixed effects for the specific position an applicant applied to. As the results are qualitatively similar, we present models in the paper without the subject area or job-specific fixed effects. Alternative versions are available on request.

8. Applicants could cover multiple themes in their essays, although correlations between themes (not shown) were low to moderate, and the largest correlations of approximately 0.45 were negative.

9. Differences are also notable by gender, credential type, experience, and education. Results available on request.

10. One might worry that prior experiences in the district or in the specific school to which a teacher was reapplying might lead them to write especially well-tailored essays. Supplemental results indicate that the themes discussed do vary somewhat between prior employees and completely new applicants. However, similar themes predict higher screening scores and successful hiring outcomes regardless of prior employment in the district or specific school.

references

Alvidrez, Jennifer, and Rhona S. Weinstein. 1999. "Early Teacher Perceptions and Later Student Academic Achievement." Journal of Educational Psychology 91(4): 731.
Baker, Courtney N., Marianne H. Tichovolsky, Janis B. Kupersmidt, Mary Ellen Voegler-Lee, and David H. Arnold. 2015. "Teacher (Mis)Perceptions of Preschoolers' Academic Skills: Predictors and Associations with Longitudinal Outcomes." Journal of Educational Psychology 107 (3): 805.
Balter, Dana, and William Duncombe. 2006. "Staffing Classrooms: Do Teacher Hiring Practices Affect Teacher Qualifications." Paper presented at the annual meeting of the American Education Finance Association, Denver, Colo. (March 23–25, 2006).
Bettinger, Eric, Jing Liu, and Susanna Loeb. 2016. "Connections Matter: How Interactive Peers Affect Students in Online College Courses." Journal of Policy Analysis and Management 35(4): 932–54.
Bireda, Saba, and Robin Chait. 2011. "Increasing Teacher Diversity." Center for American Progress (blog). November 9, 2011. Accessed October 25, 2018. https://www.americanprogress.org/issues/education-k-12/reports/2011/11/09/10636/increasing-teacher-diversity.
Bourdieu, Pierre. 2000. "Cultural Reproduction and Social Reproduction." In The Structure of Schooling: Readings in the Sociology of Education, edited by Richard Arum and Irene Beattie, 56–68. Thousand Oaks, Calif.: Pine Forde Press.
Boyd, Donald J., Hamilton Lankford, Susanna Loeb, and James Wyckoff. 2005. "The Draw of Home: How Teachers' Preferences for Proximity Disadvantage Urban Schools." Journal of Policy Analysis and Management 24(1): 113–32.
Brown, Catherine, and Ulrich Boser. 2017. "Revisiting the Persistent Teacher Diversity Problem." Center for American Progress (blog). September 28, 2017. Accessed October 25, 2018. https://www.americanprogress.org/issues/education-k-12/news/2017/09/28/415203/revisiting-persistent-teacher-diversity-problem.
Buehl, Michelle M., and Jori S. Beck. 2015. "The Relationship Between Teachers' Beliefs and Teachers' Practices." In International Handbook of Research on Teachers' Beliefs, edited by Helenrose Fives and Michele Gregoire Gill. New York: Routledge.
Calarco, Jessica McCrory. 2011. "'I Need Help!' Social Class and Children's Help-Seeking in Elementary School." American Sociological Review 76(6): 862–82.
California Department of Education. 2015. "Largest & Smallest Public School Districts." CalEdFacts, September 21. Accessed October 25, 2018. http://www.cde.ca.gov/ds/sd/cb/ceflargesmalldist.asp.
———. 2016a. "Facts About English Learners in California." CalEdFacts, September 29. Accessed October 25, 2018. http://www.cde.ca.gov/ds/sd/cb/cefelfacts.asp.
———. 2016b. "Fingertip Facts on Education in California." CalEdFacts. September 29. Accessed October 25, 2018. http://www.cde.ca.gov/ds/sd/cb/ceffingertipfacts.asp.
Coburn, Cynthia E. 2001. "Collective Sensemaking About Reading: How Teachers Mediate Reading Policy in Their Professional Communities." Educational Evaluation and Policy Analysis 23(2): 145–70.
———. 2004. "Beyond Decoupling: Rethinking the Relationship between the Institutional Environment and the Classroom." Sociology of Education 77(3): 211–44.
Coburn, Cynthia E., Heather C. Hill, and James P. Spillane. 2016. "Alignment and Accountability in Policy Design and Implementation: The Common Core State Standards and Implementation Research." Educational Researcher 45(4): 243–51.
DeArmond, Michael, Betheny Gross, and Dan Goldhaber. 2010. "Is It Better to Be Good or Lucky? Decentralized Teacher Selection in 10 Elementary Schools." Educational Administration Quarterly 46(3): 322–62.
Dee, Thomas S. 2004. "Teachers, Race, and Student Achievement in a Randomized Experiment." Review of Economics and Statistics 86(1): 195–210.
———. 2005. "A Teacher Like Me: Does Race, Ethnicity, or Gender Matter?" The American Economic Review 95(2): 158–65.
Downey, Douglas B., and Shana Pribesh. 2004. "When Race Matters: Teachers' Evaluations of Students' Classroom Behavior." Sociology of Education 77(4): 267–82.
Egalite, Anna J., and Brian Kisida. 2018. "The Effects of Teacher Match on Students' Academic Perceptions and Attitudes." Educational Evaluation and Policy Analysis 40(1): 59–81.
Egalite, Anna J., Brian Kisida, and Marcus A. Winters. 2015. "Representation in the Classroom: The Effect of Own-Race Teachers on Student Achievement." Economics of Education Review 45(1): 44–52.
Ferguson, Ronald F. 2003. "Teachers' Perceptions and Expectations and the Black-White Test Score Gap." Urban Education 38(4): 460–507.
Fernandez, Roberto M., and Colette Friedrich. 2011. "Gender Sorting at the Application Interface." Industrial Relations: A Journal of Economy and Society 50 (4): 591–609.
Fernandez, Roberto M., and Brian Rubineau. 2019. "Network Recruitment and the Glass Ceiling: Evidence from Two Firms." RSF: The Russell Sage Foundation Journal of the Social Sciences 5(3): 88–102. DOI: 10.7758/RSF.2019.5.3.05.
Fox, Lindsay. 2016. "Seeing Potential: The Effects of Student–Teacher Demographic Congruence on Teacher Expectations and Recommendations." AERA Open 2(1): 1–17. DOI: 10.1177/2332858415623758.
Gamoran, Adam. 2001. "American Schooling and Educational Inequality: A Forecast for the 21st Century." Sociology of Education 74:135–53.
Gentzkow, Matthew, Bryan T. Kelly, and Matt Taddy. 2017. "Text as Data." NBER working paper no. w23276. Cambridge, Mass.: National Bureau of Economic Research.
Gershenson, Seth, Cassandra Hart, Constance Lindsay, and Nicholas W. Papageorge. 2017. "The Long-Run Impacts of Same-Race Teachers." IZA discussion paper no. 10630. Bonn: Institute of Labor Economics.
Golann, Joanne W. 2018. "Conformers, Adaptors, Imitators, and Rejecters: How No-Excuses Teachers' Cultural Toolkits Shape Their Responses to Control." Sociology of Education 91(1): 28–45.
Goldhaber, Dan, Cyrus Grout, and Nick Huntington-Klein. 2016. "Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools." Education Finance and Policy 12(2): 197–223.
Harris, Douglas N., and Carolyn D. Herrington. 2006. "Accountability, Standards, and the Growing Achievement Gap: Lessons from the Past Half-Century." American Journal of Education 112(2): 209–38.
Hollingsworth, Sandra. 1989. "Prior Beliefs and Cognitive Change in Learning to Teach." American Educational Research Journal 26(2): 160–89.
Jacob, Brian, Jonah E. Rockoff, Eric S. Taylor, Benjamin Lindy, and Rachel Rosen. 2016. "Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools." NBER working paper no. 22054. Cambridge, Mass.: National Bureau of Economic Research.
Jockers, Matthew. 2017. Syuzhet: Extracts Sentiment and Sentiment-Derived Plot Arcs from Text. Version 1.0.1. Accessed October 25, 2018. https://cran.r-project.org/web/packages/syuzhet/index.html.
Jones, Dawn S. 2012. "How Do I Handle Those Online Application Essays?" Education Week—Career Corner (blog). March 28, 2012. Accessed October 25, 2018. http://blogs.edweek.org/topschooljobs/careers/2012/03/how_do_i_handle_those_online_a.html?cmp=SOC-SHR-FB.
Kelly, Sean, Andrew M. Olney, Patrick Donnelly, Martin Nystrand, and Sidney K. D'Mello. 2018. "Automatically Measuring Question Authenticity in Real-World Classrooms." Educational Researcher 47(7): 451–64.
King, John B., Jr., Amy McIntosh, and Jennifer Bell-Ellwanger. 2016. "The State of Racial Diversity in the Educator Workforce." Washington: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service.
Labaree, David F. 1997. "Public Goods, Private Goods: The American Struggle over Educational Goals." American Educational Research Journal 34(1): 39–81.
Lindsay, Constance A., and Cassandra M. D. Hart. 2017. "Teacher Race and School Discipline." Education Next 17(1). Accessed October 25, 2018. https://www.educationnext.org/teacher-race-and-school-discipline-suspensions-research/.
Liu, Jing. 2018. "Measuring Beneficial Teacher Practices at Scale: A Novel Application of Text-as-Data Methods." European Journal of Education Research 3(2): 87–109.
Ludwig, Jens. 2018. "Administrative Data and Prediction Policy Problems." Paper presented at the Russell Sage Foundation conference Using Administrative Data for Science and Policy. New York (January 18, 2018).
McFarland, Daniel A., Dan Jurafsky, and Craig Rawlings. 2013. "Making the Connection: Social Bonding in Courtship Situations." American Journal of Sociology 118(6): 1596–1649.
McGrady, Patrick B., and John R. Reynolds. 2013. "Racial Mismatch in the Classroom: Beyond Black-White Differences." Sociology of Education 86(1): 3–17.
Murphey, David. 2014. "The Academic Achievement of English Language Learners: Data for the U.S. and Each of the States." Publication no. 2014–62. Washington, D.C.: Child Trends.
National Center for Education Statistics. 2004. "Schools and Staffing Survey (SASS)." 2004. Accessed October 25, 2018. http://nces.ed.gov/surveys/sass/tables/state_2004_18.asp.
Okonofua, Jason A., and Jennifer L. Eberhardt. 2015. "Two Strikes: Race and the Disciplining of Young Students." Psychological Science 26(5): 617–24.
Okonofua, Jason A., David Paunesku, and Gregory M. Walton. 2016. "Brief Intervention to Encourage Empathic Discipline Cuts Suspension Rates in Half among Adolescents." Proceedings of the National Academy of Sciences 113(19): 5221–26.
Opfer, V. Darleen, and David Pedder. 2011. "Conceptualizing Teacher Professional Learning." Review of Educational Research 81(3): 376–407.
Pajares, M. Frank. 1992. "Teachers' Beliefs and Educational Research: Cleaning up a Messy Construct." Review of Educational Research 62(3): 307–32.
Petersen, Trond, and Thea Togstad. 2006. "Getting the Offer: Sex Discrimination in Hiring." Research in Social Stratification and Mobility 24(3): 239–57.
Reardon, Sean F. 2011. "The Widening Academic Achievement Gap Between the Rich and the Poor: New Evidence and Possible Explanations." In Whither Opportunity: Rising Inequality, Schools, and Children's Life Chances, edited by Greg J. Duncan and Richard Murnane. New York: Russell Sage Foundation.
Reardon, Sean F., and Kendra Bischoff. 2011. "Income Inequality and Income Segregation." American Journal of Sociology 116(4): 1092–153.
Reardon, Sean F., and Ann Owens. 2014. "60 Years After Brown: Trends and Consequences of School Segregation." Annual Review of Sociology 40(1): 199–218.
Rimm-Kaufman, Sara E., Melissa D. Storm, Brook E. Sawyer, Robert C. Pianta, and Karen M. LaParo. 2006. "The Teacher Belief Q-Sort: A Measure of Teachers' Priorities in Relation to Disciplinary Practices, Teaching Practices, and Beliefs about Children." Journal of School Psychology 44(2): 141–65.
Rist, Ray. 1970. "Student Social Class and Teacher Expectations: The Self-Fulfilling Prophecy in Ghetto Education." Harvard Educational Review 40(3): 411–51.
Rivera, Lauren A. 2016. Pedigree: How Elite Students Get Elite Jobs. Princeton, N.J.: Princeton University Press.
Roberts, Margaret E., Brandon M. Stewart, and Dustin Tingley. 2014. "STM: R Package for Structural Topic Models." Journal of Statistical Software 10(2): 1–40. Accessed October 25, 2018. https://cran.r-project.org/web/packages/stm/vignettes/stmVignette.pdf.
Rochmes, Jane. 2018. "The Power of Seeing Potential: Teachers' Beliefs about Overcoming Social Disadvantage and Student Achievement." Stanford, Calif.: Stanford University,
Rockoff, Jonah E., Brian A. Jacob, Thomas J. Kane, and Douglas O. Staiger. 2011. "Can You Recognize an Effective Teacher When You Recruit One?" Education 6(1): 43–74.
Saavedra, Juan, Dario Maldonado, Lucrecia Santibanez, and Luis Omar Herrera Prada. 2017. "Premium or Penalty? Labor Market Returns to Novice Public Sector Teachers." NBER working paper no. 24012. Cambridge, Mass.: National Bureau of Economic Research.
Senter, R. J., and Edgar A. Smith. 1967. "Automated Readability Index." AMRL-T-66-220. Cincinnati University, Ohio.
Smagorinsky, Peter, Leslie Susan Cook, Cynthia Moore, Alecia Y. Jackson, and Pamela G. Fry. 2004. "Tensions in Learning to Teach: Accommodation and the Development of a Teaching Identity." Journal of Teacher Education 55(1): 8–24.
Spillane, James P. 1999. "External Reform Initiatives and Teachers' Efforts to Reconstruct Their Practice: The Mediating Role of Teachers' Zones of Enactment." Journal of Curriculum Studies 31(2): 143–75.
Templin, Mildred C. 1957. Certain Language Skills in Children: Their Development and Interrelationships. Minneapolis: University of Minnesota Press.
Villegas, Ana Maria, and Jacqueline Jordan Irvine. 2010. "Diversifying the Teaching Force: An Examination of Major Arguments." Urban Review 42(3): 175–92.
Wilkins, Jesse L. M. 2008. "The Relationship among Elementary Teachers' Content Knowledge, Attitudes, Beliefs, and Practices." Journal of Mathematics Teacher Education 11(2): 139–64.

Share