publisher colophon
  • Ninety-Nine Problems:Assessment, Inclusion, and Other Old-New Problems

Writing at the end of the nineteenth century, W. E. B. Du Bois opens his seminal The Souls of Black Folk with a simple formulation that encapsulates the workings of structural inequality even still today: "How does it feel to be a problem?" The question is striking in its disingenuousness, masking the questioner's complicity in the interrogee's predicament. When I ask you to talk to me about a thing that has happened to you, "your" thing, the fact of my asking asserts my ignorance as innocence, even as I know enough to presume it is safe to refer to you as a problem. The fact of my asking also weaponizes my imagination of good will: I am asking because I am concerned about you, which transforms a thing that is happening to you into a thing discussed squarely in terms of how it has been processed by you and can be narrated to me. Even as it ostensibly expresses concern for you, again, how are you feeling, the question transfers the responsibility for making meaning onto you—you who are special, tell me more. To be a problem is to carry an identity that a structure is unable or unwilling to accommodate. To ask the question is to acknowledge the dissonance while abdicating responsibility for its resolution.

Developing less burdensome and more equitable ways to support scholarly difference is a preeminent challenge when thinking about the future of assessment and promotion in higher education. At stake in this is the very capacity of institutions to do the work of scholarly inclusion, to recognize the range of approaches well captured in the digital humanities caucus of the American Studies Association's succinct 2016 characterization of humanities work that is "innovative, critical, boundary-pushing, justice-based, and experimental work—scholarship that takes a diversity of forms, that reaches and is produced by thinkers, teachers, practitioners, and makers from a wide range of communities and contexts."1 Assessment potentially shadows or highlights scholarly identity at every institutional juncture, and this is as true for undergraduate research work as it is for matters of promotion, tenure, or contract renewal for faculty and staff. In 2018 digital work is still often an unreasonably risky [End Page 677] pursuit for many faculty, staff, and students, unless that work is undertaken as additional to the other kinds of scholarship already vetted by any given field or discipline, or that work comes after a scholar has an established record of nondigital publication. At many institutions, digital and other kinds of new or experimental scholarship have become the latest examples of adding qualities that make job candidates desirable without actively subtracting from the list of expectations historically attached to any given position. In many cases the oldest DH cliché still holds true, that if you want to secure your appointment, you must "do double the work."2

What would it mean for a college or university to encourage or support digital work without a willingness to demonstrate equal and immediate commitment to building a larger or deeper institutional conversation around that work? To do so would be to support the neoliberal tendencies identified in some of the most trenchant critiques of digital humanities projects and programs, hiring in the name of "innovation" that is ultimately superficial to the institution and destructive to faculty and staff enfranchisement.3 In such cases, untenured scholars and contract professionals working on some of the most telegenic and financially supported projects, for instance, would be explicitly asked to do so from a position of institutionally advocated precarity. In 2011 Kathleen Fitzpatrick, writing as director of scholarly communication at the Modern Language Association, noted the increase both in graduate students becoming interested in new kinds of scholarship and in faculty being recruited to bring digital approaches to campuses. Despite her optimism for institutions broadening humanities approaches, Fitzpatrick was also concerned about "what provisions are being made for supporting those new faculty members, particularly on campuses where the positions represent a first foray into the digital humanities."4 Under what kinds of institutional conditions does it become possible to successfully address and transform an institution from a weakly enfranchised position, even when that person holds knowledge and insight? In the worst version of this scenario, scholars become simultaneously desirable and unassimilable, initially characterized as instructive or useful to the institution, but also as an example of possible exception, not rule. What does it mean to create positions that are supposed to teach institutions how to value the very idea of a position, or that are intended to illuminate institutional problematics that do indeed need solving? Many flavors of postdoc, visiting positions, alt-ac, and even grant-seeded tenure-line positions are ultimately asking scholars to enter institutions as problems, but with few indications of how institutions will actively produce solutions on their behalf. [End Page 678]

When thinking about institutional assessment, we should welcome the opportunity to narrativize ourselves as scholars, to tell the story of our work and of its meaning, and to set the terms of its circulation. Yet the very possibility of a fair assessment of digital scholarship remains a deep concern for so many: I am worried about being legible to my institution. I am not sure my department knows how to read my work. Will this count? In many cases, institutional representatives would likely note that faculty can find answers, because they can be extrapolated from faculty handbooks or from institutional tenure and promotion guidelines. Regardless, it is clear that something else is at work, that there is a disconnect between institutional procedure, constituents' ideas of what they should be doing or have been asked to do, and difficult questions about who is ultimately responsible for the institutionalization of innovative or cutting-edge humanities scholarship. That so much of this negative feeling swirls around a career moment that is technically dialogic—the institution sets and communicates benchmarks, candidates respond—reveals assessment as a site of miscommunication and unacknowledged institutional disinterest in transformation. It is the site of ideology, which is to say that assessment is the place onto which other underlying social forces are displaced.5 Because assessment is about institutional power—the power to shape its membership, to support or not support disciplinary interests, and to platform students' intellectual opportunities—this miscommunication carries differential consequences for everyone involved.

For a variety of reasons, it can be difficult to determine who is responsible for clarifying communication and establishing pathways to success, especially in units and departments that otherwise prefer to imagine themselves as self-critical and committed to working in minimally hierarchical ways. In the past several years, many institutions have adopted the language of mentorship to help build better roads to tenure and promotion. On the one hand, this is a great institutional value: in all kinds of ways it is important that colleagues have opportunities to share their hard-won expertise, to help make sure that lessons are not unnecessarily repeated across generations. Also, ideally, mentorship should be allowed to move in a variety of directions, which allows for "mentoring up rather than down," as Fitzpatrick puts it, while also attending to Kerry Ann Rockquemore's deeply strategic sense of networked mentoring.6

At the same time, when procedures are unclear, or even just slowly disinte-grating under the pressure of institutional change, mentorship itself becomes an exercise in individuals engaging herculean tasks to make up for structural deficiencies, a kind of work that often falls disproportionately on women and [End Page 679] people of color, who also in many ways do such work over, above, and in spite of institutions. From this perspective, whole swathes of faculty and staff have always done "double the work." Such additional labor, for instance the kinds of recuperative advising practices Katrin Schultheiss has characterized as "ghost advising," is often supported by departments. As Schultheiss notes, differential workloads are allowed to continue because overwhelming numbers of faculty and staff "are complicit in perpetuating norms of masculine ambition and feminine helpfulness."7 And, perhaps most ironically, practices like ghost advising are also perpetuated because such labor can be held up as exemplifying how individual relationships supersede institutional ones, which thus supposedly bespeaks the human side of the institution. And yes, this should be read as good thing, a laudable value. But as Roxanne Shirazi has deftly shown, providing the affective labor that makes other kinds of institutional labor possible also means choosing "unquantifiable," unassessable, feminized work.8 Indeed, as Caroline Elerding and Roopika Risam note, in the context of affective labor doing a kind of diversity work, this work that on principle seeks to do well by others "is not valued for its transformative possibilities. This is reflected in performance evaluations where, at best, it might be understood as 'service.' Yet, 'service' … does not adequately capture the ways this diversity work is performed in hallways, during office hours, over lunch, by the water cooler."9 In this paradigm, even the most well-intentioned desire to labor outside institutional constraint or limitation ultimately supports structural inequality, particularly as the idea of "the institution" is made to carry the material and symbolic weight of fundamentally differentialized relationships, which is to say that the institution is that for which no one takes responsibility, even as each of its members' own daily acts are in fact its machinery. This displacement of responsibility onto the idea of the institution, its weakness and its vicissitudes, also sets the stage for a kind of burden transfer, wherein people on the lowest tiers of unarticulated hierarchies are forced to take material and symbolic responsibility for the discomfort of people situated more highly in the hierarchy. They are asked to remain silent, or at least to frame their complaints without ever localizing blame. Meanwhile, the structure itself continues with minimal intervention from the people most able to transform it, at the expense of those most vulnerable to it.

In both cases one might also think here of Sara Ahmed, who notes "how the presumption of our own criticality can be a way of protecting ourselves from complicity. As Fiona Probyn-Ramsey has observed, complicity can be a starting point; if we start with complicity, we recognize our 'proximity to the problems [End Page 680] we are addressing.'"10 Indeed, during my time as director of Five College Digital Humanities (5CollDH), I had become increasingly uncomfortable with what it meant for us to ask faculty and staff to develop digital projects, but without having much to say when they wondered if their intense work would be institutionally recognized as proper academic labor. The growing accessibility of digital technologies at our five member institutions had been gamely met with an increased willingness on the part of scholars to integrate new digital methods into their various interpretive and presentational practices, and 5CollDH was itself a large Mellon-funded initiative.11 We were in the strange place of having access to considerable financial resources, but undertaking an enterprise with limited access to the structures that ultimately underwrite the meaningfulness of the labor pursued with our funding.12 During my tenure as director I watched as several prominent digital humanists were caught up in tumultuous tenure cases, even despite the success of their digital work by all reasonable metrics.

The New Rigor

In 2015 Five College Digital Humanities undertook a yearlong process to reevaluate evaluation itself. Even as much as we loved doing the work of project development, we had quickly come to appreciate the growing complexities our project participants were facing in regard to contract, tenure, and promotion. We called our process "The New Rigor," which encapsulated our sense of what was at stake in getting institutions to reconfigure how they talked about academic rigor, especially as rigor is a term used by many to signify scholarship's seriousness and meaningfulness, and as evidence of labor. From our perspective, the term risks being emptied out whenever forms are allowed to be substituted for contents, for instance as institutional reviewers could claim that because they did not know how to look at a digital project and "see the work," the work simply was not there. Or, as Mark Sample has usefully outlined elsewhere, reviewers can too easily characterize much of the scholarly work of the digital humanities as yet more examples of "service."13

Our invitation to participants opened with the premise that "every scholar cares very deeply about assessment, but rarely do we talk about it as a space for active intervention."14 For even as much as the 5CollDH staff enjoyed doing innovative work, it also became apparent that what might feel like the most mundane and institutional aspects of scholarly work—labor, employment terms, and assessment—were also the most important, as such processes [End Page 681] operate at the intersection between who we are, what we make, and how those are allowed to thrive in institutions.

With all of this in mind, "The New Rigor" conversations were framed as an opportunity for "a variety of academic stakeholders to self-consciously approach matters of evaluation and assessment," to account for the structures to which we as scholars are held accountable. We asked our participants two questions: if you could start from scratch, what structure of evaluation or assessment—in terms of peer review, tenure and promotion, or student research experience—would encourage you to do digital work? How might we construe evaluation and assessment as generative processes, rather than as merely restrictive ones? We staked the conversation this way because we wanted to take digital projects' various resistances to standardized assessment as fundamentally instructive, for the problematics exposed by digital scholarship are part and parcel of larger problematics inhering in how academic institutions characterize, or refuse to characterize, different kinds of scholarly labor. As the result of a three-part process—an initial conference, the editing of community documents resulting from that conference, then a follow-up workshop—we were able to collect "The New Rigor" findings as a draft report, designed to give departments some insights and starting points for thinking more broadly about hiring, promoting, and otherwise retaining scholars whose portfolios would include digital projects.

In talking about rigor, it also became clear that the problematics arising in the evaluation of digital scholarship were structurally similar to those arising in the evaluation of much interdisciplinary work, or in general of work that does not become a printed journal article or monograph: edited volumes, bibliographic investigation and curation, public scholarship, digital work, creative scholarship, community engagement reports, the list goes on. Much of what we think of as problems of assessment in digital humanities are not new problems, particularly from the perspective of scholars in women's studies, Black studies, or American studies, for instance. But because their labor had never quite been accounted for as part and parcel of institutions' work, beyond a sort of representational function, digital humanists have been historically left with fewer signs that point us toward building coalition and constituency via the work of previous generations of structurally similar or thematically adjacent scholars. We have lost, therefore, numerous opportunities to see how our enterprises are connected not only through subject matter but also through how we do it.

In other words, the problem is not that assessing digital work requires institutions to develop new modalities, workflows, and values. Institutions [End Page 682] do actually change all the time. The problem is that asking departments and units to identify the values inhering in current institutional practices would require many faculty and staff to reckon with how so many people have been historically underserved by them. To give voice to the flexible, networked, and transformative processes of guidance and support that so many faculty and staff already hold dear would be to make larger institutional transformation possible, for to do so would allow more institutional members to contribute their heretofore institutionally localized—marginalized—value systems to a larger shared enterprise. This is not to suggest that heaping process on process is the path to the transformative inclusion; the point is that even the most mundane bureaucratic processes, often characterized as the especial bane of tenured or supervisory faculty and staff, must nonetheless be taken as an opportunity for the evaluation and assertion of the human values underwriting that work. It is the difference between what we say we want the world to look like and what we actually carry out in our smallest acts. Carrying, how we carry ourselves in our relationships and how we carry each other, is the real place of transformation. Many in the academy have already experienced the consequences of the most enfranchised faculty and staff not being required to carry and to take care, and it is clear that every member of the scholarly enterprise must be given more opportunities to come into this work, if the institutions in which we labor are to survive at all.

Working merely to map extant systems of assessment onto new kinds of scholarship not only assumes that such translation is possible; it also implies that we are in fact satisfied with what we already have. Such processes therefore risk producing maps not of the terrain but simply of extant and unsatisfactory maps. What constitutes the terrain in today's academy? Who, actually, do we want to be able to be, and how might assessment practices support that growth? Further, insofar as engaging in the assessment of others represents the ur-site of academic enfranchisement, how might the last decade of insights from scholars like Katherine D. Harris, Cathy Davidson, and Jesse Stommel come to be understood as practical institutional interventions beyond digital humanities? It is clear that digital humanities has already arrived at its next iteration, its 2.0, while many systems of institutional assessment, evaluation, and valuation often feature the most calcified practices. Questions arising at the intersection of American studies and digital humanities are especially urgent today because they carry concerns that have already historically dogged scholars working at a variety of kinds of margins and institutional problematics. [End Page 683]

Marisa Parham

Marisa Parham is professor of English at Amherst College and directs the Immersive Reality Lab for the Humanities. She is the author of Haunting and Displacement in African American Literature and Culture (Routledge, 2009) and coeditor of Theorizing Glissant: Sites and Citations (Rowman & Little-field, 2015). She is also a faculty diversity and inclusion officer at Amherst and a former director of the Five College Digital Humanities Initiative, serving Amherst, Hampshire, Mt. Holyoke, and Smith Colleges, and the University of Massachusetts Amherst.


1. "DH Caucus Advisory Committee Statement on AQ's Digital Projects Review," April 6, 2016, American Studies Association,

2. Sydni Dunn, "Digital Humanists: If You Want Tenure, Do Double the Work," Chronicle of Higher Education, January 5, 2014,

3. Melissa Dinsman, "The Digital in the Humanities: An Interview with David Golumbia," Los Angeles Review of Books, June 30, 2016,

4. Kathleen Fitzpatrick, "Do 'the Risky Thing' in Digital Humanities," Chronicle of Higher Education, September 25, 2011,

5. I am thinking specifically here of Stuart Hall's articulation of ideology and representation. See Stuart Hall, "Signification, Representation, Ideology: Althusser and the Post-structuralist Debates," Critical Studies in Mass Communication 2.2 (1985): 91–114,

7. Katrin Schultheiss, "Ghost Advising," Chronicle of Higher Education, March 4, 2018,

8. Roxanne Shirazi, "Reproducing the Academy: Librarians and the Question of Service in the Digital Humanities," July 5, 2014,

9. Carolyn Elerding and Roopika Risam, "Introduction: A Gathering of Feminist Perspectives on Digital Labor," First Monday 23.3 (2018),

10. Sara Ahmed, On Being Included: Racism and Diversity in Institutional Life (Durham, NC: Duke University Press, 2012), 5–6.

11. Five College Digital Humanities began as a Mellon-funded initiative to seed digital humanities projects and to promote DH approaches across five partnered institutions: Amherst, Hampshire, Mount Holyoke, and Smith Colleges, and the University of Massachusetts Amherst.

12. To be sure, we had in many cases been able to secure demonstrations of institutional good will for projects and individuals at opportune moments, but we nevertheless observed that such goodwill by default assures that the value of faculty and staff work is ultimately tethered to the whim of ambassadors, be they chairs, deans, supervisors, or the like.

13. Mark Sample, "When Does Service Become Scholarship?," Samplereality, February 8, 2013,

14. Marisa Parham, Kimberly Bain, Jeffrey Moro, and Mariel Nyröp, The New Rigor Report, March 2015,

Additional Information

Print ISSN
Launched on MUSE
Open Access
Back To Top

This website uses cookies to ensure you get the best experience on our website. Without cookies your experience may not be seamless.