Hacking the Academy: New Approaches to Scholarship and Teaching from Digital Humanities

Making Digital Scholarship Count

Mills Kelly

As more and more scholars do work in the digital environment they are expecting this work to count toward tenure, promotion, and other types of formal evaluation. It seems to me that the first step is to define what we actually mean when we say that digital work should “count” in higher education. At most colleges and universities around the United States—and, to varying degrees, elsewhere in the world—there are three domains of activity that faculty members engage in: research, teaching, and service. Most of us have to turn in an annual report that is organized into three sections corresponding to these domains. In varying ways at various campuses, what can be claimed in each domain is defined by the institution or by departments. Sometimes, those things that count are defined in union contracts; sometimes they are defined as they come up. In short, there is no standard practice in academia, other than to generally rely on research, teaching, and service as the main categories for faculty evaluation.

A thornier issue is how activity in each of these domains is evaluated. Here we see even more variation in practice from one campus to another; from one department to another. What counts at one place, is ignored or even penalized at another. At one institution, research trumps all; at another, teaching is the coin of the realm. In some history departments, it is enough to have published a book; in others, that book needs to be published by some relatively short list of prestigious presses. Context is everything in this discussion.

Does this mean it is hopeless to even take on the issue of how digital work might fit into such a heterogeneous set of practices? By no means.

In the history business, we have a very informal and fluid set of standards for determining what is and is not meritorious. We all know that an article published in a journal judged to be prestigious is probably more praiseworthy than one published in a backwater journal with little or no reputation. We know that a book published by a university press that has a great reputation is almost surely better than one published by a press no one has ever heard of. Or at least we think we know these things.

Whether book or article X published by a prestigious journal/press is actually better than book or article Y published by a journal/press we've not heard of is an open question. We assume in advance, though, that X is probably better than Y.

We do so not without good reason. Those things submitted for publication to a prestigious press/journal are more likely to go through a more rigorous peer-review and editorial process than something published in an underfunded and little-known press or journal. The competition to publish in the prestigious venues is keen—submissions of lesser quality get weeded out; thus it has been for generations.

As long as historians produced scholarship that was in a form that fit neatly into this model—books or journal articles published following a peer-review process—all was well, and the system functioned fairly smoothly. Then digital technology invaded the cozy confines of our discipline and things got a lot more complicated.

You may have noticed that I use the term “digital work” rather than “digital scholarship.” My choice of words was in no way accidental. Digital work encompasses everything historians do in the digital realm—scholarship, teaching, and service. “Digital scholarship” is a precisely defined—or should be precisely defined—subset of “digital work.”

Before we can even begin to claim that something called “digital scholarship” should count in the research domain of our professional lives, we would do well to define exactly what constitutes “scholarship.” Here, I think we have an easier task. In almost any discipline, scholarship has the following characteristics: it is the result of original research; it has an argument of some sort and that argument is situated in a preexisting conversation among scholars; it is public; it is peer-reviewed; and it has an audience response.

There are exceptions, of course. A novel, a collection of poetry, a work of art, or a piece of music may all count as scholarship in certain contexts. By and large, though, the characteristics I've described hold for most forms of scholarship. This means that for digital scholarship to be scholarship it has to have all of these characteristics.

We'll return to this crucial issue later. But for now, I think it's easier to define what digital scholarship isn't than to define what it is—especially because, as we'll see, it is an inherently moving target.

I think we would all agree that a course website or a series of lectures created in one's favorite slideware program do not constitute scholarship. They may well be very scholarly, but on any campus I can think of, this sort of work falls clearly and unequivocally into the teaching domain.

Where it gets trickier is when we consider digitization projects—whether small in scale, or massive, like Tufts University's Perseus Project, or the University of Virginia's Valley of the Shadow.¹ Each of these excellent and heavily used projects offers scholars, teachers, students, and the general public unique access to their content. But, as Duke University's Cathy Davidson said in an interview, “the database is not the scholarship. The book or the article that results from it is the scholarship.” Or, I would add, the digital scholarship that results from it. In other words, I'm not willing to limit us to the old warhorses of the book or scholarly article.

I also want to emphasize that I have tremendous respect for the scholars and teams of students and staff who created these two projects—both of which I use often in my own teaching. But I also have to say that I don't think either project can be considered scholarship if we use the definition I've proposed here.

Why not? you may ask. The reason is fairly simple in both cases. Neither project offers an argument. Both are amazing resources, but neither advances our understanding of particular historical questions. They make it possible for that understanding to advance in ways that weren't available before, but as Davidson says, it is what results from a project like these that is the scholarship. Thus, for instance, though the Valley of the Shadow database does not qualify as scholarship, the resulting article, “The Differences Slavery Made,” published by the database's creators William Thomas and Edward Ayers in the American Historical Review rises to the level of scholarship in our working definition.²

I think that almost all historians would agree with this definition because it's the one we use all the time. We're comfortable with it, it works for us, and given how used we are to it, many historians—including many I know and respect—argue that there is no need to change it. After all, if it ain't broke, why fix it?

Alas, for our current definition of scholarship, the digital world is undermining our certainties.

The big sticking point is the next-to-the-last part of my definition—peer review. For a century or more, peer review in our discipline has meant that the historian produces his or her work—book or article—and submits it for publication. Then, after waiting months—or, more likely, many months—the historian finally receives feedback on his or her work and either has a little more work to do, a lot more work to do, or finds the work rejected entirely.

Why won't this process survive in the digital world? The answer is pretty simple. It just takes too long and does not work in a medium where gatekeeping makes no sense. By its very nature, digital scholarship happens in a dynamic space—one where the work is often self-published in the sense that a scholar or a group of scholars creates historical work in the digital environment, and then it is made available when it's done—or close enough to done to show other people. Not after a lengthy process of peer review—but when it's ready to be seen.

Then, and only then, does peer review begin. The Internet is an open environment, not the closed environment of the publishing industry that we have lived with for many generations. Anyone can publish anything online and that, of course, means that a lot of dreck appears. But the fact that dreck is scattered all over the Internet does not mean that quality work cannot also appear through the same process.

The American Historical Association is proposing to try to act as some sort of gatekeeper for digital historical scholarship, but this proposal is doomed because it is trying to find a way to fit the old system into a new technological environment where gatekeeping as we've known it doesn't—and can't—work.

Already in other industries we have seen what happens when the guardians of the old ways try to hold back the tide of change. Sales of music CDs continue to drop like a stone while sales of individual songs through services like iTunes continue to rise rapidly. A decade ago, Kodak employed four times as many workers as it does today (when was the last time you bought a roll of film?). And while Amazon hasn't killed off all local bookstores, there certainly are far fewer than there used to be.

So what, you might ask? Why do we have to change?

Because if we don't, we'll eventually become irrelevant. Already other disciplines that are not as resistant to change have embraced the digital world to a much greater extent. For example, work posted on the online Social Science Research Network “counts” in many academic departments around the country, despite the fact that peer review takes place after the fact, not before.³ And in other disciplines—computer science, biology, physics, etc.—peer review increasingly takes other forms entirely. So why are we so hung up on keeping a system that made good sense a hundred or fifty years ago, but makes less and less sense today?

I wondered what a provost might think about this issue, so I spoke to Peter Stearns, provost at George Mason University, a past vice president of the American Historical Association (Teaching Division), the founding editor of the Journal of Social History, and the author of more than 100 books—so, he knows something about peer review.

He told me that being a provost meant that he had to take a much more capacious view of peer review, because each discipline at the university has its own standards for what constitutes proper peer review. What Peter cares about is not how the peer review happens, but that it does happen. “It can be either before or after publication,” he said in our interview.

Other disciplines do it, so what is so particular, so unique about historical and humanities scholarship that it must be reviewed prior to publication? Upon reflection—nothing.

I'm not proposing that we throw out a system that has worked for so long in one fell swoop. I am suggesting, however, that there needs to be a serious discussion in our profession about what peer review means, what its value is to the process of advancing knowledge, and how it can change to take into account the new realities of the digital world. If we don't have this discussion—and soon—we're in danger of losing touch with a rising generation of young scholars who will see us as nothing more than cranky old scholars who are hanging onto an old system because it serves our interests—not theirs.

Notes

1. Gregory R. Crane, “Perseus Digital Library,” http://www.perseus.tufts.edu/hopper/. “The Valley of the Shadow: Two Communities in the American Civil War,” http://valley.lib.virginia.edu/.

2. William G. Thomas III and Edward L. Ayers, “An Overview: The Differences Slavery Made: A Close Analysis of Two American Communities,” American Historical Review 108, no. 5. “The History Cooperative,” American Historical Review, December 2003.

3. Social Science Research Network (SSRN), http://ssrn.com/.