The Myth of Information Effects in Human Rights Data: Response to Ann Marie Clark and Kathryn Sikkink

I. INTRODUCTION

In August 2013, Ann Marie Clark and Kathryn Sikkink published “Information Effects and Human Rights Data: Is the Good News about Increased Human Rights Information Bad News for Human Rights Measures?” in this journal.¹ Their article examines important issues related to indicators of government respect for physical integrity rights from the Cingranelli-Richards (CIRI) Human Rights Data Project as well as the Political Terror Scale (PTS) index of physical integrity conditions. Using a data set about Latin America and the case examples of Brazil and Guatemala, they make an argument that an increased amount of information over time in the US State Department Country Reports on Human Rights Practices, as evidenced by growing word counts, is responsible for a downward bias in scores indicating government respect for human rights over time.²

While the article has already been cited in a number of prominent venues,³ its conclusions appear to have been accepted without scrutiny. However, given the extremely important impact that Clark and Sikkink’s argument, if true, would have on the large number of published studies using CIRI and PTS in time-series form, it seems prudent to begin holding their claims to some inspection. Before settling into that task, I wish to say that I applaud without reservation the impetus for their article, which is that we should understand everything we can about the data we use to examine the world. With data, as with food, we are what we eat. Thus, caring as much or more about our data as we do our models is imperative for producing high-quality quantitative scholarship. Their article contributes to a highly important conversation.

In this article, I use a data set containing word counts of US State Department Country Reports on Human Rights Practices reports, as well as the CIRI Human Rights Data, to empirically address the viability of some of Clark and Sikkink’s core arguments. I find the assumption that CIRI scores are declining in recent years, the conclusion that longer State Department reports have a substantive role in these lower scores, and the argument that changing standards affected the CIRI time series data, to be suspect.

II. DO THE DATA REALLY SHOW THINGS ARE GETTING WORSE?

Clark & Sikkink’s key worry is that human rights scores are declining over time, while actual human rights practices may be getting better:

[S]cholars should keep in mind that the data are based on contemporaneous documents and understand the changing characteristics of the source materials over time. Because of increased quality and quantity of information, the data may skew toward worse scores in later years.⁴

They examine in detail the cases of Guatemala and Brazil as the basis for a generalized warning about the possibility of information effects downwardly biasing any country’s data. However, if the information effects posed by Clark and Sikkink are in play at large in the CIRI and PTS data, we should see significant, downwards trends over time in the global data. Or, one could also make the argument that stagnant trends in data -- in effect hiding real-world improvements-- might also be demonstrative of information effects. Both of these possibilities will be considered here.

Figure 1.

CIRI Physical Integrity Rights Index, Global Average, 1981–2011

A. Physical Integrity Rights

Let us look at a simple graph of the CIRI physical integrity index over time. This index combines measures of the level of government respect for extrajudicial killing, disappearance, political imprisonment, and torture into a nine-point additive ordinal index, ranging from 0 (no respect for any of these four rights) to 8 (full respect for all four rights).

What we see in Figure 1 is that the global average on this aggregate index of government respect for physical integrity rights is very stable over time. It is decidedly not decreasing over time, as one might expect were Clark and Sikkink’s version of information effects at work. There is a small dip in average respect...