Working with Freedom House Scores

It has been some time since I have posted something new here due to my ever-increasing work load. However, in the course of working on one of my projects (a conference paper examining rival mechanisms of the diffusion of political liberalism), I came across an efficient solution to a problem that seems to be pervasive in  comparative and international relations political science: how to reshape Freedom House data into a useable format.

I should first be clear about something: Freedom House (FH) is not usually where I go to get data. For my research I typically use the POLITY data, since I am usually interested in operationalizing democratic political institutions. POLITY was created by political scientists rather than a NGO, and has a more transparent data collection process, and utilizes less controversial (and more stable) measures such as institutional constraints on the executive. While it is inappropriate to consider either POLITY or FH scores as true measures of democracy- as many comparativists have discussed at length in the literature- when political scientists need to measure concepts like democratization, they usually end up using one of these. Despite its useful qualities, the POLITY project is hampered by a slow updating process, as noted in a post by  political scientist Jay Ulfelder. For researchers interested in getting the most recent data, FH can do the trick. For my diffusion project, I needed data that covered the recent unrest in the Middle East and North Africa. Since the 2014 update to FH has been released, I was in luck.

However, as many who use Freedom House scores know, the data is not formatted for the typical political science application. In order to merge FH scores with economic indicators or conflict data, you need to reshape the data so that rows are by country-year and columns are the included measures of political rights and civil liberties. You need to attach Correlates of War country codes, and you also probably want the combined freedom score that informs FH’s status measure (free, partly free, not free). Most of all, you do not want to have to do this from scratch each year when FH updates their data. Ideally, there would be a package in R that could do this for you. Since there is not (that I am aware of), I am sharing the following code (adapted from Dave Armstrong’s solution to this issue), which automates most of the process.

There are two steps which you must do before running this code. The most recent version of the FH data should be saved as a CSV, removing the “Notes and Clarifications” section at the end of the spreadsheet. Since this information is unpredictable, I did not write its removal into the automated code. Also, the apostrophe must be removed from the “Cote d’Ivoire” country name, or the code will break. Once that is done, you can run the following code. The cleaned 2014 FH scores are available here for those interested.

[code language=”r”]

## read in column names
fh.names <- read.table("FH1973-2013.csv", header=F, skip=5, sep=",")[1:2, -1]

## read in data
fh <- read.table("FH1973-2013.csv", header=F, skip=7, sep=",")

## years covered by FH data
years <- na.omit(unique(t(fh.names)[,1])[-1])
years[10:17] <- c(1981, 1983, 1984, 1985, 1986, 1987, 1988, 1989)

## countries in the FH data
cn <- c("country", paste(gsub(" ", "", t(fh.names)[1:123,2]), rep(years, each=3), sep=""))
fh <- fh[1:205,1:124]
colnames(fh) <- cn
rownames(fh) <- NULL

## reshape the data
redat <- reshape(fh,
idvar = "country",
timevar = "year",
varying = list(PR=grep("^PR", colnames(fh), value=T),
CL = grep("^CL", colnames(fh), value=T),
Status=grep("^S", colnames(fh), value=T)
rownames(redat) <- NULL

## recode the year variable, making sure to skip 1982
redat$year <- redat$year + 1971
redat$year[which(redat$year > 1981)] <- redat$year[which(redat$year > 1981)] + 1

## recode the missing value
redat[which(redat == "..", arr.ind=T)] <- NA

## rename the variables
redat <- rename(redat, c(PR1972="political.rights", CL1972="civil.liberties", Status1972="status"))

## create COW country code for matching with other datasets, manual fix for Serbia
redat$ccode <- countrycode(redat$country, "", "cown")
redat$ccode[which(redat$country == "Serbia")] <- 345
redat <- redat[-which(redat$ccode == 345 &$status) | redat$status == ""),]

## create region using countrycode
redat$region <- factor(countrycode(redat$ccode, "cown", "region"))

## fix the S. Africa issue of dual coding in 1972
redat[which(redat$ccode == 560 & redat$year == 1972),3:5] <- c(5,6,"NF")

## set PR and CL scores to numeric and combine
redat$political.rights <- as.numeric(as.character(redat$political.rights))
redat$civil.liberties <- as.numeric(as.character(redat$civil.liberties))
redat$freedom <- (redat$political.rights + redat$civil.liberties)/2

Foreign Policy Contradictions and Undetermined Democratic Transitions

A few days ago I was perusing the BBC News site and came across an article by Tara Mckelvey that interested me. Suggestively titled “Is President Obama cosying up to dictators?” this piece is concerned with assessing how the Obama administration compares to its predecessors on democracy promotion. The author cites evidence that Obama administration has changed tactics to a more subtle approach rather than the overt posturing and direct funding of democratic activists that marked the G. W. Bush administrations. Despite this, Mckelvey notes that the total amount of funding for democracy promotion under Obama is actually higher than it was under G. W. Bush, however despite these democracy promotion efforts the U.S. remains close with several authoritarians in the Middle East. The author also claims that although overall democracy in the world is increasing, why it is so remains unclear:

“Still transitions are hard. And no-one really knows why they happen or what makes an authoritarian nation turn into a democracy. Meanwhile the US has been a bundle of contradictions.”

It is important that the public know about U.S. foreign policies such as democracy promotion through funding of democratic reformers and the work that IFES does. Mckelvey provides useful information in the article in this respect. However, the above excerpt reveals a two serious problems when viewed from the perspective of a political scientist.

First, Mckelvey is missing a key point about U. S. foreign policy that is essential to understanding the apparent contradiction between democracy promotion and aiding authoritarians. Well it is often broadcast to the American public and even to other actors in the international system that the U.S. is primarily interested in promoting human rights, the rule of law, and participatory democratic institutions throughout the world, the actual foreign policy agenda of the Obama administration reflects a different end goal: stability. It is well known to political scientists that rational actors in the international system desire stability for political, economic, and geostrategic reasons; political scientist Samuel Huntington has analyzed this extensively in his 1968 book Political Order in Changing Societies. While human rights and democracy are admirable goals, they cannot substitute stability goals for an administration that inherited two complicated wars/statebuilding projects, is challenged by rampant instability in the Middle East among allies and potentially hostile state alike, and faces political challenges at home. In terms of military commitments, the price of oil and other tied commodities, and the increased potential of state-sponsored terrorism, the U.S. currently cannot afford to forcefully promote democracy in the MENA region at the expense of insuring stability.

Second, Mckelvey rightly points out the overall increase in democracies worldwide since 1989. The figure below shows aggregated democracy scores by region and for the entire world (dark blue line). It is clear that democracy has indeed been increasing globally, with transitions in Eastern Europe (fall of communism), Sub-Saharan Africa (decolonization) and Latin America (transition from military dictatorships) driving global average increasingly higher. However, the author’s statement that the causes of democratic transitions are basically undetermined is not exactly true. Transitions are indeed “hard” but they are not unexplored in the scientific literature.

Average aggregate POLITY institutionalized democracy scores 1960-2012. Higher values indicate stronger democratic institutions.
Average aggregate POLITY institutionalized democracy scores 1960-2012. Higher values indicate stronger democratic institutions.

From the 1980s onward, political scientists have studied regime transitions in abstract and in specific cases (O’Donnel and Schmitter 1986, Linz and Stepan 1990, Przeworski 1991, Yashar 1997, Colomer 2000, Brownlee 2007). The comparative democratization literature and the persistent authoritarianism literature have contributed several relevant explanations as to why transitions occur and why they succeed or fail including rational decisionmaking among elites, the effect of mass social movements, political culture, economic conditions, institutional design, and international pressure. Detailed explanations of these factors are beyond the scope of this blog post, but suffice it to say we (political scientists) do have suggestions why transitions occur and why authoritarian regimes may turn into democracies. We also have many empirically-based claims on why transitions are particularly hard in persistently authoritarian places like the MENA region (see green line in above figure).

It is important to note, however, that Mckelvey and other journalists are likely not to blame for the misunderstanding that the causes and correlates of democratic transitions are unexplored. Instead, if there is blame to spread in this scenario, it rests mostly with political scientists themselves.

Few political scientists actively advertise the knowledge that we accrue in areas like transitions to democracy. Most scholarly work is simply that: written by scholars for other scholars. Empirical work on democratization seldom makes it past the small audience of other comparativists and international relations scholars. Some political scientists will try to engage policymakers, even fewer will try to engage the public. Often we assume that empirical work will be lost on non-political science audiences (which admittedly may be true) and so appeals to wider audiences come across as goal-oriented suggestions or partisan accounts rather than scientific explanations and predictions.

I am not suggesting that scholars write for a wider audience when presenting results of their research in papers and (most) books, as this is not always appropriate.  I do think that political scientists should start using newspaper and magazine contributions, blogs, and social media to reach journalists and the wider public. A great example of a political scientist doing this is Jay Ulfelder, who does forecasting on political development and instability. Another example is The Monkey Cage blog which features several political science contributors and is now located at the Washington Post, a good way to expand readership beyond other political scientists. Hopefully in the future journalists and others will at least be aware that topics like transitions to democracy have been studied at length by political scientists, so that features and editorials can better address such topics.

Officers’ Club: The Persistence of Military Regimes in Egypt

Naguib, Nasser, Sadat, Mubarak, and now Sisi. The officers’ club of Egyptian authoritarians continues to grow, but does this mean that democracy in Egypt is unattainable? History holds some clues to whether we can expect Egypt’s military to ever return to the barracks, and comparative analysis of the role of military authoritarianism in other states’ transitions to democracy has implications for the future of persistent military regimes in Egypt.

Historical Roots of the Officers’ Club

Even before the 1952 coup started a series of military-controlled authoritarian regimes in Egypt, Ottoman and Arab military officers charged with combating European colonization and Arab subjugation led popular revolts. It should then be unsurprising that the military has played a leading role in the turmoil unfolding in Egypt over the past few years. When unemployment once again became pervasive and the price of bread rose, Mubarak’s regime saw protests against the corruption and inefficiency which embodied the status quo; the lack of a meaningful opposition pushed the population to look to a new group of military officers to facilitate the transition to a new regime.

While it is clear that Islamists and democrats alike did not anticipate a return to military governance, historically it makes sense. In the past when Egypt suffered from corrupt or inefficient regimes the military provided the necessary muscle to oust the status quo government and provided the new leadership to steer the country back to economic security. This is, after all, how Naguib, Nasser, Sadat, and indeed Mubarak all gained membership in the Egyptian authoritarian officers’ club.

Club Membership in Comparative Perspective

The phenomenon of military-led coups and revolts is not particular to the Middle East and North Africa. Comparative political scientists have noted occurrences other regions as well; southern European, Latin American, and southeast Asian democratization efforts have often been spurred by military officers. Juan Linz and Alfred Stepan’s (1996) well-known volume discusses democratic transitions in Spain, Portugal, Greece, Latin American states, and Eastern European post-communist states. While the differences in process, timeline, and success are important, it should be noted that many of these comparative cases shared one factor in common: the presence of military actors at the inception of democratic transition. It should be noted that today all cases that Linz and Stepan cover, with the exception of Russia are strong, consolidated democracies despite histories of authoritarianism and military regimes. Why then do commentators continue to forecast failure of democracy where there are military dictatorships?

When elected institutions are removed by military force, past patterns show that the outcome is almost never favorable to democracy: outright military dictatorship, military-domination of politics with a civilian facade, civil war, civil unrest or a mix of all of the above. A few highlights include Spain in 1936, Iran in 1953, Chile in 1973, Turkey in 1980, Sudan in 1989, and Algeria in 1992. Dr. Omar Ashour

The problem with this logic is its temporal frame: of the six cases referenced by Dr. Ashour, half of them (Spain, Chile, Turkey) are now consolidated democracies that have been able to deal with regional separatist actions, economic crises, and domestic and international terrorism without reverting to military rule in the long run. While elements of the military were more passive and implicitly permissive in Spain (as opposed to active and explicitly instigating in Portugal) the potential of another reverse transition like what Ashour suggests was a real concern in the early years due to regional separatism. In Chile the military remained in a strong bargaining position during the transition, and despite the trials for human rights violations and other retribution against the military post-transition, Chilean democracy since 1989 has remained stable. Turkey perhaps provides the best example of the danger of the domination of domestic politics by the military that Ashour indicates, yet even Turkey has seen relative institutional democracy stability since the 1983 constitution. Despite persistent terrorist attacks, the Kurdish separatist movement, economic crises, and the return of conservative political Islam, civilian governments in Turkey have been able to govern without military intervention to change the constitution or restructure government as it has done historically (Barkey and Taspinar 2011). Additionally, a fourth case, the Iranian coup in 1953, should not be considered a military intervention in the same sense as the other five cases. Few Iranian military officers participated (they were bribed by CIA operatives) and the coup was not to create a military regime but rather to consolidate the power of the Shah. Therefore, only Sudan (1989) and Algeria (1992) really corroborate Ashour’s statement.

Is Egypt the new Turkey?

The evidence from southern Europe and Latin America especially over the past thirty years has revealed that regime stability and democratic institutions can grow from transitions influenced by military officers in countries with histories of military rule. So why does it seem that some commentators are claiming that military-led authoritarian regime persistence in Egypt at the expense of democratization is inevitable?

Again, this is a matter of temporal frame. The current view of democratization in Egypt is much the same as it was for observers of Turkey in the early 1980s. The political reach of the officer’s club in Turkey seemed to preclude any positive assessments of democratization, especially with the recency 1980 coup. The likelihood of a conservative Islamic party (AKP) controlling government and adding seats over three successive elections seemed near zero in the frame of the early 1980s. The 1983 constitution that the military put in place allowed for the rise of the AKP, and the military (despite public threats) refrained from intervention even as the government steers domestic policy dangerously close to violating the secular constitutional commandments.

The lesson here is that transitions to democracy from military rule do not happen overnight. In Egypt, the military will retain a powerful position even if free elections are allowed in the near future. Like in the Turkish experience, the officer’s club in Egypt will be reticent to embrace a democratic institutional arrangement where they have no power, especially to prevent their most coherent challenger, here the Islamist Muslim Brotherhood, from usurping state power. Majority Muslim societies struggle with the tradeoff between the principles of representative democracy and the principles of political Islam. This is not to say that these concepts are incompatible, but the negotiation and compromise must take place in a setting where democrats and Islamists cannot exclude each other from the political process. In Turkey this “neutral” space was enforced by the military, who following the tradition of Turkish culture since 1920s, understand their role as guardians of the Turkish republic. In Egypt the military has an opportunity to create a “neutral” space for negotiations, but it is unclear for now if al-Sisi is playing the role of protector or el presidente.

Does Rouhani Represent Real Change for Iran?

The implications of the recent Iranian presidential contest last month seem to have faded away in the face of mounting violence in Syria and related spillover in Lebanon, Yemen terror threats, and a rising death toll in Egypt. As journalists and policy experts scramble to address these immediate and violent concerns, a pseudo-democratic election in a country that is not mired in civil war or domestic unrest is left for discussion another day. Yet Iran is a pivotal state in the region for both U.S. and European foreign policy agendas as well as general regional stability. It is then important to ask if Rouhani represents real practical political change for Iran, or is it simply a symbolic gesture by Ayatollah Khamenei to avoid the protests following the 2009 presidential election?

Iran is a very unique state in the Middle East in several ways. With a population greater than 77 million that is only 3 percent ethnic Arab and 89 percent Shi’a Muslim, the country is both larger and more diverse than many states in the region. Yet ethnic and religious divides (aside from small segments of Kurds and Baha’is) is coded into the Iranian constitution – seats for Jewish, Christian, and other groups are by law free to practice (Ch. 3 Art. 23) and have seats in parliament reserved for them (Gasiorowski 2011). Another difference between Iran and other states (particularly the gulf states) is extensive electoral structure in accordance with the constitution. Local elections are common, and political parties have organized in the last few decades. While the national political structure remains an Islamic republic headed by the Supreme Leader and controlled by a series of institutional checks and vetting procedures, elections routinely occur and parliament does produce laws.

Rouhani (a moderate cleric) as the pick in the presidential contest reveals a few important details about the likely future of Iranian domestic politics and relations with regional neighbors and the west. It is incorrect to assume that Rouhani is a liberal reformer or that he would be able to implement a liberal reform agenda if he was. The political structure of Iran prevents even the most adept politician from achieving policy changes unfriendly to the principles of the Supreme Leader due to judicial and clerical review processes. However, this is not simply a symbolic gesture by the Supreme Leader, it is a strategic decision. Khamenei’s acceptance of Rouhani sends some important signals to the international community on what to expect from Iran in the near future.

First, it is not a coincidence that Rouhani was chief nuclear negotiator in Ahmadinejad’s first presidency. The west has a chance to reopen negotiations in the face of presidential turnover, and with Rouhani at the helm the west can expect sincerity and commitment to the process. The U.S. has picked up on this possibility already, offering a tentative olive branch to the new leader. This reveals a marked change in priorities for the Supreme Leader, who now seems interested in hearing what the west has to offer on the nuclear front after Ahmadinejad’s vitriolic “nationalist” approach fell flat.

Also, Rouhani’s diplomatic past is a signal to other states in the region that Iran may work towards improving ties with its Arab neighbors (Such as Saudi Arabia) and relaxing its provocation of Israel. While Israel remains skeptical of any real change, Rouhani seems genuine in his push back the isolationism of his predecessor. This too reveals that Khamenei is interested in altering Iran’s global image.

Ultimately, Rouhani is deeply committed to improving domestic conditions through negotiating an ease of sanctions, reforming economic policy, and improving relations with neighboring states and the west. While nothing that we know about Rouhani indicates that he is interested in liberalism, it is clear that the Supreme Leader has made a strategic choice in accepting Rouhani as Iran’s new president. Whether real change will accompany Rouhani largely depends on how we measure that change. Fundamental alterations to the political system will not occur. Diplomatic progress is not assured, despite Rouhani’s past. What we can expect, however, is a commitment among Iran’s leaders to reshaping the country’s image from radical isolationism to cautious integration in the international system. This will occur through toning down rhetoric and a focusing on domestic problems like the economy rather than defiance of the west and Israel.

Parsing textual data from urls using R

I have developed several unique approaches to data gathering while working on the Local Elections in America Project (LEAP), and I could not resist sharing a function that I recently created in R to gather elections data in character-separated value text format. The function can take either a single url or a list of urls, and can be customized to handle different separation characters and column headings easily.

The following code can be altered to handle diverse data in character-separated value format, not just election data. While this function is intended to collect text data located at a web address, it might be altered to collect from html code with the addition of regular expressions removing common html tags.

[code language=”r”]
## Created by Nicholas Davis on 2013-08-07.
## set up the data frame
dat <- matrix(, 0, 5)
colnames(dat) <- c("precinct", "office", "", "vote.pct", "")

## the urls targeted contain first 4 columns, no date
## function that takes a datafame, url or list of urls, and an election date
parse.url <- function(data, url,{
d <- data.frame(matrix(,0,5))
colnames(d) <- colnames(data)
for(i in 1:length(url)){
tmp <- NULL
tmp <- readLines(url[[i]])
tmp <- gsub("’", "", tmp) ## remove troublesome characters from names
tmp <- gsub("\#", "", tmp) ## remove comment character from lines
write(tmp, "tmp.txt")
d <- rbind(d, read.table("tmp.txt",
sep=";", ## this can be changed to "," or "\t"
d <- cbind(d, ## append election date manually
rbind(data, d)

## call the function and set dataframe to equal the output
dat <- parse.url(dat, c("url1", "url2" . . . "urlx"), "11-06-2012")

The What and Why of DemocracyObserver

I started this blog in part as an exercise in communicating my scholarly research and in part as an outlet for my interpretations and reactions to news stories and other developments within my substantive areas of interest. It is my sincere hope that those persons (academics as well as casual observers) that find the history, politics, and culture of the Middle East and North Africa and other states in the Muslim world will find this blog a valuable source of information and a place to engage in meaningful discussion of the issues that face the region, United States foreign policy interests, and diaspora communities across the globe.

This blog will also be a space for me to update interested parties on developments in my scholarly research and news about projects that I contribute to. I encourage others who have methodological innovations or announcements about their research to contribute here – I firmly believe that as social scientists, we need to make the greater academic community aware of our research as it develops as well as engage in discussion when we have methodological problems.

More to come in the (probably near) future…