Jul
30

The Tyranny of Metrics

Jerry Z. Muller's "The Tyranny of Metrics" (2018) has a title suggestive of an essential read, but in detail it runs thin. The author may have aimed for a broad audience, providing a brief overview. For those versed in the challenges of metrics, this is not your book. Nonetheless, there are some interesting points, including:

  • "gaming the metrics occurs in every realm: in policing; in primary, secondary, and higher education; in medicine; in nonprofit organizations; and, of course, in business. And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction. There are things that can be measured. There are things that are not worth measuring. But what can be measured is not always worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge - knowledge that seems solid but is actually deceptive." (p. 3)
  • "What has largely gone unnoticed is the recurrence of the same unintended negative consequences of performance metrics, accountability, and transparency across a wide range of institutions... while they are a potentially valuable tool, the virtues of accountability metrics have been oversold, and their costs often underappreciated. It [the book] offers an etiology and diagnosis, but also a prognosis for how metric fixation can be avoided, and its pains alleviated." (p. 6)
  • Education: "The unintended consequences of NCLB's [No Child Left Behind] testing-and-accountability regime are more tangible, and exemplify many of the characteristic pitfalls of metric fixation. Under NCLB, scores on standardized tests are the numerical metric by which success and failure are judged. And the stakes are high for teachers and principals, whose raises in salary and whose very jobs sometimes depend on this performance indicator. It is no wonder, then, that teachers (encouraged by their principals) divert class time toward the subjects tests - mathematic and English - and away from other subjects, such as history, social studies, art, music, and physical education. Instruction in math and English is narrowly focused on the sorts of skills required by the test, rather than broader cognitive processes: that is, students too often learn test-taking strategies rather than substantive knowledge." (p. 92)
  • "Ultimately, the issue is not one of metrics versus judgement, but metrics as informing judgement, which includes knowledge how much weight to give metrics, recognizing their characteristic distortions, and appreciating what can't be measured. In recent decades, too many politicians, business leaders, policymakers, and academic officials have not sight of that." (p. 183)
  210 Hits
210 Hits
May
02

A Diplomat’s Memoir of South Sudan

When I first came across Nicholas Coghlan's "Collapse of a Country: A Diplomat's Memoir of South Sudan" (2017), I passed it up. Memoirs can be interesting, but not always great (unless you are interested in the day to day activities and experiences, often without in-depth contextualization). However, while on route to South Sudan myself, I read the book, and highly recommend it. It is a fascinating read, and quite well contextualized in issues well beyond a typical memoir.

On governance, one of the journalists he speaks with explains "Yes, there is a tendency towards dictatorship in South Sudan," he admit. "But you know what? This will never be a dictatorship like Eritrea's. To be a dictator you have to be efficient and you have to have a vision. Neither apply in our case." (p. 47).

On conflict resolution: The resolution of the Jonglei Crisis had followed a well-worn pattern. A Big Man becomes dissatisfied with the status quo and finds himself unable to get his way by peaceful means. He takes to the bush and assembles an armed militia. He creates mayhem. In the end, he accepts an offer of cash and promotion and comes back in. Until next time. The practice often brought peace in the short term, but over the medium to long term, It encouraged and rewarded rebellion." (p. 70). See also De Waal (2015) on this point. There are some challenging reflections on the future, such as Coghlan's reflection that "it would take more than a generation for South Sudan to get over this situation [lived experiences of conflict]" (p. 32). As the peace negotiations enter into new rounds, with similar faces making few compromises, the prediction continues to be a likely one.

On aid and priority setting: "A particularly interesting finding of the in-country surveys was that most communities identified inter-ethnic reconciliation as their top priority for donor support (this with the caveat that polling and surveys are notoriously problematic in South Sudan). I was intrigued but not surprised by this after years of observing the civil war, which as often as not pitted southerners against each other rather than against northerners. But for newcomers to South Sudan, this seemed aberrant. More to the point, how could you achieve "reconciliation" and how did you establish benchmarks? When we huddled with the government to reach a consensus over priorities, reconciliation shifted near the bottom of the list" (p. 108).

On the (lack) of accountability: "A very large convoy of World Food Program trucks carrying mainly food supplies north to the POC camp in Bentiu was hijacked and looted near Mundri, Western Equatoria State. When they were released, the drivers described their assailants as armed and uniformed. There was no doubt about where at least three of the trucks were taken; GPS tracking showed them to be inside the SPLA barracks in Yei. WFP supplied all donors with a list of the value of their goods that had been stolen; in the case of Canada, the total was US$ 300,000. But WFP insisted that we not a make public statement, let alone press the government for an explanation. They were more concerned with getting the trucks back in tact – forget the food seized – and not endangering further their already difficult relationship with the government." (p. 199)

On Canada and staffing challenges (and some self-reflection): "It seemed that the younger generation in the Canadian foreign service were not motivated by what had attracted me: the prospect of travel to exotic places, a whiff of danger and excitement, and being a big fish in a small pond. They preferred the classic "cushy" posts – London, Rome, Washington – if they wanted to go abroad at all." (p. 197)

  678 Hits
678 Hits
Aug
29

Encountering Development

One of the books commonly cited and recommended in critical development studies circles is Arturo Escobar's Encountering Development: The Making and Unmaking of the Third World (1995). Escobar is a Colombian-American Anthropologist, who interestingly started his academic career in chemical engineering. The award-winning book is typically summarized as reframing 'development' as a tool of control akin to colonialism. As a central idea, however, this was not entirely new: Amin (1976) has waged critiques of this nature, Weber (1976) had done similarly with the history of France, not to mention Scott (1985), Mintz (1985) and Ferguson (1990). The strength of this book, I believe, was not its originality but its bringing together of ideas.

Escobar begins by explaining that Encountering Development "tells the story of this dream and how it progressively turned into a nightmare. For instead of the kingdom of abundance promised by theorists and politicians in the 1950s, the discourse and strategy of development produced its opposite: massive underdevelopment and impoverishment, untold exploitation and oppression. The debt crisis, the Sahelian famine, increasing poverty, malnutrition and violence are only the most pathetic signs of the failure of forty years of development. In this way, this book can be read as a history of the loss of an illusion" (p. 4). Recognizing the book was written in the 1990s, it is interesting to contrast the framing of development presented by Escobar and that of Kenny in his 2011 book Getting Better – not to be contrasted here, but a worthwhile comparison for students of development studies.

"To understand development as a discourse, one must look not at the elements themselves but at the system of relations established among them. It is this system that allows the systematic creation of objects, concepts, and strategies; it determines what can be thought and said. These relations – established between institutions, socioeconomic processes, forms of knowledge, technological factors, and so on – define the conditions under which objects, concepts, theories, and strategies can be incorporated into the discourse. In sum, the system of relations establishes a discursive practice that sets the rules of the game: who can speak, from what points of view, with what authority, and according to what criteria of expertise; it sets the rules that must be followed for this for that problem, theory, or object to emerge and be named, analyzed, and eventually transformed into a policy or a plan." (p. 40-41)

While critiquing the defining of development, Escobar also offers his own definitions, ones that also disfranchises those who experience poverty from contributing their voice. For example, Escobar has a rather romantic vision of life before or outside of market influence (also assuming that all people before capitalism did not engage in market trade), in saying: "it is true that massive poverty in the modern sense appeared only when the spread of the market economy broke down community ties and deprived millions of people from access to land, water, and other resources. With the consolidation of capitalism, systemic pauperization became inevitable" (p. 22). This vision neglect social differentiation within communities, and also the serious challenges faced. This includes, for example, the poverty associated with enslavement and of famine, long pre-dating capitalism. Using Escobar's approach, it ought to be people themselves that define poverty, for their local circumstances, which we unfortunately do not find, and thus some of the critiques Escobar makes of others, also apply to his book (e.g. speaking about and for others).

As with all books, many of the concerns raised within them are time specific. Rather than focus on those that have changed, consider one that all development students, practitioners and academics can reflect on: "The underlying premise of this investigation is that as long as institutions and professionals are successfully reproducing themselves materially, culturally, and ideologically, certain relations of domination will prevail; and to the extent that this is the case, development will continue to be greatly conceptualized by those in power" (p. 106).

On accountability: "If 'the bank' does not have clear answers, nobody else does. Being 'the bank,' however, it can take some risks, and if 'some of the experiments fail,' they will bow to the difficulties of life (in the Third World) and humbly start all over again. Quite a comfortable position, especially if we consider that it is not they who have to suffer the consequences of failure, because the loans are paid back by Third World people" (p. 160-161).


  730 Hits
730 Hits
Subscribe to receive new blog posts via email