Thousands of evaluations have been conducted across Africa, producing large amounts of knowledge. However, these reports are not often captured and shared nor are they easily accessible, nor are always made available to the public. There is no search platform, like Web of Science for academic publications, for evaluations. Some donors have created their own databases, but none operated sector-wide. CLEAR-AA tried to address this, and developed a platform, which is updated until 2015, the African Evaluation Database. While much more work is needed, this is a good step. Those involved with the project wrote a book to present the trends of what they have seen: Evaluation Landscape in Africa: Context, Methods and Capacity (2019), edited by Mapitsa, Tirivanhu and Pophiwa. The book is an excellent resource for those interested in this type of material and/or work. As far as I know, no similar landscape assessments are available elsewhere, making this quite a unique resource. 

In development studies and practice we can get excited by new ideas, and over-stretch them. Participation was a cure all, then it became tyranny, and now we have more informed 'split ladders' that help determine when, where, why and how participation can work well. The rise of results- and evidence-based decision making was at its peak (the randomistas ran the day), there emerged naysayers, and now we have 'navigation by judgment' – an assessment of thousands of evaluations to understand when, where, why and how top-down processes work (and alternatively when more flexible approaches are warranted).

Enter Dan Honig's "Navigation by Judgment: Why and When Top-Down Management of Foreign Aid Doesn't Work" (2018). He writes that the argument "is not, then, that Navigation by Judgement is always superior; nor is it that Navigation by Judgement allows IDOs [international development organizations] to improve their absolute level of performance as environments become less predictable or projects less verifiable. It is simply that Navigation by Judgment is sometimes a good idea, particularly as contexts become harder to navigate using top-down controls and measurement… Navigation by Judgement is a second-best strategy – a strategy to employ when it is less bad than the distortions and constraints of top-down control" (p. 9).

A mixed method approach found that there is "strong evidence that Navigation by Judgement is frequently, but not always, useful. We have evidence that at least some of what determined when Navigation by Judgment is useful relates to the nature of the environment and the tractability of the project to top-down controls, which in practice often means reporting against quantitative output targets… As predicted, econometric analysis drawing on the PPD – the world's largest database of development project outcomes – suggests there are greater returns to Navigation by Judgement in less predictable environments. This is not because Navigation by Judgment actually leads projects to be more successful as predictability falls. Greater propensity to Navigate by Judgment simply cushions the falls, with high Navigation by Judgment-prone IDO performance declining less as environmental predictability rises" (p. 133).

Ideas on where different approaches might be suitable? Consider sectors: "from 2000 to 2012, 64.9 percent of all rigorous evaluations focused on the health sector with an additional 23.1 percent of studies focused on the education sector. By contrast, on 3.3 percent of studies focused on attempts to improve public-sector management. This is quite probably because it is very difficult to identify a plausible counterfactual and/or externally verifiable outcome measures for many public-sector management projects. Some of the same factors make Navigation by Judgement more beneficial for a particular project also make impact evaluations more difficult, precluding an econometrically rigorous examination of a particular project's results. Navigation by Judgement is most helpful where rigorous evaluation is most difficult and where rigorous evaluation is the least likely to build a robust knowledge base" (p. 153).

Ways forward? "One way forward is for an IDO attempting to implement a project that is difficult to effectively manage using measurement of either outputs or outcomes is simple, if somewhat radical, for IDOs: Stop using measures for the purposes of evaluating interventions or managing agents. There is no need to eliminate measurement; measures simply need to be repurposed. Measures can still be good for organizational learning. Learning is often put forward as a primary goal of IDO evaluation. International development organizations could deepen this focus on learning, sometimes putting aside the use of measures as tools of management control" (p. 155).

However, it is not always as easy to do so. "Moving toward greater Navigation by Judgment where appropriate is not without challenges; changing organizational management strategy involves risk for those IDO managers and political authorizers who might push for its adoption. But these risks need to be weighed against the benefits of better performance. To do otherwise is to condemn some foreign aid efforts to meaningless numbers and a façade of success that does little for aid's intended beneficiaries. In many contexts, political authorizers and IDOs are likely to achieve better development results by simply letting go" (p. 168).

Jerry Z. Muller's "The Tyranny of Metrics" (2018) has a title suggestive of an essential read, but in detail it runs thin. The author may have aimed for a broad audience, providing a brief overview. For those versed in the challenges of metrics, this is not your book. Nonetheless, there are some interesting points, including:

• "gaming the metrics occurs in every realm: in policing; in primary, secondary, and higher education; in medicine; in nonprofit organizations; and, of course, in business. And gaming is only one class of problems that inevitably arise when using performance metrics as the basis of reward or sanction. There are things that can be measured. There are things that are not worth measuring. But what can be measured is not always worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge - knowledge that seems solid but is actually deceptive." (p. 3)

• "What has largely gone unnoticed is the recurrence of the same unintended negative consequences of performance metrics, accountability, and transparency across a wide range of institutions... while they are a potentially valuable tool, the virtues of accountability metrics have been oversold, and their costs often underappreciated. It [the book] offers an etiology and diagnosis, but also a prognosis for how metric fixation can be avoided, and its pains alleviated." (p. 6)

• Education: "The unintended consequences of NCLB's [No Child Left Behind] testing-and-accountability regime are more tangible, and exemplify many of the characteristic pitfalls of metric fixation. Under NCLB, scores on standardized tests are the numerical metric by which success and failure are judged. And the stakes are high for teachers and principals, whose raises in salary and whose very jobs sometimes depend on this performance indicator. It is no wonder, then, that teachers (encouraged by their principals) divert class time toward the subjects tests - mathematic and English - and away from other subjects, such as history, social studies, art, music, and physical education. Instruction in math and English is narrowly focused on the sorts of skills required by the test, rather than broader cognitive processes: that is, students too often learn test-taking strategies rather than substantive knowledge." (p. 92)

• "Ultimately, the issue is not one of metrics versus judgement, but metrics as informing judgement, which includes knowledge how much weight to give metrics, recognizing their characteristic distortions, and appreciating what can't be measured. In recent decades, too many politicians, business leaders, policymakers, and academic officials have not sight of that." (p. 183)
  

Royal Roads University is seeking a Post-Doctoral Researcher to contribute to the program of the Canada Research Chair in Sustainability Research Effectiveness.

The CRC program aims to

1. Develop an international network of collaborators with a focus on impact and effectiveness in sustainability research
2. Develop and test evaluation approaches and methods for transdisciplinary research, building on emerging ideas and theoretical and practical experiences in related fields
3. Analyse and evaluate selected livelihoods, community resilience and sustainability research projects/programs
4. Recommend improved approaches and methods for transdisciplinary research evaluation and impact assessment

As the Post-Doctoral Researcher working with Dr. Brian Belcher, Canada Research Chair (CRC) in Sustainability Research Effectiveness, you will focus on assessing the effectiveness and the impact of research, especially research that aims to improve livelihoods, community resilience, and environmental sustainability. You will contribute to the development of theory and methods for conceptualizing and evaluating problem-oriented and trans-disciplinary research and to individual case studies and comparative analyses. The CRC has an ongoing international portfolio of research activities and there will be scope for you to develop and lead new projects as part of the larger program.

Additional details.