Navigation by Judgement
In development studies and practice we can get excited by new ideas, and over-stretch them. Participation was a cure all, then it became tyranny, and now we have more informed 'split ladders' that help determine when, where, why and how participation can work well. The rise of results- and evidence-based decision making was at its peak (the randomistas ran the day), there emerged naysayers, and now we have 'navigation by judgment' – an assessment of thousands of evaluations to understand when, where, why and how top-down processes work (and alternatively when more flexible approaches are warranted).
Enter Dan Honig's "Navigation by Judgment: Why and When Top-Down Management of Foreign Aid Doesn't Work" (2018). He writes that the argument "is not, then, that Navigation by Judgement is always superior; nor is it that Navigation by Judgement allows IDOs [international development organizations] to improve their absolute level of performance as environments become less predictable or projects less verifiable. It is simply that Navigation by Judgment is sometimes a good idea, particularly as contexts become harder to navigate using top-down controls and measurement… Navigation by Judgement is a second-best strategy – a strategy to employ when it is less bad than the distortions and constraints of top-down control" (p. 9).
A mixed method approach found that there is "strong evidence that Navigation by Judgement is frequently, but not always, useful. We have evidence that at least some of what determined when Navigation by Judgment is useful relates to the nature of the environment and the tractability of the project to top-down controls, which in practice often means reporting against quantitative output targets… As predicted, econometric analysis drawing on the PPD – the world's largest database of development project outcomes – suggests there are greater returns to Navigation by Judgement in less predictable environments. This is not because Navigation by Judgment actually leads projects to be more successful as predictability falls. Greater propensity to Navigate by Judgment simply cushions the falls, with high Navigation by Judgment-prone IDO performance declining less as environmental predictability rises" (p. 133).
Ideas on where different approaches might be suitable? Consider sectors: "from 2000 to 2012, 64.9 percent of all rigorous evaluations focused on the health sector with an additional 23.1 percent of studies focused on the education sector. By contrast, on 3.3 percent of studies focused on attempts to improve public-sector management. This is quite probably because it is very difficult to identify a plausible counterfactual and/or externally verifiable outcome measures for many public-sector management projects. Some of the same factors make Navigation by Judgement more beneficial for a particular project also make impact evaluations more difficult, precluding an econometrically rigorous examination of a particular project's results. Navigation by Judgement is most helpful where rigorous evaluation is most difficult and where rigorous evaluation is the least likely to build a robust knowledge base" (p. 153).
Ways forward? "One way forward is for an IDO attempting to implement a project that is difficult to effectively manage using measurement of either outputs or outcomes is simple, if somewhat radical, for IDOs: Stop using measures for the purposes of evaluating interventions or managing agents. There is no need to eliminate measurement; measures simply need to be repurposed. Measures can still be good for organizational learning. Learning is often put forward as a primary goal of IDO evaluation. International development organizations could deepen this focus on learning, sometimes putting aside the use of measures as tools of management control" (p. 155).
However, it is not always as easy to do so. "Moving toward greater Navigation by Judgment where appropriate is not without challenges; changing organizational management strategy involves risk for those IDO managers and political authorizers who might push for its adoption. But these risks need to be weighed against the benefits of better performance. To do otherwise is to condemn some foreign aid efforts to meaningless numbers and a façade of success that does little for aid's intended beneficiaries. In many contexts, political authorizers and IDOs are likely to achieve better development results by simply letting go" (p. 168).
