The OECD Quality Standards for Development Evaluation and the related „Better Criteria for Better Evaluation“ are major references for evaluations in international development cooperation. They have been crafted and revised (in a global consultation process 2018-2019, hence „better“) to support rigorous, nuanced analysis of development interventions that can feed into broader learning and deliberation on development effectiveness. Great – but the criteria must be applied thoughtfully, as recommended by the OECD in a dedicated publication (2021). What does that imply?
As explained in a blog post I wrote in 2018, those who develop evaluation questions must choose carefully which criteria need applying in an evaluation – and which not. The OECD recommends starting with just one question: „If we could ask only one question about this intervention, what would it be?“ That questions might touch on just one or two OECD criteria – which is fine in most cases (unless in the specific case where there is a need to obtain findings on all criteria across a whole set of interventions).
No OECD criterion can replace careful reflection about what an evaluation must find out to fulfil its purpose. Too often, I see evaluation terms of reference (ToR) that don’t use their own evaluation questions but simply use the definitions of the OECD-DAC criteria, e.g. „to what extent has the project achieved its objectives“. Sounds like a legitimate question. But if the evaluation is supposed to inform decisions as to what aspects of the project need strengthening, or to generate learning for future interventions, it is not enough to determine whether the intervention has fully met its objectives, or partly, or not. You need „what“, „why“ and „how“ questions to encourage evaluators to find that out.
Also, the OECD criteria are tailored to publicly funded development cooperation in low to middle income countries. Removing them from that ecosystem means that they lose their anchor. For instance, in an evaluation of a cultural festival in Berlin, assessing coherence with international commitments would make no sense. Impact, defined by the OECD as long-term effects on society is not a useful criterion when you evaluate a public transport provider’s new ticketing system. Applying the OECD criteria would generate irrelevant questions and distorted findings. You need to choose criteria that fit the context of an evaluation for evaluations that are meaningful, credible, and useful.