The other day a friend told me something slightly shocking. They felt it made no sense anymore to commission external evaluations – because evaluation quality was shoddy and hardly anyone read the reports, anyway.
To see someone draw the consequences from such a sorry state of evaluation is shocking. But it does not surprise me that the quality of their evaluations is poor. So many evaluation TOR are flawed in international cooperation: sky-high expectations (dozens of difficult questions, e.g., on impact directly caused by a very limited intervention in a complex sector, and on long-term sustainability while the project is still running), a tiny timeframe easily counted in weeks, and a budget that is just enough for “two dudes, two weeks, 20 interviews” (that is a quote but I don’t know the source). That kind of set-up makes it very hard to come up with any new or deeper insights for people in the project or programme. Of course, an evaluation team can try to make the best of a bad situation and stimulate joint reflection within the short process, so that there is at least a bit of process use of the evaluation. Or they could compare existing evidence reviews with the project logic or theory of change to hypothesise about potential impact and sustainability. Or, if they are experienced specialists in the project’s subject matter, they could reframe the evaluation as some kind of specialist feedback. It is possible to draw some use from small evaluations. But small evaluations will not provide robust evidence across the six evaluation criteria (relevance, coherence, effectiveness, efficiency, impact, sustainability) used in international development cooperation contexts.
If we need robust evidence, then it may be a good idea to go for bigger evaluations that look at broader sets of interventions in their specific contexts. More use should be made of existing evidence (evidence reviews, academic research…) instead of trying to reinvent the wheel/find out from scratch. We can still have quicker, smaller, advisory-type evaluations to answer a handful of specific, practical questions that support decision-making. We can also inject more evaluative thinking into organisations, for example by developing a culture of internal evaluation or a system of regular smaller reviews.
There are so many possible ways of making evaluation more useful. Evaluation is an opportunity to gain insights that help us make things better. But you are going to miss that opportunity if you frame evaluation as a routine exercise, or as some kind of afterthought. Doing evaluation just because it has to be done is a waste of money indeed.