A theory of change is a wonderful instrument to explore the „why“ and „how“ of an intervention. So why do evaluations make such patchy use of theories of change? Often, it is because evaluation questions ask mainly“how much“. This blog narrates how I have come to this conclusion.
Earlier this month, the Methods Circle of the German Evaluation Society (DeGEval) held its spring meeting under the caption „nothing is as practical as a good theory“. I was one of the few specialists in international cooperation in the room, as the Development Policy Circle met in a different town on that day. It was a delight exchanging with colleagues from other sectors.
Anna von Werthern shared her insights from developing theories for theory-based evaluation. Right in her introduction she pointed out that a programme theory or theory of change allows you to explore „why“ and „how“ questions. The theory of change explains the little arrows that connect activities and effects in those programme diagrams. When evaluating international cooperation, we often (re-) construct a theory of change in an inductive manner, reviewing programme documentation and gathering data from stakeholders to understand how they want the desired change to happen. Ideally, we should also use deduction, checking external literature for evidence-based theories that could be used to explain cause-to-effect chains within theory of change. But when there is no science-validated theory at hand and those implementing the programme cannot or do not want to explain how their programme is supposed to work, then you need to develop theory. That is called abduction (sounds a bit weird in English).
In Anna’s example, she developed a first TOC after reviewing the programme documentation. Then, she carried out a series of success case interviews, exploring courses for success and failure. After reviewing the emerging TOC, she followed a semi-structured interview protocol to deepen insights and fill gaps with her interlocutors, ending up with a workable, context-relevant theory of change. Most evaluators in international cooperation – especially those working on the many smaller, shorter evaluations – will not have enough resources for such thorough work. Even reconstructing a theory of change with an overworked programme team that has not asked for the evaluation to happen in the first place, can be a major challenge. And then, the question is – do clients who ask for a theory-based evaluation always really want or need it?
I review dozens of evaluations every year. Very rarely, the evaluation terms of reference ask questions about the “why” and “how” of an intervention. Most of the time, most questions are structured along the time-tested OECD-DAC criteria (relevance, coherence, effectiveness, efficiency, impact, sustainability). Most questions start with the phrase “to what extent did/ have…” That means that the evaluation primarily concerns performance – the degree to which certain criteria have been met. Such questions do not encourage evaluators to unpack the black box about how an intervention works and why it contributes to certain effects (or not). Unsurprisingly, many evaluations seem to make little use of theories of change. If a client only wants to whether certain things have happened that are (likely to be) linked to the project, there is only so much you can do with a TOC. Hence my appeal to those who commission evaluations: if you want the evaluation to use a TOC, make sure you ask “why” and “how” questions – and provide the resources it takes to answer them!