In 2012, I published a post with the title above on my former blog. And I still see major evaluations with budgets running into hundreds of thousands of euros that come with a four-week inception phase, or that are supposed to start basically the day after the evaluation firm or evaluator has been selected. That is wasteful, because an evaluation that is not tailored to its users‘ needs risks being… useless.
Ideally, one should start planning evaluations right when the project/programme that is supposed to be evaluated starts. Back in 2012, I recommended to start recruiting evaluation teams at least six months ahead of the field work – at that time, the evaluations I had on my mind were evaluations of individual projects run by civil society organisations (CSOs). With anything bigger or more complicated, I’d plea for much, much more time for finding the evaluation team, briefing it and developing a robust evaluation design with instruments that fit their purpose. But the gist of my 2012 blog is still valid – and I had promised to re-publish a few of my earlier posts. Here it is:
There has been an extraordinary flurry of calls for proposals for external evaluations. This is good news; it suggests that people find it important to evaluate their work. But, upon closer examination, you’ll notice that many calls envisage the evaluations to begin just a couple of weeks after the deadline for offers, and to end within a month or so. That is frustrating for experienced consultants, who tend to be fully booked several months ahead. Narrow time-frames may also make it difficult for those who commission the evaluation to identify sufficiently skilled and experienced candidates. If you take evaluation seriously, then surely you want it to be done in the best possible way with the available resources?
Over the years, I have come to appreciate time as a major element of evaluation quality. Most development organisations (not only CSOs) cannot and do not want to afford full-fledged scientific-quality research, which typically involves plenty of people with advanced academic degrees and several years of research. That is perfectly reasonable: if you need to make programme decisions on the basis of evaluations, you can’t afford to wait for years. (The programme would be over, the context would have changed, your organisation would have changed their priorities, to quote but a few likely problems.) But what one can afford – even on a shoestring budget – is to allow plenty of time for thinking and discussing during the preparatory phase of an evaluation. In that way, you can make sure (among other things):
- the terms of reference (TOR) express exactly what you need
- the participating organisations are well-prepared and welcoming (which they are more likely to be if the TOR have been worked out with them and take their wishes into account)
- the evaluation team understands what they are supposed to evaluate
- the evaluators can reflect on different options, discuss these with key evaluation stakeholders, and let their thoughts mature over a few weeks before deciding on the final design
- there is enough time to sample sites & projects/components so as to achieve a maximum of representativeness or a good choice of cases – to avoid visiting only what a Chinese expression calls „fields by the road“
- data collection tools can be pre-tested, adjusted, those collecting the data trained and so forth
Extra time for these activities does not necessarily mean more consulting-days – just spreading out the days budgeted for, and identifying and finding ways of making better use of existing data in the project, can make a big difference.