Evidence vs impact: The basics

In this series we explore the use of evidence for understanding impact and the issues surrounding its use. We look into how it can inform charities’ work and guide donors in deciding what to fund; the difficulties in finding good evidence and what you should be looking for as a donor, as well as the moral and ethical considerations around its use.

The ultimate goal of any donor is to make a difference with their funds, no matter what the cause is.

But with so many different organisations working on the same issues, how do you decide which organisation or programme is the best use of your donation?

Perhaps you’re already donating and wondering how to monitor whether your donations are making a difference.

In both cases, this is where evidence comes in.

Do impact reports demonstrate evidence?

Your first port of call for finding evidence may be to look at a charity’s latest impact report. There are plenty being produced by charities. But Rhodri and Matthew both think this is because donors expect them and are asking for them, not because charities see the benefits of producing them for themselves. This is leading to several consequences.

The first, according to Matthew, is that charities tend to view impact reporting as a way to secure more donations. This has led to the rise of ‘snake oil salesmanship’ and a proliferation of organisations claiming to measure donor impact, while simply sharing outputs, stories and short-term outcomes, without robust long-term evidence.

The second, according to Rhodri, is that it’s the donor rather than the charity who is deciding the impact a charity should have. Donors often come to an organisation offering significant donations for a specific programme or idea in mind, expecting diligent reporting and results. In reality, the initiative may not fit the charity’s mission, nor be how they’d choose to use the money given the choice.

What do we mean by impact and evidence?

Impact’ is the word of the moment in philanthropy, the charity sector and even investing. While everyone wants to be impactful, what’s missing is an agreement or understanding of what that means. At CAF, we define impact and evidence as:

Impact: the final, often long-term, positive result of a programme, such as eradicating polio.
Evidence: proof – ideally independently assessed – that a programme or intervention works in its intended way.

Why evidence is important for achieving impact

To truly understand your impact, you need data to show that the particular programme or intervention you’re funding works the way it was supposed to, and makes a positive difference to beneficiaries. This can be obtained by finding the counterfactual – what would have happened were it not for the particular programme or intervention – as well as other methods detailed below. You also need to be able to tell whether any results have been sustained, bringing long-term benefits for the beneficiaries.

The only real way to know this? Hard evidence.

If a charity can provide robust evaluation data to show that their programme is working as designed, or that it’s following a well-evidenced approach, you know they’re likely to achieve a good impact. On the other side, a lack of evidence can lead to programmes that are ineffective or, at worst, even cause harm. There are some situations, however, where evidence is harder to come by or simply won’t warrant an evidence base – such as brand new interventions, systems change or campaigning work – which we explore in our Evidence vs impact: questioning the use of evidence article.

But solid evidence can help charities – and you – understand what works, for whom and why, so that they can realise their missions. Collecting evidence about their programmes helps charities make better decisions, become better equipped to deal with the complexities of social issues, improve efficiencies and resource management, and ultimately learn from experiences to improve in the future.

Evidence doesn’t have to accompany a specific programme, it could be that external data is used to inform the approach of a programme. For example, numerous studies have shown that cash grants are more effective for poverty reduction than in-kind payments, despite the popularity of in-kind payments among some donors. Charities can therefore use this external evidence to inform their programmes and use cash grants.

Looking at an organisation’s external evidence base is always important when deciding which organisation to support, particularly if they lack their own. When they don’t have in-house or external monitoring and evaluation processes in place, donors should find out more about how an organisation informs their work and how their programmes have been set up. If they don’t have the relevant data but you know that they are following an evidence-based approach, such as the cash grants example above, then you can be somewhat confident that the approach is one demonstrated to be impactful.

Identifying the good evidence from the bad

There are several different kinds of evidence, indeed there is a whole hierarchy, and different types will be appropriate for different circumstances.

So what types of evidence really matter?

There are three main types of evidence that are good to look out for:

Randomised control trials: these are the gold standard in understanding whether an intervention works in its intended way and the difference it’s made to beneficiaries. This is because they establish a causality, a link between the output and the longer-term impact of the outcomes, such as the counterfactual. They don’t, however, include how the programme is implemented or the way the beneficiary feels about it.
Consistent qualitative evidence: obtained for example through a survey, qualitative evidence will be most useful to understand the beneficiaries’ perspective of a programme. It’s quite low on the hierarchy of evidence, but when it comes to reporting beneficiary experience, it’s good evidence to use.
Process evaluations: focused on how interventions work and why they deliver certain results, process evaluations allow you to ‘go behind the results’ of an evaluation. They are particularly useful when understanding how complex interventions work, what may have gone wrong in the delivery of an intervention and how it could be changed for a different environment.

Looking for the long-term benefits. Ask about how the evidence shows, if any, the long-term benefits for beneficiaries.

Beware, however, of other information in impact reports or charities’ websites presented as evidence but lacking actual data.

One example is the use of reach (reporting on outputs) as evidence of impact. A charity may claim to have reached a certain number of beneficiaries, but what they’re missing is insight into the impact they’ve had on those people – and evidence to prove it. Independently assessed evaluations will be able to demonstrate the link between what the charity has done and how it has helped those beneficiaries.

Information on the charity’s outputs, stories and short-term outcomes, while interesting, also don’t amount to useful data for truly assessing long-term impact.

How to marry evidence and impact in your giving

The first step in adopting an evidence-based approach to your giving is to look for a charity’s evidence base and identify how they’re monitoring and evaluating their work, so you can understand what’s working.

Choosing to donate to organisations that use evidence effectively in their decision-making process will ensure you make a greater impact with your funds and can help you develop trust in the organisation before you support them. Once you trust that the organisation uses donations effectively and is making a long term difference, you may be more inclined to provide unrestricted funding that charities can use more flexibly.

Be mindful, however, of when evidence is necessary and when it may be harder to come by. Of course, a charity may simply not have the evidence base needed due to a lack of funds or skills. If this is the case, ask if their programmes are being informed by externally-produced evidence. If so, you can be somewhat more confident than if they’re not following a well-evidenced approach.

Another option may be to use the opportunity to step in and facilitate that work. Consider funding the evaluation of a programme’s work to produce independently-assessed evidence for a better understanding of the impact it is having. This can be a good way to have philanthropic leverage.
You can find out more about the use of evidence in measuring your impact in our other articles:

Ecological Land  Co-operative

Environment
●
£23,000

Working to widen access to land for sustainable use by creating affordable ecological smallholdings.

Find out more

Local Food Links

Children and Young People
●
£425,000 (three unsecured loans)

Local Food Links used social investment to expand their kitchens and now produce 3,700 school meals daily across Dorset, using local produce and employing people in the community.

Find out more about Local Food Links

About the author

Aurelia Kassatly is a senior private client manager at CAF. She leads on the impact and philanthropy advisory work for CAF’s private clients, helping them to take an evidence-based approach to their philanthropy.

T: +44 (0) 3000 123 299
E: philanthropy@cafonline.org
Linkedin: Aurelia Kassatly

Give anywhere safely

CAF group enables individual and corporate donors from the UK, USA and Canada to make tax-effective strategic, impactful gifts to carefully validated charities, social enterprises and charitable projects on a global scale.

Learn More