Evidence vs impact: Finding the evidence

As we’ve discussed, the use of evidence when measuring impact is extremely important for both charities and donors. But how can you find the right evidence to understand the impact of an organisation or programme? Do impact reports have the answers? We explore what you should be looking for and why it’s often so hard to find.

We thank philanthropy expert Rhodri Davies; Matthew Whittaker, CEO of Pro Bono Economics (PBE); and Caroline Greenhalgh, PhD Candidate at the University of Birmingham and Head of Development at the Childhood Trust, for sharing their expertise on impact and evidence in philanthropy for this article.

Do impact reports demonstrate evidence?

Your first port of call for finding evidence may be to look at a charity’s latest impact report. There are plenty being produced by charities. But Rhodri and Matthew both think this is because donors expect them and are asking for them, not because charities see the benefits of producing them for themselves. This is leading to several consequences.

The first, according to Matthew, is that charities tend to view impact reporting as a way to secure more donations. This has led to the rise of ‘snake oil salesmanship’ and a proliferation of organisations claiming to measure donor impact, while simply sharing outputs, stories and short-term outcomes, without robust long-term evidence.

The second, according to Rhodri, is that it’s the donor rather than the charity who is deciding the impact a charity should have. Donors often come to an organisation offering significant donations for a specific programme or idea in mind, expecting diligent reporting and results. In reality, the initiative may not fit the charity’s mission, nor be how they’d choose to use the money given the choice.

Why is good evidence hard to come by?

In Caroline’s PhD research, she found that there was not nearly enough robust evidence available, while some donors felt overwhelmed by the amount of information provided and didn’t have the skills to evaluate its quality.

So why is good evidence so difficult to find?

1. Lack of good data

Many charities simply don’t have the evidence base to prove their impact. The format of information shared by charities is also often inconsistent, claims Matthew, with little indication of how the programme helped beneficiaries over time. This means that charities may be able to report on outcomes but find it difficult to prove longer-term impact.

2. Lack of skills and knowledge

To develop a strong evidence base, you need data scientists and economists – skills that charities don’t normally have in-house. Equally, funders often don’t have these backgrounds and so don’t know what to look or ask for.

3. Lack of funds

Most charities don’t have the funds to recruit for these skills. It’s also more difficult for charities to secure general funds that can enable them to step-back and assess what and how they are doing. Donors also tend to give restricted funds to deliver specific programmes. The result is that charities have little room to take on additional work beyond programme delivery. The pandemic has increased donor awareness on the importance of unrestricted donations, particularly during the initial emergency phase, something that we hope will continue.

4. Lack of infrastructure

As there is no centralised platform to share evidence on charitable programmes, its use and dissemination is severely hampered.

How can impact reporting be better for charities and donors?

Evidence building should not be done to secure more donations. It should primarily be about helping the charity understand the benefit of their work and how they can improve.

It’s also very common for different donors supporting the same charity or project to request different impact reports all with their own requirements. This naturally places a heavy burden on the charities. Ultimately, there needs to be a more joined-up and efficient way of demonstrating impact.

If donors have researched the evidence base for a charity’s work and feel confident in the long-term benefits of that work before they donate, they may feel more confident in making unrestricted donations and attaching fewer reporting requirements to the grant itself.

What evidence should donors be looking for?

When looking for evidence of impact from a charity, our contributors recommend:

Finding out the methodology or logic behind how they have collected and interpreted their programme data.

Looking for the long-term benefits. Ask about how the evidence shows, if any, the long-term benefits for beneficiaries.

Asking yourself how fantastical the claims are. If they are making claims that are absolute, rather than relative, explore that with them.

Looking for the counterfactual. Most importantly, ask if the charity has a control group that their programme can be compared against so you can understand what might have happened otherwise.

Look out too for the three main types of evidence we covered in our Evidence vs Impact: the basics article: randomised control trials, consistent qualitative evidence, and process evaluations.

You can find out more about the use of evidence in measuring your impact in our other articles:

Learn how we help our clients make a difference with their giving