Philanthropy: Evidence vs impact

Unsure whether your philanthropy is making a difference? Aurelia Kassatly explores what evidence is, why it is important, and how donors may use it to inform their charitable giving.

‘Impact’ is the word of the moment in philanthropy, charity and even investing. Donors want to have an impact with their giving, charities want to know if their work is making a difference, and investors want their investments to have an impact.

While everyone wants to be impactful, what’s missing is an agreement or understanding of what that means.

To address this gap, we spoke to experienced academics, researchers and advocates of philanthropy to get their take, in a two-part piece exploring impact, evidence, and how that relates to philanthropy.

We thank Rhodri Davies, Head of CAF’s think tank, Giving Thought; Matthew Whittaker, CEO of Pro Bono Economics (PBE); and Caroline Greenhalgh, PhD Candidate at the University of Birmingham and Head of Development at the Childhood Trust, for their time and contributions.

Defining impact

The first thing to keep in mind is that impact usually has positive connotations.

The second point is that impact is defined as the final, often long-term, result of a programme.

This means that in order to really know whether you have had an impact or not, you need to compare, or understand, what would have happened otherwise were it not for that particular programme (known as the counterfactual). You also need to be able to tell whether any results have been sustained, bringing long term benefits for beneficiaries. 

The only real way to know is through the use of evidence. Evidence is independently assessed proof that a body of work, works in its intended way.

There are several different kinds of evidence, indeed there is a whole hierarchy, and different types will be appropriate for different circumstances.

Why is good evidence hard to come by?

The only real way to know if you are having an impact, or not, is through evidence; but whether sufficient evidence is available is debatable.

In her PhD research, Caroline found two perspectives from philanthropists – that there was not nearly enough robust evidence available, while on the other hand some felt bombarded with information but did not have the skills to evaluate its quality.

While there is certainly a lot of information and reporting out there, good evidence is difficult to find for several reasons:

1. Lack of good data

Matthew highlighted that the format of information shared by charities is often inconsistent, with little indication of how the programme helped beneficiaries over time. This means that charities may be able to report on outcomes but find it difficult to prove longer-term impact.

2. Lack of skills and knowledge

Matthew and Caroline highlighted that to develop a strong evidence base, you need data scientists and economists – skills that charities don’t normally have in-house. Equally, funders often do not have these backgrounds and so don’t know what to look or ask for.

3. Lack of funds

Most charities don’t have the funds to recruit for these skills. Matthew shared that it is more difficult for charities to secure general support funds, enabling them to step-back and assess what, and how, they are doing. Donors usually like to give ‘restricted funds’ for the delivery of specific programmes and plans. The net result, as Caroline puts it, is that charities have little room to take on additional work beyond delivering their programmes.

It is important to highlight that due to the COVID-19 pandemic, we are seeing an increase in awareness from donors on the importance and use for unrestricted donations, particularly during the emergency phase of this virus. This is something that we hope will continue after the pandemic subsides.

4. Lack of infrastructure

Caroline also pointed out that because there is no centralised platform to share evidence on charitable programmes, its use and dissemination is severely hampered. We will explore this more in part two of this article.

So what is impact reporting?

If good evidence is hard to come by, then why is so much impact reporting being done, and how can it be better for the charity and the donor?

Rhodri and Matthew are of the view that charities are compiling impact reports because donors are asking for, and expect it, rather than because charities see the benefits of doing so. The consequences of this are several.

The first, according to Matthew, is that charities tend to view impact reporting as a way to secure more donations. This has led to the rise of ‘snake oil salesmanship’ and a proliferation of organisations claiming to measure your impact, when in fact they are sharing stories and short-term outcomes, without robust long-term evidence.

The second consequence, according to Rhodri, is that it is the donor and not the charity who is deciding the impact that a charity should have. Donors come to an organisation with a specific programme or idea in mind, and expect diligent reporting and results on that, when in fact it may not be in line with a charity’s mission, nor what they would choose to direct the money toward if given the choice.

Finally there is the question, if capturing and understanding the evidence is rarely done, what is recorded in impact reports and what are donors asking for?

Firstly, Caroline mentioned that, at the moment, it is very common for different funders supporting the same charity and the same project to ask for different impact reporting with different reporting requirements. This naturally places a big burden on charities. 

Second, when we use the word impact in this context, if many charities do not have the evidence base to prove their impact, many funders do not have the expertise to know what good evidence looks like and what they should be asking for; most of what gets reported on are outputs.

Ultimately, there needs to be a more joined-up and efficient way of demonstrating impact and evidence building should not be to secure more donations. First and foremost, it should be about helping the charity understand the benefit of their work, if any, and how they can improve.

If donors have researched the evidence base for a charity’s work and feel confident in the long term benefits of that work before donating, then perhaps they will feel more confident in making unrestricted donations and attach fewer reporting requirements to the grant itself.

What information could donors be asking for?

We have put together some recommendations from our contributors that we hope are helpful to you. Here’s what donors should ask for from a charity that they are interested in:

  1. Find out the methodology or logic behind how the charity has collected and interpreted their programme data;

  2. Ask about how the evidence shows, if any, the long term benefits for beneficiaries;

  3. Ask yourself how fantastical the claims are. If an organisation is making claims that are absolute, rather than relative, then that is worth exploring with them;

  4. Most importantly, ask if the charity has a counterfactual or control group that their programme can be compared against so that you can understand what might have happened otherwise. 

Ultimately, the gold standard for most types of interventions to understand whether they work or not are randomised control trials because they establish causality.

There are other important points to bear in mind. For example, as Rhodri points out, some measures of impact place all the emphasis on the end result. The risk here is that the way the programme is implemented, or the way beneficiaries feel about it, is totally ignored.

To capture this information, Caroline tells us that you may wish to look at two different forms of evidence:

  • Process evaluation: This looks at a step-by-step process and how that process delivers certain results.

  • Consistent qualitative evidence: If you want to know what beneficiaries think, this is the most useful way to do that. Generally speaking, qualitative evidence is quite low on the hierarchy of evidence we referred to, but when it comes to reporting beneficiary experience is good evidence to use.

Final thoughts

The world of impact and evidence is a complicated one, because achieving social change is hard. After all, if it were easy then there would be no need for philanthropy. Ultimately, before deciding to fund any charity, it is useful to stop and think more deeply about the problem you wish to solve. Evidence can go a long way in helping donors and organisations to answer these questions.

In the follow-up piece to this article, we will explore the bigger questions and issues surrounding impact and evidence and how they relate to the role of philanthropy. The CAF Private Client team is always on hand to help our clients use evidence to inform their donations, so please get in touch if you have any questions or would like to discuss how we can support your giving.

Learn how we help our clients make a difference with their philanthropy.

Read the second article in this series.

Aurelia Kassatly

About the author

Aurelia Kassatly is a senior private client manager at CAF. She leads on the impact and philanthropy advisory work for CAF’s private clients, helping them to take an evidence-based approach to their philanthropy.

T: +44 (0) 3000 123 299
E: philanthropy@cafonline.org
Linkedin: Aurelia Kassatly