Effective impact measurement: why the SDGs or what your investors want should never be your starting points

As an organisation that develops software to help social organisations learn from data to maximise impact, it would be hypocritical not to learn from our data too. 

So, when we wanted to measure our impact, what did we really do?

Rest assured, we didn’t look for some standard metrics to track, nor did we convince ourselves that we don’t have the skills or the technology to implement an impact measurement and management process – two issues we often see among our clients. 

Instead, we  thought deeply about the problems we were solving and the outcomes we were pursuing. Then, we thought about what would tell us we were achieving our outcomes (custom metrics). We needed time, technical skills and some pretty sophisticated software to compute and keep track of those custom metrics. We had to build our software in such a way that even non-technical people on our team could analyse the data in ways that helped them maximise the impact of their activities. For the impact measurement and management (IMM) process to be sustainable, we also had to ensure that our software automatically keeps the data up to date.

Your programmes are personal, your activities are personal, and yet, we think impact measurement should be standardised

For a regular for-profit company, this continuous learning is business intelligence. Our position: it shouldn’t be any different when it comes to social organisations.

Despite this, many social organisations object to investing in this kind of business intelligence. Their reasons include (not an exhaustive list):

1. I don’t have the money
2. I don’t have skilled people
3. I don’t have time to develop new skills
4. I find it hard to measure the impact
5. I cannot work with data

All the problems listed here are real, but this cannot be an excuse for organisations not to attempt to measure and manage their impact. Companies like ours can solve most of these problems – but organisations still have to have a positive mindset and a willingness to put in the effort. 

• Read more impact management insights from Sopact and its partners

Over-reliance on standards

If mindset is the first hurdle we see, the second hurdle is a problem with this industry in general. An over-reliance on and even obsession with so-called standards has become the norm. We see example after example of organisations thinking about standard frameworks, alignment with sustainable development goals, or ESG (environmental, social, and governance) criteria when they think of putting a solid IMM process in their organisations. 

Why is this a problem?

Let’s take a look at a real example. One of the organisations we are engaged with is working to improve students’ maths and science skills by matching them to volunteer tutors. One of the outcomes they are keeping track of is “students increase maths confidence”. A standard metric for this is “% of students that feel more confident in maths”.

An over-reliance on and even obsession with so-called standards has become the norm

This has a high chance of misleading an organisation to rely solely on a survey question post-programme, such as “How confident do you feel in your ability to solve maths problems?” with some options for students to choose from.

Let’s say 85% say they are “extremely confident”. This data is meaningless without hard evidence that their confidence is justified. In the case of this organisation, a ton of data on maths diagnostic scores is being collected. Hence, we literally triangulated the confidence data from the survey with actual score improvements from the Salesforce platform. 

Now, we can actually dig deeper and find correlations between tutoring sessions and the maths diagnostic score and confidence improvement in students – making the whole IMM process more credible and effective. This raises the odds of programme success, of applying the right interventions, and of raising funds to do the same.

The wrong starting point

IMM should allow organisations to learn about the outcomes their projects and programmes have on the people experiencing them. This requires the right data, the right technology (similar to what for-profit organisations use), the right skills, and, above all, the right mindset.

Does this mean that you totally discard any helpful frameworks or standards?

Absolutely not, but it means that you won’t make that your starting point or the only thing your organisation does. Because it won’t help your organisation improve its programmes and make any real difference in people’s lives. 

Impact management and measurement is an enterprise’s responsibility; they should prioritise it, fund it, and execute it – irrespective of what their investor may expect or do

A lot of this is driven by impact investors: in looking to keep track of their investments’ health, they ask organisations to report on certain standard metrics. This works just fine for investors, but for enterprises, it will mean they are doing the bare minimum rather than real IMM. 

IMM is an enterprise’s responsibility; they should prioritise it, fund it, and execute it – irrespective of what their investor may expect or do. You are not doing it FOR the investor; you are doing it for yourself to understand better the programmes you are running, to ask better questions, to have a small chance at growing your organisation and making a real difference in the stakeholder’s life. 

It baffles us to this day that organisations are looking for some magical one-size-fits-all kind of standardised process to measure impact. Your programmes are personal, your activities are personal, and yet, we think impact measurement should be standardised. The real need is for organisations to think in terms of hard evidence, which is backed by quality data and processes that allow organisations to put a system in place to draw insights from that data.

Overcoming IMM hurdles: three steps for every impact organisation

So, how do we overcome these common hurdles to implementing an effective IMM process?

Our experience of working with 100+ organisations tells us that they need to develop three key traits: a learning mindset (rather than proving); access to skills and technology that are on par with for-profit companies; and a tailored measurement framework.

1. Learning mindset

It might sound preachy – but we cannot state strongly enough that having a learning mindset is at the foundation of putting an IMM solution in place. 

Just start with one question: How do I know whether my programme is working at all?

You can have someone external to your organisation to help brainstorm the answer, but you simply cannot outsource it all. It only works when the organisation owns the core thinking process.

Having the quality of  “high agency” is a must to establish a strong IMM process. 

You don’t have to think of implementing anything sophisticated early on. As you scale – which you will as a result of learning from data – you can start looking at improving efficiency through technology

The economy might not be ideal, the donor landscape may not look pretty, and you may not have the right technology or skills within your team, but do not let this not stop you from attempting to implement an IMM process. 

Be scrappy if you are a very small organisation or running a small programme like a workshop. You don’t have to think of implementing anything sophisticated early on. As you scale – which you will as a result of learning from data – you can start looking at improving efficiency through technology. 

2. Access to similar skills and tools that for-profit companies use

Midsize organisations with established programmes need access to similar skills and technology that a for-profit organisation can access. Nothing short of that is going to cut it.

Let’s take the tutoring example that we described above. 

To correlate the confidence data (from surveys) with actual score improvements (from the Salesforce platform), we need the following in our toolkit:

1. Ability to bring data in from the survey and Salesforce platforms
2. Ability to correlate data from both sources together (join the data in a meaningful manner)
3. Ability to clean up data for anything that prevents accurate measurement
4. Ability to modify the data if needed to draw richer conclusions and insights
5. Getting the data autonomously and continuously applying data transformations needed without a lot of manual intervention (kind of like putting IMM on autopilot)

In our experience, IMM is not sustainable without having the core abilities listed above. Otherwise, it becomes a manual process that is extremely hard to follow through – and ultimately falls apart.

3. Tailored measurement framework

A tailored measurement framework is at the heart of learning the right things in a project or a programme. So how do you create this? 

Lean on expertise and knowledge within your teams to think about what to measure (ie, what  tells you about the impact) and how to report to your funders on the same. This is important because it’s not focusing on what your organisation can currently measure. Instead, it ensures that at the very least your organisation is thinking about the right things to measure. This means that you won’t start with standard metrics but will deeply think about the context and come up with something that is more tailored. 

This also means that you have to own the process. At Sopact, we help many organisations achieve this, but the important thing to note here is we “help”, but we don’t own the whole process. After all, no one is a bigger expert on your projects or programmes than you.

This article was produced in partnership with Sopact. Find out more about how Sopact helps organisations to implement an end-to-end impact measurement and management process.

Image: Mikhail Nilov (Pexels)

Thanks for reading our stories. As an entrepreneur or investor yourself, you'll know that producing quality work doesn't come free. We rely on our subscribers to sustain our journalism – so if you think it's worth having an independent, specialist media platform that covers social enterprise stories, please consider subscribing. You'll also be buying social: Pioneers Post is a social enterprise itself, reinvesting all our profits into helping you do good business, better.