Time to change Theories of Change?

Having a theory of change is the must have accessory for any self-respecting social (or environmental) purpose organisation.

Of course, it is important that any organisation seeking to create change should be clear on how that is going to happen and should be able to collect some data to test whether it is happening. This will be useful for external communications, for the board discussions on strategy and will provide a focus for the organisation and its staff.

However too much focus on theories of change and the well-known associated logic model of ‘inputs outputs outcomes impact’ has become a problem. Theories of change can become the evidence to support goals rather than a frame to test those goals.

Proponents of these approaches will argue that, if done well, the risks that I am about to raise can be avoided. However, my proposition is that being clear about goals and having a theory of change for those goals is not the same thing as considering the consequences of an activity.

The traditional programme cycle of plan, do, analyse and review sometimes, but not always, includes consideration of the consequences of action under analysis.

But goals and consequences are so different in culture, data, rigour, systems and assurance that they need to be kept apart, and only come together to inform discussions on operations and strategy.

Although theories of change are not the same as logic models, they are often used interchangeably and can blend into each other. Logic models work back from goals to show how activities led to outputs and outcomes.

Theories of change (ToCs) explain how and why the links between inputs, outputs and outcomes work and may show a bigger picture including other factors that contribute to change.

ToCs should, in theory, be addressing the issues and risks below by considering the larger picture but it’s the focus on goals without separately considering consequences that is the main problem.

In this essay I will concern myself with simple ToCs (a logic model, a log frame and/or the data supporting any links in the model).

The issue with ToCs is that their use, alongside the focus on having and managing goals, without separately thinking about consequences, hold a number of risks, most of which arise from our human condition and our cognitive dissonance.

Cognitive dissonance is the tendency for individuals to seek consistency among their beliefs and opinions. When there is an inconsistency between attitudes or behaviours (dissonance), something must change to eliminate that dissonance.

It’s quite frightening how many of these psychological traps we can fall into, explored and summarised brilliantly here. From one perspective, the purpose of impact measurement is to protect us from these risks. Unfortunately, in practice ToCs can too easily reinforce these risks.

Ironically, this may be more of a challenge for social purpose organisations (SPOs) than other organisations because the focus on intent to do good makes it harder to see when this isn’t working and makes it more difficult for people to criticise our well meaning actions.

These risks are sometimes recognised but the common argument is that building a simple model of how your inputs lead to the outcomes that are related to your objectives is the first step, or as much as can be expected. It’s the idea that going further than that is so much harder that is so dangerous.

There is no reason why an organisation cannot think about both goals and consequences from the start. In fact, all small private businesses need to consider both.

They will have a product, a great product, a product that they believe in, but if they don’t listen and ACT on consumer feedback they won’t be in business for long.

That social purpose organisations are generally less accountable to their users than private enterprises are to their customers may be accepted. But it’s the consequences of this and how this as resulted in the primacy of ToCs that is the problem.

The problems 

Outlined below are some reasons why we should be concerned about an over dependence on ToCs as the basis for measuring and managing impact in SPOs. The implication of these challenges is that organisations will not be able to maximise the value they are creating until they consider, account for and manage the consequences of their actions.

Problem 1: the name

Perhaps the simplest issue to fix is to not call it a Theory of Change, Unfortunately, this limits people immediately to a change, the change is the goal and this closes down thinking. One simple thing that would help would be to call it a Theory of Changes or perhaps even theories of changes.

Improving means making decisions to change existing activities to increase the value that is being created

Problem 2: the assumption that proving and improving can be done with the same data or even the same culture

Obviously, it is always a good idea to both prove and improve your social value or impact. The activity of setting out objectives, developing a theory for how activities link to outcomes and hence to objectives, then collecting data to test that theory all makes sense. And if all goes well it means we prove that our activities are meeting objectives.

But how many organisations do you know that have generated information which disproved their ToC? Not so many? This is one example of cognitive dissonance. We believe that our intervention is the right one before we start; if we didn’t we wouldn’t start.

There is therefore a risk that we will over focus on information that supports this belief, we will draw patterns where none exist or we will find causality where there was none. Contradictory information will only tend to affirm our original belief.

This isn’t just a problem for an organisation but also for any evaluation. You only need to check badscience.net to see that these issues remain challenges in peer reviewed academic research.

Despite all this we will spend a lot of time getting good samples, controlling for bias, doing the stats – all designed to increase the rigour and the credibility of the ‘prove’ part of our analysis. Improving is very different.

Improving means making decisions to change existing activities to increase the value that is being created.

Any proposal to change something has to be assessed against risk of data being incorrect but also the implications and consequences of that being incorrect.

If you make a small change to a programme which can easily be reversed, the consequences are very low and so a higher risk is acceptable.

This is just as well since improving means a relentless process of trial and error, whereas timely information will often mean less accurate information. Proving can take more time, improving means there is never enough time.

This is a much more entrepreneurial mindset, asking the people whose lives we affect to tell us their experiences, looking within those stories for patterns that allow us to test new products and realising that segmenting people into groups means we can provide more specific services.

This means aligning the culture of enterprise with the culture of social impact. A culture where board and management constantly review and update strategy, operations and services, testing improvements to create more value.

There are a growing number of organisations, both investors and businesses, starting to do this. Acumen’s Lean Data Initiative is a great example.

Problem 3: The myth that a ToC will always help you improve

This can be true (subject to 1 above), but importantly not as much as it could and should be. This is another consequence of lower accountability. Again, understanding outcomes from the perspective of stakeholders will reveal other outcomes than those in the ToC.

Obviously negative outcomes provide opportunities to change services. But it’s less often recognised that even positive outcomes, where they are experienced by only some of the stakeholders provide possible opportunity for more bespoke services.

Those other outcomes may even be preferred to your objective; what is unintended to the organisation is in fact valued highly by the stakeholder. These insights can help underpin your goals and exceed your targets.

This type of information is critical to moving towards maximising value – it is gold dust for social innovation. Such information can be gleaned by talking and listening to stakeholders on a regular basis and responding to what you hear, even before you have developed a fully-fledged ToC.

Developing ToCs should involve stakeholders but because the purpose is to build the ToC it will be easy for that involvement to be limited in scope as well as quantity.

Accurate information on one outcome and on the links between outputs and outcomes is not as useful as less accurate information on several outcomes with more detailed understanding of the outcome, including segmentation of people who experience different outcomes or have different preferences of those outcomes.

This information can be good enough to trial a new product or test a new way of doing the same thing. If it doesn’t work you can always go back to the old way.

It’s part of our human condition to see causation in data where none exists all the time

Problem 4: theories of change (with measurement of the outputs and outcomes) provide evidence of causality

Yes they do. Unfortunately, it’s not great evidence.

If you want better (but NOT perfect) evidence of causality you will need a control group. Preferably one in which the control group has been randomly selected, starts from the same point and has the same experience before the intervention. The cry goes up that this is expensive, time consuming and difficult.

But relying on a ToC and some data means that the risk that you will over claim your impact will be very high. The other factors that are causal will be missed, even if you drew up a ‘bigger picture’ at the start.

It’s part of our human condition to see causation in data where none exists all the time. So whilst this may not be possible to start with, it needs to remain the aim. The risk is that having a ToC becomes the aim of impact management and seen as a sufficient approach to the activity.

That said, for the organisation seeking to improve its impacts, simply asking stakeholders what would have happened anyway, (though also not ‘rigorous) can reveal groups of people with shared characteristics that respond to the question in similar ways.

And yes, for the statisticians reading this, there are risks here. But back to the consequences of getting it wrong; it may still be worth trying some more targeted services.

That’s not of course to say that organisations shouldn’t be constantly having to consider whether their activities are effective, what net change they create, and should start with Peter Rossi’s metal laws before they start.

Problem 5: the findings of meta research

This is another example of cognitive dissonance. We don’t change our activity or our objective when we have evidence that the intervention doesn’t work.

Meta research on reoffending in the UK by the Justice Datalab concluded that some interventions had helped but the majority had had no effect, and a small proportion had had a negative effect.

Another example would be homeopathy. The meta studies of the studies have been done and it doesn’t work. But it doesn’t make any difference. Those who believe will continue to believe despite this evidence. Evidence that refutes what we believe only makes our beliefs stronger.

Imagine realising that many of your stakeholders are worse off after your intervention

Problem 6: ToCs do not test whether we have the right ToC

They definitely should but our ToC is there to give us clarity on our goals. And we already believe in our goals.

Imagine realising that many of your stakeholders are worse off after your intervention. After all, they thought your intervention would solve their problem but for some it didn’t. You haven’t thought about this as the majority of stakeholders achieved your stated objective. Then you realise that there was another contributing factor and most of those that did achieve your objective would have anyway, even without your intervention.

On the face of it you have a ToC and you are achieving it. But in reality you are wasting resources where change would have happened anyway and are making some people worse off than you were.

Problem 7: There should be a stronger relationship between social innovation and social impact

Or so you would have thought. And having a goal and perhaps a ToC will be an important piece of your initial strategy and your initial innovation.

But only if you are relentlessly talking to people and thinking about how to change what you do to create more value will this relationship continue.

Social innovation doesn’t need to be big changes, it can also be the continuous process of updating and renewing services as users and their context change.

Problem 8: all these other things may be true but it’s too hard to think about and so it’s better to start with a ToC

Businesses start with an idea, then they make a product and then they recognise that they need to keep improving (which means also stopping some doing some things, not just improving the same things). They do this with imperfect data.

They are perfectly able to do this, seeking opportunities in the stories they hear from customers, and the patterns of consumption. They don’t start with a ToC. However, if it doesn’t work because customers don’t buy their services they either change or die. Instead of ToCs and social science, they are held to account by their investors.

It’s the lack of accountability by social purpose organisations to those whose lives are being improved that has led to other tests, and ToCs, to show effectiveness but they are no replacement. Necessary perhaps but not sufficient.

Organisations need to start with both goals and a culture of listening to stakeholders and relentlessly considering improvements to products and services as a result. Hand in hand!

Problem 9: standardised measurement will become easier if organisations have a ToC

Perhaps, but again it will risk providing less useful information. Organisations need to understand change as it is experienced by stakeholders in order to make design decisions. This can be collated with a taxonomy and standardising taxonomies will help. There are growing examples, the Big Society Capital outcome matrix.

Even if those experiencing outcomes agree a set of outcomes, fixing this as the set list of measures will inhibit innovation. SPOs need to be adaptable and innovative to adequately serve their beneficiaries.

What now?

So what next? How do you begin to tackle the issues with dependency on ToCs? ToCs have an important place in setting strategy and in pulling together the team around a goal. But accounting for, and managing what happens as a consequence of your activities is critical for maximising impact.

●    Talk to people, the importance of this cannot be overstated. Only by continuously talking to, and asking the right questions of, your stakeholders will you see patterns and develop ideas for new, better or more targeted services

●    Develop understanding of how the organisation creates change for different people. This will involve recognising the end users have different characteristics and how these cluster against outcomes and the perceived importance of outcomes provides opportunities for innovation

●    Recognise that measuring, managing and maximising is a journey, and that control groups will be needed somewhere in this journey.    

●    Review all this against those goals and objectives, regularly at an operational level and also, but less frequently, at a strategic level. Recognising that it may be necessary to amend goals as much as change services to do the good you set out to.

Being reactive to the needs of stakeholders underpins being able to maximise value and reduces the risk that you become trapped within your own ToC. A key part of this is learning how to measure, manage and maximise the value you are creating. Social Value International can help you along this journey, with support, resources and guidance. Meanwhile perhaps we can start talking about theories of changes!
 

Photo credit: Dietmar Becker