The Challenge Of Measuring Social Impact

On Thursday (24th), I’m one of the panellists for The Challenge Of Measuring Social Impact event at The Hub Westminster. The organisers have asked us to blog our thoughts in the run up to the event, so these are my initial responses to the questions we’ll be attempting to answer on the night – although obivously I can’t promise not to have changed my mind by Thursday.

My fellow panelist, Dr Pathik Pathak, has already blogged some of his thoughts.

What is social impact and how can it be measured, does it influence funding decisions and how do small organisations with limited resources manage to do this?

For social enterprises and social entrepreneurs, social impact is generally short for ‘positive social impact’. It’s the positive change that takes place – or that we hope will take place – as a result of what we do.

Monitoring and evaluation – which has always been part of the voluntary sector landscape, at least over the 12 years that I’ve been voluntary sector professional – can be a way of measuring social impact but isn’t always.

Keeping a record of how many people have turned up to use your service  – for example, a creative writing course delivered in your local community – and who they are is monitoring but is not social impact measurement. Measuring the impact of your course might involve asking the participants some questions at the beginning of the course – do you they enjoy writing? are they confident about writing sonnets? – and then asking the same questions at the end of the course.

Opinions will be divided on the level of social impact delivered by some people in your local community being more confident about being able to write a sonnet. Is society a better place as a result? It’s (relatively) easier for people who services involve working directly with people to stop them doing crime to explain how their work delivers positive changes for society and (often) harder for people whose services are websites that provide people with generic information.

The promise of social impact definitely does influence funding decisions. Most grant funding applications have questions about why the thing you’re promising to do is needed and what will happen as a result of you doing it. In terms of commissioning, wider social impact – beyond the idea that the service that (usually) a public sector body is paying to deliver, delivers positive social change by virtue of its existence – isn’t currently of interest to most commissioners. Many social entrepreneurs hope The Public Services (Social Value) Act 2012, which comes into force this month, will begin to change that.

For small organisations, the danger is that they get bamboozled by questions such as ‘what is our theory of change?’ and miss the the underlying point that demonstrating social impact means explaining why what you do is useful and why someone, whether it’s a customer, a grant-funder or a public sector commissioner should give you money to do it. And also, equally importantly, knowing yourself whether your work is succeeding or whether you’re wasting your time.

Being small is not an excuse for not being able to do that at all. You don’t necessarily need to set up complicated measurements system but you do need to have some way of working out whether what you do is working.

Social investors state that measuring the impact of the products or services is just as important as increasing the capacity of an organisation to actually deliver the product or service, but what does this mean in practice?

The premise of this question is slightly optimistic. While Nesta’s impact fund, Joe Ludlow is on the panel, is an honourable exception, the vast majority of social investors do not yet have a specific commitment to supporting organisations that deliver measurable social impact (as opposed to being organisations committed to doing social good in a general sense). As the recent Big Lottery-backed report Investment Readiness in the UK noted, when discussing the perceptions of voluntary sector organisations seeking social investment: “Investees appear to think that their ability to create social impact will be more significant to investors than seems to be the case on the evidence of those who have received investment. This perception amongst potential investees appears to weaken as they get closer towards securing finance.”

There is a way to get money if you can deliver positive social change but can’t do so as part of a sustainable business. Apply for a grant. So far, the social investment market is far less sophisticated than the traditional grant-funding market in terms of defining the connection between measurable impact and cash.

There are difficulties on both sides. There are (so far) very few social investors who can currently explain what effect, if any, investees’ ability to measure social impact will have on their investment decisions. There are equally few trading charities and social enterprises who are measuring their social impact in any meaningful way. Although clearly there’s also no agreement on what ‘a meaningul way’ might mean. It’s a chicken and egg situation but with a lack of clarity about what exactly a chicken or an egg might look like when either or both arrives.

How are organisations developing the evidence that a particular product or service is having a positive impact, to ensure that what is being funded  is making a difference?  

Some (relatively) large social enterprises produce social impact reports. HCT Group is one of them. FRC Group is another. Matter & Co’s RBS SE100 provides an annual guide to the social enterprises who are best at measuring social impact (and have entered their competition).

I’ll have a bit more to say on this on Thursday. A few further questions, though, that I think are important are:

Is it useful for social impact to measured using systems – such as SROI – that attempt to put cash figures on the social impact generated by particularly activities?

What role does social impact measurement play in the overall battle for resources? Should resources be targetted at generating the greatest possible volume of positive social impact – if we were in the position to agree on what that meant – or are certain types of positive social impact a higher priority than others?

Advertisements

6 Comments

Filed under Uncategorized

6 responses to “The Challenge Of Measuring Social Impact

  1. Hi David,

    I very often wonder in the cause of impact measurement whether the job should be done not by the organisation, whether charity or social enterprise, delivering the service/s but by an independent body or bodies.

    In many years of work in the homeless sector, impact reporting through monitoring and evaluation has always seemed to me to be almost completely inefficient. I remember delivering a programme very early on in my professional career where there were 3 different funders and so three different sets of monitoring and evaluation criteria, the completing of which had the end result of limiting the time I had to spend in actual programme delivery! The constant search for a ‘means of measurement’, ‘tracking the money’ ‘SROI’ etc has in my view more of a negative impact on service delivery than most people realise or care to admit. Perhaps if we paid for outside, independent evaluation and coupled that with transparent practise, opening up the results for all to see we might get somewhere.

    We live in the age of social media, so let beneficiaries, patients, service users etc make open comment. Let funding, whether grant, loan, investment or other reward transparency of practise, because ultimately that transparency will show us what’s working and what’s not.

    Martin
    @network2013
    @failureconsult

    Like

    • Like you, Martin, I see no real value in SROI, particularly when it comes to that which seems immesuarable. Muhammad Yunus refers to the bottom line of Grameen Danone being the number od children removed from malnutrition, a human impact which might actually increase the financial burden of the state.
      Radical tranparency is certainly a means by which others can assess where impact is being made. Where I see problems with the use of social media is that those with the least ability to project their brand will be disregarded and the impact of their efforts attributed to others.
      Jeff

      .

      Like

      • Hi Jeff, I think your absolutely right re the limitations of social media not only in the sense of the organisational ability regarding weight and clarity of message but also in relation to the ability of beneficiaries [disabled children for example] to use the medium.

        A method of ‘radical transparency’ however, if we’re going to aspire to something regarding measurement and proving our organisational worth is I would suggest much more achieve able than matching money to social outcomes.

        Funding on Transparency of practise criteria negates innovation of course but in my honest opinion (which is probably a little arrogant) there’s no further argument about it’s efficacy in comparison to SROI.

        Like

  2. Pingback: What is your big question about social impact?Join the #hubmsi debate | Connecting Social Care and Social Media

  3. Jenni Inglis

    Great post David

    I particularly like that you opened with an explanation that for social entrepreneurs social impact is generally shorthand for “positive social impact”. This rings true to me, but I also think it is a problem that it is the case because in rushing to try to prove things social purpose organisations may miss learning about how they can improve what they are doing and they may misrepresent what they are actually achieving.

    The SROI Network has been making a fuss about PRINCIPLES of measuring social value- which includes impact- for some time. A lot of the rest of the discussion in this space has so far focused on one part of impact measurement- measuring outcomes. In your example above this would be analogous to asking questions before and after the sonnet writing workshop. It is good that most people have got beyond thinking that outcomes are not measurable and are now mostly thinking about how they can be measured.

    Principles (of SROI which is about accounting for value ) would suggest that before you start to measure (i.e. before you choose which questions to ask people before and after your sonnet writing workshop) you would seek to understand what is actually changing for different groups. And that you would do this by involving a sample in a conversation about what changes. You could then design or chose survey questions that most closely reflect the types of outcome you discovered, this could include validated scales or personal outcome tools where available. To be clear I’m not saying there’s anything wrong with just picking a couple of questions to ask attendees- doing so is absolutely better than just counting how many people came to the activity but I suppose I’m saying now that it is more accepted that outcomes are measurable why not stop for a minute and see if we understand what these outcomes actually are. For outcomes that are effects on people, those people are well placed to help us understand what the effects actually are. So it is a practical consideration but I’d argue also an ethical one.

    The Social Impact Analysts Association has been doing some work to compare different sets of principles that seek to underpin how one might social impact. One thing that I gather has come out of this is the insight that, whilst there is a lot of agreement on principles, there is divergence primarily on one thing- whether impact measurement should be (1) focused only on checking whether you have achieved what you set out to do- your intended positive outcome- or whether it should be (2) additionally focused on what impact you actually had- what actually changes, positive, negative, intended, unintended for the groups of people you affect. This latter category tends also to include other stakeholders more, not just immediate beneficiaries.

    I expect there will be some debate at the event about proving causality. NESTA helpfully published standards of evidence for their own impact investing which includes much on this. Perhaps I missed something but it does all look rather focused on intended positives. Having recently done some SROI principles based work for a University who were also running a randomised control trial (of something that there had been many previous RCTs of) I conclude that the choice of what to measure in an RCT is vitally important but that basing this only on the delivery organisation’s intended positive effects can miss something important.

    Lastly on the point about valuation. The principle of “value the things that matter” is not really about putting a cash sum on things. It is absolutely not only about- as Jeff suggests inaccurately- savings to the state. It is in fact about addressing the last point you make- that perhaps certain types of impact might deserve a higher priority than others. At the moment decision makers allocate resources to some extent on the basis of an understanding of the volume of positive change because that is generally all they have to go on, and they tend to feel happy if they’ve got even that. In doing so they almost certainly apply some of their own value judgements in an assessment of relative importance of these outcomes. The principle of “value the things that matter” demands that we better understand importance of an outcome to the people who experience it. So in fact, putting this principle into practice we’d focus on the value to children and families of addressing malnutrition, which would dwarf any costs. A focus only on costs (relative inputs required) leads to perverse decisions but a focus on value to stakeholders (value of outcomes) should lead to better decisions. I don’t know anything about decision making around stopping malnutrition but I do know that where the value of an outcome to a marginalised group is brought into an account of value I’ve been involved in it really makes decision makers stop and think.

    Hope you enjoy the event

    Jenni

    Like

  4. Well it’s Davos time again and I’m reminded of what happened 4 years ago. when Sir Richard Branson spoke at the Davos Ukrainian Lunch, aka, the Davos Philanthropic forum . Here’s what he said to world leaders:

    Quote:

    “Capitalism is the only economic system that really works”, he underlined, saying that the downside of the capitalist system is accumulation of great wealth in hands of relatively small number of people. “Not all of these people use their assets to create new jobs and new opportunities.” Branson stated.

    This is precisely why social role of business, as said by Richard Branson, is so important in modern world. “A modern company should focus not only on making money, but also on solving social problems and investing in protection of environment.”

    http://pinchukfund.org/en/news/361/

    We all know Branson uses the internet. To illustrate the impact of radical transparency, here’s what we made very public two years ealier by publishing a stategy paper in an online magazine.

    “An inherent assumption about capitalism is that profit is defined only in terms of monetary gain. This assumption is virtually unquestioned in most of the world. However, it is not a valid assumption. Business enterprise, capitalism, must be measured in terms of monetary profit. That rule is not arguable. A business enterprise must make monetary profit, or it will merely cease to exist. That is an absolute requirement. But it does not follow that this must necessarily be the final bottom line and the sole aim of the enterprise. How this profit is used is another question. It is commonly assumed that profit will enrich enterprise owners and investors, which in turn gives them incentive to participate financially in the enterprise to start with.

    That, however, is not the only possible outcome for use of profits. Profits can be directly applied to help resolve a broad range of social problems: poverty relief, improving childcare, seeding scientific research for nationwide economic advancement, improving communications infrastructure and accessibility, for examples – the target objectives of this particular project plan. The same financial discipline required of any conventional for-profit business can be applied to projects with the primary aim of improving socioeconomic conditions. Profitability provides money needed to be self-sustaining for the purpose of achieving social and economic objectives such as benefit of a nation’s poorest, neediest people. In which case, the enterprise is a social enterprise.”

    I followed this up with Virgin Unite introducing our work and offering to
    lead the way with their help in Ukraine. That ended the conversation.

    But then:

    http://www.virginunite.com/Templates/screwbusinessasusual.aspx

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s