Metric martyrs

For the few of us still bothering to take an interest in the impact measurement industry’s geostationary orbit of planet earth, there’s been a couple of divertingly divergent interventions from top academics at London School of Economics recently.

First up, my friend Julia Morley, from the Department of Accounting, incurred the wrath of impact consultants’ umbrella body, Social Value UK, with her research blaming “an elite group of social investors” for providing “the main impetus for the adoption of social impact reporting” which “may create dysfunctional incentives for social enterprises” as a result.

Perhaps unsurprisingly, Social Value UK offered a storming social media media rebuttal:

Julia’s full paper, Elite networks and the rise of impact reporting in the UK social sector, merits further reading but I’m respectfully sceptical about parts of her abridged argument, albeit for different reasons to Social Value UK.

Who cares?

For me, the idea that social investors are forcing charities and social enterprises to adopt impact measurement over-estimates both the extent to which most social investors are interested in impact measurement and the extent to which most charities and social enterprises care what social investors think.

Maybe we’re seeing more organisations using the word ‘social impact’ on their website and/or calling their annual report ‘Impact Report’ but, as last year’s Oranges & Lemons report illustrated, there’s not much actual impact measurement being demanded by social investors. And, for most charities and social enterprises, social investment remains an obscure siding in the wider market for finance where nothing of practical relevance is on offer.

Impact measurement is a fundamentally important component of social impact bonds (SIBs – see previous) because it’s baked into the business model. Irrespective of our view on whether SIBs provide good value for money, the value proposition makes sense. The organisation (and investors) need to measure impact so that they get paid, the government needs to measure impact so that it knows whether the contract has succeeded on its own terms.

In terms of conventional social investment into organisations, it’s not clear how impact measurement creates value for anyone other impact measurement consultants. Organisations don’t get a cheaper loan for measuring their impact – and that’s understandable because (in general) social investors don’t get a better deal from wholesale investors for proving impact. It’s often claimed that public sector commissioners are more likely to buy from organisations who can demonstrate their impact but, SIBs aside, I’m not aware of any published evidence that this is true.

What’s the point?

I should make clear that I’m not putting this forward as an argument against the social value of impact measurement. I don’t believe it’s acceptable for charities and social enterprises to have no way of determining whether what they do is/remains socially useful – and the reporting systems advocated by impact measurers are one way of doing that. So there be may good reasons why government, grant-funders or others should (continue to) pay for measurement – but (SIBs aside) the commercial drivers for most organisations and investors to fund measurement themselves are (still) not obvious.

Fortunately, as Dan Gregory noted yesterday, some of Julia Morley’s LSE colleagues have some ideas:

There’s a shorter version here.

As Dan suggests, it’s genuinely wonderful to encounter people so optimistic that their response to the fact: “there has been a rise in the tools available for measuring the social impact of business – to the tune of more than 150 impact assessment methods” is: “In an effort to move impact assessment forward, we propose a holistic, transparent platform – called the External Rate of Return (ERR).

The full situation seems to be that with people primarily on social change failing to do much social impact measurement but creating loads of complicated, highly contested measurement systems that no one uses – so the answer is to create an impact measurement system based on a complicated, highly contested financial measure Internal Rate of Return (IRR) and attempt to get businesses who are not primarily motivated by social change to use it.

As if that wasn’t fantastical enough, the added twist is that the model requires interested members of the general public to turn up a website and vote on the impact data that companies are providing.

On what possible basis would this actually happen? 

If you are not an impact measurer yourself, you probably don’t need to be told that there’s no logical reason why a statistically significant number of members of the general public would turn up to a website and express an opinion on a company’s impact data.

As with many of the specifically social-focused impact measurement approaches, the creators of ERR are apparently focused on creating a theoretical model that would work really well in the event that large numbers of stakeholders on all sides of the equation were interested in it, while seemingly ignoring the question of why those stakeholders might be interested (or not).

While the impact measurement industry is unremitting in its demands, not just for funding but also for changes in the law, there’s ongoing confusion about what problems it’s solving and who it’s solving them for.

Advertisements

9 Comments

Filed under Uncategorized

9 responses to “Metric martyrs

  1. Appreciate this well written article. May I offer an alternative viewpoint? The commercial drivers for impact measurement are clear and compelling: those organizations that can show clear, compelling, concrete ROI, or (social)ROI, are far more competitive when it comes to raising generating and growing revenue. This isn’t an abstract conversation as you seem to present it here– simple, compelling metrics on impact are key drivers of growth and performance.

    Like

  2. Beanbags admin

    Thanks. Appreciate the comment. This is still an abstract conversation at the moment, though because your response is an abstract assertion.

    Can you point me in the direction of some evidence that SROI or other forms of social impact measurement drive growth?

    Like

  3. Thanks David, this is an interesting debate.

    I would like to challenge your assertion that not much evidence is demanded by social investors. At Big Society capital, we require impact evidence as part of our legal requirements in deals. Our starting point is that organisations should gather what is useful for their own work and then simply share it with us This helps us to build up a picture of how social investment is helping organisations to achieve impact and to share examples about how it can be used to address social issues which are through our impact report, case studies, website and insights series.

    The Oranges and Lemons report which you cite is now over a year old and a third of the organisations in the sample had not received Big Society Capital investment at that time. Practice has definitely moved on and improved since then but there is always room for improvement as outlined in our revised social impact plan which we recently published. As part of it, we have also set up an in independent impact peer forum in partnership with the Esmee Fairbairn Foundation which is for intermediaries to share learning and good practice and is run by Social Enterprise UK in partnership with Investing for Good.

    I also disagree with your comment that impact measurement only creates value for impact measurement consultants as my experience has been quite different in the social investment, funding and charity sectors.. The real value from impact measurement is when it can help organisations to improve their services based on the views of the people that they intend to help. I do agree though there should be better rationalisation across all the different approaches which can be confusing and so that the Impact Management Programme funded by Access and Power to Change is a great step forward to address this issue.

    There is an emerging evidence base about the value of impact measurement. When I was at the Big Lottery Fund we commissioned research to understand the benefits of monitoring and evaluation which is much the same as impact measurement and it provided some rich information. Nesta published a report last year also highlighted helpful learning about impact measurement in practice. Both of these reports are publicly available. We are about to publish a review of how organisations use impact measurement in practice to build on the existing evidence base and it will be interesting to see the results of the Buzzacott Social Impact Survey which will be launched at Good Deals 2016 to see what more action is needed.

    Like

  4. Beanbags admin

    Hi Marcus,

    Thanks for this. Really interesting and useful.

    In terms of the value of impact measurement. In this blog, I’m specifically (but possibly not very clearly) talking about commercial value. It creates commercial value for consultants because they get paid for doing it.

    I’m absolutely not taking the position that impact measurement is not *useful* to anyone other impact measurement consultants, I’m taking the position that, for most organisations, there’s not a direct, commercial case for doing it – that would justify paying for it out of reserves/earned income if a funder wasn’t paying for it.

    (This is another blog but… ) Beyond SIBs, there is a wider group of social organisations for whom well directed impact measurement could deliver commercial value. That’s organisations for whom measurable social impact is the same thing as or fundamentally integral their commercial product(s). But for most organisations it isn’t.

    It’s (possibly) ironic that impact measurement is partly designed to tackle the problem that commercial value and social value are not the same thing but is in danger of being unsustainable without outside subsidy because, for many social organisations, social and commercial value are not closely enough aligned to make it worth spending money on.

    In terms of the SIFI activity/interest stuff, I’m not sure that what you’re saying contradicts the underlying point that impact measurement – in the sense of paid consultants using recognised frameworks to measure post-deal outcomes – is not fundamental to the business model of non-SIB social investment.

    The vast majority of investees don’t get a better deal from SIFIs based on measuring their impact or face any financial penalties for not measuring impact. I’m not suggesting they should do but, if they don’t, the connection between impact measurement and social investment is fairly woolly.

    That said, more generally, if there are different ideas emerging about what impact measurement for non-SIB social investment, I’m keen to hear more about how that’s developing.

    Like

  5. David – as ever a really thoughtful and thought-provoking piece. I just want to add a few reflections on why the ‘impact/value of social impact measurement’ may be rolling like water off the oily feathers of many I know in the charity sector (and I mean specifically the charity sector, cos that’s what I really know, but also has some distinctive cultural factors that I find are different, or at least differently viewed/expressed in the social enterprise world).

    1. The idea that proving ‘social impact’ through quant metrics and £s saved by intervention may (theoretically at least) appeal to a public procurement audience, but if you rely on charitable fundraising for income, then general wisdom has it that donors want emotional / highly subjective / human stories of lives changed, rather than stats and figures. The evidence you gather to describe your achievements is driven by your primary fund(rais)ing audience, and I’m not clear at all that ‘social impact measurement’ is a demand that comes from the general public/charitable donors.

    2. Many charities wrestle, culturally and ethically, with over-claiming about their causality for changes in ‘beneficiaries’ lives. If your practice philosophy is one of empowerment of individuals to determine and change their own lives (rather than ‘doing to’ them), it sits uncomfortably (to say the least) to then market and promote your work as having ‘delivered’ the outcomes in those people’s lives.

    3. A lot of charitable activities and their ‘modalities’ are not fit for causal claims of impact analysis at all. If you offer a completely confidential ‘momentary’ service (helpline counselling, drop-in confidential sexual health advice), then you make a clear, fundamental commitment not to inquire, let alone to ‘follow up’ the people who contact and their subsequent lives and outcomes. The best that you might be able to do offer by way of assessing their social value in such cases might be to ask what on earth might happen if to people in anonymous need or personal despair the organisation hadn’t been there….which is an important question to imagine an answer to, but definitely not actual social impact analysis!

    4. Even if, in theory, there is a strong case for public spending (and/or ‘social investment’) to be influenced/driven by having great impact evidence (compared with other similar organisations who are vying for the same funds), it is rare to see that kind of regular reward for better evidence happening in procurement practice. Among my very experienced (and not unsuccessful!) members, it is overwhelmingly ‘price competitiveness’ that wins out, time and time again. Many professional service providing charities are bang up for putting the effort in to demonstrating their impact, their ‘social and economic benefit’ and much more. But to be frank, that’s not because experience shows it will bring them greater success in their relevant marketplace….it appeals, rather, to their equally important interest in proving and promoting their worth more generally, defending against criticism and prevailing cynicism about why they think they’re such a ‘good thing’….

    Don’t know if I’ve strayed too far off the topic path!

    Kathy

    Like

  6. Thanks David, I believe that the type of impact measurement for organisations should depend on the business model and the purpose, audience and use of the information and be proportionate, useful and relevant.

    There are some examples where impact evidence can have commercial value and help drive sales or raise finance such as Oomph which Geetha mentioned yesterday. There are others too which I will shortly be sharing in a new report. The GIIN also recently published a paper called the ‘Business Value of Impact Measurement’ which some interesting examples and the survey about impact measurement run by Pioneers Post recently also asked a specific question about the use of impact data by organisations so it will be interesting to see the results at Good Deals and find out more about whether it generates commercial value.

    Programmes like the Impact Readiness Fund run by the Social investment Business were designed to help organisations improve their measurement approach to win investment or contracts so in that sense it can help generate commercial value. I have met a few organisations that have told me how useful if was to have focused external support to develop their impact approach both to help them improve their own delivery and also for raising investment or contracts.

    On the point about intermediaries, most of them do not pay external consultants for their impact measurement work and instead employ an impact advisor whose role is to assess deals, gather data and provide measurement support. Some intermediaries do draw on external help as well. The decision about which approach to take I usually comes down to what is right resource needed for a particular investment.

    Most frontline organisations in my experience gather data themselves rather than using external consultants as resources can be limited. Again it comes down to what information and approach is right for them. There is also pro bono support in some deals from universities so a real range of approaches.

    In terms of impact penalties, I have seen examples of where a social investor has decided to exit a deal becuase they felt that the frontline organisation was drifting away from its social purpose so it can happen but it has not been widespread.

    There is always more to be done and the Access and Power to Change programme is a great example of a new approach to support organisations and it would be great to see more examples of peer to the peer learning and support in the sector.

    Your blog has raised some interesting challenges and it will be good to see what others have to say about it too.

    Like

  7. At the risk of being seen as a personal friend of Julia Morley (we’ve never met) I feel you might have done her a disservice.

    Back in time, the original blueprint for the bright new world of social investment did make a prominent feature of social impact measurement. A financial market based on social as well as financial returns needed new measures.

    And so the same political hype and sector ‘oxygen’ which has followed social investment has also, too frequently, dominated the progression of impact measurement.

    The fact that these efforts have not had the desired effect (“there’s not much actual impact measurement being demanded by social investors”) does not mean that they haven’t had any effects at all.

    Like

    • Beanbags admin

      Hi Paula,

      Thanks for the comment. I don’t disagree with that point at all. I’m making a relatively narrow point about what’s happened and how it relates to business models, there’s a different point about what social investors are trying to do.

      And that’s not just the original blueprint. Even now the OECD defines what it calls ‘Social Impact Investment’: “the provision of finance to organisations addressing social needs with the explicit expectation of a measurable social, as well as financial, return”.

      Depending on how you interpret the wording, it could mean that our world leading social investment market in the UK is made up of fewer than 15 Social Impact Investments.

      Beyond SIBs, I do think the effect on organisations’ impact activity specifically caused by the fact that organisations have taken on investment is small enough to be filed under ‘negligible’ – but I don’t disagree that social investment has been part of the creation of an overall climate where impact measurement is talked about more.

      Like

      • paulawoodman

        I would understand Julia’s point being that the “dominant aspirational norm”, as she calls it, has been significantly affected. I read aspirational as being irregardless of whether an organisation has taken on investment or indeed has any current impact measurement. So it’s how things stand for those considering taking part. It’s an interesting discussion.

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s