Measure twice, cut once: Why social impact measurement is an essential skill

Australian Social Value Bank (ASVB) impact specialist Andrew Callaghan has analysed new Australian government strategies for "commissioning for outcomes" and says grantmakers should be ready to lead when it comes to understanding impact and achieving measurable results.

After a decade in the rapidly developing field, Mr Callaghan continues to be surprised by the government and corporate sectors' lack of strategic leadership and general lack of understanding about the work, even as more governments are adopting impact measurement as an essential part of funding.

It's a worrying observation, and it comes as Our Community's executive officer and "chaos controller", Kathy Richardson, pegs outcomes-oriented grantmaking as the biggest single shift in the sector in recent times.

Mr Callaghan says funders can stay in front of their problems by formalising a social measurement process early with this quick checklist:

Impact specialist Andrew Callaghan

• What is the procedure for communicating to the people you're giving grants to?
• What is the support mechanism to ensure recipients are able to collect the data needed?
• What is the process for auditing the data to ensure it is sound?

Don't waste your time, collect the right information

"I've worked on evaluations of large-scale government programs and grant-giving programs in Australia and New Zealand, and I've seen that lack of clarity and consistency in what we're asking for leads to a diverse range of types of reports," Mr Callaghan says.

"There's a lot of wasteful information collected - and time wasted - that could have been spent running the programs that were given the grants, if the grantmakers were clear about the information they wanted people to collect in the first place.

"This includes being clear on the kind of outcomes you're looking to achieve from those grants and having a simple and cost-effective way of being able to give provide that evidence."

Too often, he says, evaluations are conducted after the money has been spent, putting grantmakers under pressure to justify spending with low-quality assessments.

"In some cases, I wouldn't say organisations are making it up … but not far off it."

This is where grantmakers can step in and step up.

Why government grantmakers have the power

"Grant providers should understand that there are not a lot of people who would know how to do this well. And the people who do know how to do it well realise how hard it is to continually measure effectively across different types of programs and to get the data out within appropriate time frames," Mr Callaghan says.

Grantmakers in local, state and federal government organisations are among those at the coalface of the new measurement economy, either driving change or adopting new techniques.

They are front and centre partly because they have access to the cheapest and most effective way to conduct data analysis: using data that already exists, instead of collecting it from scratch.

Government grantmakers have access to data banks others can only dream of, and the power to work at the scale that's needed.

"Grant providers can be the leaders in showing how it can be done, partly because of the data sets that they have available," Mr Callaghan says.

He cites the example of imaginary government grantmakers wanting to boost the financial resilience of beneficiaries of government support, where agencies can both track any reduction in benefits claimed and watch the effects over the longer term.

Governments can access statistics on crime, health, school attendance and more, backed by systems to protect that data. Governments are also able to partner with others by drawing on data collected by grant recipients, which can be confidentially or anonymously analysed "behind the scenes".

New Zealand's Social Investment Agency is using big data to tackle wellbeing.

This enables a government entity to assess a program's effectiveness without releasing sensitive information, allowing grantmakers to inform recipients, "keep doing what you're doing, because we can see changes happening with the people that you're targeting."

This method, Mr Callaghan says, was used successfully in the United Kingdom through a joint government and privately funded "social impact bond" that aimed to reduce recidivism rates by tracking subjects' court appearances, bail commitments and returns to prison.

Proper understanding of social impacts has massive implications for the funding of government programs. In New Zealand, the government continues to develop its Social Investment Agency, which uses big data to help the government to prioritise its spending by targeting the most vulnerable.

Mr Callaghan describes that approach as "borrowing money from the future", or taking a "forward liability" approach, based on the premise that spending now - on programs to reduce crime, for example - will save the government millions, and perhaps billions, in the longer term.

Mr Callaghan accepts he's not just an observer of the various implications of adopting social impact measures. He has skin in the game, as a practitioner who would like to see more organisations use the ASVB's approach, a variation of the cost-benefit analysis (CBA) methodology adopted by many governments globally.

The wave of the future: Governments establish outcomes frameworks

Governments and departments of all persuasions as well as foundations and corporates are creating new guidelines requiring partner organisations to prove the efficacy of their work.

Last year, for example, as the AIGM reported, the Department of Social Security moved to revamp its $217 million families and children program, focusing on outcomes at the expense of the former program-related funding model.

The ASVB has been examining the release of outcomes frameworks for service areas in New South Wales and Western Australia.

The ASVB's basic argument is that organisations pitching for funding should be moving to "align their organisation's outcomes" to those frameworks.

New South Wales, for instance, has adopted a Human Services Outcomes Framework (HSOF) for its funding strategies, with outcomes now embedded into service contracts too.

The ASVB predicts, "In the future, it is highly likely that any organisation … unable to measure outcomes will find it hard to meet government selection criteria for contracts or funding."

There are significant similarities and important differences in the way the Western Australian government has approached the issue, Mr Callaghan says.

The overall aim of measuring achievement in the human services and social sector domains is the same, but in the west the government has not yet made outcomes a requirement of contracts.

More significantly, the WA Department of Premier and Cabinet has commissioned the state's social sector peak body, the Western Australian Council of Social Services (WACOSS), to develop that framework, rather than develop it internally.

Mr Callaghan supports the decision. "You're essentially allowing the sector - who are out there working on the ground - to get consensus and to build up an agreed framework of collective impact for the sector."

For tackling large-scale social issues, "that is a really good way of making sure everyone has agreed to that model".

The NSW approach of creating those frameworks internally mirrors Victoria's process, he says. In both cases, those states have developed a "bank of metrics" for organisations to employ.

But he predicts a challenge for those states in translating and applying what he describes as "population-level" metrics. He's referring to measures that test for improvements at state level but could prove problematic for assessing smaller-scale grants.

He cites the imaginary case of a not-for-profit working with schools to combat absenteeism across several schools. While the government stipulates the use of indicators by postcode, this fails to capture results that might apply to just a few dozen students in a string of different postcodes.

"How does that relate to individual organisations that get a grant for $20,000? How are they meant to relate those metrics to the impact they're having? That's where the disconnect is."

Instead, he says, funding organisations must aim for a set of principles that can capture information consistently across organisations and can also be combined across programs.

Not surprisingly, Mr Callaghan says that which models you choose for measurement and evaluation depends on the use case, but it helps to consider the purpose.

"At a more basic level where traditionally there was a pot of money, whether it's a grant or whether it's to deliver services, consider: what's the thinking going on within a department and how they're formulating that offer out to the people who win those grants or that funding? What are they getting back? How useful is it?"

Once those basic questions are answered, it is easier to improve that system and get on with the main game of "maximising impact through the use of that data".

Can we agree to disagree? Models for measurement

These NSW and WA approaches are but two of countless frameworks being tested across the globe.

Which begs the question: which model is best?

Perhaps the answer will be provided by the Universal Commons project, spearheaded by social investor Alan Schwartz, tied to a generous prize for anyone who can crack the code of valuing social good. (See our December 2018 report)

A single dominant model is perhaps a long way off, but there is a growing sense of the need to seek agreement on basic principles and practices.

"The methodology can be agnostic," Mr Callaghan says. "It doesn't have to be cost-benefit analysis (CBA), or social return on investment (SROI), but the reality is there has to be agreement on the principles, there has to be agreement around a certain level of kind of academic rigor, and especially that creation of value, which should be a kind of monetary value if we want social value to be easily understood."

Earlier this year the OECD highlighted the problem, painting a picture of 590 different social impact investment policies across 45 countries, with most of the $US228 billion invested targeting "easy returns".

The OECD believes the current mix is less than desirable in the battle to achieve the UN's stated social development goals (SDGs).

"The challenge lies in defining and measuring impact," said OECD development co-operation director Jorge Moreira da Silva in a recent announcement.

"Different countries, public and private organisations are using different yardsticks to measure different elements. To counter the risk of 'impact washing', public authorities have a responsibility to set standards and ensure they are adhered to."

Mr Callaghan agrees, drawing a parallel with accounting standards.

"You have different peak bodies in accounting which make different judgments around value, depreciation percentages and other measures. That's an example of reaching consensus. And eventually people will start signing up to different principles and practices.

"If you want to take impact measurement and evaluation to a level where it becomes a profession, like accountancy, then you need to start understanding that it needs large-scale guidance."

That guidance includes pushing organisations to measure the whole picture of the impact they're having instead of selectively reporting the most positive aspects.

Again drawing a comparison with accounting, Mr Callaghan believes organisations should regularly report on their social impact, ideally alongside their financial reporting.

"The company I work for is based on a certain methodology, using cost-benefit analysis, which is the way government goes about looking at the effectiveness of social programs.

"Our perspective is that we're looking to create something that's cost-effective, light-touch resource-wise for organisations, but is rigorous enough to meet the standards of what government and other funders want."

He accepts that some academic institutions and others "fundamentally disagree with putting monetary value on social outcomes", but he believes it is still "early days".