In my December 2012 entry recapping the year in performance measurement, I shared two studies that suggest nonprofits and funders continue to struggle with the kind of measurement that drives continuous improvement. But viewed over a longer time period, I’d argue that the social sector has made some pretty impressive gains in this area.
So in honor of SSIR’s anniversary, let’s raise a glass to celebrate five improvements in performance measurement during the last 10 years:
- From overhead to outcomes. Prevailing wisdom used to be that the best way to judge the efficiency of nonprofits was by looking at the proportion of their budgets that they spent on overhead. Today, websites such as GiveWell and Coalition for Evidence-Based Policy are rating nonprofits on the quality of their outcomes. Even Charity Navigator has dramatically changed course—“results reporting” is now a pillar of its rating system.
- From ideology to evidence, particularly in global development. In the early 2000s, fierce ideological debates about how to bring developing countries out of poverty were the norm. Then a new breed of economists at the Poverty Action Lab and its sister networks proposed letting the evidence decide. Though the meteoric growth in the use of randomized control trials is controversial, it has helped the largest global development funders make more decisions based on evidence about what works—a shift we are also starting to see in some domains, such as education, in the United States.
- From isolated to shared measurement. The past decade has seen renewed interest in partnership and collaboration across agencies and sectors to achieve common goals. Especially exciting is that many of these local collaborations, as well as national networks, are now setting common indicators, reporting through a common data system, and analyzing and learning from data together.
- From straightforward to complex interventions. Innovation in measurement used to focus mainly on direct-service interventions with fairly linear logic models (for example, how to rigorously measure outcomes of a summer literacy program for children). But lately there has been an explosion of innovative approaches (such as developmental evaluation, outcome mapping, and policymaker ratings) for measuring what’s happening in more dynamic environments, such as advocacy work, systems change, or neighborhood revitalization. These and other approaches are allowing organizations and initiatives with complex interventions to learn more from their measurement, undergirded by more sophisticated theories of change.
- From external evaluation to performance management. In the past, as described by the 2006 SSIR article "Drowning in Data" social sector measurement was dominated by funder requests for endless lists of metrics or proof of program impact from external evaluations. While many nonprofits will tell you these requests haven’t necessarily slowed, there has been a more concerted effort by evaluation firms and other measurement providers to develop techniques and tools that support performance management. For example: the adaptation of the balanced scorecard to the nonprofit sector, the proliferation of performance management data systems tailored to nonprofits, and the launch of the PerformWell website.
And now for you “glass half empty” types, five areas where we need to make much more progress. We need:
- Greater focus on long-term outcomes. Nonprofits still get by with measuring only what’s easiest to measure: short-term outcomes such as school attendance or job placement rates. But if nonprofits want to change lives, they (and their funders) should also care about whether their programs are helping people achieve more meaningful goals, such as completing college or making sustained economic gains. If more nonprofits asked the question of whether the people they serve actually end up in a better situation over the long haul, there would perhaps be a greater focus on holistic interventions or collaborations that create solid pathways to those more meaningful outcomes.
- Fewer organizations getting big on the basis of limited evidence. A recent study by Veris Consulting found that only 39 percent of nonprofits that are scaling their programs have evaluated the impact of their work in any way; less than a fifth had a third-party outcome or impact evaluation. While having multiple sites can help an organization test and improve its programs, more extensive growth should require rigorous evaluation.
- Greater recognition of organizational and contextual factors that drive strong performance. In the rare instances when models are proven highly effective, funders understandably get excited about scaling those models. But most evaluation studies devote little, if any, attention to underlying organizational factors (such as culture and leader characteristics) and contextual factors (such as regulatory climate and the presence of high-capacity partners) that play a role in success. In their absence, organizations or funders often require that replicators follow the original model with full fidelity; yet this precludes important adaptations and improvements that could increase the odds of success.
- A bigger role for an organization’s constituents in performance measurement. Nonprofits have a hard enough time implementing a measurement system that works for senior leaders and program staff; they rarely tackle the question of how measurement can work for the individuals, families, and communities they seek to benefit. But nonprofits often do a real disservice to these constituents—and their own success—when they fail to involve their constituents in reflecting on results, setting goals, and deciding how best to achieve those goals.
- As much attention given to “performance audits” as financial audits. Though exact requirements vary by state, most nonprofits with revenues greater than $500,000 are required to obtain an annual independent financial audit. Yet only a handful ever get an outside assessment of their self-reported program performance data. Over time, the social sector would benefit from independent social impact analysts to help clarify for funders and nonprofits the extent to which program data is accurate and representative.
As you reflect on the past decade, what other gains or gaps in social sector measurement would you put on the list?
