This entry is part of a series of posts around measuring to improve the impact of nonprofit and other social sector organizations. Visit the blog's home page for other entries.
The most common question we get at Bridgespan when an organization wants to talk about performance measurement is: What exactly should we measure? As in most aspects of measurement, there isn’t a cookie cutter answer. That said, we believe three principles that relate to what to measure, how to bound your effort, and how to clarify what you want to achieve can help you mark the trail. (Quick caveat: this post is focused on measuring program performance and answering what types of measures to collect, not how many.)
Throughout, I will reference the components of an organization’s theory of change, or its beliefs about how it can achieve the impact it strives for. (For more about theory of change, please see our paper "Measurement as Learning.")
1. Measure at least the inputs, outputs, and intermediate outcomes of your programs
Learning organizations don't just care about whether they achieve results, but why and how they can get better. As such, at a minimum, they track inputs, outputs, and intermediate outcomes, for each participant. For example, a math tutoring program for disadvantaged middle school students should not only know how much math grades increase for each youth (intermediate outcomes), but also how many sessions each youth attends (output), the incoming grades of each youth (input), other tutoring support each youth may have received (input), and so on. Without this data, how would the organization discover, for instance, that its program works best for youth who arrive with math grades in the ‘C’ range, or for youth who have been tutored by a star teacher?
Even when implementing a proven program or approach, we believe there is still merit in measuring outcomes as a means of learning whether changes in how or where a program is delivered makes a difference. For example, research overwhelmingly shows that bed nets properly deployed and used in mosquito-infested areas in Africa (the output) lead to reductions in the incidence of malaria (the outcome). Yet an organization distributing bed nets still may want to measure outcomes so it can test, for instance, whether training clients in how and when to use bed nets is a cost-effective means to even further reduce malaria.
2. Reference measurement literature to decide whether to track ultimate outcomes
Sometimes your program alone cannot influence the ultimate outcome, which makes measuring it less relevant. Consider, for example, the tutoring program discussed earlier. The organization may define its ultimate goal as "participating youth graduate from high school." Yet literature reviews of hundreds of studies [1] show that much more than improved math grades in middle school is necessary for high school graduation. Unless the organization offers programs, or partners with others, to tackle issues such as absenteeism, deviance, and educational expectations, chances are the youth they serve will not graduate at a higher rate; therefore, the organization may not want to invest the time and money in measuring ultimate outcomes. (The inability of single-program organizations to be accountable for long-term outcomes is a reason why holistic programs such as Harlem Children’s Zone and collaboratives such as Strive Partnership in Cincinnati are receiving so much attention).
Another example where it is difficult to "own" ultimate outcomes lies in the measurement of complex systems. Take advocacy, for example, where there are many outside and unpredictable factors that can derail an ultimate outcome (e.g., a policy change) even when an organization delivers the activities it plans with perfection to the constituents it is targeting. The organization even may see intermediate outcomes materialize, such as winning the support of important lawmakers, but still fail to influence the final outcome. In these cases, the literature can suggest alternate approaches [2].
3. Clarify what you are trying to achieve and whether an impact study is needed
Though definitively proving impact is possible only for interventions that can be studied using rigorous evaluation designs, all interventions can be externally evaluated in some way to try to discern whether and why impact is being achieved [3]. Thus, it is important to ask yourself whether the investment of time and money—typically hefty for both—is worth the benefits. And only an organization itself can clarify the answer. If the organization wants to dramatically scale a program, encourage others to replicate or fund it, inform others who offer similar programs, and so on, the organization will likely need to measure its impact. Otherwise, outcome measurement may be enough.
At the end of the day, deciding how deeply to measure your theory of change is more art than science, but we’ve shared these principles to provide at least a starting point. What do you think? Are these principles aligned with how your organization has decided what to measure? How far do you currently go? Are you interested in doing more?
Matt Forti is Bridgespan’s Performance Measurement Practice Area Manager. He can be reached at [email protected].
