This week, Matt Forti is excited to welcome Bridgespan Manager Jacob Allen to the “Measuring to Improve” blog. In this guest post, Jacob discusses the concept of ‘embedded measurement,’ an intentionally integrative approach to measurement for improvement by building data collection, analysis, and action into the routine activities of front-line staff and participants.
Too often, measurement feels like a burden to front-line staff and falls short of its potential to empower, engage, and improve in real time and at every level.
Consider the example of a large philanthropy embarking on a multimillion dollar training program for disadvantaged participants throughout the US. This was a large initiative and the organization was serious about measuring for both results and learning. To do this, the pilot included detailed participant surveys before and after the training, conducted and analyzed by an external measurement team who provided reports to the initiative’s leaders.
But it wasn’t all smooth sailing. Front-line staff (site managers and instructors) and program participants felt burdened by requirements to provide data and verify its quality. And while leaders were learning a lot from the initial reports, staff received the analysis long after each group had completed the program—too late to make improvements that would benefit current participants. While there were benefits for some, for many, measurement was a drag.
It’s not news that measurement is far from frictionless. But what’s the solution?
Consider an analogy. Along the spectrum of measurement approaches, at one end rigorous impact evaluations are akin to external audits of an organization’s annual financials. In the middle, ‘typical’ measurement that collects data along the way is more like quarterly financial reports conducted by accounting staff. Though audits serve an important purpose, quarterly reports are much better for creating accountability and facilitating regular decision making by managers. But even this more frequent approach can fall short—it’s hard for front-line staff and participants to own and benefit from something that feels like it’s done to or through you rather than by and for you.
Consider a third, more integrated approach. In many effective organizations, staff in each division assess ‘key indicators’ each month (or more frequently). They may review budgeted versus actual expenses, fundraising or sales figures, and customer or participant satisfaction. By developing frequent reports to meet their own needs and sharing data throughout the organization, everyone owns a piece of both the measurement process and using the data to improve. Measurement becomes integrated with the organization’s work, not an extra exercise.
Similarly, one solution to measurement’s frictions and missed opportunities is what we call embedded measurement—an approach that spreads the ownership of measurement efforts, results, and learning throughout an organization by integrating data collection, analysis, and action into the daily processes, of the program itself. When this happens, measurement shifts from feeling burdensome to being beneficial for everyone involved.
Embedded measurement has several key characteristics. First, data collection is part of the standard service delivery. In other words, data requests are timed and structured to feel part of (not in addition to) services provided to participants. Instead of separate surveys, for example, staff use ongoing interactions with participants to collect data. Second, when data is collected by front-line staff in streamlined ways, it directly benefits their daily work. This typically includes real-time, simple analyses that lead to immediate decisions and actions, primarily oriented toward improving the very next interaction with the participant. And third, participants themselves receive rapid feedback and analysis they can use to accelerate their own progress.
Let’s return to the previous example to see how embedded measurement helped. After piloting the typical measurement approach, the program and measurement teams went back to the drawing board. First, both staff and participants reviewed the surveys, refining them to ensure data would directly inform their key decisions and actions as well as leadership’s. More important, the teams changed the processes, enabling the program’s instructors to collect data as a part of regular meetings with participants rather than burdensome, extra interactions. Finally, the foundation created online, real-time reports so that both advisors and participants could review data immediately and use it to identify their next steps for business improvement.
The result? Staff bought in across the board. Statements like “It’s so much easier” and “This information will be so helpful for my next session with the participant” became commonplace. The instructors and other front-line staff became more invested in the program, resulting in more frequent discussion of what they were learning both within and across sites. On the participant side, feedback was similarly positive, as participants now see measurement as a powerful diagnostic tool to help them improve—rather than an extractive tool designed to give someone else their data. And though it’s still early, participant responsiveness to data requests seems to be improving. Perhaps just as important, since everyone benefits from and has bought in to measurement, site managers have a powerful platform to review performance and encourage improvement.
Designing a system that’s truly embedded into your day-to-day processes does take more work upfront. It also requires a new mindset—one that acknowledges that your own staff and participants are the most important audiences for your measurement. But in many cases, the payoff is worth it.
What’s your experience—where have you seen embedded measurement? What’s required to pull it off? And when is it worth the investment?
