When Measurement Creates False Confidence

Most systems rely on measurement to understand performance. Dashboards, reports, and metrics are used to define whether something is working. And because of that, measurement creates confidence.

But not all confidence is earned.

In this episode, I break down how measurement can create a false sense of control when it becomes disconnected from how the work actually happens. When numbers are trusted without understanding the reality behind them, systems can appear stable while quietly drifting underneath the surface.

This episode explores:

•  Why performance on paper is not the same as performance in practice 
•  How systems begin responding to metrics instead of reality 
•  What “drift” looks like in real-world operations 
•  How measurement changes behavior, even without intent 
•  Why leadership requires better questions, not more data 

Because numbers don’t improve systems. They reflect them.
And reflection without understanding can be misleading.

The views and perspectives shared in this podcast are my own and do not represent the views of my employer or any organization I am affiliated with. 

Transcript

SPEAKER_01 0:02

Welcome back. This is Behind the Measures, a podcast about public sector leadership, quality, and accountability, and the work that doesn't show up in dashboards or reports. My name's Jeremy Hurley. I work inside the system building programs, fixing what's broken, and navigating a space between compliance and real improvement. This isn't about theory, it's about the work.

SPEAKER_00 0:30

Measurement is everywhere.

SPEAKER_01 0:33

Dashboards, reports, performance metrics, targets that define whether something is working or not. And because of that, measurement creates confidence. Confidence that things are being tracked, confidence that things are being managed, and confidence that the system is performing. But measurement can also do something else. It can create confidence that isn't actually earned. And it's not because the data is wrong, but because it's incomplete, because it's misunderstood, and because it's disconnected from how the work actually happens. And when that happens, the system starts to trust the number more than the reality behind it. And that's where false confidence starts. False confidence doesn't show up as a warning sign, it shows up as a reassurance. It shows up in meetings where no one is asking hard questions because the numbers already answered them, right? It shows up in reports that move forward without discussion because everything is on track. It shows up in systems where people stop looking deeper. And that's not because they don't care, but because they believe they don't need to. And that's what makes it dangerous. Because nothing feels wrong. On paper, everything can look stable, targets are being met, reports are being submitted, metrics are trending in the right direction. You know, from a distance, it looks like the system is working. But performance on paper is not the same as performance in practice.

SPEAKER_00 2:27

That gap is where false confidence grows.

SPEAKER_01 2:40

Most measurement systems are built around what can be counted: completion, timeliness, the volume of that particular thing, outcomes that can be defined clearly and tracked over time. That structure matters. Without it, there's no consistency, there's no shared understanding. But what gets measured is only part of what actually happens. The system doesn't capture every delay, it doesn't capture every workaround, every moment where something had to be fixed just to keep moving. It doesn't show how hard the work actually is. Or how much effort it takes to keep things looking stable. So the data shows one version of reality, and the people doing the work experience something different. The people doing the work know where things are fragile. They know where steps are being skipped just in order to keep up. They know where something only works because someone is paying extra attention. And then they also know where the process breaks and what it takes to hold that part of that process together. But that knowledge rarely shows up in a report, right? So now you have two realities running at the same time. One that's documented and one that's lived. And over time, that documented version starts to carry more weight. And it's not because it's more accurate, because it's but it's it's because it's it's visible. And when decisions are made based only on what is measured, the difference becomes a problem. Because the system starts responding to the metric and not the reality behind it. Over time, that shift changes how the system operates. Instead of asking whether the system is improving, the focus becomes whether the numbers look acceptable. Instead of understanding the process, the attention stays on the outcome. And instead of asking why something is happening, the system focuses on whether it meets the target.

SPEAKER_00 5:33

And once that target is met, the conversation slows down.

SPEAKER_01 5:37

Sometimes it slows down almost instantaneously. Because a form of measurements, because from a measurement standpoint, the system is performing, but systems don't fall all at once. A lot of times they drift, and drift is usually subtle and it doesn't trigger alarms. It usually doesn't show up as failure, it shows up as small adjustments that make things work just enough to keep going. A step gets shortened, check gets skipped. A workaround becomes standard practice. And none of that looks like a problem. As long as the outcome still meets the target, but over time, the system becomes dependent on those adjustments. It relies on effort instead of design. It depends on attention instead of structure. And that dependency is invisible. You know, a system can meet a target and still be unstable. It can look consistent while depending on the same people to fix the same problems over and over again, right? It can perform well as long as attention stays high. But the moment attention shifts, things start to break again. And that's the part measurement doesn't show. It doesn't show what happens when pressure increases, when resources change, when the system has to operate on its own. And that's where false confidence grows. Because the system starts to believe it's it starts to believe that it's it's stable. When it's actually being held together. Measurement also changes behavior, even when no one is trying to game the system. Not because they're trying to manipulate anything, but because they're trying to meet expectations. No one has to tell them to adjust, they just kind of figure it out. They learn what gets reviewed, they learn what gets questioned, they learn what matters for the report. And naturally, their attention shifts. It doesn't shift away from the work, but toward what gets seen. And over time, the system aligns itself to the measure. Sometimes that that can be a good thing.

SPEAKER_00 8:48

If the measure reflects what actually matters.

SPEAKER_01 9:31

Because the measure is being achieved, the problem isn't being solved. Over time, that creates a different kind of risk. Confidence increases, leaders trust a process, pressure decreases, attention moves somewhere else. And the system is left to run on its own. That's when the gaps start to matter more. Because if the system was never truly stable, it won't hold. It'll drift slowly and quietly until someone forces attention back to it. And when that happens, it feels unexpected because the data said everything was fine. And that's the danger. False confidence doesn't feel like a problem. It feels like control. So what does this require? It requires a different way of looking at measurement. Not as an answer, but as a signal. Not as proof that something is working, but as an indication that something needs to be understood. Because a number tells you what happened, but it doesn't tell you why it happened, right? It doesn't tell you how it was achieved, whether it can be sustained, or what would happen if conditions changed. You know, that's that's where leadership comes in. Not in asking for more data, but in asking better questions about the data that already exists. Questions like, what did it take to get this result? Where did the system struggle? What is this number not telling us?

SPEAKER_00 11:48

What happens if the attention shifts somewhere else?

SPEAKER_01 11:56

Because if those questions aren't asked, the system won't answer them. And without that, measurement becomes a ceiling. Once the target is met, the system stops pushing.

SPEAKER_00 12:14

Because from a measurement standpoint, it's already succeeded.

SPEAKER_01 12:22

But as we all know, or as we all should know, improvement doesn't stop at acceptable. It continues as long as a system can become more stable, more consistent, and easier to operate.

SPEAKER_00 12:45

Measurement should support that, not limit it.

SPEAKER_01 12:49

Because numbers don't improve systems, they reflect them. And reflection without understanding can be misleading. Measurement matters, it always will, but it only creates value when it's connected to how the work actually happens when it's used to explore, not just confirm. When it starts conversations, instead of ending them, because systems don't improve from data alone. They improve from clarity, from understanding, from decisions that are grounded in reality, and from staying with those decisions long enough to see if they hold.

SPEAKER_00 14:04

Because those aren't the same thing. And knowing the difference is what prevents confidence from becoming false.

SPEAKER_01 14:18

Next time I want to talk about something that sits right underneath this. I want to talk about bottlenecks, where work actually slows down, and why most systems don't see them clearly. Because the work doesn't end at the measure.