The Pilot Trap: What Health Systems Get Wrong About Implementation

Your pilot worked. You got approval to scale. Six weeks into full implementation, something is wrong: not catastrophically wrong, just wrong enough to make you wonder whether "it worked in the pilot" meant anything at all. In this episode, Dr. Sarah Matt breaks down the pilot trap: the gap between proving a tool works under controlled conditions and proving an organization is ready to run it under normal operating conditions. Pilots and implementations are not on the same continuum. They measure fundamentally different things. The health systems that scale successfully treat the pilot as a learning vehicle, not a prediction. The ones that get stuck assumed pilot success equaled implementation readiness. What you will take away from this episode: - Why pilot success measures tool performance, not organizational readiness - The three gaps that trap most health systems: motivation, workflow integration, governance - Why resistance to clinical AI is usually not irrational, and how to diagnose the source - How the physician champion role shifts from pilot advocate to implementation steward - Why building governance before implementation begins is the single variable most predictive of scale-up success Website: https://drsarahmatt.com | Book a Discovery and Clarity Session: https://calendly.com/sarahmattmd | LinkedIn: https://www.linkedin.com/in/sarahmattmd/

—

Resources & Links:

📖 Get the Book: "The Borderless Healthcare Revolution" is available now on Amazon and major retailers.

💼 Work with Dr. Matt:
Looking for a keynote speaker or strategic advisor?
Visit: drsarahmatt.com

🔗 Connect on Social:
LinkedIn: https://www.linkedin.com/in/sarahmattmd/
YouTube: https://www.youtube.com/@DrSarahMatt-ClinicalRealist

📧 Subscribe to The Briefing: drsarahmatt.com/newsletter-signup
 
—
Disclaimer:
The views expressed on this podcast are those of Dr. Sarah Matt and her guests. They do not necessarily reflect the official policy or position of any affiliated institutions. This content is for informational and educational purposes only and does not constitute medical advice or a professional consulting relationship.

Transcript

SPEAKER_00 0:00

Welcome back to the Clinical Realist. I'm Dr. Sarah Matt. Over the last few weeks, we've been building through a framework for clinical AI adoption. We talked about why pilots don't scale. We talked about what vendors are not telling you, and we talked about physician authority and the governance process. And we talked a lot about what it looks like to get the diagnosis right before you start. Today, I want to go deeper on one specific thing. What happens after the pilot ends? Because this is where I see health systems struggle the most. Not in the evaluation, not in the negotiation, and the transition from this works to this is part of how we operate. I call it the pilot trap. And if you're in a health system right now that's six to 12 months past a successful pilot that has not fully scaled, you are absolutely in it. A clinical AI pilot is a constrained environment. That's its purpose. You constrain the variables so you can isolate the tool's performance. In a typical pilot, you're running the tool with a very motivated physician, nurse, clinical population. And these are the early adopters, the people who actually wanted to use the tool, who believed in it, and who are a patient with the rough edges. But they're not representative of your broader physician or nurse population. They are the enthusiasts. You're running it with dedicated IT support. The implementation team is present. Issues get resolved quickly. And that's not what normally happens in IT support. Normal IT support is a ticket queue at three-day resolution time at best. You're also running it with protected bandwidth. The pilot gets resources that the full implementation will not have. You're treating this as a project. Full implementation actually treats it as an operational reality. And you're measuring the right things for the pilot, not necessarily the right things for the implementation. So you're measuring accuracy, clinician satisfaction, data throughput. And these are the right things to measure for a pilot. They tell you whether the tool is actually worth scaling. They don't tell you whether your organization is actually ready to scale it. So here's a thing that trips up most health systems. A pilot proves the tool is ready. It does not prove the organization is ready. And readiness is not about the tech. It's about process, governance, accountability structure, and workflow integration. A 93% accurate AI tool deployed into an organization that hasn't built the workflow integration layer is going to perform at whatever level your least prepared clinician can extract from it, which is going to definitely be below 93%. So when I work with health systems that are stuck in the pilot trap, I consistently find three gaps, almost always the same three. So gap one is the motivation gap. Remember, in the pilot you had enthusiasts. In the implementation, you have a full range. So you have enthusiasts, you have skeptics, and you have people who have not been briefed on why this tool is being deployed, and you have people who are actively hostile to the idea that an algorithm can inform their clinical judgment. Nobody planned for the skeptics. Nobody built a strategy for clinician adoption that accounted for the full distribution of attitudes. So the communication plan was aimed at the people who are already on board. You can't scale a clinical AI tool to a skeptical clinical population without deliberate adoption strategy. And this doesn't mean just training slides, not just an email announcement, a real strategy that understands where the resistance comes from and addresses it directly. Resistance to clinical AI is usually not irrational by any means. It's usually one of three things. A clinician who has not seen evidence relevant to their specific patient population, a clinician who does not understand the tool's failure modes and therefore does not trust its edge case performance, or a clinician who is not involved in the evaluation and feels like the decision was made without them completely. Each of these has a different response. The health systems at Scale Well figure out which type of resistance they're dealing with and address it at the source. The next gap is the workflow integration gap. So in the pilot, someone thought about how the AI recommendation might fit into the clinical workflow. But in the full implementation, that thinking did not propagate. So what happens in the full implementation is this the tool goes live, each clinical team integrates it differently. One team treats recommendation as a starting point for discussion. Another team ignores it completely unless it contraindicates their instinct. And a third team follows it without appropriate scrutiny because they trust the algo more than their own judgment. And none of these is the right behavior. But none of them was actually addressed in the implementation rollout because the workflow integration plan might not have been part of the rollout at all. The tool was deployed. The workflow was not designed. And the last gap is the governance gap. And we've talked about this a lot. In the pilot, someone was responsible for the AI tool's performance. There was an implementation team. There was a physician champion or a nurse champion, and there was a feedback loop. In the full implementation, yeah, the implementation group, yeah, they moved to the next project. And that physician champion went back to clinical practice. And that feedback loop is probably closed. So no one's aggregating the data on how the tool is actually performing in production. No one's identifying the edge cases where the tool is failing. And no one's accountable for the ongoing governance of the tool in a live clinical environment. Six months after GoLive, the tool is still technically running, but no one knows whether it's actually working. The solution to all three gaps starts with the same thing: a clinician steward, not just a champion. Champions belong to pilots. Stewards belong to implementations. A champion sells the idea of the tool and they build enthusiasm. They clear early blockers, they make the case to skeptics. Champions are essential during the evaluation phase. They're not the right model for the implementation phase. So a steward operates the tool in a live clinical environment. They own the ongoing governance. They're the person who gets called when the tool is not working as expected. They're the person who determines whether an unexpected behavior is a tool failure or a workflow failure. And they're the person who has the authority to say, we need to pause this until we figure it out. And the accountability to say, here's what we learned and here's what we're going to change. So stewardship is not a volunteer role. It requires protected time, organizational authority, and a clear accountability structure. Most health systems do not build this role before go live. They just kind of assume the champion's going to transition naturally into the steward function. And some champions do, but most do not. Advocacy and stewardship are different skills, and assuming continuity between them is one of the most reliable ways to end up with a tool that works technically and fails operationally. So build the steward role before the pilot ends. Define it, fund it, name the person, give them the authority they need before the implementation team leaves. So I want to close with something practical. How to bridge the gap between pilot success and implementation success before it opens. The health systems I've seen do this well do three things before the pilot ends. First, they run a parallel workflow design track. While the clinical team is evaluating the tool's accuracy, the operations team is designing the workflow, not just the go live protocol, the actual clinical workflow. Where does the AI recommendation appear in the decision-making sequence? Who sees it? What action does it trigger? What happens when the clinician disagrees with the recommendation? These aren't implementation details, they're clinical design decisions. So they need to happen before go live, not during it. Second, they run a clinician adoption segmentation exercise. Before the full rollout, they map the clinician population into segments. Which doctors and nurses are enthusiasts? Which are neutral? Which are skeptics? And for each segment, they design a specific communication and engagement approach. The enthusiasts get trained in more like train the trainer roles. The neutrals get evidence-based briefings and access to the pilot physicians. And the skeptics get direct conversations with the steward about the specific concerns they have. It's not complicated. It's a week of work, but most health systems skip it and then spend 18 months dealing with variable adoption. They also build a governance structure as part of the pilot closeout. They document the tool's current performance state. They establish the feedback mechanism and they name the steward. Then they set a 90-day review and they write down what this is not working looks like. So that when it's happening, the organization has a response protocol that doesn't start from scratch. This messy middle is not an inevitable part of clinical AI implementation. It's a consequence of assuming pilot success means operational readiness. So do the work before the pilot ends. The transition is much cleaner. So if you're in a health system that has a successful pilot sitting on the shelf that never fully deployed, the problem is almost certainly one of the three gaps I described today: the motivation gap, the workflow integration gap, or the governance gap. All three are solvable. None of them require more tech. They require organizational design, clinician leadership, and a governance structure built before the pilot ends. So that's the advisory work I do. And if you're in the middle of this right now, the fastest path forward is a direct conversation. Feel free to call me directly. My information is down below. So I'm Dr. Sarah Matt. This is the Clinical Realist, and I will see you next week.