The Wrong Diagnosis: Why Health Systems Keep Failing at AI

Most health systems that fail at clinical AI adoption are not failing because the tool is bad. They are failing because they diagnosed the problem wrong before they ever touched a vendor pitch. In this episode, Dr. Sarah Matt breaks down the core misdiagnosis that derails AI implementation: health systems treat AI failure as a technology problem when the actual issue is a question problem. The organizations whose AI implementations succeed are not the ones with the best tools. They are the ones who learned to ask the right question first. Dr. Matt draws on direct clinical and advisory experience, including a session with ophthalmologists at SUNY Upstate, where she stopped a room full of smart physicians mid-conversation and asked them what framework their department used to evaluate whether an AI tool was safe for their workflow. The silence that followed is exactly the gap this episode addresses. What you will take away from this episode: - Why tool failure is almost always governance failure in disguise - Why 'should we use this AI tool?' is the wrong first question - The correct first question: do we have the framework to evaluate whether this tool is safe for our workflow? - Why physicians have authority in this conversation that they are not using - How getting the diagnosis right at the start saves 18 months of remediation Website: https://drsarahmatt.com | Book a conversation: https://calendly.com/sarahmattmd | LinkedIn: https://www.linkedin.com/in/sarahmattmd/

—

Resources & Links:

📖 Get the Book: "The Borderless Healthcare Revolution" is available now on Amazon and major retailers.

💼 Work with Dr. Matt:
Looking for a keynote speaker or strategic advisor?
Visit: drsarahmatt.com

🔗 Connect on Social:
LinkedIn: https://www.linkedin.com/in/sarahmattmd/
YouTube: https://www.youtube.com/@DrSarahMatt-ClinicalRealist

📧 Subscribe to The Briefing: drsarahmatt.com/newsletter-signup
 
—
Disclaimer:
The views expressed on this podcast are those of Dr. Sarah Matt and her guests. They do not necessarily reflect the official policy or position of any affiliated institutions. This content is for informational and educational purposes only and does not constitute medical advice or a professional consulting relationship.

Transcript

SPEAKER_00 0:00

Welcome to the Clinical Realist. I'm Dr. Sarah Matt, physician and digital health strategist. This week I want to talk about something I see repeatedly when health systems hit a wall with clinical AI. Not a tech wall, a thinking wall. I was at SUNY Upstate speaking to some ophthalmology residents last week about AI evaluation, smart physicians, smart physicians, generally engaged, asking good questions about specific tools. And I stopped the conversation and asked them, well, before we talk about a tool, can you tell me what framework your department uses to decide whether a tool is safe for your workflow? They didn't know. Not because they weren't intelligent, but because no one ever asked them that question. And that's exactly the problem I want to talk about today. When a clinical AI initiative fails, the postmortem always, always sounds like this. The tool is not ready, or the vendor overpromised, or adoption was lower than expected. These are findings, not diagnoses. And they describe what happened. They don't tell you why. So here's what usually actually happened. The health system started probably with the wrong question. They probably asked, is this tool good? And then they spent six to 18 months trying to answer that question through a pilot. The tool was good. The pilot worked. And then the implementation fell apart. Because the tool being good is not the relevant question at the point of adoption. The relevant question is, does this organization have the capacity to integrate this tool into the way we actually make decisions? Those are the same question. And conflaying them is the misdiagnosis. So let me be concrete. A diagnostic AI tool has 91% sensitivity for a specific condition. That's generally impressive and it outperforms the average doc on this task. The pilot data confirms it. You run it in your, say, cardiology department, and it works. But now you try to scale it. And you discover that the physicians in your system who are not pilot enthusiasts don't know how to weigh the AI recommendation against their own clinical judgment. They either ignore it completely or defer to it without appropriate skepticism. And neither behavior is what you wanted. The tool's not the problem. The tool's still 91% sensitive. The problem is that you did not build a framework for how your physician should integrate an AI recommendation into a clinical decision before you deploy the tool. You assume the training was explicit. It wasn't. That assumption is the misdiagnosis. So what's the right question and when should you be asking it? The right question comes before you even talk to a vendor, before the demo, before the pilot proposal, before the RFP. The right question is: does this organization have the governance infrastructure to make a sound decision about clinical AI tools? And that question has two to four components. First, do you have a defined evaluation framework? Not a pilot protocol, a framework. A set of criteria you apply to any AI tool before you actually agree to run a pilot. What evidence standards do you require? What validation data set do you need? And what failure modes are you disqualified? If you don't have this framework, you're not really ready to evaluate any tool. You're just ready to be impressed by a vendor pitch. Second, do you have a clinical decision maker with explicit accountability? Not a physician who's kind of involved, a clinician with a title, a scope of authority, and explicit accountability for the decision and its downstream clinical consequences. This person's career path is connected to this outcome. That connection creates that kind of rigor that enthusiasm doesn't. Third, do you have a workflow integration plan that actually predates the pilot? Not a go-live plan, a plan for how clinicians are going to incorporate the IA recommendation into an existing decision-making workflow. This is the gap between the tool makes a recommendation and the clinician uses the recommendation appropriately. This is not filled by training slides. It's filled by deliberate workflow design. Fourth, do you have a governance structure that can say no? And this is one most health systems completely skip. They build an evaluation process that's designed to approve tools, not reject them. The innovation or digital office, they want to win. The vendor wants deployment. So everyone in the room is incentivized to say yes. The governance structure needs a physician, a nurse, a pharmacist, a clinician with a standing authority to say, this tool is not ready for our workflow and have that decision hold. If your governance structure can't say no, it's not from the governance structure, is it? It's theater. So I want to spend a few minutes on the clinician's role in this because I think clinicians consistently underestimate their authority in this conversation. When a health system forms a committee to evaluate a clinical AI tool, the typical composition is usually the CIO or a tech leader, the innovation or digital officer, a data scientist or IT director, and one or two doctors, nurses, providers. In most of those rooms, the technology leaders dominate the conversation. They have the vocabulary, they have the slide deck, and they have an evaluation framework that they use for technology procurement. The clinician sits in those meetings and often feels like an advisor, a clinical consultant who gets asked, does this make sense from a clinical perspective? Rather than a decision maker who is standing to actually shape the entire process. That's wrong. It's wrong in a specific, demonstrable, and documentable way. A clinician is the only person in that room who's professionally and legally accountable for a clinical AI failure. The CIO is not going to be named a malpractice suit if the AI gives a wrong recommendation. The innovation officer doesn't have a license on the line. The data scientist is not standing in front of the medical board. The clinician is. That accountability creates authority. The physician or nurse or pharmacist, they don't need permission to ask hard questions at the evaluation table. They don't need the CIO's buy-in to say, we need a better validation standard before we run this pilot. Their professional accountability gives them the authority to shape those decisions. Most clinicians are not claiming that authority at all. They're showing up to the meeting, listening to the vendor pitch, then asking questions about clinical relevance. They're not saying, before we look at any vendor data, let me tell you what evidence standard I require to recommend this tool to my colleagues. That sentence said at the first meeting changes everything. It establishes that doc, that nurse, that pharmacist, as a gatekeeper, not just an advisor. That the evaluation standard is clinical, not just technical. And that the approval authority rests with the person who carries the accountability. You have that authority. Use it early. So let me close with what it looks like when an organization actually gets it right. The health systems that are scaling clinical AI well are not the ones with the biggest innovation budgets or the newest tech infrastructure. They're the ones that have built their governance structure before they even started talking to vendors. They have a right evaluation framework. It was built by a clinician-led team before any vendor comes in. And every vendor that enters the evaluation process is evaluated against that framework, not just against its own pitch. They have a clinical executive who owns the AI governance function, and that person has a title, a seat at the senior leadership table, and explicit authority to delay or halt implementation if the evidence standard isn't met. And they're not just a committee chair, they're a decision maker. They have a workflow integration process that happens in parallel with the pilot, not after. And while the clinical team is evaluating the tool's accuracy, the ops team is designing the workflow changes. The two tracks converge before go live. And they have a governance body that's structurally capable of actually saying no. And they've said no before. The organization knows that saying no is a legitimate outcome of an evaluation process. So these organizations don't have more tech talent than appears. They have better diagnostic instincts. They ask the right questions first. That's the diagnosis. So the right question is not, is this tool good? The right question is, do we have the framework to evaluate whether this tool is right for our workflow? Get the diagnosis right before you touch a vendor. Build the governance structure before the pilot. Put a clinician in the room with authority, not just presence. That's what I help health systems do. The framework, the governance design, the evaluation criteria, the clinical leadership coaching. If your organization is in the early stages of a clinical AI eval, that's the moment to build it correctly. So I'm Dr. Saramatt, and this is the clinical realist. I'll see you next week.