#54 - The Cancer Exam That AI Failed

What happens when the models everyone keeps calling "doctor-level" have to actually take the test? We put that claim under pressure with a study that lands uncomfortably close to real life: six popular large language models were handed 137 multiple-choice questions on colorectal cancer and forced into a strict zero-shot format — no examples, no reasoning shown, just the letter. Performance came in around chance, or below. In other words, the models often sound certain while effectively guessing.

We break down why colorectal cancer is such a revealing stress test for medical AI. Guidelines evolve, screening recommendations shift, and a single case can move across primary care, GI, surgery, pathology, oncology, and radiation — so "knowing" this disease means tracking a moving, country-specific target, not reciting one fixed fact.

Then we turn to what these systems are really doing under the hood — next-token prediction, instruction tuning, reasoning-style layers — and why none of that guarantees a reliable guideline lookup. We walk through the failure modes that matter for patient safety: basic fact-retrieval errors, hierarchical logic breaking down in cancer staging, and hallucinations that could spawn unnecessary tests or wrong recommendations.

For clinicians, trainees, and curious patients leaning on chatbots for health questions, the takeaway is blunt: don't trust an answer that can't show its work and cite the guideline.

References:

Performance of next-generation AI chatbots in colorectal cancer knowledge assessment: a comparative pilot study of ChatGPT-5.1, Gemini-3Pro Preview, DeepSeek-V3.2, Kimi K2 Thinking, Qwen3-Max and Claude Opus 4.5
Chen et al.
Updates in Surgery (2026)

Credits:

Theme music: Nowhere Land, Kevin MacLeod (incompetech.com)
 Licensed under Creative Commons: By Attribution 4.0
 https://creativecommons.org/licenses/by/4.0/

Transcript

Smart Chatbots, Shocking Test Scores

SPEAKER_00 0:00

The newest AI chatbots are smarter than ever. So, why did they score under 30% on basic colorectal cancer questions?

Why This Result Should Worry Us

SPEAKER_01 0:17

Hello and welcome back to Coding Cure, the podcast where we discuss decoding health in the age of AI. And my name is Vasant Sarathy. I'm a cognitive scientist and AI researcher. And I'm here as always with Vara Hagopian.

SPEAKER_00 0:30

I'm an emergency medicine physician.

SPEAKER_01 0:32

Alright, so I find that to be fascinating. That, you know, we hear in the news, right, all the time about AI systems passing all these medical exams.

SPEAKER_00 0:42

Doing better than doctors.

SPEAKER_01 0:44

With the boards, even, right? Doing really well with the boards or doing better than doctors. And uh so we hear those stories a lot. We see we see we hear on the internet about all these things, but we also hear at the same time, oh, be careful, it gets things wrong. And so people are often left wondering, okay, what is it that it gets right? And what is it that it gets wrong, and why does it do one or the other? And I'm not saying that this particular study answers that question directly, but there are some there are some clues. And I think that it's worth kind of, you know, looking into that and diving

How Six Chatbots Were Tested

SPEAKER_01 1:16

deeper into that.

SPEAKER_00 1:16

Yeah. And in this study, basically what they did was they provided um questions about colorectal cancer, detailed questions, you know, things about surgical treatment or pathological evaluation, things like that, a a uh clinician might want to know to a bunch of different chatbots, six different chatbots in their like native state. Um, and tried to figure, you know, basically gave them each 137 questions and saw how how well did they do answering those questions. And they're multiple choice questions. There were four four choices for each.

SPEAKER_01 1:50

Very similar to what you would see on uh uh what they might, what these models might have seen on medical exam uh questionnaires and so on. Yeah. Yeah, exactly.

SPEAKER_00 1:58

So I mean, if you're if you're like rolling the dice, you're gonna get a 25%. And these models did about that or worse, which is which means they thought about it and did their wrong thing.

SPEAKER_01 2:09

Right.

SPEAKER_00 2:09

Well, I think, well, it's interesting because I don't think they thought about it. That that's one of the things that I want to talk to you about. Because, well, backing up a little bit, um, in the real world, as a clinician, I might just go to one of these chatbots and say, like, hey, what's, you know, what's the answer to this question? Like, what's the, you know, how far do I need to resect to? Or um, what sort of pathological evaluation do I need to do, or whatever? I'm gonna ask that question and I just like want a straight up answer. Yeah. Right. Yeah. And maybe I'll use a medical specific one, maybe not. I think a lot of people go to the quick and easy, you know, I've used this chat bot before, whether that's Deep Seek or Claude or Gemini or whatever. Um, I'm gonna use it for everything. And they just go to where their comfort zone is and they get an answer out of it, and they're like, Yeah, that sounds good. I'm gonna move forward.

Zero Shot Prompts In Real Life

SPEAKER_00 3:01

But what they did in this specific scenario is they said, okay, a lot of times clinicians like me are like too busy to want all the reasoning behind it. They don't want that. They just want the answer to the question. And so in this scenario, they put a zero shot constraint on it. So I want you to explain that. What is a zero shot constraint and and like how might it have affected results?

SPEAKER_01 3:24

Yeah, and and and and I think that's the straightforward enough a question in terms of um what that means, because zero shot just means you've given it zero examples. You've just straight up asked the question. So it used to be the case. So stepping back a little bit in the world of machine learning, the idea in machine learning has always been hey, give the give the model a bunch of examples to train on, like uh questions and right answers. And then it knows the pattern and it uses that pattern to answer the next question that comes up. Um, and so large language models uh were also trained on a lot of data beforehand. But people found that sometimes, even in your prompt, if you gave it a couple of examples of um the types of answers you want, right? People still kind of do that. They say, Oh, you know, here's a couple of things that I've written. Can you write something else that's kind of like the same style? Like that's giving a couple of examples uh up front and saying, write this in my style.

SPEAKER_00 4:18

And so you're giving it like more information.

SPEAKER_01 4:20

Yeah, and that's called few shot because in the old machine learning world, in order to train a model from scratch, you had to give it thousands of examples. But now with just a couple of examples and relying on the large language models already trained information, you just have to give it a few examples and maybe it catches on what it needs to do. And that's called few shot. Um and so a zero shot is basically uh you give it no examples.

SPEAKER_00 4:45

You're just like, do just do answer the question. Just answer the question, and that's it. I don't want any other information.

SPEAKER_01 4:49

Yes, you are this massive holder of all of the world's knowledge. Just answer my question, right? And so that's that's what zero shot means. And it's a very common way of operating for us. That's how we use it, most of us use AI on a daily basis. Yeah, I'm not a good thing. Right, right. So that's that's a very naturalistic setting for

Why Colorectal Cancer Is Complex

SPEAKER_01 5:08

sure. I'm wondering, especially in this study, like why um it seems like they focused on colorectal cancer. And I'd be curious to understand a little bit better as to what makes it what why that and why is that interesting, why is that complex, and so on. I I know that they talked about a little bit in the paper where they talked about how it's dependent on not just static information.

SPEAKER_00 5:32

Well, for sure. I I think this is true of a lot of clinical topics is that the guidelines get updated over time, right? And so what was true 10 years ago is not true now. Not exactly, right? We have more information and so we act on things differently. So, one example in the United States is that, you know, semi-recently the colorectal cancer screening age was decreased from 50 down to 45. And that was based on data that younger people were getting polyps or getting colorectal cancer. And so let's let's start screening people earlier. So that's the evolution of science is that our truth changes over time. And that's that's normal, that's natural, and the guidelines then change over its time. So the guidelines that were true 10, 15, 20 years ago, 30 years ago, they might be out there, right, for these LLMs to see, but they're not what we clinically are using presently. We're using the new and updated guidelines because they were updated based on new research and new information that was available. Yes. And so there's a lot that goes into this. And colorectal cancer, in my mind, is as a complex topic. So I think it's a great topic for this because there's so many different pieces that come together. If someone has colorectal cancer, first of all, there's there's screening for it. So there's like, yeah, can we detect it early? Can we prevent it? Um, that involves primary care. Then you think about, you know, who's doing, for example, a colonoscopy. That's your GI doc. Then you may have your surgeon coming in to do any sort of resection or surgery that needs to be done. You have, you know, an oncologist, maybe a radiation oncologist coming in to recommend chemo or radiation. And now you're talking about all these different specialties coming together. And the guidelines would apply to all of them. But each other.

SPEAKER_01 7:28

Or differently, right? There might be separate guidelines for each of these separate groups as well, in addition to the general guidelines for colorectal cancer.

SPEAKER_00 7:34

Yeah, depending on on how they're sort of arranged. They looked in this paper specifically, it was done in China. So they looked at um the the guidelines from there. And a lot of the guidelines from there were actually, you know, all cemented in one location. Okay. Yeah. For for the LLMs to look up if they sort of chose to do so. But who knows where they were going for information, right? It's like they could go, I don't know, anywhere. I think that's one of the issues with it. And because the prompt, like we talked about, was zero shot, we don't have information about where they went to. The prompt they gave said, Hey, you are a medical exam assistant, read the following colorectal cancer question. Do not explain your reasoning, do not provide a rationale, only output the correct option letter like A, B, or C. So I think that's interesting because one, it totally makes sure it's a zero shot prompt, right? Just give the answer. There's no examples.

SPEAKER_01 8:35

They're not providing any examples. Yeah.

SPEAKER_00 8:36

And two, like they're actually saying don't don't give an explanation. I don't think I never write that when I when I'm putting a feels a little artificial, yes, yes. A prompt in. I don't say like don't provide rationale. I don't necessarily want the rationale or the reasoning. Sometimes I just want the answer, but I don't actually like explicitly say don't do that. And interestingly, and you could probably speak to this better than I can, but when you actually ask for this like reasoning chain from the LLM, the studies I've seen show it does better.

What LLMs Actually Do

SPEAKER_01 9:11

Yes. Yes. So speaking to that a little bit, I think it's worthwhile just thinking about what happens when you're typing in these questions into Chat GPT or Claude or whatever. What's happening behind the scenes? And part of that is actually a very complex picture, some of which is unknown to us because these are private companies doing uh proprietary things that you know make their model better than others, and they're not going to share that with the world. But part of that is the general infrastructure that goes behind modern AI. So we've talked about LLMs. This is the core neural network model that takes in text and puts out text, predicts the next word, predicts the next word. And you put in all that, you know, you put in five words, it predicts the sixth word. And then you put in those six words, it predicts the seventh words, seventh word. And that's how you get a sequence of words that come out of these systems. So they sound reasonable. So they sound reasonable, but they've been trained on trillions of internet data, uh pieces of internet data, and and the idea is that they've learned something to not just complete grammatically correct sentences, but also it appears that um in that trillions of data uh there was encoded some human knowledge, some reasoning. And so when these um language models are outputting those words, they are also incorporating the same type of reasoning. And so that's why you see that these models actually doing something really, really powerful, right? That's why they're useful in so many different respects, because they're they've already encoded patterns of our reasoning uh that capture much of our thinking, which is you know kind of incredible to think about, right? That's that's I think the big breakthrough revolution of these large language models is the fact that just next-word prediction actually encodes, if you but give it enough data, actually encodes a lot more. And it's not just the statistical patterns, but maybe there's something more complex. And there's whole bodies of researchers working on just figuring out what these language models actually have internally. But that's not all that exists. That that's not the only model that's being hit when you type in a question into Chat GPT. These models today are very sophisticated because they take that core base and they do a lot of interesting things to it. They teach it how to be more general, question answerer. It's not just completing the next word, but it's also if you asked a question, it should know to answer the question, not just ask another question or you know, do something else, which people, you know, in a in a common piece of writing, there might be a question, but there might be other things right after. It might the next word prediction might not be enough. So they've trained the model on top to be a good question answerers. Uh, that's called instruction fine-tuning. They also generally, and and all of these things are called fine-tuning, ways to like make this model even more usable and useful for us. And then there's a whole set of uh things that they do to make the model answer questions in a way that humans like and humans prefer. And we've talked about this extensively on our podcast before about sycophency and the idea of models telling you things you want to hear. Yeah right. Uh, but in addition to all that, these models also have uh now, today's models also have uh fine-tuned for reasoning. So what they do is they're also trained to like lay out um potential next steps. And this is what happens when you see um, you know, when you type in something and it says thinking. Um, and that's really that's really kind of not what it's doing, right? It's not thinking. What it's doing is producing in-between words that are based on prompts and other things that it then uses as context. So remember I said you give it six words, it predicts the seventh word. Well, what if you gave it six words and then it predicted the seventh word, but then you gave it eight, nine, and ten, and then it used all of that and predicted eleven, right? So now you've inserted things in there to make it to kind of drive it and steer it in a particular direction. And that's uh kind of a lot of the reasoning models help steer or drive it towards uh incorporating or externalizing reasoning. That is instead of just giving you the answer, have it specifically like lay out what the derivation might be first before it gives the answer. Does that make sense? So it's like saying, What is you know, what is four times four, right? Instead of just ask having it give the answer, if you forced it to have what's called chain of thought reasoning, you're going to force it to first say what does times mean. Times means you add four, four times, right? If you had it write that out, now those words become part of that input to the model for the next round, which didn't have before. So you're just forcing it to do things extra. So my point is these models are doing a lot more in the back end than they used to before. And as a result, they, you know, are more complex. And also, some of these models might have access to external um databases and such. We don't know that.

SPEAKER_00 13:50

Yeah, well, I mean, that was going to be my sort of follow-up question was hey, wouldn't it be great if these models were looking for the colorectal cancer guidelines in their location? And this was this study was done in China. So for for this study in China, but you know, if a study were done here in the United States, wouldn't it be great if it said, okay, like because these questions are clinical questions about colorectal cancer, I'm gonna go to the guidelines and get the information from there.

SPEAKER_01 14:19

Yeah, and it might do that, right? It might here's the thing the how do you know? Well, this is the this is the challenge. So the prompt said, no one don't think about it, don't you know do any reasoning. But I'm not sure that that's necessarily going to stop it from reasoning or doing those things. Because these systems, again, like I said, are proprietary systems that are pretty complex in the back end. So it's not clear what exactly are the capabilities that are manipulable by the by the user and what they're what what are not. But regardless, you know, I think the results are stark because if you assume that it's doing all of that in the back end, it should have gotten better results than

Fact Errors, Logic Failures, Hallucinations

SPEAKER_01 14:54

it actually did.

SPEAKER_00 14:54

I mean, yeah, it was basically like rolling the dice. And a lot of times the errors, and we can go through some of the errors because I think they're interesting, but like a lot of the most of the errors were actually fact retrieval errors, which I would have expected the AI to get those if it's gonna go back to the guidelines and look them up. And maybe, maybe the zero shot prompt is part of what prevented it. But there were other, you know, there were some reasoning errors, there were some hallucinations, there were other things, but like fact retrieval errors were very high up there.

SPEAKER_01 15:26

Right, right, right. And it's it's it's amazing. So, like one of the questions I thought was very interesting was the very first question on screening, which was according to the 2020 China Cancer Statistics Report, what is the ranking of the incidence of colorectal cancer among all malignant tumors in China? And the option was well options were first, second, third, fifth.

SPEAKER_00 15:50

Yeah, that feels like a just like a very lookup answer. Yeah, you just look up the answer in the guidelines. Do I know the answer for China off the top of my head? No, but I could go to the guidelines and look at it.

SPEAKER_01 15:59

Oh, the report, yeah, statistics report in this case. Right, exactly.

SPEAKER_00 16:02

And so this this is an example of like uh almost like a probabilistic trap, right? Yes. Where um, you know, the AI often said second. And yeah, the AI got it wrong.

SPEAKER_01 16:14

It said second, it said, it said second, right? What the right answer was I I forget what the right answer was, but the yeah.

SPEAKER_00 16:20

But it is common for GI cancers to be second. That's a common ranking for GI cancers throughout the medical literature, but that was not the answer in the 2020 China Cancer Statistics Report, which it literally tells you to refer to, it tells the AI to refer to that to get the answer, and it still got it wrong.

SPEAKER_01 16:37

That's right. And I I've, you know, again, uh not to knock the study too much, but the the prompt may have prevented that from happening, right? If the prompt had not said, don't think about it, don't reason about it, it might have looked it up, surfaced that report, and then used that to answer the question, right? In which case it's it might have gotten it correct. I I don't know that. But the it I think it's interesting that it got it wrong at all, right? Independent of anything, which is exactly what you said, which is that probabilistic error, which is that if there is a lot of data in the system, in the tokens, in the words, in the original trillions of web interface data pointing a particular way, it's going to be very hard for it to overcome that. Because again, like I said, these systems are based on predicting the next word, and they do that by extracting the pattern, statistical patterns of the words before. So if something is more likely to follow something, right, that's what's going to be selected.

SPEAKER_00 17:34

Yeah, it's interesting because you've talked about it with words a number of times in the past. It's like, oh, you know, we went to the or we went to uh, and most of the time it's the 90% of the and 10% of the time it's uh, it's gonna give you the as the answer. So here it's the same sort of thing. Like most of the time it's listed as second, but in this specific scenario, it's it should have had a different answer. Right. And it wasn't able to, it's not going it, it we don't know how it did it, right? We don't know where it got the answer from, but we know probabilistically, a lot of times it's listed differently in the scientific literature. In this particular set of cancer statistics, it maybe it didn't go back to it. Maybe it just took what was out there and said, hey, this is this is like what I'm used to seeing. Yeah. And I'm gonna stick that number in there. Yeah.

SPEAKER_01 18:23

And so generally the more unusual the question-answer is, the more likely it is to get it wrong.

SPEAKER_00 18:30

Yeah, that makes sense. It's sort of this regression to the main kind of thing. Just the average, yeah.

SPEAKER_01 18:36

Everything else washes out, all the nuance washes out.

SPEAKER_00 18:39

And you could on you could see how getting misinformation like this could be dangerous. Like if you either over or under screen for cancer based on not knowing, you know, how prevalent it is, for example. Right. Um, there were other different areas where it failed too. Um, another example of this was like hierarchical logic failure. There was a question about what the definition um of early colorectal cancer PT1 stage was. This is not something that like, you know, everyone would know, but if you go back to the guidelines, you know, what happened is that uh PT1, PT1 stage colorectal cancer has invasion into the submucosa. You don't you don't even have to really know what that means. But what the AI said was that it's confined to the mucosal layer, meaning it didn't go any deeper. And the truth is that PT1 does go deeper, it goes deeper than what the AI said.

SPEAKER_01 19:39

Is that a guideline thing or just a factual like anatomical thing?

SPEAKER_00 19:43

A factual anatomical thing that was in the guidelines. And so you could imagine if if you have a clinician who uses this and thinks, oh, it's confined to this area versus it goes deeper, well, that's that's gonna have different treatment. Oh my god, yeah.

SPEAKER_01 20:01

Right? Yeah, for sure.

SPEAKER_00 20:03

Uh you might need surgery versus you could just do um a resection on a colonoscopy, for example. You could have completely different treatments based on how deep this cancer goes. Right. And so this is another example where like, oh, maybe that sounds reasonable, but like that's not that's actually not reasonable. And maybe it was kind of the wording, maybe it was the wording of this concept of early colorectal cancer that made the AI sort of think that it was in site two, but it's not in site two. It specifically says PT1 stage and not in site two. Yeah. And so I I think this is another example of like, hey, the logic is failing here, and it could have dire clinical consequences for a patient if it were followed.

SPEAKER_01 20:49

It's very nuanced, it's very specific, and it requires almost like a an accurate sort of causal model of the mechanics of what's happening, as opposed to just predicting the next word. You want to understand, okay, what is the mechanism by which this, you know, situation occurs.

SPEAKER_00 21:05

Yeah. And they did go into a ton of detail in some of these questions. I mean, there's another example here where um there was just like a plain old fact retrieval error. It's like, okay, well, the recommended distance of the distal margin of the rectal mesentery from the lower edge of the tumor is what? This is like, you know, again, you don't really have to understand this, but basically it's like it's like what a surgeon would need to know is how far, how far do I do the surgery? How, how far or deep do I go in, or how far past the margin of the tumor do I go? And there's guidelines for that for a reason, right? Do you go three or four or five or six centimeters? You want to try to make sure you get all the tumor out, right? And so the AIs came back and said three centimeters. But the correct answer in this case was five centimeters. So you can imagine if you don't get large enough margins around. Your cancer, then you might not actually get the whole tumor out. Right. Right. And you could have, you know, recurrent cancer from that. And so you can see that these like little nuances actually could turn into much bigger problems for a patient.

SPEAKER_01 22:16

Right.

SPEAKER_00 22:17

And then, of course, I would be remiss not to mention number four, the fourth area of failure, which is hallucinations, something that we have seen and talked about many times on this podcast. Um, where, you know, for example, they were recommending checking a blood test for all colorectal cancer patients for a tumor marker, but it's actually only used for a certain subset of them. That could lead to like, you know, just paying more for the healthcare system, uh getting unnecessary testing, leading to extra anxiety, et cetera, et cetera.

SPEAKER_01 22:46

And hallucinations is is is a is a known failure mode, and frankly, a fan a fancy word for just when it makes things up, right? That's kind of where we are with hallucinations. And it's it makes things up. That that that it's known to do. Um yeah. So I mean, I think the this paper was very interesting because A, they constrained it sometimes, you know, some might argue in an artificial way.

SPEAKER_00 23:07

Too much, maybe.

SPEAKER_01 23:08

Maybe, but it also is a bit what they say also, which I agree with, which is that it's a baseline for these models. And sure, they can get better than this, but this is pretty bad to begin with. This is a modern day 2026 set of models that are meant to have certain information correctly. Correct. And they don't.

What Safer Clinical Use Requires

SPEAKER_00 23:31

Yeah, and it's tempting, I think, for everyone to want to go to these models and just use them to answer your questions. But the truth of the matter is that like they are not necessarily necessarily look up the guidelines and give you accurate answers, right? And this is where you need, like, either like you were talking about chain of thought prompting to try to improve the accuracy, and also this like human who's verifying that information. For me, like, I'm gonna go back to the guidelines and look up the answer because this makes me very nervous. This makes me think this is not reliable, this is not safe, this could cause patient harm. Right. And that's not a place where we want to be.

SPEAKER_01 24:12

Yeah, or a tool that is can can can verifiably show that its answer came from the guidelines, right? Can show explicitly where in the guidelines it is and show ex make the argument explicitly why it is the right answer, right? That's again, I mean, I know it's said don't think, but it should be able to do that anyways, right? That should be part of the mechanism by which it works, but it it is not. And the thing that makes these large language models really powerful is the statistical patterns. But, you know, we're sort of implicitly saying there is more to um answering these questions than just identifying a statistical pattern. Um, and there is, you know, the there are the patterns might be wrong because they change with the guidelines, or patterns might not be enough because they require some kind of hierarchical reasoning. Um, or a pattern might be saying something that you never actually meant to say, like that causing hallucinations. So these are all problems with a purely pattern-based system like these large language models.

SPEAKER_00 25:09

Yeah, I mean, I think that was very well said. And I think um moving forward, I think this is just adds to a little bit of a probably healthy dose of skepticism for using these models for very detailed clinical information. I think at this point, it's better to go back to the guidelines if you have questions about something complex like this to figure out how am I gonna treat the patient best. Um, because at the end of the day, you know, physicians are on the line for the care that they provide. And you can't be like, well, ChatGPT told me three centimeters, so that's all I resected. Right. So I gotta hold up.

SPEAKER_01 25:47

Yeah, exactly.

SPEAKER_00 25:49

So, all

Final Takeaways And Closing

SPEAKER_00 25:50

right. Well, I think we can uh leave it here, and we will see you next week on Code and Cure.

SPEAKER_01 25:54

Thank you for joining us.