Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 10 - Above All, Do No Harm
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Send us comments and questions.
Today we are speaking with Dr Rohan Shad of Stanford University Medical Center, now an Integrated Cardiothoracic Surgery Resident at the University of Pennsylvania. He has been exploring the application of AI techniques to help surgeons anticipate and reduce risks in certain heart surgeries. [2023-05-30]
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that in addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you. Our fantasies of how artificial intelligence might appear in medicine have been shaped by science fiction devices, like the medical tricorer wielded in Star Trek. This amazing device could sense everything in your body, even if you were a Klingon, and instantly diagnose what was wrong with you. Medical decisions are sufficiently weighty that one might imagine that they will be the very last subjects for the application of artificial intelligence techniques. In fact, the medical profession is quite advanced in the application of AI. So there is no resemblance to Dr. McCoy's tools. In this and in other episodes of Mind the Gap, Dialogues and Artificial Intelligence. We will speak at length with medical experts who are applying or studying the application of AI techniques in medicine. Today, we will be speaking with Dr. Rohan Schad of Stanford University Medical Center. He is a postdoctoral research fellow in cardiothoracic surgery. He has been exploring the application of AI techniques to help surgeons anticipate and reduce risks in some very complex surgeries.
SPEAKER_01Good morning, Dr. Schad. Could you tell us, please, a little bit about yourself?
SPEAKER_04Hi, yeah. My name is Rohan Schad. I'm a medical doctor. I'm currently a postdoctoral fellow in the Department of Cardiothoracic Surgery or heart surgery at Stanford University. And our work in the lab lies at the intersection of computer science and the life sciences. So we're always trying to find ways to innovatively integrate AI or physics-based simulations into how we treat or diagnose heart disease.
SPEAKER_01Could you tell us how you got interested in AI? And by the way, since you started working in software at a very early age, could you give us a sort of a trace from your interest in that software development to your interest in AI? Right.
SPEAKER_04Well, you know, I've been coding since I was 13. So I think it's one of those things which is a skill that finds use no matter what you want to do today in the medical field, whether it's clinical research, you know, whether you're doing statistics or randomized control trials. But for me, what really triggered my interest in AI was the fact that in the cardiovascular sciences, we rely on a lot of non-invasive imaging to determine if somebody has a disease or if somebody's going to do poorly in the future. And AI, therefore, was quite a natural choice in trying to quantify many of these things which humans huddle around on a screen and make sort of diagnoses by.
SPEAKER_01Now you referred to a kind of imaging. Could you explain what that imaging is? Is it like an X-ray? Is it like a CT scan? Just so uh we all are on the same page on that. Right.
SPEAKER_04So the vast majority of cardiovascular imaging is ultrasound-based. Um, well, it's an ultrasound of the heart in motion. So we called it an echocardiography. But there are obviously other imaging modalities as well, which are you know like CT scans for the chest. Uh, we have cardiac MRI, which is a time-resolved MRI of the heart. So you can see in these different imaging modalities um the heart beating and contracting uh as a video. Um, we do have some other imaging modalities which are more X-ray based, uh, in that you will shoot a dye through a catheter and there'll be an X-ray sort of video reader that will show how the dye is moving through the heart. Uh, that's what we call angiography. So whether somebody's you know considered for surgery, or somebody's come in presenting with chest pain and we're worried that they have a heart attack, or somebody's come in for routine follow-up after getting heart failure. Um, a lot of the diagnostic and prognostic process revolves around accurate imaging and accurate assessments from these imaging douses.
SPEAKER_01So um, how do you get AI applied to those images, whether they're a video from an uh echocardiogram recording uh or the other images you described, how does AI get trained for that and how do you use it? And really, I should ask those as separate questions. How does AI get trained to be useful to you, and then how do you use it?
SPEAKER_04So all AI systems, um, or at least the modern ones, which we will sort of talk about in terms of deep learning, you train a large corpus of medical imaging files. And the traditional approach is that you give it a target of either a disease or no disease. So let's say we were interested in somebody getting a heart attack, we would say heart attack and no heart attack, and we would train a neural network of a deep learning neural network to try and identify which of these scans that it's been fed um have disease and which of them don't. And the training process is usually very intensive from a clinical standpoint. It requires curating a very high-quality data set uh with the right kind of label. So each image and each video uh has to be looked at by a radiographer or someologist to make sure that it does actually contain the disease or it does actually show that somebody doesn't have a disease in definite terms. And then we train these algorithms, it requires usually a tremendous amount of computational hardware. Um, these training processes are very time-intensive. Uh, but once it's trained, you know, there's this whole other world of quality control and making sure it does what we think it does. Um, the second part of the question on how it's actually deployed in clinic, this is this is actually quite tricky. Because it's one thing to have a code base that works well for a research paper or you know, works well in a siloed sort of environment, but you know, the clinic is by definition not a siloed environment. We have all sorts of patients who come in with a variety of different diseases, and um, many of these diseases might not be seen by the AI system. So not only is there a requirement for clinical judgment in when you deploy these AI systems, but also the technical details of actually how it's deployed is also quite challenging. Healthcare systems have electronic health records, you know, and these digital sort of enterprises, but for the most part, it does feel like we're stuck in the 70s with these technologies. And AI doesn't fit very well within these hospital infrastructures. That is changing slowly, but I think it'll be a while before we see routine adoption of many AI tools within hospital workflows.
SPEAKER_01But when you say stuck in the 1970s, what do you mean by that?
SPEAKER_04Well, any hospital informatic system is is built on a very strong basic legacy code. And when you bring in a new sort of paradigm where you want imaging files to be streamed into an AI system continuously and be giving outputs, it's very challenging to do that when the system was designed as a point-and-click operation to retrieve data or type in clinical notes or you know, administer billing, for example. Uh, there have been some movements in that. Um, I think the companies like Google, which have been developing healthcare infrastructure sort of back ends, and there are many hospitals around the country that now are utilizing that. Uh, and those sort of backends are much more compatible, let's say, with prognostic AI.
SPEAKER_01Well, let me sort of summarize what I understand you'd have said to us. There'll be uh a huge corpus of uh images that the AI is trained on, and you have to be very careful when you transition from the research siloed use into the clinic and to make sure that the AI is going to look for and is trained to detect what you want it to look for. What sort of treatments with that background do you envision that AI might be able to improve or has been found to improve?
SPEAKER_04Well, I think some of the, if you if we look at the current literature in research on cardiovascular AI in specific, there's a lot of work that focuses on automating things that humans are good at doing. So if we are taking measurements from um the ultrasound of the heart, for example, if we're trying to quantify um what we call as ejection fraction, which basically is what percentage of the blood going into the heart is pushed out by the heart. It's it's a measure of the pumping capacity, basically. So things like this, you know, they're very um humans are very good at measuring this, but it helps in a health system standpoint, in a democratization of expertise standpoint, if you can develop tools that can make very accurate measurements without the need for trained staff to do it. So that's that's one sort of aspect. And then the other side of AI is looking at well, can you diagnose diseases? Can we diagnose heart failure? Can we diagnose things like amyloid? Um and that's where you know the value sort of shifts from automating things that humans are really good at doing to maybe detecting subclinical features that humans might have missed. And on the far end is then AI systems that try to predict disease states or outcomes that humans are today very bad at doing. So, can we use AI to detect some subliminal signs of disease that humans have really not picked up on? Or does AI look at an image in a high-dimensional space that humans simply can't perceive that is predictive of disease in some social thing? And there are unique challenges to each phase of this.
SPEAKER_01To pick up on that continuum that you've mapped out between using AI to diagnose a particular existing condition and to forecast one that might come into existence in which there may be some development. Uh, it would seem to me from having looked at some of your articles, there's an in-between state where you're trying to use AI to predict when a particular treatment might not be successful. And the case that you were talking about involved uh implanting a uh pumping device in the left ventricle. Could you kind of describe that research and how you used AI or think AI could be useful to recognizing when that treatment might not be successful in the few days following it?
SPEAKER_04Sure. So I think I'll give a little background to the field that my interest really lies in. So we in cardiac surgery do a number of heart transplants and a number of procedures called left ventricular cyst device implantations. Um these are treatments reserved for end-stage heart failure. What that basically means is that when somebody develops end-stage heart failure, their heart cannot pump blood to sustain their bodies anymore. And now they are in desperate need for either a heart transplant, which gets them a new heart, or they need some other mechanism to keep the blood moving. Now, we are capped in the United States by the availability of donor hearts. There are far more people waiting on the transplant list than there are available hearts to be transplanted. So in the early 2000s, the FDA approved a device which is called the left ventricular resist device, which is essentially a pump that takes blood from one of the chambers in the heart and pushes it around the body, basically doing the work of the heart as a mechanical sort of device. And these devices have gotten better and better uh in the decades that have come. However, one thing that we noticed in the clinic was that there was a certain percentage of patients who would develop failure of one side of the heart that isn't really supported and isn't usually a problem. Um, so the right side of the heart in this case. Um but the problem with right heart failure after an LVAT implantation is that the mortality risk is about two and a half or three times higher than those who don't develop it. And we were struggling to find ways to predict which of these patients that we implant these LVAT devices into would actually end up developing right heart failure after the operation. And there were many clinical risk scores and many sort of um you know papers that came out, but we really couldn't find a way to do this reliably. So we thought maybe we could use AI in the pre-operative phase, before somebody's taken into the operating room. We tried to take their imaging scans and we fed that into an AI system and to basically see if it could look at the heart and find some subliminal sort of signs of disease that may be predicted that 14 days from now this individual is going to get right heart failure. And the importance of that is because the treatment decision window really is in that phase. If we waited for 14 days and waited for this person to get right heart failure, by then the kidneys start shutting down, the liver starts shutting down. There are so many downstream effects that whatever treatment you try to institute becomes less and less effective. So to try and get into this treatment decision window, you really have to predict it before you take the patient into the operating room at day one. And this is sort of the research that uh we were talking about here, where we deployed an AI system, a video-based AI system, to take in scans of the ultrasound of the heart, predict what the probability is for this individual to get right heart failure, and then the dream is that we will use such a system to basically prophylactically treat these individuals for their right heart failure, preventing them from going into this downward spiral of disease.
SPEAKER_01And has the research yet concluded whether AI can provide that predicted capability, or you're just at the earlier stage of it?
SPEAKER_04So this paper is actually under review. We showed that we can predict it, and we can predict it actually better than a team of blinded clinicians. So what we did was we assembled a team of clinicians to look at these imaging scans and make their own predictions, and they were blinded to the outcome. And in parallel, we ran our AI system on the same images, and we found out that our AI system actually beat the human clinician team by a pretty large margin as well.
SPEAKER_01So this is a uh this is really interesting to me because I I've read that in reading uh radiographic images, AI is more accurate and can do it faster. But this is really uh uh sounds like a considerable advance. But one last question I'd like to ask in this area. From your description, it sounded like the right heart, the right side of the heart failing could occur, whether or not you implanted the device in the left ventricle. Did you do any looking to see whether the implantation itself caused it, meaning that you you put in this wonderful pump and it pumps so well that the right side of a diseased heart simply can't keep up with it?
SPEAKER_04That's actually a really interesting question. Uh, not something that I would expect from uh somebody who's actually not trained in this specific heart failure field. Because you're right, the the left side of the heart uh heart pumps, um, they can actually induce right heart failure. Um, but you know, this is the this is the problem with this disease space. All of these patients need an intervention, otherwise, they are looking at death. So either they need a transplant or they need a left ventricle resist device. But the left ventricular resist device, it seems, can put some strain on the right side of the heart. The problem is how much strain is it putting on, and two, how much strain can the right side of the heart in that patient tolerate? And these are questions which then sort of, you know, philosophically, you could then use to sort of talk about well, these two issues together might be able to predict who might be getting right heart further down the road. But in a clinical standpoint, it's almost impossible to tell how much specific strain your device is putting on that specific patient's heart, which is why I think AI comes in really handy, because it can really pick patterns that might be more challenging for a human to perceive.
SPEAKER_01Mark, why don't you take uh uh the next set of questions?
SPEAKER_03Okay, that's that's fascinating. Um, you've you've uh, as you uh outlined earlier, uh have a background in AI and in uh software in general, even though you're uh uh a doctor. Uh how does you think having a background in AI helps uh doctors use systems of this sort?
SPEAKER_04I think Richard Feynman really put it the best in me. Um, you know, if you can build it, you can understand it. So, you know, being able to construct these systems, being able to code these systems allows one to understand not only where these systems can do really well, but also many of the considerations you must take in ensuring you are accounting for when these systems could really go wrong. Um, and you know, as a clinician, my interest is in deployment in the clinical practice, ensuring that we can take this to the patients and improve the standard of care. But if you don't understand the limitations of your tools, you'll probably run into many problems down the road.
SPEAKER_03So um assuming these things become into clinical practice, uh if a patient then, you know, understands the use of AI and is uh uh skeptical about it uh and challenges your use of AI in treatment, what what do you say? How do you respond?
SPEAKER_04Well, I think before any AI system goes into human practice, um, you know, nowadays we're looking at regulations by the FDA, where many of these AI systems will be regulated as software as a medical device, where based on what the software will be used for, whether it's for diagnosing a very simple skin disorder, for example, uh, you know, the regulatory burden might be lower versus when you're doing something like trying to decide whether or not to intervene invasively for right heart failure. You know, this is a very invasive procedure, and if you're using AIA to guide this, the FDA might want to regulate it much more heavily. So part of it is the regulatory process will require clinical trials, just as any other device will or any drug will. So by the time it gets to a patient, there should be and ought to be a large amount of uh literature, prospective literature that shows it worked in a randomized trial setting. And the second part is well, you know, healthcare in general and the medical field, we've embraced new technologies. Um, I wouldn't say at the most rapid pace, but we have embraced new technologies as they've come. You know, so gone are the days when you would find doctors tapping away at somebody's chest and proclaiming a diagnosis. Today you will use uh echocardiography, you will use a CT scan, you will use these diverse different modalities. Um at the end of the day, it's these are all tools in the in the sort of alimentarium of a physician. Um, for patients who are concerned about the use of AI, I think one thing might be helpful is to know that all these AI systems, at least for the foreseeable future, will be used as tools by physicians rather than ways to replace physicians.
SPEAKER_01Dr. Shod, it's fascinating to listen to you on this because my first experience with surgeons uh decades ago when I had major. Surgery. They didn't want to tell you anything other than we're going to do this surgery. This will be the result. And if you ask them anything about what you'd feel post-op or what techniques they were using, they really felt the less you were told, the better. And for a patient to challenge the use of AI, you'd actually have to be telling them that AI is being used. Do you think as AI starts to, you know, is increasingly used, that patients will, in fact, as part of their informed consent uh procedure before surgery, will they be told, as they're told about anesthesia, you know, you're going to be given this and you have to sign off on it. Will they, do you think it'll be important to tell them that AI is being used as part of the treatment? Uh, or would that be kept in the background?
SPEAKER_04That's an interesting question. I mean, I think this is something that will evolve with time as the capabilities of these AI systems evolve. I think it will be very challenging to offer informed consent when the lay patient might not fully understand what AI is. You know, with anesthesia and with many surgical procedures, the informed consent is always to try and explain these systems or these procedures in ways that the patient can truly understand. Otherwise, it really isn't informed consent, is it? So with AI, as it gets more and more complicated, as it gets more and more entrenched into the system, there may be a day where it may be considered normal for AI to be deeply rooted in medical practice. But it's one of those things that I think will require constant um readjustments and revisits by both medical practitioners and legal practitioners alike.
SPEAKER_01Well, let me transition from that to sort of pull back the curtain to the kind of thing that you probably wouldn't want to include in informed consent, because it might make a patient either confused or so agitated they might decline to go forward. But it's important for our audience to talk about not only what AI can do, but some of its current limitations. And I want to emphasize the word current because as a caveat to almost anything we say about AI, we always want to say as yet, because AI is advancing so rapidly and much more rapidly than probably previous technologies have. In a recent article, you highlight what is currently a limitation in machine learning AI and how it reminds us that AI does not generate predictions by reasoning in the same way that humans do, even though the output may resemble it. You note, for example, that humans tend to err on the side of caution when unsure about a task, and I certainly agree with that. And you explain that by contrast, machine learning AI and radiographic imagery interpretation lacks, as yet, the capability to generate an output that would be the equivalent to admitting, I don't know, I can't figure it out. Could you explain a little more about what that limitation is and how it differentiates what AI does in its process from the way a human would reason their way to the same output?
SPEAKER_04Right. So this is this is actually something that we find a lot of in contemporary medical AI where you know the way these AI systems work, when you feed it an image or imaging scan, the output is a probability between zero and one. So if you've trained your AI system to try and pick up stroke, for example, differentiate between stroke and no stroke from a CT scan. Uh the output probability is a number between zero and one. So a number which is a 0.05 might mean that individual probably really doesn't have a stroke. Whereas a number output such as 0.97, that might mean the AI system thinks the individual does have a stroke. The challenge is that sometimes these systems will give you an output regardless of what the input is. So if you instead of a CT scan provided an image of a cat, the AI system still would give you a number between zero and one. It wouldn't tell you that it doesn't know what it's looking at. And this is extremely challenging because let's say this AI system was trained in this example just to predict stroke or diagnose stroke. The training data might have just had patients who are normal and had strokes. What if somebody comes into the emergency department with a fracture in the skull? The AI system would have no clue what that is because it's never seen it in its training data. Now there's there's more work being done in sort of uncertainty quantification. And it's something I also bring up in the paper. But the long and short of it is trying to basically tell a system to be okay with saying, I don't know, when it sees an input that seems far afield from what it was trained on. And you know, these sort of like uncertainty quantification systems are used now, I think, more as a research tool. Um, but I think we're seeing a transition now where clinical applications of AI that come out in these research papers, uh, they're starting to employ some of these methods. So as of yet, AI systems that we see in nature, nature communications, science, translational, medicine, all these big journals, they probably don't use uncertainty quantification as a core tenet. Moving forward, I think as AI systems, um, as authors of these sort of systems want to make AI that's more generalizable, uh, it'll be essential for these AI systems to be able to say that I don't know.
SPEAKER_01Well, as a doctor relying on an AI output, even if uh as a support or check on your work, does the fact that AI is compelled to give an answer even when it doesn't know it? And uh do doctors then have to be careful of over-relying on something that is giving an answer when it does not yet have the capability to say, I don't know, or I'm uncertainty. I'm uncertain about the response. Of course.
SPEAKER_04Um, there's actually some research that came up from Stanford. It was another group that works most mainly on chest x-rays. Um they actually did a study where they had clinicians of different training expertise and paired them up with AI systems. And they found that clinicians who were very unexperienced or very early in their training um actually did worse when AI was paired with them because they started relying so much on the AI to do the work for them. However, expert clinicians who were sort of at the top of the game, um were able to intuitively see where AI was going wrong and reason why it might be going wrong. So when you paired AI with clinicians in that context, um the end result was better. So I think it's interesting how automation bias plays into medical, uh medical sciences. Um but I think understanding how these systems are built uh will help reduce that. Um overrelying on AI systems just like over-relying on a chest x-ray uh is fraught with problems. A clinician who overrelies on his tools and without sort of contextualizing the clinical presentation is is bound to run into issues.
SPEAKER_01Now I'm gonna be a little bit of a devil's advocate here. I admire your insight. Humans tend to err on the side of caution when unsure about a task. But I can also recall experiences in law firms where young lawyers asked a question before they had had a chance to fully research it, felt compelled to give an answer, and didn't want to say, I don't know, because the response from somebody senior would not be confidence building. Um, and this happened often in the intelligence area, and one of their ways of dealing with that was to have answers come with an expressed level of confidence uh in the answer. Uh in the field of AI, is it possible to give a score from an AI where it's giving an answer of, and you said a scale of one to five, was that scale giving a level of confidence or was that giving a different kind of response?
SPEAKER_04Right. So one is the output probability. So the probability of having a disease. Now, in in sort of the way we would understand it traditionally, is a probability is a score of confidence, right? Um, but not in this case. In this case, it's just a number that says, well, it could be a disease or is not a disease. It doesn't tell you how confident it is in that uh in that it being a correct prediction.
SPEAKER_01Ah, which is exactly my point. Is there a way of adding a level of confidence to this method?
SPEAKER_04So the zero to one score is simply what it thinks is gonna happen. Is it healthy or is the individual have dis does the individual have disease? Um but you can be um you can give a highly probable output of let's say 0.98, uh, but be hilariously wrong. So how confident are you that the output that you've given is correct? That is a second score that um that is the sort of realm of uncertainty quantification and such. Um anecdotally, I think when DeepMind, Google DeepMind, when they worked on Alpha Fold, the protein folding uh AI system that that really sort of changed the game. Uh they too had a uncertainty score in build uh that would tell the research team how confident it was in the protein structure it had sort of predicted, so that the research team then was you know alerted into maybe we need to do some additional studies on this structure that it doesn't think is very correct.
SPEAKER_01So has that has the the procedure for expressing a level of confidence that was developed uh in that example, has that migrated or propagated into other fields of AI in your own work, or is that not really something that's transferable?
SPEAKER_04It is quite transferable, and um there's plenty of work, uh, the theoretical basis of this has been around for a while. Um I think deep learning practitioners or AI practitioners were initially we were sort of focused on trying to see if we could even predict disease or if we could even uh diagnose disease via an AI system that could beat a clinician. And it's only now I think that as the natural progression to maybe wanting to transition into a sort of live clinical setting, um, the requirement for these sort of these concepts of uncertainty, quantification, and such, these have become more acute. So I think there is an active transition towards that, even as we speak. Um, and I think we'll see that come out in the next few years in the papers that that come out.
SPEAKER_01Well, to go back to the example you gave earlier of the AI system that predicted with a higher degree of accuracy than uh clinicians uh whether there would be a failure of the right ventricle after the procedure. Has that procedure yet been uh supplemented with a level of confidence, or do you think that's an area where that would be a promising application?
SPEAKER_04That's an area that would be where it would be promising. It's there are some methods of doing this which are relatively trivial. Um, there are some methods which are much more involved from a technical standpoint, but I think all AI systems moving forward will need to have some sort of inbuilt metric where it can say how confident it is in its prediction. Um two outputs, not just the number from zero to one, but a second output that says how confident am I in the number that I spat out.
SPEAKER_01And do you think, as this gets applied in a clinical context, that you're going to want to design a machine learning AI system to alert physicians when they should not trust on rely on an AI prognosis for deciding on whether a particular therapeutic intervention is justified or good medicine? Uh, or do you want to rely instead on an experienced senior physician to alert younger staff that use this only up to a certain point, but don't over-rely on it?
SPEAKER_04That's a pretty challenging sort of question. You see, the advances that we've seen in healthcare over the past decades, for the most part, has been taking away the sort of responsibility on senior, knowledgeable, key opinion leaders to a data-driven, sort of randomized control, trial-based specialty. Right. Whenever we try and institute treatments now, we we call it evidence-based medicine where you know we want to see how many trials have been done, what was the effect of those trials, how randomized were they? It's not enough anymore to rely on the knowledge of one person who may be very good, you know, but where the reality is not everybody can be that good. So, how do we bring at a system level um an acceptable and uh high sort of quality of healthcare? How do we bring at a system level a uniform degree of quality? And that's by following the sort of evidence-based um mindset. I think what will end up happening is that as AI systems transition from the research lab into the clinical realm, there will be demands from both clinicians, from legal experts, from the companies trying to market them to um to show in randomized prospective control trials that the AI system can improve things. And short of that, I think it would be irresponsible for anybody to deploy an AI system that doesn't really help. So the question of whether this AI system will help in the clinical scenario, you know, whether you have junior doctors on service, whether you have senior doctors on service, it'll depend on how the trials go.
SPEAKER_01In other words, it might be limited to use by senior physicians because they would have the breadth of knowledge and experience to oversee it.
SPEAKER_04Or you have the AI system give an output that even junior physicians uh can be sort of trained on that, hey, when it gives you a confidence score of you know, two, don't trust it. If it gives you a confidence score of over five, you can trust it then.
SPEAKER_01Am I right in thinking that um younger clinicians who grew up as digital natives and relied on technology, may have started coding early as you did, um, may be more at risk of over-relying on an AI output, whereas a senior physician who has that longer time span and has seen transitions to other technologies with and discover later that there had been an over-reliance on that. Does it are the younger, am I right in thinking the younger clinicians may be more at risk of over-relying on it and have that technology bias that you mentioned earlier?
SPEAKER_04I don't really know. Um it's possible it could go both ways. If we, if we, you know, if we look at younger sort of teenagers today, they're very good at detecting, you know, which videos on social media are faked and which ones are real. They're very good at telling, telling a faked video versus a real one. That's something that people my father's age, for example, might might actually struggle with.
SPEAKER_01Um, am I right?
SPEAKER_04Yes.
SPEAKER_01Okay.
SPEAKER_04Um it's it's challenging because humans intuitively learn from patterns, right? So as we use these tools, it's possible that we might learn ourselves certain patterns that are indicative of failure modes, or certain patterns or or sort of clinical presentations where the AI system could be trusted a bit better. So it's very interesting because even the interplay between an AI system and a human is not static. A human learns intuitively from experience, and this dynamic sort of back and forth is quite interesting. On one hand, you can train the AI system more and more based on what it sees. On the other hand, the humans are training themselves subconsciously or consciously on how it to best deploy and use these systems.
SPEAKER_01Well, I want to correct a gender bias that crept into our conversation there. If I'm not mistaken, it's not only your father that's a surgeon. Isn't your mother also a physician? Or do I have that? Am I not recollecting?
SPEAKER_04Yes, my my mom is a physician, she's she's a radiologist, yeah.
SPEAKER_01Right. So let's make sure we get that balanced. Um Mark, yeah, there are questions that you'd like to ask now?
SPEAKER_03Absolutely. Um so um one of the interesting things about all this is that uh all of these new these new technologies uh require uh regulatory approval before they can actually go into clinical practice. Could you walk us through the process of validating the performance uh for AI-related medical devices? Sure.
SPEAKER_04So in the research setting, the aim is to show that an AI system can do what it's been trained to do on a data set that it hasn't seen before. So the way we set it up usually is we'll have a training set where let's say we have a thousand patients in our data set, we will maybe train the AI system on 800 patients. And then in the remaining 200, we split that further into two. We call it a validation data set, and then we call the the other one the holdout test data set. And the reason we do this is because when we're training the AI system on the training set, we want to continuously monitor how it's doing on data it hasn't really been trained on. So that's the validation set. Now, when we get an a level of performance that we are happy with, whatever that metric might be, we then use the testing set, and we just do this once, where whatever the the AI system, whatever it's it's trained on, it's finalized, we just run it on the test set to see how it performs. The reason it's so important to have that third data set, the data set that I'm calling the test set, is even when you are developing a system and you're tracking how it performs on this validation data set, there's still potential for us to overfit on that validation set. What I mean by that is when you're training your algorithms and you're sort of benchmarking it against this data set over and over and over again, you might just inadvertently pick the version of the model that performed really good on the validation set. That's that's a sort of human-introduced bias. And to account for that, if you have a holdout set where you never tested on it before, you've never seen the performance on that before, you just run it once. That will give you the sort of real world picture. So that's in the research setting. That's how we define whether the AI system works, the performance metrics that we publish, that you know, we get the accuracy of X or an area under curve of Y. That is all on the test data set. But then when we transition into clinical practice, when we want to take an AI system into the clinic, um the process looks a bit different because now not only are you feeding it in data sets that have been collected previously, you might have new patients streaming into your clinic, new patients coming into your operating room. Um, and there then needs to be levels of quality control. How do you decide when to use the AI system, when not to use the AI system? So that's sort of analogous to the, you know, in medical devices. When you're building a medical device in the lab, in the engineering unit, and then you take it to animal research and you see that it performs reasonably well, then you transition into maybe taking it to humans. So the regulatory hurdles and the regulatory sort of um overhead almost logarithmically increases as you start taking things to first achieve it. As we move forward from there, it looks more and more like a clinical trial, less like a sort of AI software development project, and much more like a trial. Of any other medical device.
SPEAKER_03So one of the interesting things about modern AI systems is having trained it, let's say, on a on a supervisory training set, one of the common things to do is to have it then evolve over time as it handles more and more cases, and you uh learn about how those worked out. Um, how would the regulatory folks deal with the fact that the underlying model may be changing as the clinical practice improves?
SPEAKER_04So here it's very important to differentiate the underlying model from the trained weights. Um, this is analogous to separating the brain from the mind. So the model architecture, uh, what we define as a deep learning system, might not change. But the weights, as in what the mind is learning, might change. Right. And um, there were some documents that the FDA released in part of its sort of software as a medical device AI regulatory uh framework, where there may be provisions where as long as the underlying model stays the same, as in if the brain stays the same, um, you can still update the weights based on whatever new information it's learned and not require such a tedious re-sort of regulatory process. Um, however, if you change the neural network architecture and you you bring in a completely different set of brains, say to address a program uh problem, uh that would probably invite fresh regulatory approvals. So I'm not fully sure whether these have been finalized yet. Uh these these these white papers were released, I think, a year and a half ago. Um but there is a sort of understanding within the regulatory bodies of how these AI systems work and how um the continuous evolving paradigm can be brought into the real world of sort of regulatory approvals.
SPEAKER_03As you encounter your colleagues uh in the profession more broadly, uh what kinds of comments are they making to you about the prospect of using AI in this way?
SPEAKER_04I mean, I think there's a lot of optimism. Um there's a lot of optimism that AI can maybe help um you know predict disease states much earlier than we can, and maybe we can institute treatments uh much more effectively. If you ask radiologists, um, you know, many of them are very optimistic that AI can streamline their workflows. Uh, anesthesiologists, there was a lot of optimism about how you know maybe AI-based sort of automated systems could offload some of the workloads there. But at the same time, there's a lot of skepticism because some of the recently approved um AI systems that are in the market today, people are learning the hard way, right? Um, not by software engineering training, but by seeing it fail in the clinic that AI systems are not infallible. So, I mean, to summarize, it's I think people are very optimistic, but the optimism is I think slowly people are realizing that there are limitations of this sort of technology.
SPEAKER_01When you talked about learning the hard way, you suddenly reminded me of something, and maybe this is an ignorance bias on my part. But when new surgical techniques are introduced, my understanding as a layman is that the surgeons often try these on mammals that are not humans, maybe a sheep, for example. Um has AI progressed through use on other clinical animals before it's been used on humans? Is there are there certain uses of AI where that is going to be the procedure, or am I misanalogizing here?
SPEAKER_04Um I think the analogy is somewhat incorrect here, because with surgical techniques, um there's no retrospective data set to play with. With surgical techniques, you know, when when the new cardiac surgery procedures were being invented in the 60s and 70s, they did not have a large data set that they just had lying around, which they could sort of test things out on. The only way you can really test new surgical techniques is by doing it. Doing it in in mammals was, you know, the the way they sort of develop these different procedures. Um with AI, the difference, the key difference is that you can train on data that's already been collected. So before you move into a setting which is prospective, where you are sort of prospectively deploying it on new patients, you can take imaging scans or clinical health data from thousands of patients in the past, and then see how your AI system behaves on that.
SPEAKER_01As you look ahead to the future of AI in your field, could you share your thoughts with us on what challenges will need to be addressed? Uh and I'm thinking uh, as examples, bias in data, the inherent uncertainty in predictions, and of course, what we just were discussing, the regulatory validation and approval process.
SPEAKER_04So I think there are multiple sort of um, let's say, phases to this question as well, because there's the software development phase, but there are many challenges, such as uh, you know, do we have enough data? How do we ensure that we can secure high-quality multi-center data under the purview of the you know the HIPAA act? Like, how do we ensure that we can share, you know, hundreds of thousands of patients' worth of data, train on it in a secure fashion? So access to data is a big thing. The second is, of course, as you mentioned, bias. How do we ensure that our algorithms are not subconsciously biasing towards one population or another? This is a very large area of research right now. As we're learning again the hard way, that AI systems that would that were designed to do something, you know, let's say maybe predict disease or diagnose disease, aren't doing it the way we think they are. They might be doing it by biasing against certain demographic data it's picked up. As an example, if an AI system is predicting that women um have less heart disease than men, uh when it sees a scan that comes from a woman, inadvertently it might just say this person does not have heart disease, even when this individual does, simply because the background sort of demographic suggests that women are less likely to develop disease. Of course, these have many ramifications down the road. Um, and many tech companies are seeing this as we speak, where AI systems that they deployed for a certain feature they find is biased towards a certain race or gender or income group. There are algorithmic ways of dealing with this. Um just as we sort of account for many variables when we do clinical research. Um, there are many ways you can somewhat address these biases, but it really does require a lot of, I think, prospective um sanity checking and validation to make sure that people who understand how AI systems can be biased are constantly monitoring the situation.
SPEAKER_01Well, given the way you've addressed those challenges, I want to give you an opportunity as we wrap up to sort of look ahead. Um with new technologies, there's a tendency at its introduction to overestimate in the short run what it can do and underestimate in the long run what it actually will eventually be improved to do. Could you give us your view on how you think AI at present may be uh described in overpromising ways in the short run? But in the long run, where you think it may be uh enabling medicine to go that people at this time may be underestimated?
SPEAKER_04That's a very difficult question. So I think the early sort of experiences with with people who are naive to how AI works, if you ask them how what they hope to see from AI, um what they hope to see is not feasible for today's AI. It is, you know, clinicians, if you ask them today, they want something which can predict certain things in the future which cannot be done. Um, you know, they they might want to to predict certain disease states with 100% accuracy, which again might not be feasible. And uh the perception as AI as this sort of almost supernatural intelligence um is something that comes from individuals who may not be fully aware of how these AI systems are built. In the end, it's just code. Um the long run question is much more challenging. I think uh I would be um at risk of wronging myself by making uh such predictions that inevitably will be wrong.
SPEAKER_01Uh I I don't want to invite you to do that.
SPEAKER_04Let's uh go ahead. I think what we will find is that AI will be a greater and more subliminal presence in in the medical field. You know, our cell phones today have so much AI baked into them. When you are searching something on Google, you know, there's autocomplete there. If you are autocorrecting something, a message that you're sending to a friend, there's AI involved there. You're clicking a picture, there's AI involved in sharpening certain regions of the photo versus not. And these are very subliminal. You're not actively asking AI to do things for you, uh, it just happens in the background. So I think this will be the if we if we take this analogy uh as something that's happened in one industry, we might see something similar happen in the healthcare industry uh given a few decades. But uh anything more specific than that, I think I I would be um I'd be uncomfortable with making such predictions.
SPEAKER_01Yeah, but you know, you've prompted one another thought. I'm gonna indulge in asking you one additional question. Um, because of your description of the presence of AI in our cell phones and other places where we're not aware of it, the ordinary user is not aware that it's there in the background. Um it's my understanding that AI um applications go back decades. For example, in the military field and aeronautics, we've had autopilot uh for decades, and that was an AI application, early signal processing. Um and in the sort of consumer area, as I would call when uh tax software preparation programs first came out, they were described and marketed as artificial intelligence. And then as they became more widely adopted and used, they were renamed as uh tax preparation software. And it has seemed to me that frequently an application of AI, when it first comes out, we marvel at it and we call it AI, but as we find it becomes familiar and uh ordinary, we start labeling it as software. Do you see that happening also in medicine?
SPEAKER_04I think what's happened now is that you know the key distinction that I find between what used to be called AI at different stages in the past and what is called AI now is that the AI systems in the past or whatever systems in the past were based on heuristic rules. Somebody had to actively say, if this happens, do this. So if you see X income in your tax preparation software, maybe put this income tax bracket or something of that sort. Whereas now the what we call AI is uh systems that are able to learn from the data it's given and update itself and give you a new output based on whatever data it's been fed in. So this ability to learn, um, where it changes the weights of the model uh on the fly, that's what I think really differentiates what we now know as AI versus what previously was marketed as AI. And of course, you know, as things move forward, maybe there'll be some other um sort of variant where what we call as the models that we might think are a constant tomorrow, might be flexible and constantly evolving. You know, maybe AI will be that, and today what we're doing is just deep learning. Um very difficult to predict the march of progress in this field. Um, what we thought were uh ground realities, you know, just two years ago have been debunked already.
SPEAKER_03What one amusing little uh side uh story. When I was a graduate student back in the 1980s, uh studying AI, uh it was a common a common comment among the the uh computer science community that uh things that were uh hard to understand and uncertain in their uh framework were called AI. And then once they settled down, uh they were no longer called AI. So compilers and text editors and all kinds of things that we think of today as very generic tools and utilities uh were AI early in their uh in their genesis. So uh I think we can expect that uh as things mature, we will we will we will demote them from the category of AI to simple tools.
SPEAKER_04Right. I mean, I think from a legal standpoint, maybe it might be worth using the AI term as more viewing the AI term as more of a um marketing sort of tool, and uh instead of focusing more on what the actual code base does, whether it's a vision network, whether it's a natural language processing system.
SPEAKER_03Aspirational rather than uh Exactly.
SPEAKER_01Well, I can looking forward to some of our podcasts on the application of law, I can tell you now that in recent executive orders and in federal regulations applying to intellectual property, there is already perceived to be some significant gaps between regulations that apply to software, but don't specify AI. And the fact that AI applications may start out as AI and be called software later is going to be a challenge for the drafters of those regulations. Uh, Dr. Schott, I want to tell you that this was a fascinating discussion. Uh you're you're amazingly articulate, very responsive to questions, and it was a privilege and honor for us to have you on the podcast. Mark, did you want to add anything?
SPEAKER_03Thank you. This was uh really a great conversation and I enjoyed it tremendously.
SPEAKER_04Thank you very much.
SPEAKER_01We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention the specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.
SPEAKER_00Thank you for listening to the AVA Business Law Sections podcast series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit anbar.org slash disloc. That's P I D L A W.