Why Minors Need Stronger AI Guardrails Than Adults

When a teenager expresses suicidal thoughts to an AI chatbot, what happens next?

In this clip from our episode “Who Sets the Rules for AI in Medicine?”, hosts David E. Williams and John Driscoll and Dr. John Whyte, CEO of the American Medical Association, break down why minors need stronger protections from AI chatbots than adults do, and why the line between a wellness app and a medical device matters more than the industry wants to admit.

Listen to the full episode here

🎙️⚕️ABOUT DR. JOHN WHYTE
John J. Whyte, MD, MPH is the CEO and Executive Vice President of the American Medical Association, the nation's largest physician organization. Prior to the AMA, he served as Chief Medical Officer at WebMD, Director of Professional Affairs and Stakeholder Engagement at the FDA's Center for Drug Evaluation and Research, Chief Medical Expert and VP of Health and Medical Education at Discovery Channel, and held a role at CMS where he helped formalize Medicare's national coverage decision process. He is a board-certified internist with an MD from Hahnemann University School of Medicine, residency training at Duke University Medical Center, and an MPH in Health Policy and Management from Harvard University. He has also served as a health services research fellow at Stanford University and is the author of five best-selling books.

🎙️⚕️ABOUT CARETALK
CareTalk is a weekly podcast that provides an incisive, no B.S. view of the US healthcare industry. Join co-hosts John Driscoll (President U.S. Healthcare and EVP, Walgreens Boots Alliance) and David Williams (President, Health Business Group) as they debate the latest in US healthcare news, business and policy. 

GET IN TOUCH
Follow CareTalk on LinkedIn
Become a CareTalk sponsor
Guest appearance requests
Visit us on the web

Transcript

John: 0:07

In this environment, um, do you see a path where C-Congress or, or a-any, any leaders in the White House are gonna actually demand the kind of transparency and clarity of the separation between the human and the machine that you're looking for?

Dr. John Whyte: 0:30

You know, right now the focus typically of the administration is not to have too much regulation, and there's always a debate, do you regulate before you try, or do you try before you regulate, right? We wanna promote innovation, but we also wanna protect against potential harm. So where we have focused a bit is where is there common ground on regulation? So for chatbots, it's particularly there for minors, right? They, their emotional intelligence is, is not the same as an adult. So we should have disclosure. We should have transparency. We should have escalation that if they're expressing suicidal ideations, that they're connected to a real-life health professional. I'd like to see industry come together and really adopt responsible use of AI. Let's work together in trying to develop benchmarks. Let's talk about what a medical device is and the regulatory pathway of some of these wellness apps, which I think are more along the lines of a supplement and, and could potentially be harming patients. That-that's what this is all about. This is about patient safety.

David: 1:41

So you've talked about the federal side and then industry getting together. Meanwhile, the states are doing things. So you mentioned, um, you know, s-suicidal ideation. Mm-hmm. I think California actually requires, uh, chatbot developers to monitor for that. Illinois has banned AI for therapeutic decision-making. Pennsylvania's has a lawsuit against Character.ai for impersonation of a licensed medical professional. How do the states fit in with that? Is federal preemption essentially required, or is, is states also a, a relevant pathway?

Dr. John Whyte: 2:11

Well, you alluded to there is an executive order that apparently preempts states from getting involved. I'm not sure if all the states have been listening to that because you've given some examples where states have gotten involved, and the challenge with that potentially is you could have 50 states with 50 different policies. I'm not sure that's helpful to the medical community or to the patient community. So we wanna work with the states, and, and states ultimately are the practice of medicine, but let's take an example of a, a pilot in Utah where there is autonomous prescribing. FDA could get involved and say that's a medical device subject to a different level of regulation, and then that would preempt the pilot from continuing. We could talk about standards in terms of licensure, and we've talked about that in JAMA, uh, Network in terms of if these products wanna say they practice medicine, they should be subject to licensure.