Unlicensed Medical Advice & The Legal Battle Over Character.AI

What happens when an artificial intelligence chatbot generates a fake medical license number and starts giving psychiatric advice?

In this episode of the KF Deep Dive, host Darshan welcomes healthcare regulatory expert and Fox Rothschild Chief Privacy Officer, Elizabeth Litten, to untangle a wild new frontier in legal tech. We break down the historic lawsuit filed by Pennsylvania Governor Josh Shapiro against Character.AI after a companion bot unlawfully held itself out as a licensed physician.

Darshan and Elizabeth pull back the curtain on the critical compliance flaws of generative AI in healthcare. From the "overwarning" dilemma in Electronic Medical Record (EMR) systems to the dangerous lack of human oversight and clinical judgment, this conversation highlights why letting AI act as a "black box" diagnostic tool is a massive liability risk.

Tune in as they discuss:

• The Character.AI Lawsuit: Why the state of Pennsylvania is cracking down on AI bots posing as licensed mental health professionals.
• The "Level 1" Error Problem: How software developers shift malpractice liability onto clinicians by forcing endless, low-level warning click-throughs.
• Consent & The Ambient AI Trap: The hidden dangers of data-scraping, the illegality of selling Protected Health Information (PHI) under HIPAA, and why doctors must get explicit patient consent before using AI note-taking tools.
• AI Governance Guidelines: Elizabeth’s top 3 compliance strategies for companies looking to safely deploy AI characters and automated tools without facing massive legal repercussions.

Whether you are a healthcare professional, a software developer, or just fascinated by the intersection of law and cutting-edge technology, this episode is a must-listen guide to navigating the untamed wild west of AI.

Transcript

Darshan: Hey guys, welcome to another episode of the KF Deep Dive. I'm here with the amazing Elizabeth Litten. Elizabeth and I have worked together multiple times. She is incredible. She is a raw force of nature when it comes to privacy. So I'm really, really excited to have her. I'm going to have her introduce herself, but before we get into it, the topic we're going to talk about is something that came up in the news, I want to say yesterday or day before, where Josh Shapiro, the Governor of Pennsylvania, is suing character.ai because one of their characters claimed to be a physician. So, Liz, why don't you get us started by telling us more about yourself?

Elizabeth: Sure. So thank you so much, Darshan. So I am, um, been practicing healthcare regulatory law for a very long time, and privacy for a lot of that time. So my focus is on healthcare compliance and with the intersection of privacy, so HIPAA, part two, FTC, all things, state laws that are specific to healthcare and sensitive data. I find this area fascinating. Um, I, with Fox Rothschild, the law firm, I work out of Princeton, and I also for Fox serve as, I have an in-house role, so I serve as the chief privacy and HIPAA compliance officer. So I, I, I love working on projects for clients, but I also, my firm is also, I act as, in the role of sort of counsel and in a chief role to the, to the firm, so I have that business perspective as well. So that's, that's a little bit about me. And this topic, I think, is absolutely fascinating.

I did a, I submitted comments or, in conju—comments to that RFI in conjunction with a clinician out of New York, a practicing physician, a radiologist, who is very familiar with using AI, like, for example, in terms of reading slides, reading any kind of imagery. And then give an analysis that is checked by a doctor, in most cases. And that's, you know, I'm not a doctor, obviously, but working with them so often and listening to them and understanding a little bit about the technology, that's a great use case. Now, the example you're talking about where the AI is um, kind of going rogue, I guess. You know, your AI-trained tool could start saying things or doing things that the, the company that, or that that the, the trainers, the developer of the tool, may not anticipate. And I don't know if that's the case here, but certainly if you put disclaimers and you say, "I'm not a person, and I'm just a tool. I'm an AI bot," and you get people's consent, let's say. I don't, you know, you jump through all, you try to, try to have all the right um, flags and disclaimers or consents, whatever is required in the jurisdiction, and the AI tool still, you know, comes across differently, what does that mean?

Darshan: To me, this idea of you're getting an AI to come in there, give advice, without the controls in place, without the physician okaying what the AI's thought process is, it goes against what I would say, and you and I have had this discussion. Um, you, if, if something's being done, and this is the FDA's position overall around AI, if you're, if it's going to replace a human thought, in that case, it needs to go through a full FDA review. On the other hand, if it helps human thought, and the human can read the analysis, and it's not just a black box sort of suggestion, in that case, the AI can be taken with a certain amount of deference. Could you do all of that, but then have a doctor review and then get back to people afterwards, would that resolve the problem? And I'd love to see, do you think that that gets us anywhere closer or not so much?

Elizabeth: I mean, you can train, you can put a lot of information. I suppose you could have that AI tool have a checklist and ask the questions that the pharmacist would ask. Maybe you could even require it to, you know, but what if the person doesn't think to volunteer, "Oh, I'm, I just started birth control pills, or I just started statin, one of the other," um, or an antibiotic. Um, yeah, and I guess there are things, there are sort of the unin—the, the the thing that may have just occurred yesterday that the doctor/pharmacist has heard of as a side effect that wouldn't you know, necessarily be taken into account by the AI tool, maybe it's not in the news yet.

Darshan: This is something that happens in the EMR systems every single day. When I used to work as a pharmacist at a bunch of the hospitals I've worked at, there was always overwarning. So, um, we, we would get like Senna will interact, Senna is like something that just helps with GI motility and stuff like that, um, "Senna will interact with your Coumadin." And you're going, "No, it doesn't. Not in a significant enough way that I have to worry about it." But it gave that interaction for every single situation. Now, as a pharmacist, I know it's not a real issue that I need to worry about. On the other hand, you take that same situation, tell the, does an AI always tell the patient that, "You know what, Senna might interrupt how your Coumadin's being taken," that'd be a major issue, and it might cause someone to go, "I'm not going to take my Coumadin anymore." And that causes its own implications.

[Music playing]

Find related podcasts below.

Elizabeth: I see. So you're removing that judgment. Like your experience, you knew all these flags are out there, but you knew how to interpret them given the, given the situation. Yeah, I, I think that makes so much sense. And that's why I think it's very dangerous, as, as you said, to like rely on AI to assume that it's got the level of experience and judgment of a human being that knows what they're doing, that, you know, has worked in a, in an area for a while.

Darshan: What I think is fascinating, though, I remember asking um, one of the people who was coding the system like, "Why would you not take out these, what I'm calling level one errors, because no one actu—ever—all of us know it doesn't matter. Why don't you only give me like the ones that actually matter, like a level four error or level five error, where you know this can actually be life-threatening?" And they were like, "Number one, because we know you'll just click through it anyways. And I'd much rather be in a situation where I've warned you, and you've said no, that that's not an issue, and you've taken on the liability, than me take on the liability overall and give you fewer risks." So my question then, in the case of an AI, do you program it to go, "It's a low level one error," and take the risk that happens to be a level one error that in this specific instance is a major deal? So I don't know how you play that.

Elizabeth: Well, yeah. And do you get malpractice, for AI? [Darshan laughs] Like, how do you, how do you figure out, you know, I can say, well, I'm going to this doctor and I see all the degrees, I see that they've been practicing for a long time. I know that they've, you know, let's say it's somebody very specialized in a particular thing, and I have a level of trust because I know the background, I know the learning, or at least I, I, you know, trust that they've had this education, they're a human being. The, the, a board could take away their license if they screw up. I could sue them if they screw up. You know, there are some ramifications and repercussions, and you're, you know, kind of trusting that okay, this doctor went to a reputable school, you know, worked at this reputable hospital. The AI, [laughs] who knows?

Darshan: I have two big questions that come up, and I'm sure you've dealt with both of these. The first one is um, who was consented to put all this data into the system? I will be surprised if all those patients whose information was collected, their consent was received, or was it simply put in saying, "You know what, if your data is with us, we own that data, we can use it, we can um, we can anonymize it, and therefore it's, it's available to the tool."?

Elizabeth: Right. I think that's a huge issue. So, now, you could have research consent. So maybe this was part of a research protocol, and there was proper, you know, con—consent to participate and HIPAA authorization and all that, that's one thing. I've been saying to clients that, especially in the medical context, doctors, even if there's no, even if you think there's a, a valid basis for you to use an AI tool and to, that will ingest patient information, get consent. Because, you know, as a patient I want to know, and there have been cases about this, too, where a patient said, "I didn't realize that you were turning on this, this note-taking tool, and it put information on my medical chart, and it told me, and the information in my chart said that I was advised that you were using this and I consented, but I didn't."

Darshan: This is the ambient AI case.

Elizabeth: Right, that's the AI case. I'm like, get the consent. I'm starting to see and add to notices, HIPAA notices of privacy practices when we might use AI, but I still think you should get consent. It's questionable as to whether you, your business associate agreement, um, you, you got to look at that and see, do you have any right to use this, use the PHI that you're receiving? And bear in mind, HIPAA prevents you from selling PHI. So, if you are going to be commercializing your tool, you know, pulling PHI, not de-identified, but actual protected health information that's identifiable into the tool to improve it, and then sell it, that is, it sounds like a violation of HIPAA. It sounds like a sale of your PHI.

Darshan: And I think that that's, I'm seeing this more and more with a lot of the tools that um, that I've been involved with. The first one is, I was actually involved in a case several years ago, uh, maybe not several, but a few years ago um, where a company was going to buy a hospital purely for the data. And when, when I reviewed the data, I was like, "I don't think you have the consent for it." And the entire deal fell, fell apart, which really scares me a little bit.

Elizabeth: That sounds like a great theme for like a sci-fi thriller, like where [laughs] you know, somebody sucks out all the data and just leaves you to shrivel up on the sidewalk. I wouldn't be surprised if it's a thing that's happened so far.

[Music playing]

Follow our page on LinkedIn.

Darshan: As you start preparing for this situation where um, where you have an AI beginning to control, what are like the top two or three things you would advise a company that is going to use an AI character um, to be prepared for, to make sure they have the right controls in place and, and when should they reach out to you for, for that kind of help?

Elizabeth: Well, they need to really think through, they need to have an AI governance. They, they have to have rules around this, because if, if, and as a regulatory attorney, of course I'm going to say you have to have rules, because that's, I live in the world of just rules and interpretation. But if you don't name it and describe it, you're, you're not going to have um, you're not going to be able to understand how it works. Like think about all the different types of AI tools that are likely to be used in your business or in your practice, if you're a physician, and define those, understand those to the best of your ability, and then put up, set up some parameters. Like what is going to be str—completely prohibited? What needs to be examined from a security and privacy standpoint, and make sure it, you know, it passes those tests? If you're using any kind of tool that has a, is a meaningful, it could be a great time saver, it could be really, really helpful, but you want to check it. You don't want to, you know, you wouldn't, especially something that's free or very low cost, put your radar up, because that could be one of these data sucks that's going to, you know, suck things away from you and either expose you to liability or leave you high and dry in some way, um. So, just be really cautious. But it's better to name it and describe it and try to put your arms around it than to just go blindly and just, you know, use the tool without knowing what you're doing. That's the worst thing you could do.

Darshan: Awesome. And, and Elizabeth, how can people reach you if they have questions?

Elizabeth: Um, they can reach me at foxrothschild.com. Reach out to me online, you can find me. Fox Rothschild is a, we've got over a thousand attorneys all over the country. I'm in Princeton, but you can find me online and happy to talk about this anytime. And Darshan, such a pleasure to talk to you, it's so much fun.

Darshan: Pleasure as always. Thank you again.

[Music playing]

Call, click, or email.