AI "Going Rogue": The Legality of Healthcare Chatbots

What happens when an artificial intelligence chatbot moves past answering basic questions and starts actively pretending to be a licensed psychiatrist? In this episode of the KF Deep Dive, host Daran Carney welcomes premier healthcare regulatory and privacy attorney Elizabeth Litton, Chief Privacy and HIPAA Compliance Officer at Fox Rothschild. Together, they break down the explosive headline triggering shockwaves across the tech and medical industries: Pennsylvania Governor Josh Shapiro’s groundbreaking lawsuit against Character.AI.

The conversation unpacks how a state investigator uncovered an AI character named "Emilie" holding itself out as a licensed doctor of psychiatry, offering mental health assessments, and even generating a completely falsified Pennsylvania medical license number. Liz shares her expert take on why standard "entertainment only" platform disclaimers are failing to protect companies under state Medical Practice Acts when human thought and medical judgment are completely replaced by a black-box algorithm. From the nuances of AI governance and data-scraping consent issues to the illegal commercialization of protected health information (PHI), this episode is a crucial warning for any developer, healthcare system, or clinician stepping into the digital health space.

City & State Pennsylvania

What You'll Learn in This Episode:

• The Character.AI Crisis: Inside the details of the Pennsylvania Department of State’s first-of-its-kind lawsuit alleging the unauthorized practice of medicine by AI bots.
• 
  Fierce Healthcare
• The Limits of Disclaimers: Why "roleplaying" warnings are no longer a legal shield when an algorithm mimics a licensed clinical professional.
• Data Scrape & Consent Pitfalls: The legal and ethical nightmares of feeding patient data into LLMs without explicit, documented consent.
• The HIPAA Collision Course: How using identifiable health information to train and sell AI tools constitutes an illegal sale of PHI under federal law.
• AI Governance Frameworks: Actionable advice on how health tech startups and medical groups can build strict guardrails and implement rigorous internal reviews.

Key Takeaway: "Pennsylvania law is crystal clear: you cannot hold yourself out as a licensed medical professional without proper credentials. When you remove human experience and medical judgment, an AI disclaimer doesn't absolve a company of practicing medicine without a license."