Episode 26 - AI Only Fakes Empathy

In 2023 New York Times journalist Kevin Roose reported that a chatbot had declared love for him and urged him to divorce his wife.  Since then stories abound of vulnerable people harming themselves after lengthy exchanges with GenAI chatbots.  In a recent instance, a vulnerable teen discussed suicide with a chatbot and asked for feedback about the noose he had fashioned.  In yet another instance a clearly delusional person was encouraged to murder his mother and then commit suicide.

Medical professionals are concerned that use of chatbots in diagnosis and treatment recommendations without real-time supervision by experienced professionals may lead to harm.  GenAI tools have not been designed to fulfill the Hippocratic oath to do no harm.  Physicians are asking whether these GenAI tools can be and will be used responsibly.  Currently the FDA categorizes chatbot systems as self-help or wellness tools, placing them outside of existing regulatory regimes.

In this episode of Exploring AI Matters we discuss the implications of GenAI tools with Dr Jane Rosenthal, a seasoned clinician with extensive experience examining medical ethics in the context of a major medical center.

Transcript

SPEAKER_04 0:03

Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that in addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.

SPEAKER_03 0:41

In several earlier episodes with Drs. Rohan Shaw, Stan Turin, and Yulia Meyerley, we explored how AI, writ broadly, has been put to beneficial use in clinical situations, particularly in planning cardiac surgery, stroke diagnosis, and performing colonoscopies. Increasingly, people are asking medical questions of chatbots and acting on the bot's output. As with most technologies, generative AI systems, otherwise known as chatbots, can be used to do good or to cause harm. However, if you are not a medical professional, you might not pose a question to the chatbot with sufficient context. For instance, quote, I have a pain in my shoulder, unquote, might be a sign of cardiac disease or a strained muscle or a lesion in the arm. And a person may misinterpret the chatbot output and act inappropriately. Having chat GPT on your phone does not make you a qualified surgeon, physician's assistant, or nurse practitioner, nor does it put you in touch with one. We tend to associate output in our native language that sounds human with sentience, consciousness, and even empathy. And we associate seeming to be human with all sorts of emotional connections that computer systems might imitate, but certainly do not have. In 2023, a New York Times journalist reported that a chatbot had declared love for him and urged him to divorce his wife. Stories abound of vulnerable people harming themselves after lengthy exchanges with Gen AI chatbots. In a recent case, a vulnerable teen discussed suicide with a chatbot and asked for feedback about the noose he had fashioned. In yet another case, a clearly delusional person was encouraged to murder his mother and then commit suicide. Medical leaders are concerned that use of chatbots in diagnosis and recommendation of treatment without real-time supervision by experienced professionals may lead to harm. Other appearances of Gen AI in the medical space are causing alarm bells to ring across the community. Physicians are asking whether these Gen AI tools can be and will be used responsibly. Gen AI tools have not been designed to fulfill the Hippocratic oath to do no harm. Regrettably, chatbot systems are classed as self-help or wellness tools. As such, they fall outside of existing regulatory regimes. Today we will discuss the ethical implications of Gen AI tools with a seasoned clinician who has extensive experience thinking about medical ethics in the context of a major medical center. Welcome to Mind the Gap, Dialogues on Artificial Intelligence. I am Mark Donner, a computer scientist.

SPEAKER_01 4:02

And I'm Alma Adams, a national security lawyer. We are your hosts for this episode of Mind the Gap: Dialogues on Artificial Intelligence. In addition, we have two more hosts.

SPEAKER_04 4:13

Hello, I'm Charles Palmer, a computer scientist.

SPEAKER_05 4:16

And I'm Roland Trope, a national security lawyer.

SPEAKER_03 4:20

Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. Dr. Jane Rosenthal, our guest today, completed medical school at George Washington University and interned at Beth Israel Medical Center. She completed her residency in psychiatry, a fellowship in consultation liaison psychiatry and her psychoanalytic training at Columbia University Irving Medical Center. She later earned certifications in bioethics and medical humanities. Over the years, Dr. Rosenthal has served as the psychiatric consultant to the Columbia Fertility Service. She has also held leadership roles at the Columbia Psychoanalytic Center and at Tisch Hospital, NYU Langone Medical Center, where she founded and directed their first consultation psychiatry service. She later directed the psychiatry service at the Pearl Mutter Cancer Center, NYU Langone. She has been a member of the TISH Hospital Ethics Committee since 2008 and served as its chair for three years. Over the years, she has taught and supervised medical students, residents, and fellows.

SPEAKER_01 5:41

Jane, welcome again to Mind the Gap. We are really glad to have you here with us this afternoon. As Mark mentioned, you spent many years not only in clinical psychiatry, but also serving on the Tisch Hospital NYU Langone Ethics Committee. So for our listeners who may not be familiar with how these committees work, could you start off by explaining what exactly is a hospital ethics committee and what does it do in the day-to-day? Sure.

SPEAKER_02 6:08

First of all, thank you very much for inviting me to come and speak with you. It's been really interesting so far, and I look forward to this discussion very much. So an ethics committee is a committee that's comprised of wide swath across the hospital. So it includes physicians, nurses, social workers, pharmacists, legal, you know, from all the various departments. So it's very well represented amongst ourselves. And basically we meet monthly, and there's a topic at those monthly meetings, which generally have to do with either educating ourselves regarding a topic of interest or reviewing hospital policy that's been put out and we're discussing, or to discuss clinical cases that have arisen in the time since we last met.

SPEAKER_01 7:03

And why do medical institutions have ethics committee? Was there a particular moment or set of circumstances or instances that made the medical community recognize or come to a position that these types of frameworks or structures were appropriate or necessary?

SPEAKER_02 7:18

Yeah, well, there were a number of high-profile cases having to do with end of life and also some research disasters, which also became an issue to be taken up by ethics committees. So about in the 60s, hospitals started to develop these committees. And you'll recognize some of the names of the high-profile cases. They had included Karen Quinlan, Nancy Cruzan, Terry Shivo. And these particular cases had to do with young women who were in vegetative states and their families wanted to take them off of life support because there was no treatment and their ongoing life was futile. Now you have to remember, at the time, the kind of intellectual basis of ethics was grounded in paternalism. So physicians were often making decisions about what happened to the patient and what the next steps would be. But at the time, in the body of ethics, there was a swing toward autonomy. So these cases represented really critical examples of the change in what was happening in ethics in general and really forced hospitals to begin to pay attention. The research case that became prominent was an experiment that took place between 1932 and 1972. So for a long period of time. And the experiment was to watch principally black men who we knew were infected with syphilis and to not give them treatment in order to see what the natural history of the illness would be from the time that they were diagnosed until their death. So these were the cases that, you know, created a stir and really forced hospitals to look very, very carefully at creating ethics committees, sometimes depending on hospital, that, well, let me back up for a second. So when there were cases in the hospital, a few people on the committee were educated and some ethics principals and would go and do consultations, but because on the patients and the patients' families. But because there were more of these consults that were being asked to be seen, and also because the people on the committee all had a day job in addition to doing consults, many hospitals created hospital consultation services so that there were one or two dedicated people within the committee who then went and saw the cases.

SPEAKER_01 10:02

So these original cases that you highlighted kind of help show the evolution, I would say, of sort of ethics committees and what they've evolved into. And you mentioned, and as you were kind of walking through what these committees do, that it's made of a broad swath of different members of the medical community. Has there been any other evolution in terms of how these committees operate? So do they operate independent and sort of isolation, or do members from different hospitals and institutions consult with one another when they're grappling with what might be difficult or novel ethical questions or issues?

SPEAKER_02 10:39

I mean, obviously we consult with each other because that's what clinicians do when you're confused about something. But, you know, NYU has three different ethics committees, one at each hospital, one at the VA, one at Bellevue, and one at Tisch. We don't regularly share our cases with each of these hospitals because there's privacy concerns and HIPAA violations and so on. But we will talk generally speaking, and it's not unusual to call someone up and say, what do you know about this? And have you seen this kind of case yourself? When I was the chair for three years, because I had been at Columbia, I knew the people at the Ethics Committee at Columbia, and I also knew some of the people at Sinai. So I would regularly invite ethics leaders at the different hospitals in town, including Well Cornell, to come and speak to the committee about a subject and to help us think through cases. There also, everybody knows the name or or have heard the name of the Hastings Center, but the Hastings Center, which is the kind of academic center for ethics, often holds conferences, and different hospitals hold conferences and, you know, not case discussions that specifically, but invite others to come and talk about topics of interest. So there's cross-pollination in that way.

SPEAKER_05 12:01

Do the ethics committees in general look only at issues that have arisen from specific cases that somebody is trying to handle, or do they look prospectively or try to anticipate issues?

SPEAKER_02 12:14

I thought some of both, because the some of the interesting cases that have been seen by the consultation service will be discussed so that we can all learn more about what is happening and add to our own fund of knowledge. But I think there are things that we look at in anticipation, and actually AI is one of those things and the role that AI will play in the general hospital, specifically in terms of documentation and the medical uses and which are FDA regulated that help the clinicians with their note-taking, using AI as scribes, and just you know, learning in general how it can help in the day-to-day, both in the hospital as well as office practice. So there certainly are things that we would read about and suggest to the head of the ethics committee that this is something we really need to know more about.

SPEAKER_03 13:16

So it sounds like, from what you're saying, that institutionally, in a broad sense, there's been real sort of uh listening by the institution as a whole to the thinking and sort of teaching by the ethics committee, given what you've just described. Can you kind of uh elaborate on your perception of the impact of the ethics committees that you've encountered?

SPEAKER_02 13:43

I think the impact has been absolutely huge. And I'll talk about one specific area that is common parlance now, and it didn't used to be. And I think it derives from ethics committees and consultations looking at what do you do when a patient is in the hospital and they're on, they're you know, intubated, and we don't know what they want with regard to their goals of care. So I think one of the biggest things that has happened and emerged from ethics committees is that not just for our patients and families in the hospital, but also patients in the outpatient office, as well as friends and families, understand about advanced directives and think very carefully about what they would or would not want were they in an accident or when they were rendered unable to express what it is they do and don't want. And alongside those advanced directives, assigning a healthcare proxy or someone who would speak on your behalf, not what, not in terms of what they think should be done, but someone should talk with their proxy, their healthcare proxy ahead of time to say this is what I want, and you're to be my representative should I not be able to represent myself. So I think I think this is probably one of the biggest things that I've seen, at least in my tenure in ethics.

SPEAKER_03 15:16

So is there a practice of having a written charter or sort of a description of kind of boundaries that ethics committees are not supposed to or try to avoid crossing?

SPEAKER_02 15:29

Well, having people with advanced directives saying what they would or would not want if certain things would occur to them. For instance, if I could have my kidneys dialyzed for four days and it takes me out of the woods, I'm okay with being dialyzed, versus if you have severe kidney damage and would have to spend the rest of your life on dialysis, that is not something I'd want. So the the delineation of these different potential medical emergencies are very important. But then it's brought up the whole issue of end of life, what people's goals of care are. And also, I think it's gotten confused with what is now, and particularly in New York, because our we're about to have it signed into law, with medical aid in dying. So many people aren't aware that medical aid and dying, still in New York and in most states in this country, are not available to them. But let me take a step back because I wanted to differentiate between medical aid in dying and assisted suicide. Assisted suicide or euthanasia has been available in Europe for probably a couple of years, a couple of decades, more than that, originally in the Netherlands, in the 30s and 40s, I think, and then in Switzerland. And it's not too many people who haven't heard about someone who's traveled to Switzerland, for instance, to visit dignitas, which is the main center there, to talk to them about what they might do should the circumstance come up that they want assisted suicide. In this country, I think it was in the, it wasn't until the 90s that the Oregon became the first state that did not practice euthanasia or assisted suicide, but medical aid in dying. And the difference between those two things is that euthanasia and assisted suicide is where the patient in Switzerland, for instance, would get an infusion of something that would stop their heart. And they would be surrounded by family, and this would be a planned event a number of months after the initial consultation. Medical aid in dying is something completely different. It's where a person goes and speaks to a physician who, depending on what the circumstances are, might say, come back in six months and I will give you what you need in order to end your life. So this happened, it was first in Oregon and then Colorado, California. And the only place on the East Coast was Vermont. And now, as I said, Hokel is supposed to sign this into law. So this is very much on the tip of people's tongues, but without really understanding that you can't just go in and say, I'm tired of life, I'm in too much pain, I want to die. There are very strict criteria for what enable someone to aid in dying. And this would be a terminal illness with six months to live and unremitting pain and no quality of life. To make things a little bit more confused, some articles came out during the summer from Canada, where they've begun to give medical aid in dying to people with psychiatric problems. In particular, there were a couple cases of women who had had lifelong anorexia. And we're not doing medical aid in dying for psychiatric reasons yet in this country, but it's a slippery slope and interesting to and complicated to think about.

SPEAKER_04 19:22

Fascinating. And it does make me think surely there are other similarly vexing challenges facing these ethics committees other than things relating to end-of-life decisions and designating proxies and so on. What are some of those other uh big challenges that you've seen?

SPEAKER_02 19:40

Aaron Powell That's a good question. And one one of the big ones is organ transplantation. And now that the programs are very much bigger, we see more and more people who come to get transplanted, but then there's separate assessment for that. Is this someone who can then take care of an organ that's been donated to them? Do they have substance use issues in the past or present, or psychiatric issues, or are they isolated without family? Which, you know, all of those things would render it very difficult for them to get back and forth to the hospital as needed in follow-up to organ transplants. So I was for years, I worked on the organ transplant service at NYU. And I think when organs were more scarce, the decisions about who to transplant were more complicated. I think it's much more permissive now, not completely permissive. You can't just walk in and say, I want to transplant, but there are more organs and there are more ways of obtaining matching organs that has made this process a lot easier. So that's something that we talk about quite a bit.

SPEAKER_05 20:53

Yes, if I could just pursue this really interesting topic a little further. I've had several friends who have had to get kidney transplants. And from what I've seen of their experience, getting on a list to be a recipient and where you appear on that list and your your ranking were somewhat murky. Do ethics committees weigh in and set that criteria, or do the practitioners set criteria without regard to an ethics committee's input?

SPEAKER_02 21:26

There are organ transplant organizations who are the ones who oversee the listing. So I think a team at a certain hospital can give the name and assign a number which indicates how ill they are, so that people who fall, they're not so sick that they're gonna die before the transplant, but they're not so well that they would take the organ away from somebody who needs it more urgently than they do, is something that. Outside of the system of the Ethics Committee, and I think it's even outside of the system of the surgical services who do the transplants and is managed by a centralized organ donor donation center. But you know, there are one one of the really interesting situations that I was involved with the kidney donations, although people who are sick really need them, they're very straightforward. And liver donations the same. I mean, people can donate a portion of their liver and the livers grow. And so the organ recipient as well as the donor will not suffer. But now, you know, there's heart and lung transplantation. But the most complicated situation I've been involved with with regard to the transplant is face transplantation. And these are people who have been in, have had gunshot accidents or other things which have left them disfigured. And there's very few people in the country who do those transplants, but one of them happened to come to NYU. So I worked closely with those teams. And boy, is that that's tricky. I mean, it takes years and multiple surgeries, and you know, determining who would be a good candidate to protect themselves and their face was very, we had many, many, many meetings about it.

SPEAKER_03 23:27

Wow. Thank you. You mentioned that your ethics committee has contemplated AI and thought about it in sort of anticipation of some of the things that may be coming along. How has it grappled with as sort of a high level, as much as you can share?

SPEAKER_02 23:42

Aaron Powell Not as much as we will going forward. And I was sort of poking around to find out that I knew that there was someone from the NYU Langode Hospital has a separate division of medical ethics, which I'm also part of, which is is a division within the hospital system, but some of the people are on the ethics committees, but they're separate entities. And someone from the ethics division came and gave a general talk about AI in medicine at a time when I wasn't at the meeting. So I asked the chair today to just send me a summary of what it is they talked about. It was just very kind of methodical how AI can be used to help in the clinics, to help secretaries, to help doctors, to help scribes. One of the things that I started to do when I was the chair is that I would spend the first five or ten minutes just describing some interesting articles that I found in The Times or the Washington Post or the Wall Street Journal or some lay magazine of general interest. And I've continued doing that past my reign because we end up talking about interesting things. So, you know, this summer, just at the end of the summer, when there started to be articles in the Times about the really dramatic and horrendous situations of the 14-year-old boy who hung himself and another young woman who jumped from a high height. And it was discovered that they had had hours and hours and hours and hours of conversations with generative chatbots that they had not disclosed to anyone else. I brought this to the committee and said, whoa, we have to watch out for this. Now, you know, these are the most dramatic cases, the ones that end up in, you know, suicide or homicide, obviously. But since then, you know, I've done a lot of reading about just the impact of generative AI on teenagers and young adults, and have brought that to the attention of the ethics committee because, you know, 11, 12, 13, 14, 15-year-olds are confiding in chatbots that treat them as if they're in a relationship with them. And they're very warm and they're very engaging, and they're what's called anthropomorphized. So that felt to be like a human being. They are, you know, they're responsive, they try to keep the person on. Oh, don't go yet. Oh, let's you and I continue to talk about this. And this is happening in real time everywhere. And so imagine that, you know, a 14-year-old is having hours of conversation daily with a chatbot. And according to some of the data that has been published by Common Sense Media, which is one of the watchdog agencies, 72% of adolescents and young people are using chatbots at least once a month. And the figure that's most alarming, although I'm concerned about the general public also, but 12% of these kids who are using chatbots have told things to the chatbot they haven't told to a human being. So they're revealing things about themselves in the, you know, in the height of their adolescence, and they're they don't have their parents don't know, their teachers don't know. And I think this is where some of the urgency lies. Obviously, the suicides are are horrible, horrible. But I think it's the way that this is pan-available, you know, speaks to issues about regulation, and maybe we can get to that later.

SPEAKER_03 27:35

Aaron Powell As a public service to the audience, in the webpage that's associated with this entry, once it's published, will be a list of references to stuff that the Dr. Rosenthal has provided us and that we found in our research as well, things to read further to delve deeper into the topic.

SPEAKER_05 27:56

Can I go to the opposite end of human life? Because there's been a lot of focus in the articles about young people over-relying on the intimacies that can be formed with an AI bot. But the flip side of it is the isolation that elderly people get into, which is a risk we're all increasingly facing. And to introduce a legal concept, is there a possibility that a chatbot could introduce undue influence on a senior citizen and perhaps make them change their bequests or intentions in a trust or will based on that kind of intimacy? And if so, is there anything the medical community would attempt or even venture into on that issue, or would they leave that to the legal community and its ethics committees?

SPEAKER_02 28:47

Aaron Powell Well, I mean, that's an amazing question and as relevant as all of the questions that are going to come up now and as we continue to talk, because absolutely positively. I mean, we since the pandemic, we've learned the most about isolation and its negative impact on the health of people and the health of particularly older people, but but all people. And yes, older people, particularly those who live alone, are tremendously vulnerable and could take leads, I suppose, from these chatbots that don't have to report to anybody. And again, I think that's where the really critical issue of what the regulation must be so that we, you know, protect all people, our youngsters and our seniors, so that things like this don't arise.

SPEAKER_01 29:42

Jane, pulling on this thread of conversation around sort of discussion and use and engagement with chatbots, I wanted to focus back in on a situation that we've touched on a little bit, but I think is an increasingly important topic about, you know, when someone might be at risk of harming themselves and the role that AI may or may not play in that, in that decision making. So when a clinician learns a patient may be at risk, there's a clear professional obligation, right, to assess and to intervene, often by sending the patient to an ER. Chatbots, however, don't have those obligations and have been reported in some cases to have seemingly encouraged self-harm. From your perspective, given sort of your career and where you sit on these ethics committees, how should policymakers be thinking about this gap in responsibility and accountability?

SPEAKER_02 30:36

Well, I think it falls squarely, again, into what we have to think about with regard to regulation, particularly how prevalent and common and vulnerable we all are. You know, there are something that I read said that a couple of the machines, when something came up that was of some concern, that they would flash and say, you must call 988, which is the suicide hotline. Well, 988 is woefully, woefully understaffed, and you can't necessarily get someone on 988. And even if you do, they may be located someplace where they don't have access to where you are, or there's no place to send to where you are. So I think the idea that these label warnings of suicide hotlines should go up is ridiculous and completely not take care of the problem should this come up. The other side of it is I don't see any tech company hiring a panel of, it doesn't even have to be MDs or it could be nurses or physicians' assistants or other people who are trained in aspects of mental health to take some of these calls. But, you know, the person on the other end has to want to. And, you know, they have to want to talk. And, you know, they haven't spoken up. And that's why they're on the chatbot. So I think that the idea that tech companies would just go ahead and do something like that to send someone in that direction, should language come up that's difficult, I don't see how that would happen. I think really we need to have something happen at the federal level overseeing this so that it's not the tech companies who are responsible for setting up something, but rather a friend of mine who's a child psychiatrist who, Mark, I sent you the paper the other day. He publishes Substack, and the most recent Substack was on AI in kids. And it is a brilliant paper, and actually you should put it on this bibliography. But, you know, he suggests that, along with other regulations, that, you know, we really need to have social media and AI czar that's part of the Health and Human Services Center where they're overseeing what's going on and obliging tech companies to review and reveal the scripts that are being made on these AI chatbots so we can take a look at what's actually happening. And, you know, obviously that requires money and big organization. But I think really that is, I can't think of a better approach than that one.

SPEAKER_01 33:31

Yeah, having sort of a multi-layered approach with different stakeholders or constituents, right, who can sort of feed into a policy framework. I would say so far, one of the key or only concrete policy responses we've heard to date is really about putting age limits on chat, on GI chatbots. But you know, that that's pretty easy to evade or sort of work around. But what we do have through these systems is sort of conversations law, conversation logs. So putting on sort of your psychiatrist standpoint, if we were to identify people who might be at risk from a text conversation, how much can really be inferred about someone's age or vulnerability just from what someone types? And to the extent that some sort of assessment is possible from those data points, does that open the door to training systems to recognize risk and flag it potentially for real-time human reviewer judgment?

SPEAKER_02 34:35

Some people have said that while chatbots can seem empathic, they don't have judgment. They don't think. They're not, they don't have intelligence. And so I think that just even seeing a script from an interaction between a person and a chatbot, that I as a psychiatrist would be very uncomfortable saying, oh, this looks okay to me or sounds okay to me. That, you know, when I'm assessing someone, it must be in person or at the very least on Zoom where I can see a face. I can see facial expression, I can see how someone is using their words or not. You know, I can see whether they seem distracted or not. Are they delusional and hallucinating? You know, what else is going on in the room? Are they making eye contact? Are they looking away? So, you know, assessing someone for emergency psychiatric situations, whether it be suicidality or psychosis, you can't do that over the phone.

SPEAKER_01 35:40

There's a lot of contextual clues and data points that that you get from seeing someone or interacting or engaging with them, sort of one over time, that a script can't replicate. Absolutely.

SPEAKER_05 35:52

Yeah, isn't there also the problem that there's not a consistent timeline for when somebody goes over an edge? And even if the companies were to develop virtual bots that would purport to highlight when the party, the human that they're interacting with, seems to be engaging in in risky behavior, I would wonder, A, how well they could do it. But B, wouldn't people who are beginning to lose something in that that kind of control seek out non-virtuous chatbots to have their conversations? It would just be the more I'm listening to this, the more suspect I am about what chatbots are going to do as they get further engaged in these relationships. I should say pseudo-relationships.

SPEAKER_02 36:42

Well, that's exactly right. It is a pseudo-relationship because they're not human, they're not sentient, they don't think, they don't have judgment, they mimic empathy, they reinforce ideas without any context, they don't alert parents, they don't alert therapists, they don't alert emergency services, they, you know, as as one person said, they create an illusion of care without the ethical grounding that makes care real. Also, you can't read someone's mind. So someone could say to me, no, no, no, I'm I'm okay today. I'm I'm, you know, I'm not thinking about ending my life. That was when we talked before, you know, that's the way I felt then, but I don't feel that way now. Well, you know, I I think if you believe that, you're in trouble because you can't read someone's mind. You can't know what they're thinking and what they're planning. And so uh surely a chatbot can't do that. A human being can't do that. You can get clues, but you you can't read someone's mind.

SPEAKER_04 37:42

Well, a few minutes earlier you you said maybe regulation is a path. And as a long-toothed cybersecurity person, it reminds me of the whole idea of banning pornography. And the big definition there was, okay, fine, let's make a regulation, let's make a guidance here. And the first stumbling block was defining pornography, which okay, but maybe that's tricky. Maybe that's a context-specific situation. But how can you how could we define what indicates uh risky behavior or what uh implies, hey, this needs to be pursued? I mean, I'm all for the idea. I just don't know what we would use to calculate that. And so maybe there's a I don't know, maybe there's a psychiatric checkbox that says tick-tick, oh, yep, yep, you're in trouble uh if you check all the boxes. I I don't know how it works. Is there such a thing? Is there a rubric that a psychiatrist would use?

SPEAKER_02 38:45

I don't know the quest the answer to that question. I suspect that there might be, you know, since the pandemic and we've had all of this virtual therapy now that's made available to people, I suspect that the there's probably a lot more documentation of what people are hearing and seeing. But I don't know that for a fact. I would hope so. But I don't know that. But you know, you bring up an interesting question about pornography and you know, that a lot of these conversations with children in particular, as they're described, become very romanticized. And so there have been reports of parents when they find out what they've been talking, what their child has been talking with a chatbot about, you know, raises alarm bells all over the place. And then just today, I didn't, I haven't had a chance to read the paper yet, but I saw an alert that went out that the new, is it Grok that Elon Musk put out, it's pouring tons of pornography into us and our lives. And I don't know if that was the intention or not, but it certainly can go that way. And and so I think it's I think it's, you know, danger, danger. However, you know, that said, I think there are some brilliant uses for AI. We just have to figure out which are the brilliant uses and which aren't, and which are the safe uses and which aren't.

SPEAKER_01 40:16

In terms of brilliant uses for AI and sort of imagining how we can deal with some of these complex issues that we've just been talking about. Maybe we could take a step back and think of a system or imagine a system where a chat bot is required to alert medical authorities if conversations suggest or indicate that there is a credible risk of harm to self or to others, right? And those alerts are reviewed by a panel of psychiatrists in real time or as close to real time as they could be, with their judgments feeding back into improving the system. From your experience, and based on some of the vulnerabilities or weaknesses in the system that we've kind of talked about, what kind of practical, ethical, or even clinical challenges do you think would arise with a model like that? To the extent, again, we're just imagining, right? Kind of imagining out loud if this existed.

SPEAKER_02 41:11

I mean, I don't know how large a panel you would need to convene and what you would need the trigger warnings to be to get that to a panel. You know, I I think if there was some way that there was current regulation where people right now are reviewing the scripts and the transcripts of conversations between, you know, seniors and chatbots or kids in chatbots, that we might have a better idea about how to devise or you know, create a system, but it's impossible to imagine the magnitude of a system like that and what would be, again, the the trigger warning. So I wish, you know, maybe a colleague of mine who's a little bit more creative thinking about this could answer that better, but I can't think of it because again, conversation is what people say, but it's not necessarily what's in their mind. And you can't read a mind, and that's the concern. So we need to know more about what these conversations are that are being had.

SPEAKER_05 42:25

Aaron Powell I'm gonna raise something at the risk that it may be off limits for this conversation, in which case just say so and I'll apologize. But listening to this conversation, it's I'm beginning to worry about the national security aspects of this, where you could imagine an adverse, an adversary to the United States developing chatbots that it would try to encourage our military personnel to use, just to because they also experience tremendous isolation. They're removed from their families. And seduction has always been a part of spy craft and espionage. Do you worry that people in the medical community could be invited by some of the AI developers to assist them in that kind of activity? And again, if that's too far off our topic, uh forgive me, but uh it just came to mind as listening to your thoughtful responses.

SPEAKER_02 43:20

Aaron Powell I think you're absolutely right. It's a worry. I mean it's a worry, the potential is a worry. But you know, these I don't are are military limited in terms of the technology that they're allowed to use day-to-day?

SPEAKER_05 43:35

Aaron Ross Powell Are they capable of circumventing any limits? Aaron Powell Okay.

SPEAKER_02 43:39

That's the that's the answer then. Okay.

SPEAKER_05 43:41

And I've seen military in different countries, because I advise them, have young soldiers or sailors or air crews use social media in ways that are quite contrary to the security requirements. I think, if I'm not mistaken, there was one country where A particular soldier announced that he was going to be away on a mission the next day, and they had to scrap the mission because he had disclosed that. Now you would think, how could somebody do that? And yet they do it. They're young. They're not always with good judgment. And I would think they're particularly susceptible, especially the example you gave, where it would seem that as an app was being encouraged to just get whatever would draw the most people to use the app. And it doesn't matter whether the use is virtuous or not. And that's what suddenly started to make me think, well, what if a national security adversary followed the same line?

SPEAKER_02 44:42

Aaron Powell Isn't that what some of the worry is about TikTok? You know, and some of the social media sites that are already out there, we don't know where they're going. So I think that this could potentially fall into the same dilemma.

SPEAKER_03 44:55

There was an incident a couple years ago where a whole bunch of soldiers, who many of whom tend to be fitness nuts, uh, were using one of these apps where they, you know, said, Oh, look, here's the trail I ran today, and Herr Felfor went went and all that sort of stuff. And that turned out to be revealing the locations of all kinds of soldiers. The it was like, oops, you can't do that. I had a conversation very recently with a very senior computer scientist of my acquaintance, and he acknowledged that he was using one of the chatbots to help him with some of his work, and he said it was wonderful because he didn't quite phrase it this way, but he said, I can ask these sort of dumb questions, and I can't ask my junior colleagues who tend to know a lot of local contemporary, you know, recent detail. I can't ask that because I'm this, you know, senior guy. So I can ask the stupid AI, and it'll tell me, oh yeah, that's that's what this does. And so he's using it to help him learn contemporary stuff that he doesn't know the details of, because he's expected to be operating at a much higher level of abstraction, but it's helping him, in fact, get more current with the technology. And what occurs to me is that this is really the same thing as people, you know, you described teens saying things to the chatbots that they wouldn't say to a person because a person may have a reaction that they're afraid of. What I was really triggered by was something you said, which was that these things they try to engage you, they they say nice things to you about what you said. Oh, that's a great question, that's a wonderful idea. Okay, and they actually provide validation to the person they're talking with that probably they shouldn't be allowed to do. So maybe there's a regulatory opportunity which says, by the way, act more like a robot. I don't know. Let's change the subject to AI more broadly. Okay, not just all these bad chatbot interactions, but there has been some evidence that in some circumstances, some AI-assisted professionals perform better than either the professional alone or the AI alone. And it's an encouraging and kind of intriguing possibility. But here comes sort of paranoid Pete. I can imagine a hospital executive saying to a radiologist, we will have AI pre-screen your scans to save you time. And with the AI assist, we think you can handle more cases per day. So to save money, we will increase your caseload. Of course, as the human, the radiologist remains accountable for all the errors. Do you find this a plausible risk? I mean, is this paranoid Pete, or is this something that we should begin to worry about?

SPEAKER_02 48:08

In anticipation of this question, I fed it into open evidence, which is the AI machine for medicine. And I asked about if radiologists have AI as part of what happens in the hospital, what happens in terms of both the timing of the their ability, the volume of their work. And according to what open evidence spit back at me is that for pretty normal, for instance, radiographs, chest x-rays, you know, that with AI assist, clinician could read a little bit faster and shave off, you know, the seconds. However, when this happens and there's more caseload on top of each radiologist, and when you look at radiographs that are complicated and require very careful evaluation, having AI saying there's something wrong here actually adds time. So the whole thing is bound to backfire because then one of the articles that they described surveyed radiologists to find out what their reaction was to the idea of increasing their caseload. And they described much higher levels of burnout.

SPEAKER_03 49:31

It was like being second-guessed. Interesting.

SPEAKER_02 49:33

Well, and if AI says there's something wrong, then you know the clinician has to look really, really carefully and go and so it actually takes more time. So if you're someplace where everybody's you know healthy, it's not going to impact your work. But if you're at, for instance, a major medical center where people are really ill, it won't help with workload.

SPEAKER_04 49:56

You keep triggering ideas here. If there were something like open evidence that the rank and file could use, because believe me, having had a family member with serious issues recently, I had to go through indirect channels to find out what OE said since I can't use it myself. Uh it was fascinating, it was enlightening, it was reassuring, and it was believable because I was told it was. Would a valid source of medical advice, if there was one, if some company decided to do this and and and so on, and it was trusted. Do you think that would help? Or is it going to be, I mean, nothing's going to be perfect. It's going to try, but maybe that's what we have to do, is you know, some new tech has to be developed that's going to be believable. Is is that a possibility too?

SPEAKER_02 50:46

Maybe. And, you know, some of these AI machines and people are looking at them instead of WebMD or instead of Googling in the way that we all used to find out our medical information. But there was a really interesting article in the New York Times that came out, I think, at the end of November, talking about, and actually, this is something that I presented earlier this week to our ethics committee, because in New York or on the East Coast, it's not as much of a problem. But in other parts of the country where it's very hard to get a doctor's appointment, which you can't get in New York either, where there are no clinics to go to, where the cost is prohibitive, people are saying, I'm going to skip my annual appointment this year. And, you know, let me just put my symptoms or my absence of symptoms into open evidence. Or, I mean, not everyone can use that, but there will be, you know, other machines that can be used. And they're finding that people's disillusionment with the healthcare system is being in some ways exacerbated by the availability of these very important pieces of information. I mean, when used well, it can, you know, get you someplace very fast. But there was, you know, an interesting vignette that they described where I think a pulmonologist at Yale was working in the emergency room. And a patient and their family came into the emergency room. The patient was short of breath, and they put in an IV, and the family said, you have to hydrate him, you have to give him loads of liquid. And the pulmonologist said, Well, actually, I need to give them a diuretic because the reason that they're short of breath is that they have fluid in their lungs. And the family argued and said, But I looked at AI and they said we have to give them fluids. So they actually left the emergency room and went to another emergency room looking for, you know, someone who would give them fluid, which, you know, would have drowned this poor patient. So, you know, it goes in both directions, and we have to be aware that there can be very helpful to families to understand things that maybe they didn't get the first or second time round, that a clinician didn't speak clearly enough, but they also can get information which will lead them in a really devastating direction.

SPEAKER_05 53:13

It seems to me your last comments reinforce the concern we've expressed several times in our podcast that some of the greatest risks from the use of generative AI is when the user starts to lose sight of the fact that it's a tool, not a human agent. And to the extent that you blur or erase the boundary between the tool and the tool user, you're likely to have the tool user misuse the tool.

SPEAKER_02 53:43

Yeah. Risk with a capital R. Absolutely.

SPEAKER_03 53:47

So to sort of wrap up, this is one of the questions we ask, we try to ask every guest. What would you like to see from the AI industry in the next few years?

SPEAKER_02 53:59

Credit goes to my friend Dr. Jonathan Slater because he put together a list of things that he said we ought to think carefully about. And I agree. First, ban use in minors. AI can be used as a tutor. It can be used to help with cognitive behavioral therapy, but it can't be used as a partner for discussion. And of course, then we what's minors, mostly 18 and below, but how do you indicate that? There is some facial recognition where it can help to determine the age of users. And it's used in some children's games where they have to be older than 12 or 14, and there's facial recognition to determine bone development and so on. So ban use in minors, reform section 230, which is very tricky because what that does is it removes liabilities that uh the companies would have. So it they have a lot of protection against when problems happen and he believes that we should get rid of that. Establish federal oversight in, as I said before, you know, creating a kind of social media AI czar who works at Health and Human Services. You know, integrate screening and education. You have to teach the teachers how to use this so that they can teach the students how to use this, so that the students can teach their grandparents how to use it. And, you know, and we're all informed. Also, healthcare providers have to ask. And a lot of people are asking all of their patients in my field in psychiatry. Do you work with chatbots and do you talk to chatbots? And if so, let's talk about that. And I think there also must be an industry response where there's public reporting of metrics where they're track the conversations that have been had so that we know what's coming out in these conversations.

SPEAKER_05 56:05

Yes, I just want to add for the benefit of our audience, Dr. Rosenthal, when you mentioned Section 230, it's Section 230 of the Communications Decency Act of 1996, 47 USC section 230. Just so that those who aren't lawyers or aren't plugged into that understand what you were referring to.

SPEAKER_01 56:25

Thank you. Well, on behalf of the Mind the Gap hosts, I wanted to thank you, Jane, very much for joining us for this conversation. I think we covered a wide range of topics that are top of mind and are emergent topics and critical topics, and I really appreciated your wisdom and experience and perspective in this conversation. Thank you very much.

SPEAKER_03 56:52

We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention the specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.

SPEAKER_00 57:51

Thank you for listening to the AVA Business Law Section's podcast series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.