Episode 11 - AI's Invisible Hand on the Scales of Justice

Judge Katherine Forrest, a former federal court judge and our guest in this episode, observes that machine learning systems build models from historical behavior that can reflect structural bias or discrimination.  She notes that the inclusion of flawed data in AI models has serious implications for personal liberty, with particular consequences for Black defendants.

On this episode of Exploring AI Matters we have the wonderful opportunity to discuss with Judge Forrest her observations on AI and what she thinks can be done to help ensure that these tools can lead to fair and just results. [2023-06-01]

Transcript

SPEAKER_04 0:01

Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap, Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.

SPEAKER_03 0:52

State court judges also use these AI tools when determining the sentence to impose on a convicted criminal. Federal judges do not use AI tools for bail or sentencing, but do use them after conviction. Federal court judges use AI tools for post-conviction risk assessments, essentially to determine what services to provide and supervised release times. Judges have been relying on these AI tools to provide these types of risk assessment for over a decade. However, evidence shows that the AI-guided decisions do not treat all defendants equally and fairly, which raises important questions about how to appropriately use these tools. Judge Catherine Forrest, a former federal court judge and our guest today, observed in her 2021 book, When Machines Can Be Judge, Jury, and Executioner, Justice in the Age of Artificial Intelligence, that machine learning systems build models from historical behavior that can reflect structural bias or discrimination. She noted that the inclusion of flawed data in AI models has serious implications for personal liberty, with particular consequences for black defendants. Judge Forrest reasons that even judges who may be more aware of structural inequities are using a risk score that cannot and does not take any of that growing awareness into consideration. She cautions that continued use of unremediated AI tools by judges cannot continue in their current forms if we want the American ideal of justice to survive. On this episode of our podcast series, we have the wonderful opportunity to discuss with Judge Forrest her observations on AI and what she thinks can be done to help ensure that these tools can lead to fair and just results. Welcome to Mind the Gap Dialogues on Artificial Intelligence. I'm Roland Trope, a national security lawyer.

SPEAKER_05 2:55

And I'm Charles Palmer, a computer scientist. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence. In addition, we have two more hosts.

SPEAKER_01 3:07

Hello, I'm Ama Adams, a national security lawyer.

SPEAKER_03 3:11

And I'm Mark Donner, a computer scientist. Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. Judge Forrest served as a U.S. district judge in the Southern District of New York from 2011 to 2018. She is a prolific writer on issues of AI and the law, and recently completed a forthcoming book entitled, Is Justice Real When Reality Is Not? The Construction of Ethical Systems in Virtual Worlds. She is a partner in the litigation department at Paul Weiss, working on, among other things, advisory work, investigations, and litigations related to AI.

SPEAKER_05 3:52

Thank you, Judge Forrest, for giving us your time today. I understand you served eight years as a district court judge in the United States District Court of the Southern District of New York. Had being a judge been a long-term goal of yours?

SPEAKER_00 4:06

You know, it was really a goal of mine after I had become a partner at a law firm where I was trying cases and decided in that capacity that being a decision maker and really uh trying to make a big impact on issues that uh affected large groups of people was uh a goal of mine down the road. And so it took several years to put that into place, but it was a goal of mine, uh I would say, from the time I was about 34, 35 years old and on.

SPEAKER_05 4:41

Well, you certainly got there. And speaking of goals, when did um when did you start to develop an interest in AI?

SPEAKER_00 4:49

You know, it it's uh people ask me that a lot, and it's it's hard to really trace it back because I'd had a long interest in high technology issues, and I had in my early part of my career done a lot of litigation uh in the internet area. And I'd been interested in new technologies and keeping up with new technologies, read a lot of nonfiction. I had an interest in quantum mechanics and quantum physics, and uh I also uh then had an interest in theories of consciousness. And if you are interested in whether or not the same particles that make up a rock have consciousness and why they don't, it pretty quickly leads you into theories of artificial intelligence and people who are writing on just those, uh just those issues. And so I got into it really uh as a matter of my some of my professional interests informing some of my personal interests and then on from there.

SPEAKER_05 5:46

Fascinating. Um I should I'd I'd love to sit around and talk with you a lot more. But anyway, uh it was a surprise to us and probably to a lot of our listeners that some judges are actively using AI in the courts. What tools are they using? Uh used by the federal courts and state courts, and what types of information are they providing to the courts that they, I guess, didn't have before?

SPEAKER_00 6:10

Yeah, you know, it's it's interesting because AI uh is used really right now in all phases of the uh court system. Uh and I'll focus my remarks on the tools that are used by the courts in terms of the judges and the uh judicial process in the criminal justice area, but they're really used throughout the entire investigation process, supporting of a case, uh, creation of a case, et cetera, et cetera. But in terms of the courts themselves, there has always been a desire to have some form of evidence-based decision making in court. And that is something that judges have often been seeking. They are looking for the way to find out what a human will do next, how the human uh will behave in a variety of circumstances. So the tools uh that have been developed that are AI-based are really follow-ons to earlier tools that judges had used, maybe on an ad hoc basis themselves, where they would write down their personal experiences with certain kinds of criminal defendants who had certain kinds of backgrounds and uh, et cetera, et cetera. And so there was sort of a paper and pen version of what then evolved into a series of paper-based regression analyses, actuarial analyses, into eventually AI tools that are machine learning based. But now, all over the country, we have deployed in the court system uh a number of AI tools that uh function in ways that assess and predict things such as recidivism, the likelihood that a defendant will show up uh for uh a court appointment, uh, whether or not they should be granted bail because they may or may not be violent, what kinds of housing determination is best suited to a particular criminal defendant, what kinds of services are needed for a criminal defendant to try to achieve some of the goals of sentencing, whether that be, for instance, rehabilitation or something else. And so the tools themselves, I want to sort of stress this, uh, go that whole gamut. They really sort of uh are both in the uh initial stages of the criminal justice system where they are assessing defendants in terms of bail all the way through post-conviction, where they're looking at the needs of a defendant after having served a sentence in terms of what kind of uh post-conviction supervision tools might be useful. So let me give you a few examples. The most famous example is the compass tool, because that compass tool is the subject of a case that people often are cite and are talking about in the AI area, which was uh the Loomis case. But the Compass tool is only one of the tools that does some of the things that I've just talked about. There's also an LS CMI tool that California uses, and also, by the way, Canada uses that tool as well. There's the Ohio risk assessment tool called Aura, which is used in Texas and Florida, Indiana. There's the Strong R tool that's used in Washington state, and of course, there's a tool uh that is known by the acronym PICRA, which is the post-conviction risk assessment tool. And that's the tool used in the federal system. The other tools that I mentioned are used in the state court system. So there are a variety of tools that really cover this gamut of services and of decisions uh that are made in the criminal justice area. You'd also asked about what other kinds of information these tools provided that weren't perhaps provided by non-AI-based tools. Well, as we all know, what AI does better than anything else is take large data sets, huge amounts of information, and process it and give us predictive results based upon all kinds of probabilities and algorithmic analysis of the data. What these tools do is they take large data sets of individuals, whether it be arrestees in some instances, uh, individuals who simply have had contact with the criminal justice system in other instances, only convicted criminals in other instances. It can be a whole variety. It takes large data sets and it combines those data sets with a variety of academic literature, theories of justice that had been used to construct a particular algorithm with a particular uh, I'm going to call it a viewpoint for the moment, and to then work with that data to assess a particular outcome for an individual. So it's really the quantum of data and the theory that embedded in the additional work around the algorithm uh that's different, and that's giving judges additional information uh on a broader set of the population.

SPEAKER_03 11:41

I want to take a step to the side for a moment because uh it may be that I'm one of the few who didn't notice it, but I was unaware when courts started using AI. And I was wondering how well um reported or advertised or disclosed was this at the time? And what kind of a process did the courts go through before deciding they would use AI and how they selected a particular AI tool?

SPEAKER_00 12:07

Well, that's actually a fascinating question because it's one I myself tried to get to the bottom of when I was writing my book, When Machines Can Be Judge, Jury, and Executioner. And it's not so easy to find the moment when the tools turned from just sophisticated software into uh machine learning tools. And it takes some digging. But somewhere in the vicinity of, I would say, 2010 to 2013, there really was a deployment uh of AI-based machine learning tools uh in large force. Before that, there were some beta and there were some other software programs, but a lot of people would call them more along the lines of sophisticated regression analyses rather than really a machine learning based tool. So uh it's really a relatively uh recent phenomenon, and it comes again with some of the same reasons that we're seeing so much AI now during the same time frame, which is the access to huge data sets and lots of computing power all at the same time. And then you asked, Roland, well, how much awareness was there that these were AI tools that were in fact being rolled out? And the answer is there wasn't a lot of awareness that these were AI tools being rolled out. They were just tools that courts were taking in as software applications, really, and starting to use them. And your next question was well, what kind of analysis was done to assess whether a particular tool should or should not be adopted? Let me start from the federal side, which is that the federal government decided to uh itself analyze several of the tools that I've mentioned and decided after analyzing them that it did not want to use any of those tools, but it's would be better served by developing a bespoke tool. So the PICRA tool is a bespoke tool that was developed for the federal system. Now, in 2015, there was an association of national state courts that actively encouraged the adoption of these kinds of AI tools. So there were, there wasn't a prescription as to which one it had to be, but there was uh a real push towards the state courts using these tools uh and allowing each court, each state to evaluate what tool served it best.

SPEAKER_01 14:45

Just force, you've mentioned you're talking about all these different tools, which you know, me sitting here as a lawyer, it's fascinating to hear because I wasn't aware that all of these tools were being used. But you said something that raised a question with me. Are these tools all using the same sort of data inputs, whether that be sort of, I don't know, arrest history, drug use, race, um, marital status? And then you mentioned something about academic literature and how that is built in to the information that the algorithm is using as it looks as the data sets. How are the courts adjusting for these? Like who's putting in the inputs, whose viewpoints or philosophies are being sort of weaved in to try to help come up with a system that can help judges make these types of decisions?

SPEAKER_00 15:33

Uh that's an excellent question. And it's the also it's the great unknown. So each tool has got its own, what I'm gonna call, for lack of a better word, right now, literature that goes along with it. The Compass tool actually has a very extensive handbook that you can get on the web. And they've updated it from time to time. And there's a 2019 or 2020 version uh that is currently available that actually uh goes through a fair amount of the uh decision making that it has engaged in in terms of the uh theories of criminal behavior that it has utilized, the factors that it has taken into consideration, and all of that. Some of the other tools don't have as much background on what choices they have made, who's making those choices, and where the lines are being drawn. Uh, and so you really don't know. In terms of the portion of your answer about who's making the decisions, that is unknown, I think it's fair to say, for every tool. So we know what company is making the decisions, but we don't know the backgrounds of the individuals who are themselves coming up with and choosing the theories of criminal behavior, of uh criminological literature, criminal justice in general theories of justice. We don't know who those individuals are. What we do know is that there's a great deal of debate all day, every day, in any uh criminal justice class that you can take as to where people fall on which side of a particular debate as to whether a particular theory is a correct theory, is a theory that is an incorrect theory, is an offensive theory, et cetera. The last piece of what you'd asked was are all of the same factors included? Things that you'd mentioned were things like uh marital status, education, uh, we can add in drug use, we should add in race, we should add in gender, we should add in uh whether or not the individual has a stable residence, uh, whether they're homeless, whether their parents uh encounter the criminal justice system. All of those factors may or may not be a part of the particular tools. So each tool has got its own set of inputs, its own set of factors. They're each using their own data sets which may change and which are not disclosed. And as a result of different data sets, you have different patterns which float to the top, et cetera, et cetera.

SPEAKER_03 18:19

And do defense counsel, when arguing to a court, take into account the AI tools the court is using? Do they address those in their discussions with the judge, or is that left under the hood?

SPEAKER_00 18:33

Well, let me sort again, I want to separate the federal system, which has got the post-conviction risk assessment tool, which you'd mentioned, Roland, in your uh opening remarks, and the state system. And in the state system, there are some judges which uh actively articulate that they are using a score of, for instance, the potential for recidivism or violence uh based upon a tool. That's what gave rise to the Loomis case, uh, the very famous case out in Wisconsin where there was a judge who'd indicated that the score of a particular defendant increased the judge's concern about that defendant. The defense counsel then wanted to inquire deeply into what made that tool come up with a particular answer. And the judge said, Well, it's proprietary information from that tool was the compass tool, but it probably could have been any number of tools. It happened to be the compass tool. And you've got the guidebook, I've got the guidebook, that's enough. The defense made a due process challenge, that due process challenge was denied, and it went up and the denial was affirmed. So some judges articulate the utilization of a score and some do not. And so uh that may reflect whether or not they're in fact relying on it or not. It's not required in the state system that the judge articulate whether or not they are in fact relying on a particular score.

SPEAKER_03 20:12

Could you help us understand a little more about how judges use the tools? For example, how are post-conviction AI-based risk assessments provided to or obtained by a federal court judge?

SPEAKER_00 20:25

Well, the post-conviction risk assessments are often used to give a sense to a court in a narrative structure of the view of the portion of the court that's dealing with the uh pre-sentence investigation report that's preparing that as to the likelihood of a particular individual defendant in that case, a uh convicted individual, it's at sentencing, uh, to comply with the terms of supervision and potentially to experience sort of growth, personal growth in the future, and what services would be most useful for purposes of achieving the goals of their rehabilitation, trying to reintegrate the individual into society. And so it's really just a portion of the report that's given to the court by the post-trial services. And so it's not known to necessarily be an AI tool, if you will. The fact that it's a this information is generated from an AI tool is not something that is being widely discussed, at least not while I was on the bench, and I'm not aware of literature that has recently uh surfaced that in a broad way. There's teaching that can go on and judicial conferences where it's discussed, but it's not part of each and every uh pre sentence investigation report. So the way in which the technology Technology that's being used to generate some of this information. That's not front and center. What's front and center is the recommendation. Now I would also say, and it's important to stress, that at each level of this process, there can be and should be a level of human discretion, judgment, and intervention if something really seems off. And that can be from the post-trial services in staff that is putting together recommendations. It can also be by the judge who says this doesn't look right to me. And so these are not set in stone. These recommendations from the tools are in the federal system simply recommendations.

SPEAKER_03 22:46

Without going into any specific case, is it possible for you to share with us some of your experiences or post-experience reflections about AI risk assessments?

SPEAKER_00 22:58

Yes. And I think that what I want to do is step back for a moment and say that these tools, these AI tools, are really designed, they're geared towards trying to eliminate, if you will, variability as between different judges and to provide some kind of, as I said, evidence-based decision making. However, as we know, the tools end up having disproportionate impact, a disparate impact on certain uh populations, particularly men of color. Now, one thing we have to ask ourselves are is the hard question are human decision makers any better? And there are some studies which say human decision makers are worse. In my book, I cite a study from 2000 by Groves and another study by Sweat, but then I analyze why those studies are infirm, and I look at some alternative studies, one by Fareed, that says that in fact human decision making is better. But there are there's a question as to whether or not human decision making is in fact better or worse than an AI tool. So I say that because when we're looking at my experiences and observations about the utilization of these tools, I want to stress that we have to know that it's against the backdrop of the reality of the infirmities of human decision making to begin with. So in terms of the AI tools, they are less than accurate in all ways. And they, I think, have some very serious fairness issues. But human decision making has its own problems. That is for probably a different podcast for you at some point in time. But in terms of my experience, I know that there were instances when I was making a decision about an individual in front of me where I would have a recommendation about a particular series of needs for that individual that I did not think necessarily fit with what I saw in front of me, what I read in terms of uh the history of that individual, in terms of the information I had from his defense counsel, from the letters that were presented to me. I also know that there were times when the decision, the judgment from the uh folks who were putting together the PSR, I felt was about the likelihood that the individual would perhaps uh recidivate was right on or not right on, and that I would then use my judgment and discretion to change and alter uh where I would go with a particular decision. We know that in terms of some of the tools that have been used, there had been a number of studies, including the first very famous one in 2016 by Angwin of the Compass Tool going on in a series of studies after that, that there are real issues with the uh, as I was saying before, the disparate treatment of these tools with individuals and particularly men of color. And we know that, for instance, in a study of real life of what happened as between what the tool predicted and what really happened, that there was a very significant difference in the accuracy of the tool between black men and white men. So we know that in terms of what's happening with the tool, that the accuracy rate is uh a lot lower for black men. And one thing that I would say is that each of these tools that I have mentioned today have been tested for accuracy. And the accuracy rate is somewhere in the vicinity of 60 to 74 percent. And that is relatively uh it's it's much better than average uh in terms of accuracy. But you don't want to be somebody who's in the the percentage that was uh inaccurate. And that really has very significant implications for our American concepts of individualized liberty determinations.

SPEAKER_03 27:25

I I have two quick follow-ups. One is whether we've reached a point where courts would say it it's not responsible for them to proceed without using these AI tools. And the other question, it may be outside of our scope, but I was wondering if if district courts and state trial courts are using AI tools, are appellate courts starting to use them for other purposes?

SPEAKER_00 27:50

Well, the appellate court uh is reviewing really the record as it stands in the district court, and so that's not as much of an issue, really, at all. I would say it's not an issue at all in terms of the utilization down at the level that we're talking about today. There are other tools and generative AI that one can talk about being in beta testing for the appellate courts down the road, but that's some years uh to come. In terms of whether or not uh the district courts and the state courts are uh, I think your your question, Roland, was whether or not they were thinking of no longer using these AI tools.

SPEAKER_03 28:31

No, I was wondering if they feel that it it they're obligated to use them, that it's not it, they're not doing their job if they haven't used them.

SPEAKER_00 28:40

Yeah, I think that the answer is different for different judges in different districts. Certainly for the federal system, it's really we can put it to the side because I think the federal judges feel a great deal of empowerment in all aspects of the sentencing that don't use the AI tool because it doesn't exist for you know, giving a uh a duration of a sentence. But there are some very interesting studies of some of the state court judges, some of which I cite in my book, which talk about some judges feeling relieved that they are uh being given this tool that suddenly breaks through the mystery of human behavior. And they're able now to say, ah, I never knew before whether what I was doing was right or wrong, but this tool now gives me something to hang on to. And, you know, when I read uh this particular study, I was very surprised because it was not followed up with any information about the particular level of inaccuracy of the tool, uh, the particular, the particular issues that the tool has with distinguishing between a population and the individual circumstances of a single person. So I think that the states may be, there may be some judges in some states that are now more reliant on the tools and feel that they like them as a kind of lifeline to evidence-based decision making. But I do believe that judges all over are repeatedly told you should apply your own discretion. Let me just add one more point to that, though, which is that there have been studies about the number of instances when state court judges have varied from the determinations of the tools. And the finding is that there is very little variance, if you will, from what the tools have said. So while they may feel uh empowered to, and it may be it's only coincidence that they come up with the same decision, they are in fact following the tool quite closely.

SPEAKER_05 30:54

I suspect that very few judges have your kind of background in this. Which leads me to ask are the judges generally well positioned to understand and assess the value or the accuracy of these risk assessments assessments?

SPEAKER_00 31:12

Well, there is a uh an increasing desire by, for instance, the National Judicial College. I've done a lot of speaking for them on these AI tools, on exactly the issues that we've been talking about, on the potential for implicit bias. I've done a lot of speaking for the New York state system and spoken to a number of times to uh New York State judges about these AI tools, what they do. Uh, but I can't speak to what's happening on a state-by-state basis uh everywhere, and for the folks who aren't attending some of these uh conferences that I've mentioned, because uh there really does need to be training in what the tools are doing, what they're made up of, what some of their shortfallings may be, the kinds of concerns that if a defense counsel raises it, ought to be taken seriously, some of the potential due process issues, all of those are things where I believe uh the judges should be trained and want to be trained. You know, you sometimes don't know what even to ask if you've not been given the vocabulary or the very basics to understand what's being put in front of you.

SPEAKER_05 32:35

That that is very interesting. And and with a lot of AI uh applications, I don't think this one's any different. There's also the risk that the users will end up depending or deferring too much to the AI-based decisions. You're just assuming, well, you know, I don't understand this stuff. It must be really good. So they wouldn't be charging so much money for it. Um is that a real risk that we might have judges deferring too much to AI-based decisions?

SPEAKER_00 33:04

Uh I think that is a real risk. I think that that is a risk that is something that we see in some of the statements that I've mentioned about some judges feeling relieved in the state court system, uh, that they finally have a tool which sort of breaks through the mysterious code of human behavior and an over-reliance believing that AI is somehow science. It's somehow truth. It's somehow going to produce true and correct results. And the problem is, of course, these AI tools are based upon the design of an algorithm by humans. Humans are the progenitors of these AI tools. The humans may be and certainly are choosing the data sets, which will then inform how the inputs are generated or what inputs are generated. Sometimes we know there are human adjustments in the inputs that go into the algorithms, into the weightings of the algorithm of those inputs in the algorithm. And all of this human involvement in the tool suggests to us that there's a lot more than just a tool that is somehow based in some kind of pure science where there's a right and a wrong that a human can't really do anything to make worse or make wrong. But in fact, the human involvement does skew the accuracy of the tool and it can skew the output of the tool. So there is a real, I think, risk there.

SPEAKER_05 34:50

Yes. No exceptions just because it's the judge. Right. If if a judge has read all you read your your publications and feels like they understand it uh reasonably well, would they or maybe it's even just a gut reaction, would they have the flexibility to adjust these recommendations or just outright ignore them? Especially if they think there might be issues around bias or inequity?

SPEAKER_00 35:16

Yes. I mean, in the in the state court system as well as in the federal system, there's always the requirement that judges exercise individualized discretion. For some state court systems, there is a requirement that if there's a variation away from the tool, that there be a statement as to uh why that variation occurred. Now, if you just have a gut feeling that that defendant really is done with crime, really is not going to recidivate. Really, they did something stupid once because they were young and impulsive, and they're, you know, they're now in a different place in their life, and the tool tells you otherwise, and you want to vary from it. It might not seem to a judge that just putting down their gut reaction is going to be enough. So there's a uh, I think a risk in requiring judges to try to come up with too much sort of contrary evidence-based decision making to that will not necessarily seem as weighty as the almighty AI tool. In the federal system, there's a requirement that the judge ultimately exercise all discretion over every aspect. And so that's not uh the same kind of uh decision-making path.

SPEAKER_05 36:40

Sort of shifting the judge's actions from uh at the state level, you say, to uh arguing with the AI as opposed to the attorneys. So okay. Is there anything that a judge might use or an approach or some some angle to assess whether an AI recommendation is just flawed?

SPEAKER_00 37:03

You know, I think that the best tools that a judge has are based on uh their individual experience and their individual judgment. And that may seem uh as if that is not uh necessarily a satisfying answer, but it's an answer that we've relied on forever in terms of judicial decision making, which is judges have a lot of experience with human nature. Uh, are they always right? Absolutely not. Do they try hard? Every judge I know is trying hard and they're trying hard to do the right thing. So if a judge sees something that does not look right, particularly if it's skewing, I would say, in the direction of a harsher and more severe outcome, they need to really stop and ask the question of what is this based in? What am I not seeing? I don't judge it to be as this individual to be as violent as this tool is telling me that he or she is going to be. Uh, but what is this tool basing this on? And frankly, it could be that the tool is basing it on a data set that had to do with a particular meth situation that was occurring in a particular community over a period of time that resulted in a high crime rate. So it could be not, it could end up being an issue that needs to be adjusted for.

SPEAKER_03 38:27

Can you build on that? I recall, and I can't remember the page number, of course, in your book, um, you discussed how certain AI tools draw on geographic data samples that aren't the same as where the defendant was raised, grew up, has lived, and that that mismatch between the data that the AI tool is using and the individual themselves may create uh I would think some strong inaccuracies, if not biases.

SPEAKER_00 38:59

Yes. And I really uh I think it's important to think of how big our country is and how different different parts of our country are, and how every judge comes to the table uh having been born in a particular family, come from a particular community, grown up in a particular uh place, been educated in a certain way. So the way in which geographically, for instance, crimes are considered to be serious varies. And frankly, what is even a crime can vary by geography. So, for instance, California decriminalized marijuana at a state level uh far earlier than New York. And if you've got a data set from New York that is penalizing youths who are in possession of marijuana and using that data set with a California defendant, it's going to skew those results because you've now got a data set that has a whole bunch of individuals who've been convicted of crimes, and so they may be disproportionately represented in a particular demographic being used in another part of the country where that data set would be different because those individuals would not have been arrested for that crime. Same thing if you want to take, well, there are other examples. For instance, in certain parts of the country, prostitution has been decriminalized, and who is prosecuted for prostitution has been changed by both uh who's being in fact prosecuted or whether or not something is criminalized. So where you are in the country can influence who is in fact in your data set, who's being arrested for a particular crime. Another example is, for instance, policing practices. Policing practices differ all over the country. They differ by who has been chosen in a particular area to lead a particular law enforcement effort. And that can also influence who is being arrested for what. And that then informs well, what is the data set then? The data set is now informed by those policing practices. So if you're using a data set with, say, stop and frisk policing practices from New York in the 90s, and you're using it in 2022, 2023, that's a different kettle of fish. Same thing with using stop and frisk here that was that was done in New York, in uh Berkeley, California, where their policing practices were very different. So geography matters, time frame matters, and the area of the country matters because of how we view crime and how we view uh policing practices as just two examples.

SPEAKER_01 41:54

This discussion we're having about geography and time frame leads me to another question. I I don't know, I don't know if this is true, but it makes me think is the issue more than sort of geography or point of time? Is the issue also, and I don't know how AI would capture this, right? About how our notions of what is right or wrong or how we view different types of quote unquote criminal behavior changes with time, right? This concept of what is good versus what is bad. And that's something that humans and certainly judges and juries are always evaluating in the context of a case before them. But is is there a concern that AI might not be able to fully embrace that as part of sort of the input, you know, our changing notions of what is right or wrong?

SPEAKER_00 42:43

Well, so this goes, I think the most the easiest way to get at the question you're asking, and then it's a very uh important question, is to think about these tools as based on data sets, and that data sets are based upon what really happened in the real world at a historical moment in time. And what we do is we cement by choosing a data set and using a tool against a data set, we cement that historical moment in time for purposes of the utilization of that tool. So what needs to happen is you've got to, number one, understand what is the data set that you're using, because a data set of arrestees is not a data set of convicted individuals, right? And so it's a very different population. And if you're arresting people for jumping turnstiles, or you're arresting people for a variety of things that you wouldn't arrest them for in 2023, that's a that's an issue, right? Because your data set is now out of step. So you need to understand who's choosing your data set, what kinds of choices are they making in terms of the population to include. Arrestees versus convicted versus someone who may have just encountered the criminal justice system. And where what's the time frame? Is it 1990 to 1998? Is it 2000 to 2010? Well, that does not capture, by the way, when the uh, for instance, NYPD stopped arresting people for certain kinds of possession of marijuana, right? So you've got to ask yourself the data set of choices. What are the choices and does that data set best reflect our current views of what is, in fact, criminal behavior and what's a serious crime? It used to be that, for instance, crack cocaine was considered to be a far more serious crime by orders of exponential magnitude than powder cocaine. Now people still may have that view. But I'll tell you, the federal guidelines have brought that way down and evened that out greatly. So the discussion of what is a serious crime also differs. And so if you're looking at data sets that include and embed real life in it, that real life, you want it to be as close to our real life as possible. So it's actually reflective of uh the current community. And that's also geographically based and based on particular segments of a geography.

SPEAKER_04 45:31

So it it sounds to me like what one of the things we need to be able to say in the record of a case is uh what model is used, what data it was trained on, as of what specific time. Because as you observe, the data evolves. So and and you know, in the ideal world, I suppose the judge should be able to press buttons that say, I want this data included, but not that data. And I want, I actually want to be able to think about the model that is is used to provide me with feedback and record that in the in the record of the case so that that's part of the review process uh if there's an appeal. Is is that a meaningful way of framing the question?

SPEAKER_00 46:25

Well, I think that you've got a couple of different things uh embedded there. The first thing is I think that you need to know whether or not the judge is relying in any way on the output of a an AI tool. And if the answer is no, stop there. Here's your decision tree. If the answer is yes, then I think many of the questions that you've just mentioned become real questions that are important to answer in order to understand whether there are any issues with the utilization of that tool. And I think that you want to then ask, okay, if the tool was used, what was the decision that was based in part on that tool? So you can get a sense of, okay, was it one small input? How much did it factor? And so you want to get that information. And I do, I personally believe that until we have more accuracy in these tools, those decisions are absolutely uh, those questions are absolutely uh critical. And I think the judge ought to understand what they're being given and what is underlying that, because that may influence whether or not the answer to the first question is they did or did not rely upon that tool. Now you'd also mentioned, Mark, that it may be that uh the judge could then push a button about, I want this data, but not that data. I would say I only want people to be choosing data sets who are trained in choosing data sets. And because they uh may not understand exactly how that's going to influence the inputs. And there'll be a correlation between the data set and what's going to sort of float to the top in terms of the weighting of a particular input. If you pick a data set from one period of time, you may have a bunch of people who are in a particular age group. You pick a data set from another group of time, and now you've got people from a different age group. And you can imagine that that can be true of every factor. So we should have clarity on who's choosing the data set and why.

SPEAKER_04 48:36

Uh and then there needs to be a statistical agency supporting the judicial branch nationwide to reason deeply about these questions at a large strategic scale.

SPEAKER_00 48:49

I think that there needs to be, this is one of the areas where there needs to be some regulation. There needs to be some form, and it can come in different forms. I'm not saying regulate innovation and squash innovation. I'm saying that there ought to be a kind of quality testing and a kind of agreement on impact assessments for these tools and sort of environmental assessments for these tools. There ought to be when the IT groups are figuring out what tools are going to be licensed in, there ought to be a set of standardized criteria that they uh that the individuals are given that say this is what you should be looking for. And you're not looking for the speed of the interview. You're looking for the accuracy of the output. And when I say speed of interview, I'm talking about one of the tools I've mentioned markets itself as being particularly useful because it you can you can uh implement the uh data into the tool so fast. Well, frankly, if there's a liberty decision being made, I'm not sure that speed is really uh the number one criteria we should be looking for.

SPEAKER_03 49:55

You're you mentioned speed there. There was a DOD request for proposal a couple of years ago, which wanted to develop what the request for proposal was calling an AI lawyer that would serve as a JAG officer to mission commanders. And one of the problems that they started to realize was what if the commander started to say, well, the AI tool can give me the answer so much faster? The AI tool has a higher level of confidence in its own answer, whereas your JAG officer may be very circumspect, maybe take much longer to develop that level of confidence, and certainly would take longer to develop an answer. Would commanders start to shift to rely on their AI input lawyer than the JAG officer, and that for whatever reason, that project died. I don't know if it died for those concerns, but some of what you're saying, I think raises that. And I mentioned it because you have an interest in AI used in autonomous weapons also.

SPEAKER_00 51:01

Yes. And, you know, it's the AI tools used in autonomous weapons have a whole different sort of set of issues around them in terms of whether or not humans are kept in the loop or humans are out of the loop. And so it's it's quite a complicated discussion and whether or not AI tools in autonomous weapons can actually lead to more accuracy and less collateral damage. Actually, it's sort of an irony. Uh uh, if you think of an autonomous weapon, you think of destruction. And of course, there is destruction, but it might be reduced collateral damage. But let me roll and just go back to one point that you made in terms of a debate that was had at the DOD some years ago, uh, as you've described it about these uh the potential replacement of lawyers. Right now, with generative AI and chat GPT, something that's not part of my book, this is a very big discussion in law firms right now, in terms of the extent to which law firms uh or their associates ought to be using this kind of generative AI. Undoubtedly, that debate is going to move into the court system as well. And there will be a discussion as to whether or not certain aspects of decision making can be done efficiently, at least as a first step through certain kinds of generative AI tools. Of course, you can't assess credibility with a generative AI tool. And so you have a fundamental issue right there.

SPEAKER_03 52:38

Well, can I ask? Um let me refer back to an earlier episode we had with uh Dr. Meyerley, where she was describing how doctors performing uh colonoscopies use AI in Germany to help identify uh irregularities that might be indicia of future polyps. And she observed that young doctors who have been trained only to use AI as a tool with them and never have been trained without it, rely on the AI to identify what they should be looking at. Whereas experienced doctors, she said, have a habit of using, moving their eyes in a circle or semicircular oval through the uh across the image of the tissue. Uh, and she worried that young doctors would not have that extra backup. Do you worry that either judges or young associates who are when in their earliest experiences in those roles have AI already, don't come with pre-AI experiences and judgment to challenge AI?

SPEAKER_00 53:47

I I think that that's a very real issue because I think that to develop the kind of decision-making set of skills that you need, you need to be confronted with a whole host of unknowns. And you have to believe that they're unknowns and that you have to sort through them and develop the logic yourself and understand the logic very carefully and build those, put those building blocks one on top of the other. And if you've got another tool that's doing it for you, I worry that your uh logic model internal to the human won't be as sophisticated. You know, look at how many people don't even remember their own phone number anymore because their phone does it for them, right? So once you start relying on tools to do certain work for you, it becomes a uh a crutch that becomes very easy to then defer to.

SPEAKER_03 54:45

Well, and part of what has surprised me in this discussion was the discussion of data sets. I've wondered, I've always thought of updating as adding data, but part of what it seems to be saying updating should also be removing data that's no longer relevant geographically or uh for other reasons. Can I uh take us up a couple of levels of generality? A theme that's underpinned much of this discussion and uh much of the discussion in your book is fairness. Uh, could you give us your sense as a judge of how you define fairness and how you distinguish that or merge it with the concept of justice?

SPEAKER_00 55:26

Yes. I mean, fairness is, in my view, the unbiased and even-handed decision making of a judge. And that a fair judge is one who will individually assess facts that can then be used to make an even-handed and unbiased determination, not based upon anything other than facts without implicit or explicit bias. And there's a difference today between uh fairness and accuracy, as we've been talking about these AI tools, because you can have an accurate AI tool because it happens to be whizzing and whirring over a data set that's old and reflects policing practices that are no longer believed to have to be um um appropriate and uh to be uh reflective of crimes that may no longer consider to be crimes. And so you can have a tool that works as it should. It's an accurate tool, but not a fair tool. And because the fairness for an individual is based in the ability to make a decision about that person standing in front of you as a judge and the facts and circumstances relating to him or her in an unbiased way, uh, you know, a decision sort of based on all of that. So justice, I think, is when you actually have fairness applied in an even-handed way to the individual in front of you. I think that if you are using an inaccurate tool, then you may not be dispensing justice. You may be dispensing efficiency. And if you luck out and you've got the person who's in the uh 64% of the accuracy uh versus the percentage that's inaccurate, then maybe you happen to have justice coincide. But you can have fairness without justice if you use these tools without really understanding their shortcomings.

SPEAKER_03 57:50

Did your views of that change over your career as a judge and after you you retired from the bench? Because it seems looking at your writings, you're continually updating your view of these tools uh as AI changes rapidly and as you try to keep up with it, and you're doing a better job than most of us can do.

SPEAKER_00 58:10

Well, uh yeah, I don't know about that. There's so many sophisticated people out there, but I I do believe that originally my view was that that these tools should not be used until they could be made more accurate. And uh I now am of the view that whatever my view on accuracy is, that's these tools are here and they're here to stay. And that what we have to do instead is turn to a system of trying to educate judges and individuals in the court system about the tools. We have to talk to the regulators about what can be put in place to ensure the highest degree of accuracy, and that we've got to uh at the end of the day understand that AI is not going away. We are going to live with it in every aspect of our life. We're going to be living with it in the criminal justice system. And what we really have to do is try and understand how AI is then impacting individuals' lives. And then when we see the shortfallings of an AI tool, we have to address them. We can't run for run run from them. And that I think is the task before us.

SPEAKER_05 59:23

Charles? Wow, yes, yes. I'm I'm here. I'm just It's such a fascinating discussion, and uh I hear myself I find myself just you know going off in all directions. Um so you're you you've said that accuracy is is pretty much job one. What other directions or or challenges do you think we need to address as we go forward? I mean you just uh mentioned quite a few, but is there anything else that uh the researchers or the practitioners out there listening to this might uh set their sights on solving?

SPEAKER_00 1:00:01

Well, one thing I think, and this is going to sound a little bit theoretical, but bear with me for just a moment, is our American system of justice is a system based upon individual rights and liberties. And those are embedded in our constitution, and they're embedded really throughout our common law, that every single individual gets to have a determination about whether or not he or she shall be granted bail, he or she shall be incarcerated, is convicted of a crime, et cetera, et cetera, based upon their individual facts and circumstances. AI tools, as we all know, if you think about them, default to patterns and they default to what I consider to be a utilitarian theory of justice. And so when you're looking at large data sets and you're looking at what comes out of a large data set to predict the behavior of an individual, you're taking a utilitarian theory of justice and you're then applying it to an individual. So I do think that uh that is something that we have to very much uh keep our eye on because I think that they're the AI practitioners, uh lawyers, criminal, those involved in the criminal justice area ought to be keeping their eye on the fact that as we use these tools, we are trading certain efficiencies, perhaps. We're trading certain kinds of information for theories of justice that are embedded in our most fundamental American legal constitutional doctrines. And if we're going to make those trade-offs, we ought to understand that we are making that trade-off. And we ought to understand that there's a cost to that. We ought not to blindly walk in and use AI tools that are based upon patterns and utilitarianism and only be thinking about efficiency and only thinking about whether or not that particular tool happens to spit out an output that we can live with.

SPEAKER_03 1:02:12

I think that's part of what worried me about the way AI tools came from software, and courts started to use them without necessarily knowing that they were AI tools. How do we get people to be more cognizant of the fact that AI tools are being used when courts are not required to disclose that they're using them?

SPEAKER_00 1:02:33

I think that the public uh is becoming increasingly aware of AI today. I think ChatGPT, if I might, just sort of for a moment mention the uh the Chat GPT uh sort of boogeyman that's everywhere these days. You know, it's uh everywhere you go, you hear about it, but it really has raised public awareness to the utilization of AI. And I know that I speak to uh judicial officers all over the country about these issues, and others, I am sure, are doing the same thing. So we've got to have public discussion of exactly these issues, and with that public discussion will come increased public awareness. And with increased public awareness, we will then have the ability to have the judges who are then deploying these tools be able to uh understand what it is they're being handed and what the shortcomings are. So I think it's all about getting this out front and center in the public discussion, just like you folks are doing today.

SPEAKER_05 1:03:33

Uh Judge Forest, this has been absolutely fascinating. Uh, as a as a nerd uh computer scientist fine, uh, who attended exactly uh one semester of law school. This this is just fantastic. Thank you so much for your time, and thank you for writing the books that you're writing and pursuing this topic because, gosh, we really kind of need to. So thank you again.

SPEAKER_00 1:04:03

Well, thank you, folks, very much for having me on the uh the show today, and I I really appreciate it. I appreciate the opportunity to uh put some of my voice into these very important issues.

SPEAKER_03 1:04:20

We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention a specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.

SPEAKER_02 1:05:22

Thank you for listening to the ABA Business Law Sections Podcast Series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.