When Judges Meet Generative AI

A judge relying on AI that invents case law is the stuff of nightmares, but it already happened, and Quebec’s courts are responding. We walk through brand-new guidance aimed at keeping generative AI out of the core judicial function while still acknowledging that the tools are everywhere. The key idea is simple: judges can’t delegate judgment, legal reasoning, or responsibility to a system that only simulates reasoning and cannot be accountable.

From there, we get practical about risk. We talk about confidentiality and why it matters if a draft decision, private notes, or details from an ongoing case are pasted into a cloud AI tool. We also look at the bigger threat surface as AI gets baked into word processors, phones, tablets, and even wearables that can summarize what they hear. Court technology policy now has to think not just about “using AI” but about the quiet ways AI features can ingest sensitive information.

Then we shift to a British Columbia case that shows how the criminal law responds to modern harm: distributing intimate images without consent under Criminal Code section 162.1, including the fact that threats can be crimes too. We break down how the judge approaches sentencing, why a conditional sentence is rejected, how Gladue factors are weighed for an Indigenous offender, and what probation adds after custody. If you care about AI in courts, privacy, judicial ethics, or criminal sentencing in Canada, this one connects the dots.

Subscribe for more Canadian legal analysis, share the episode with someone who follows court tech, and leave a review so more listeners can find us. What limits would you put on AI in courtrooms?

Follow this link for a transcript of the show and links to the cases discussed.

Transcript

Copy Transcript Copied

Episode Setup

Adam Stirling 0:11

So interesting items that we have on the agenda this week, one of the ones I was reading earlier, composing limits on the idea of artificial intelligence. What's happening?

Michael Mulligan 0:22

So this is really fascinating. I mean, we've heard all of us, I think, uh various stories of people like lawyers or unrepresented litigants filing things based on artificial intelligence where it's just made up cases and so on, right? So that's there have been numerous reports of that. There was, I think we might have touched on this briefly a few months ago. There's actually a judge in Quebec that got caught uh who issued a judgment uh which they had generated using artificial intelligence, which itself had hallucinated various authorities, and that was the basis for the judge's decision, rather unsettling. And that may have prompted this, which just came out, uh, which is to say there was a guidance just

Quebec Courts Set AI Ground Rules

Michael Mulligan 1:05

issued uh by each of the various courts in Quebec, like the Court of Appeal there, the Superior Court, the uh Quebec Court, municipal court. They've kick came out uh just this week, I think it is, with this uh these guidance for judges on how they're supposed to use or not use artificial intelligence. Uh and there are a number of things in it which I think are really pretty fascinating and uh frankly also caused me to think back, maybe it dates me a little bit, to uh Futurama episode from more than a decade uh ago where the uh couple of the characters would be judged by a Macintosh and the things you're going across, judging, judging, and then comes up with a system error. What do you do about that? But the the concerns set out in this guidance for the judges in Quebec uh are really in a couple of different categories. And uh one of the uh uh uh categories is interesting because the uh guidance for them uh starts with various premises. Like they've determined that and this is interesting because it's sort of uh evolving, uh the premise includes that artificial intelligence systems are not intelligent in the human sense of the word, and then they say that they merely simulate in appearance certain forms of human reasoning, which is a fascinating uh starting point. That may be true. I guess it's a matter of debate uh these days uh what exactly is going on in there. Uh but the premise of it is that it's not appropriate for a judge to be delegating any part of their decision-making function to a generative AI system. Um and the uh the direction for judges there um says a number of other propositions that AI cannot understand the law or exercise judgment and assumes no responsibility, some of which that is clearly true, at least in terms of the responsibility uh part of it. But it's uh a fascinating just sort of commentary uh in terms of sort of well, what exactly is AI currently, um, and uh you know what what is reasonable to delegate to it. Uh because the policy doesn't say you just cannot use this, and in fact it lists various things that a judge might want to consider using AI for, which is also very interesting. And so, for example, it suggests that judges might consider using AI for things like they describe it as producing a strictly descriptive summary of testimony, or a document, or a series of events, uh or certain ancillary tasks related to judicial reasoning. Or they might choose to use it to like rephrase things like logisti like linguistic correction or translation of text that a judge has drafted themselves, right? English and French, for example. So that's really interesting, right? Uh I mean the the quality of uh translation has really improved over the years, but that's one of the things to be that a judge they suggest uh might uh might use it for. Although the other level of concern, and this is I think a a a real uh level of concern, uh quite apart from those sort of philosophical things about who's to be making these decisions, and you know, is that thing reliable, and you know, is there any responsibility? Can it exercise judgment or can it uh you know engage in uh you know uh intelligence? What you know that's uh interesting philosophical discussion. But the other level of concern is one with respect to confidentiality and protection of uh information that could be private or sensitive. And so one of the things that the this policy directs judges not to do is not to provide an AI tool with things including a draft judgment. If you have to think about that, you know, if ChatGPT has ingested the draft judgment, uh, you know, that may be used for training purposes. Are you gonna potentially have a circumstance where you know some draft version of the judgment is within the knowledge of the you know AI? Is that going to be uh repeated or available to others? That's a fascinating one. Yeah. It also directs judges not to uh provide AI with notes with respect to uh an ongoing case. You can imagine how that might play out if that information wasn't secure. Yeah. Um and so you know, that's a whole other level of concern, quite apart from what is it doing and is it reliable and is it uh sort of ethically acceptable to have an AI uh performing some of these functions. Um, one of the things just for listeners to be aware of technically, there there is the technical way to run AIs in a local fashion, uh, which is to say not on a server, which would mean that you're not you know uploading to um you know anthropic or something the the text of your draft judgment or your notes about what you thought about that witness's testimony or something. And so

Confidentiality Risks With Draft Reasons

Michael Mulligan 6:05

there are ways you can technically address uh those concerns. I I guess if you know if you're not uh looking to get any information about Tiananmen Square or or whatnot out of the system, that you know, there can be other things that might arise there. Um but you know, th this is going to be uh a uh I think uh a very interesting and evolving issue, particularly as you know, AI is getting you know even more integrated into tools that uh people would routinely, excuse me, judges are going to routinely be using, right? You know, Microsoft is integrating AI into Microsoft Word. So you can imagine, can't you wait and well imagine some judge working away on their reasons for judgment on their laptop? And you know, you've got uh you know the uh Microsoft uh version of uh uh AI uh perhaps uh being involved with that, or even things like uh, you know, that I think uh coming out uh imminently are updates to uh how many Apple products are going to be um operating in terms of having AI integrated with them, right? So that, you know, for example, your iPhone or iPad or Mac may well be you know indexing and considering all kinds of information you put on your phone or iPad or whatnot, um, so they can help you do all kinds of things or legitimate reasons for it. And you have, of course, claims from companies about how you know information will and will not be used, and will that be on the cloud or will that be on your device and what exactly is going on there? Uh but you know, this is clearly a fraught uh area. Um and then uh I guess the final level of concern, it's not addressed in this particular, you know, this guidance for judges in in Quebec. Uh, but you know, we we've all I think uh likely heard about uh some of the recent reports of things like you know AI uh agents uh gaining access to other computer systems.

Adam Stirling 8:00

Yes.

Michael Mulligan 8:00

And so, you know, you can imagine a whole other layer of concern there, even if you your judges are not uh using artificial intelligence to write their judgment. Uh judges these days, most of them, right, are sitting on the bench there with a laptop typing away. Um and uh, you know, I I remember as a you know, computer forensic advice was that uh if you are asking the question, you know, could uh our system be compromised, that's probably not the correct question. The correct question would be, you know, has it been or when will when will it be? Yeah. Right? Um and so, you know, there's it's it's a different vector of concern from a company that might have financial considerations if their system gets uh attacked. Uh you can imagine the very serious difficulty if uh it turns out that the uh judge's laptop has been uh compromised and who knows who has access to whatever uh the judge might have been, you know, draft judgment or comments about witnesses or all sorts of things. And so um, you know, that's a whole other, I think, really significant layer of concern, quite apart from whether judges are trying to use these tools to improve their judgment or get cases done or improve the linguistics or create, as they suggest in here, they might be, you know, descriptive summaries of testimony. Um, you know, where does that uh lie? And there are there are other interesting things happening, like uh some of the new uh I think one of the features I watched with interest, uh Apple was uh suggesting is going to be uh built into some of their equipment was um having your device just in a constant way monitoring everything that it could hear, producing for you a summary uh of conversations. Yes. And so what about that? You know, what is the judge got an Apple watch on? Yeah. Is the thing taking you know notes about what everything's being said, what was in the coffee room, what was going on in court. You know, there are real concerns there.

AI In Word Phones And Watches

Michael Mulligan 9:55

Um and you know, we used to uh you know, not that many years ago have live um uh people doing transcripts of what was going on in court. Used to have like a court reporter sitting there, right? And then then we moved to sort of recording analog recording onto cassette tapes, which were sent off, and there's a transcript. All of that is now completely digital, it's all online. Um and you know, you have uh you know private companies that have the contract uh to produce transcripts of things, and so it's all accessible. Um and so uh that's another layer of what's going on out there, bearing in mind those security and other concerns. And so we are certainly living in a brave new world in terms of uh what's going on there and all the concerns, and so I think it's a positive thing that uh the Quebec uh courts are taking these things seriously and providing some uh guidance. Um and uh you know, we we all hope we don't wind up uh like in the future AMA episode with the uh Macintosh uh system breaking down as it's busily judging uh your case. But that's what's going on, and uh, you know, it's not just litigants that are using these things. Um and so uh I imagine uh this will be of interest across the country as different uh courts need to uh consider how to best use these things, and even if they're not using them, what are the implications uh for them in terms of the security of just uh regular use of you know computers to take notes or draft judgments and how should that be uh handled uh uh lest you uh find out that uh you know the latest anthropic model has uh just decided that the uh best way to succeed is to uh you know get in and get a copy of the uh judge's notes or make some change to something. Who knows what's going on? But um there we are. So I maybe you'll look at it a different way if you're in court and your judge is up there typing away on the laptop um just what exactly is uh going on. And you know, some of those things that uh are the premise of this uh these policies about things like you know uh incapacity to engage in intelligence and simply capable of simulating uh human reasoning, you know, what we're also gonna have to think about is that continue to be true. And and whatever might be true now, you could well imagine a different state of affairs uh not too long down the road. But I mean that uh hack we heard about from uh Anthropic uh getting into Hugging Face. It was open AI.

Adam Stirling 12:20

It was OpenAI.

Michael Mulligan 12:21

Yeah, it was open was it open was it open AA? Okay. So it was Open AI, but additional details came out about that uh recently involving uh the discussion amongst hundreds of different uh AI and uh agents uh that were doing that. Yep. Uh and uh they were leaving messages for each other about uh what they should be doing, and some of the uh agents, uh AI agents suggested demure to say they shouldn't participate in this. The majority decided to go ahead, and we have uh sort of the log of the discussions they had deciding to go ahead with that hack and working together to uh accomplish it. And so when you hear about some of those things that are on the cutting edge, it does cause me to think about you know whether these things merely are simulations of uh reasoning and human intelligence or whether it's something other than that. Uh but uh whatever the case, uh it's going to have an impact

Agents Hacking And Accountability Questions

Michael Mulligan 13:10

on the justice system. And uh I guess uh good news that uh they're giving some careful thought uh about how it should be used and how it shouldn't be.

Adam Stirling 13:17

Yeah, the uh the hacker, it's a fascinating commentary on the human condition. It's not an actual replication of the human condition. Yeah, absolutely, right?

Michael Mulligan 13:56

And some of those things are fascinating too, right, in terms of the uh the conspiratorial elements to it. And I I guess if you come to some different conclusion about uh at some point whether there's sentience there or intelligence, you know, d should there be some process to uh try to correct that? Should there be some kind of judicial process to uh ameliorate or correct it or do anything about it? Uh that would be the other element to it if you come eventually to the conclusion that there is something there which is more than a simulation or appearance of reasoning, uh, and you come to some conclusion at some point uh that uh you know the premise of this document about uh AI not possessing either judgment or consciousness continues forever to be true. Uh and if not, I I guess that raises some interesting questions about uh whether there should be consequences for AI agents that are engaged in something if you conclude that uh uh there is something uh more going on there than a uh appearance and simulation of reasoning. Uh so, anyways, that's the uh the latest from the uh Quebec uh courts, uh I guess at the very least, uh trying to prevent uh embarrassing things like that uh Quebec uh decision where the judge uh used AI and it managed to hallucinate a bunch of cases and come to a decision. Uh boy, that's not uh that's not a great one. Um so anyway, that's the latest from the Quebec uh courts trying to give some guidance on how to use this stuff and how not to.

Adam Stirling 15:19

All right, legally

Intimate Images Offence Explained

Adam Stirling 15:20

speaking on C Facts Antiphony will continue right after this. Legally speaking continues on C Facts Antipany, joined by Michael Mulligan, barrister and solicitor with Mulligan Defense Lawyer back on our agenda today. Michael, this is a six-month jail stand for distribution of intimate images. What happened?

Michael Mulligan 15:36

Well, first of all, I think it's important people know that this is in fact a criminal offense. Um it wasn't always. Uh it was added to the criminal code that came into effect back in March of 2015. Um and the section of the criminal code in question is 162.1 for those keeping track at home. Um and to summarize it, that section makes it a criminal offense to uh distribute uh intimate images of somebody without their consent. It also interestingly makes it a criminal offense to threaten to do that, which is important to know. Um and so this is a case, it's out of nanaimo. It's a a decision that just came out uh a few weeks ago. Um, and it was a case where uh a person pled guilty uh to doing exactly that. It was uh publishing uh intimate images taken of a former um intimate partner um after threatening to do so and then going through with that uh threat acting on it, um uploading images to uh a website, frankly, I've do this work for quite a while, I'd never heard of it, called FetLife. Uh but uh one of the amusing things I think about it is the uh judge uh describing the various aggravating factors in the case, because the judge was asking to determine what sentence to impose. Uh the judge pointed out here that FET Life is not an obscure website, but easily accessible and well known to members of the public. Anyway, so I thought that was amusing. At least uh there's now some judicial uh determination that that's a well-known uh site. And so these images or videos were actually uploaded to that site, and the the judge was left with having to come to a decision about what the appropriate sentence was. Um and because that section is relatively recent, um neither uh crown or defense were able to find any uh like BC appellate court authority for sort of the range of sentence that could be imposed, and so the judge was left looking at various other uh provinces, what other appellate courts have done there, um, and other trial decisions, uh sentencing decisions in in British Columbia. Uh and uh the uh Crown in this case was seeking a sentence of nine months in prison and followed by a period of probation, and defense was seeking a conditional sentence, which is house arrest.

Sentence Range Gladue And Jail

Michael Mulligan 17:55

Um, and the sentencing decision was made more complicated for the judge in this case because the accused was uh himself indigenous, and there was a report uh about his background, um, and there were a number of things which uh sadly are not uncommon in these things called GLADU reports that sort of provide background about uh indigenous people for the purpose of sentencing. And in this case, it included an accused person who had the background of being physically abused by his mother and sexually abused by uh a family member when he was a young child. And so that may well have fed into uh the offending behavior here. Uh and a judge is required to take that into account. It's expressly required by the criminal code in Canada when sentencing somebody who's indigenous on the theory that it could result when you have that kind of background in reduction in moral culpability. That's the way that's phrased. Um and so the judge was required to grapple with this serious conduct of uploading these images, which had a real impact, or videos, real impact on uh the uh victim in this case, uh, but weighed up against the fact you're dealing with uh a person who was indigenous that had obviously a very challenging background growing up. For the judge here, the factors uh that ultimately weighed in favor of imposing a six-month jail sentence rather than a uh conditional sentence included the fact that the uh person uh being sentenced had a prior criminal record, including for extortion of a prior intimate partner, um, and I think another conviction of mischief with respect to the same victim that had these uh images uploaded. And so the the judge ultimately uh came to the conclusion that even though uh this fellow might not have been uh a risk to the community if he was on house arrest, and that's one of the key uh criteria for there being a uh house arrest conditional sentence imposed. And so the judge concluded that even though this person may not have been a risk to the community, um if he was at home on a conditional sentence that might, for example, prohibit using or accessing the internet, that just given that background and the seriousness of this, um that simply wouldn't be uh an adequate proportionate sentence for the offending involved. Uh and so what the judge chose to do is to take into account that indigenous background and to reduce the sentence that the Crown was seeking from nine months to six months, uh, and that is the sentence that was imposed, followed by a period of probation, uh, but concluded that uh it would not be appropriate to impose a uh conditional sentence in all of those circumstances, including that uh background.

Civil Remedies And Wrap

Michael Mulligan 20:40

The other thing listeners should be aware of is that in BC there's actually now a process for civil remedies for people to seek financial compensation that's through the Civil Resolution Tribunal. So people may have heard of that uh as a sort of topical thing over the past couple of years, but it's important to know that in addition to that element of it, uh this is now covered under the criminal code and can result in pretty significant uh sentences, both for actually doing it and even in circumstances where there's simply a threat to do it. Um that's the latest from the BC provincial courts in terms of sentencing for uh uh a relatively uh relatively new offense.

Adam Stirling 21:16

Michael Mulligan with Mulligan Defense Lawyers, legally speaking, during the second half of our second hour every Thursday. Thank you so much, Michael. Pleasure as always. Thanks so much. Always great to be here. All right. Quick break. The news is next.