Olivia Dhein: Lawyers, AI, And The New Non‑Negotiable

We chart how AI moved from “innovation” to a non‑negotiable capability for practising lawyers, and why choosing the right use case matters more than hype. Olivia Dhein shares how Big Law deploys enterprise‑grade AI, trains teams, and balances risk with results.

• defining AI as business continuity not innovation 
• mapping tasks to model strengths across drafting, review, translation, research 
• larger context windows raising draft quality and speed 
• hallucinations trending below human error on bounded tasks 
• human‑in‑the‑loop risks and measurement bias 
• confidentiality solved via enterprise tools, not consumer chatbots 
• client demands for responsible use and AI‑checked advice 
• training juniors for verification, prompting and judgment 
• future signals: reasoning research, MoE, multimodality

Transcript

Welcome And Guest Intro

Barry 0:07

Hello and welcome to the Garfield Podcast, Litigating AI, a series of conversations with the people who are working with AI to improve access to justice. In this episode, Garfield founder and CEO Philip Young talks to Olivia Dyne, the practice gen AI integration lead at Baker Mackenzie. The pair discuss how lawyers in the big city firms are deploying AI, the risks and rewards of AI in law, and even whether AI might help lawyers better understand their own decision making.

Philip 0:43

The purpose of this podcast series is to talk to lots of really fascinating, interesting people that I've met along the way in my AI journey, and they range from policy makers such as politicians, people who are founders who are building AI products, people who work in law firms who are very much working on the frontier of how AI is being integrated into law firm processes, and many other people besides. On today's episode, I am honored to be joined by Olivia Dine, who is the Practice Gen AI integration lead at Baker Mackenzie here in London, and has many other interests in AI and law, and also has over the last few years shown an uncanny ability to predict the future with amazing accuracy. Although she's yet to be persuaded by me to let me know what next week's winning Euro Millions lottery numbers are, unfortunately. So without any further ado, let me say hello to Olivia. Hello, Olivia.

SPEAKER_01 1:46

Hi, Philip. Hi, thank you so much for having me. I'm really excited to do this.

Philip 1:50

Well, thank you very much for joining me. And I should have also said that you have many other strings to your bow, but uh another one is that I believe you're a guest lecturer at King's College London.

SPEAKER_01 2:00

Yeah, that's right. Recklessly, they let me talk there about AI and law as well, um, which I love doing because you get curious students asking you interesting questions.

Philip 2:10

Such as what does the future hold, which is something you you uniquely know the answer to. Um where we normally speak with these podcasts, Olivia, is we normally start with people's backgrounds. So would you like to just take a few minutes and sort of sketch out your background, your journey, and how have you reached the the place you're now in?

SPEAKER_01 2:28

My background is as a dispute resolution solicitor. So I trained and practiced in the city. I was very much doing financial services disputes, commercial disputes, financial services investigations, and so on. And um, I certainly never gave a thought to AI while I was doing all of that. I was just uh doing my thing. I then moved into more of a knowledge role. I worked at Lexus Nexus, and then after that, I moved to another law firm to be a knowledge lawyer there, and very much black letter law, you know, building up their knowledge function. It was great fun. I really, really enjoyed it. But someone made the fatal mistake of saying to me, Olivia, make sure you know what's on the horizon. And then I did, and I started looking. And the other the other big coincidence that got me into this is that I went to a party because my neighbours had a party. And I happened to be Yeah, I happened to be chatting to some of my neighbors who all happened to be software engineers in their 20s, and we were having a chat about AI because I had seen something. I was like, oh, I think I really need to know about this. And um one of them kindly worked with me to explain what large language models are to me, and I basically fell off my chair and decided that was the thing I really needed to look at and think about for disputes, initially for disputes and then later for law.

Philip 3:54

It sounds like that party was quite a pivotal moment then in your professional career to date.

First Encounters With LLMs

SPEAKER_01 4:00

Yeah, exactly. So I think I it's fair to say I got a bean my bonnet about it and decided to first learn everything I could. Um so I went out to meet people, I read everything I could, and my friend obviously was teaching me different things um as well, which is very nice of him. He's now in San Francisco doing a startup, obviously, so he's very successful doing other doing other things, nothing to do with law. And I very quickly noticed that not many people were paying attention to this at all. So everyone around me thought this was, I think, maybe a slightly peculiar hobby I was getting into. Whereas I thought it was a really fundamental shift and how we were going to do things. So I just kept going to events, then I started speaking at events, which was um very nice of some people to let me do that and just put me on panels and so on. And then eventually I decided that I would approach the editors of the white book because I felt that the civil procedure rules needed some commentary around this as well. And they very kindly let me um write that commentary for them. So that's something I do as well now, so on the side.

Philip 5:07

Just going back to the party, uh not not not because it's a party, though that's inevitably interesting, but um roughly when was this? Because when LLMs came across my sort of knowledge, was sort of 2022. Um was it roughly the same time for you, or you were you even earlier?

SPEAKER_01 5:22

It no, no, no. It was early 23.

Philip 5:25

Okay.

SPEAKER_01 5:25

I had sort of read a little bit about it at that point, but I thought actually think that it was a bit of a marketing issue because obviously the press referred to LLMs as chatbots. Yes. And I think it made everybody think that they're these, you know, those customer service chatbots, you know, sort of this tiny little thing that doesn't really, it's just a tool. And I certainly fell into that trap because I thought, okay, it's a chatbot, okay, quite interesting. But I remember reading in the New York Times, I think there was that journalist who had this very existential um conversation with with ChatGPT. And I thought, oh, this is this is something different. And yeah, so it was early 23.

Philip 6:00

I think that's interesting, isn't it? Because a lot of us who are doing what we're doing now, we sort of spotted the sort of the moment in the sort of 2022-23 time. And I I can still vividly remember the first time I had a bit of a play with um, it was ChatGPT 3.5 then. And um, I was I knew I wasn't expecting very much and I was blown away. You know, it's just like the first time with an LLM, it's like, what is this?

SPEAKER_01 6:24

Yeah, no, absolutely. I was sitting actually right in this chair in this spot when I first tried it out, and I called my mum that evening and I said, Oh my god, like this is this is unbelievable. Because I knew it was a general model, but I was asking it legal questions just to see, and it gave me something that was uh even then it was at the level of a trainee. I was just blown away by it, and yeah, yeah, all started from there. But you were ahead of me, Philip. That was just a polite way of saying you were way ahead of me.

Philip 6:54

Only only by a couple of months, not very much. And I I I I sort of uh I sort of on the topic of mums, I I think what's interesting to me is looking at the adoption. I noticed uh over Christmas that my parents, who I mean, I'm no spring chicken, but they they're in their 80s. I'm sure they wouldn't mind me saying that. They've now got Chat GPT on their phones, and I was noticing them consulting ChatGPT about some run-of-the-mill life things. And I thought, my goodness me, if if this technology has now reached a point where people in their 80s have sort of adopted it, then it's gone absolutely mainstream everywhere.

SPEAKER_01 7:27

Yeah.

Philip 7:28

Which I think is probably the case, isn't it?

Mainstream Adoption And Legal Mindset

SPEAKER_01 7:30

Yeah, yeah. I people may have heard of ChatGPT, but they certainly wouldn't know what an LLM is. But now, about I think it was about a year later, I started to hear people talk in the street about this topic. So I was sort of fully immersed at that point. And um, but it I I was suddenly like I I would hear people say in a cafe, or like, oh, I asked ChatGPT this and that, and that certainly happens to me almost every day now here in London. I think it's maybe ever so slightly different in the legal profession, but the mindset I think across the board has changed quite a bit, quite quickly.

Philip 8:02

Let's talk a bit about your role at Baker McKenzie at sort of a high level. Did you want to just quickly sketch out what it is you do at Baker McKenzie? You know, the sort of work you're doing to integrate AI into the firm's operations.

SPEAKER_01 8:15

Yeah, I mean, I'm I'm really lucky in a way because these kinds of jobs didn't really exist very much. So um, when I was looking for an AI role, I remember I spoke to a recruiter and I sort of described what I'm doing now, and they said, What is this? We have never heard of this. It's a new job. This doesn't exist, you know. It does now. It should, it should. Yeah. And then um lo and behold. So um so I was basically looking to go full-time AI at because they were just thinking about this kind of role. So they have obviously been in the AI game for a long time. I think they've had an AI strategy since 2016 or 2017, and they have the applied AI team, which does specific customized AI workflows for clients. But this role was actually about working with the lawyers inside Baker Mackenzie and looking at what should be adopted, how, or what lawyers should do with it specifically across the practice groups. And my role is actually global. So even though I'm a disputes lawyer, I now sometimes feel like half a transaction lawyer or a quarter of a tax lawyer because I speak to those teams about what they do. But interestingly, this stuff does um translate across. So when you mix AI and law, the principles do sometimes pop up. So real estate might have something it has in common with disputes, and disputes might have something in common with MA when you think about how do I get a model to look at documents and how do we do that in a meaningful way for the lawyers. Also, looking obviously at training um lawyers, getting them excited. Um I I try to be very enthusiastic, which is easy to do because I think it's such an important exercise. So, and the other thing I've also been looking at is obviously risk and you know, when we're looking at new tools and working with our risk team to decide what tool works, what questions we need to ask. Um, I think the the providers generally have that covered, but it's just important to know what to even ask about. Or I might work with our data privacy expert and just help on the AI side, even though I'm not a data privacy expert, together we can usually work out any questions that might arise.

Philip 10:22

I've always thought that legal work, but probably life more generally as a whole, it's often knowing what questions to ask, not necessarily knowing what the answers are, because if you know what the questions are, at least you can find somebody who knows the answers or find them yourself, you know.

Inside The Baker McKenzie AI Role

SPEAKER_01 10:34

Oh yeah, yeah, yeah. That's a really good way of looking at it. And I think it's but with this stuff, I do find it's it's actually quite challenging because a lot of the concepts are just it's just new. Regulation keeps popping up all over the place, that's all new. It's definitely demanding.

Philip 10:50

Absolutely, and right at the cutting edge, very much frontier work, so to speak. So perhaps we could talk a bit more generally about what large law firms are doing in terms of AI, you know, around mapping processes and advantages and challenges. What's your sort of general perception at a high level? Do you think Bakers is sort of doing the same things as other firms, or is there a sort of general theme? What are your views on this?

SPEAKER_01 11:12

I certainly joined Bakers because I thought they were quite forward-thinking on this side. So it's a very active consideration all the time. So the first thing they did, and this is public information, is rollout co-pilot for the firm, which obviously gives you a lot of flexibility because it can access all the M365 data you have, and then there's just very active consideration of what else might be needed. I have noticed that the market is a little bit split. So there's some other firms who also have these very active statements on they're doing things, they're looking at these tools, those tools, and then there's a sort of silent other half where you don't say very much at all. So I'm I I do wonder, I think um it does seem to be a little bit a function of the risk-averse profession. So um I think there's maybe firms who are really sort of jumping onto this because they know it's really important. And I don't particularly see this as an innovation topic. So people always talk about it as oh, this is our innovation topic and this is what we're doing. And I think it's fair to say at Bakers it's more of a this is where it's going, this is where it has gone, this is what we need to do, this is part of what you need to be thinking about carefully, thoughtfully, responsibly. But if you are a lawyer in 2026, this is kind of a non-negotiable.

Philip 12:30

Yeah, exactly. It's interesting you mentioned that. So I had a couple of conversations last week with two different people at two other large law firms. Um, and one person was saying to me that, like you, they no longer see it as a sort of innovation topic, they see it as a business continuation topic. In other words, they just have to have this and they're just working on how to do it in the best way possible. And then the other person um said that they regarded it as a business survival topic. They said in order to continue to survive as a business um and not to be out competed, they had to integrate AI into as many workflows as was sensible to do so, obviously in a in a responsible way. And I thought, gosh, that's a that that's a heck of a statement to describe it as a survival topic. Now, maybe that person was overstating it, um, but clearly they were concerned that if they didn't use AI intelligently, then they would just be out competed by other law firms.

Market Split And Risk Culture

SPEAKER_01 13:26

It is interesting though, because some firms don't see it like this at all. They are very sort of cautious and you know, and the thing you always come back to is you have a human in the loop, right, to take everything. And um, we've been doing that with other humans for a very long time. So we let other humans do things and we don't know exactly what goes on in their black box brains, right? But then we just check it and where we've been happy with that for years and years and years. But um, there does seem to be a sort of I don't know if it's fear, hopefully not, um, of the technology, but people don't quite know what to do with it sometimes. Um, and they and and lawyers when I speak to lawyers outside of the firm, often I just get a lot of discussion about risk or perceived risk.

SPEAKER_02 14:11

Yeah.

SPEAKER_01 14:11

Um, which isn't it's not the whole equation. Well, you know more maths than I do, but uh Oh, I don't know about that, Olivia.

Philip 14:19

I think my my co-founder Dan might disagree with that one.

SPEAKER_01 14:24

Yeah, no, it's just this. Um, so it's always this this conversation about the risk of using AI, and that is sort of our mindset default as lawyers, because we we're risk management people, right? But the risk of not using AI is not even in the equation. And um that's what it does, I have to say, Philip, it does make me very nervous when people say to me, Oh, I'll wait and see, maybe, you know, when the tech people tell me to use this, or I tried it out and it and and it got it wrong. Even though everybody should know at this point that hallucinations does happen and you need to iterate and you need to to use it in a very thoughtful way to get the right result. But I do get very nervous when I hear this attitude still in 2026, when I would have thought if everybody's talking about it in the street, you know, maybe lawyers have, oh, maybe you know, I should should have a go. But yeah, but I'm still meeting lawyers who've never used ChatGPT.

Philip 15:17

Yeah, so I I still meet a few. And what I've noticed is that um, so obviously do a lot of demos of Garfield, but um, most people now, the reaction isn't the same as when we first did demos um a year and a half or so ago, where at that point it was sort of you could drop a brick in the room, everyone went silent, and jaws were hitting the floor. And now that's sort of slowing down a bit. Now some people look shocked, and some people like, oh yeah, yeah, this is really cool, and it looks, it looks like it's helpful and does something else that's uh adds to our sort of repertoire. But yeah, definitely there's still some people who they almost can't believe it. They come up afterwards and say, Goodness, you know, this is going to change the world, and we're like, Yeah, it will. Um, I think the interesting about new technology is that you always get a range of reactions. Humans are just humans, aren't they? And uh it was like, um I think I I give this sometimes as an analogy that when they did the first trials of steam engines, uh, was it the Rainhill trials in somewhere like that? I might probably got that wrong, probably made a terrible historical error there. But some people were saying steam engines will never work, you know, that literally no one can travel at sort of 35 miles an hour. You'll be I think the quote was you'll be blown to atoms. In the same way that there's still people that are denying that that we landed on the moon. Even though if you've got a powerful enough telescope, you could actually see the um the landing sites, you know. It's uh there's always going to be some people who are like, oh, technology, or you know, no, no. We do it. But I I think this leads on to sort of a really interesting topic, which is sort of misconceptions about AI held in the law. And I mean, this is one of my bugbears, but I think it might be one of yours as well, Olivia. So do you have do you have some particular thoughts on this, on this topic?

Risk Of Using Vs Not Using AI

SPEAKER_01 16:55

I do have a few. Well, I have concerns because there's just a few points where I just wish that, you know, it's very much 2026 and it's not 2023. And I just wish we um I don't want to blame anyone for not knowing because that there's nobody's done anything wrong. But I just wish that people had maybe moved on in the in the general conversation and the general discourse just tends to get a bit stuck sometimes on certain topics. So I think the innovation topic one is one. People think it's oh, it's innovation and it's one of many, many, many things. You know, it's a bit like crypto, and you know, with all respect to my lovely colleagues who do crypto, um, crypto work, um, which is super interesting. But you can live your whole life and not deal with crypto, but you cannot, absolutely cannot ignore AI in law. I mean, as in your profession, you will not be able to avoid it. That's just, I think, a bit naive to think that you can just surf that wave and not not just not be caught up by by AI. No, definitely. The other one that really I think needs more discuss is uh junior lawyers. So I don't know if you've heard this, but I have heard at least 15 times in discussions, panels, and our junior lawyers they will not know how law works, they will become very lazy, they will not learn attention to detail, but they really, really need to learn it the way we learnt it. And I don't agree at all with this. I think that I was, if I may say so, the most exceptional proofreader as a trainee who found every semicolon that was out of place, and um I don't think it made me a better lawyer at all. Um, I think it was something you had to do, it was something that was highly valued at the time because work product had to be polished, and that was the way we could get that done. I don't think I needed a law degree to be able to do that, and it didn't help me negotiate anything or draft better in any way, really. I think that the truth is that junior lawyers are actually in a particularly demanding position. So I think this is the first generation of lawyers now who have to very actively think about what they do. So they have to develop a special skill, and we can't teach it to them because we don't know either. So they're more senior members of the profession. So they have to learn and develop the skill of what do I need to learn and which bit do I do manually, the the old-fashioned way to make sure I really get it, and which part of it must I do with AI because it is efficient and it makes sense to do so. So, for example, if you are summarizing something nine times out of ten, I would say probably AI is something you should use. But if you're doing legal research, I would probably say to a junior, this is the sort of thing you should probably learn to do manually. So you really know how it works, but then you not only need to learn how to do it manually, you then also need to learn how probably later you can check an AI-generated legal research. So you need to learn how you do it, you need to learn how the AI will likely do it, how the AI may make a mistake, and how you find that mistake. So, you know, if anything, the juniors are now in a position where, you know, they have to do way more than we had to because they have to do it all at the same time. And we're not really there to tell them exactly how to do it because we're working it out, we're we're building the plane as we're flying it.

Rethinking Junior Lawyer Training

Philip 20:18

No, I agree with that. And I I one of the things I've been saying is that I think junior lawyers are going to have a greater challenge in some ways because they're going to be expected with the use of AI to do tasks that are a lot more, you know, strategic and complicated. They're going to be relieved of the sort of tasks you were talking about. And I remember those days where you sat in a room, there were two lawyers, you you went through each page in turn and proofread, and it was a it was a very expensive for the client, but unnecessary, but fundamentally low-value job. So those sort of jobs will go, and junior lawyers will be expected to do a lot more complicated things. I think on the point about um the right tools for the right job, I think that's sort of fundamental. So just in the same way that, for example, a carpenter is going to use a masonry drill bit in his drill to drill into brick and not a, I don't know, a wood drill bit, then as a lawyer, you've got to you've got to use the right tool for the job, don't you? And actually, if you don't, you're making a mistake. On that topic, I um I saw a LinkedIn post the other day by a Silk in the London market, and he made quite an interesting point. He said he'd had a bit of a tinker with AI, but he didn't really want to use it in his work, particularly to draft submissions, because he felt that as an advocate, you had to have read all the papers backwards in order to draft the submissions. And I could see his point sort of, but I also have to say I came away feeling, well, you can use AI to do a first draft. I agree with him that as an advocate, you've still got to know the case backwards, you still got to read everything. But you could use it in that context as a time saving tool. You could use it to prepare a first draft, and then of course, you can go through the draft and revise it as you think appropriate, and hopefully save yourself quite a bit of time and hopefully the client some money. Uh, but anyway, yes, um, I think you might have had a few other bugbears that. um you wanted to chat about.

Choosing The Right Use Case

SPEAKER_01 22:02

Yeah, so it's a little bit along those lines actually. So what I try to do when I think about what AI are we using for what is I start from the model and then I think about the tool and you know the sort of more practical things. But I start from the model. So I think about what is the lawyer doing and then I think about what is the model good at. And a lot of people don't seem to or they seem to forget it that models are good at different things to humans. So they just sort of throw the kitchen sink at the model and then the poor model's like, oh I don't know. Some sort of sycophantic answer comes out. But if you think if you split it up so if you categorize a little bit what you do. So what I do in my head is say a lawyer comes with some sort of workflow. I split it up in my mind for example into is this drafting is this querying a document set? Is there some translation involved for example because in a global firm that that might be the case. And depending on what it is, then you can advise whether and how you should use AI to do it. So for example, in my spectrum legal research is right at the most difficult end. So even the most advanced reasoning models they can do some legal research for you but not that really complicated stuff that city firms get paid to do so that there's no chance they just can't do it. But drafting is right at the other end of the spectrum. So drafting is something that these models can do really well and they can certainly spit it out more quickly than I can type it even though I can touch type. So I actually think drafting is one of those really good use cases but you have to know what you're doing with the prompting and I agree with you obviously you shouldn't draft anything if you haven't read the underlying material or the documents but with the drafting it's just the magic is in the prompting because you can get it to write like you would sound and you can get it to you you know you might have to have a few standard prompts that you develop and you know take out your M-dashes and whatever you might want to take out the delve and delving into whatever it might be. But drafting I think is a really super super use case because you can individualize it. It's I mean the clues in the name is a large language model it knows language really really well and you can you make you can make it your own really easily you can also exactly see what the model is doing. So even though it's a black box this is the safest use case in a way because you can read literally every single word that comes out and you know if you wanted to you could ask it to give you some sort of reasoning why it's put it that way or whatever it might be. But I'm a big fan of that use case and I think it's a little bit undervalued and people often go for the more difficult stuff which is you know find me the answer to this really complicated question in five million documents. We can sort of do that as well for sure but it's much harder and you know there's some acrobatics you have to do to make that to make that work. But um so yeah I think it's a bit of a shame um I'd love to speak to that person to see what they were trying to draft and why it didn't why they thought it didn't work. Because I think that's one of the best ways also to get familiar with how the model works.

Context Windows And Draft Quality

Philip 25:02

Yeah exactly exactly I mean there has been an evolution hasn't there so for example the context windows of LLMs have got bigger and bigger and bigger and that's been I think very helpful for lawyers and particularly for people who are not very familiar with how LLMs work because when the context windows are much smaller somebody who was new to an LM might try in one shot some gigantic corpus of information and then find that the LM does a relatively bad job because of course it's not been able to take everything into its context window. Or alternatively there was a long stream of prompts and responses and and material that was put in at the beginning dropped off the context window. So the LLM almost forgets what it was told. And then I I would occasionally and we're going back a a year and before that now I'd hear people complaining about that and I would say well no you've got to work with the tool you've got and you've got to appreciate that you've got to almost summarize things to get them into the context window if you're trying to do a really big sort of exercise. Now of course that's changing because one of the areas in which all the Frontier labs are competing is trying to make the context windows bigger and bigger whilst also retaining what I think they call attention, a very high level of attention so that all of the material within the context window is sort of gets the same degree of um you know bits of it aren't sort of lost sight of by the model in in drafting. I mean I I have to say this is not the way that we operate Garfield for various sort of architectural reasons, but I have to say that whenever I give one of the leading models a really big chunk of information and I ask for a document, I'm usually impressed with what it produces. Sometimes it's you know it needs some work but sometimes it's pretty close to being a respectable final draft that I'm happy myself signing off on. And actually this takes me to one of my bugbears so we've done some of yours now it's one thing. Hallucinations. So I think it's an important topic. Yeah okay okay well I get to steal this one Olivia can I steal this one? So it is an important topic uh everyone should be very alive to it but I also think it's a very time limited topic and I I think that um although the incidence of hallucinations will never drop to zero percent because LLMs are a probabilistic system, it's going to drop to point where the the incidence of them is far lower than the incidence of human error. And actually it's going to get to the point where LLMs are superhuman. And at that point I think hallucinations cease to be an issue. So I do think that a lot of the attention that's being spent on hallucinations is it's almost heading in the direction of becoming passe. I think we all know the principles you know someone's responsible for the work and you've got to check things and there'll come a point where even that's not required because these models are superhuman. But I'm beginning to slightly zone out of that discussion because I think that as long as you're alive to the issue, you've built safeguards it's going to disappear. What do you think? Are you the same mind or do you think that I'm being too forward thinking?

SPEAKER_01 27:46

No you would never be too forward thinking. I think that's the best thing about you Philip forward thinking but I can't I can't predict do you have any lottery numbers yet that's that's my next skill no I I think that's super interesting how you put it and I've not heard it put like that um mainly because everybody always goes on about fake case citations which is not an AI issue that's a human issue where the lawyer hasn't checked their work properly so it's really important that they do and bad consequences if you don't but it's not actually an AI point particularly if somebody doesn't do that. It's like doing bad legal research. But yeah I think it's interesting what you say about it almost becoming like below the rate of error that a human would make because it reminds me of a discussion I had with a colleague who was saying to me sometimes it doesn't matter so much if you have an AI product that's 70% correct or 85% correct because you still have to take the same amount of time to check it.

SPEAKER_02 28:41

Yes.

SPEAKER_01 28:42

But it's true once we get to 99% or whatever whatever we decide the threshold is it's true. I mean it it it is a risk allocation point again and with humans we accept that risk all the time. I mean I'm not sure I can say I am 100% correct on every single piece of legal work I've ever done in all my time being solicitor. So I think you know human error does creep in and I think we've never measured that so we actually don't know what we would be measuring against. But we but we certainly accept it we accept it and we don't measure it. And then we're hypercritical of a technological solution that that we can at least measure how many errors it makes and does have quite good output. So yeah I do think I I I I'm coming round to your view Philip is all I'm trying to say.

Philip 29:25

I I I think it's just going to be really interesting to see how this technology evolves in the next few years particularly as the frontier labs are are throwing huge resources at trying to minimize the risk of hallucinations and build architecture to provide even more sort of protections and safeguards. One thing I've noticed and this this amuses me in an ironic way is that actually we might get to point where the human in the loop is the risk. And there's been a a couple of I know I know it sounds ridiculous, doesn't it? But it's it's true. There's been a couple of moments in Garfield when I've checked documents and I've stopped and thought how does Garfield know that I've not seen that. Maybe Garfield's wrong and I've gone and reread all the documents and I think I'm a pretty thorough careful chap. And I've spotted that Garfield has spotted something in the small print in a document that as a human I wouldn't necessarily my eyes wouldn't necessarily have gone to because it's very tiny font or something and it's and found something that is relevant and useful that a human in both cases I overlooked. And so I thought to myself hmm so now Garfield's checking me effectively so I do think there will come a point where human in the loop actually will become the risk area rather than the the safeguard. I imagine that there are many other technologies where that that is already the case outside of AI.

Hallucinations And Superhuman Accuracy

SPEAKER_01 30:37

I think it's actually also already the case in some parts of legal AI so just from conversations I'm having with people in the e-discovery space, you know, who are doing because you can, you know, for disputes you have your your F-scores and you you know for first level review documents relevant not relevant to this issue in the case there's well developed well established metrics and what I'm hearing is that I think the market standard is something like 0.7 meaning about 70% correct and um the models are achieving between 0.7 and 0.9. So it's actually better than the first level human already at first level review. The more senior level no but at that junior level yes.

Philip 31:17

That's really interesting.

SPEAKER_01 31:19

Somebody told me about an exercise they were doing to find passwords. It was a sort of cybersecurity context and obviously the model is particularly well suited for that kind of task where you can't quite define what it looks like but you sort of know it when you see it. You know again this is something where people are probably think oh yeah but this is a typically human thing but actually the model performed really well and found things as passwords that the humans well they did a parallel review I think they didn't find. So this is already happening.

Philip 31:46

In my time in practice one of my concerns always in a big disclosure exercise was ensuring that actually the smoking gun documents were actually found and you were reliant in a big disclosure exercise on essentially an army, a pyramid structure but an army at the bottom of paralegals and sometimes trainees and they'd be clever people hardworking diligent but they wouldn't have necessarily the experience and also they'd be working in silos or looking at particular chunks of documents and wouldn't necessarily see the connections and I was always concerned actually that there was a risk that documents would get overlooked. And you ended up almost having to do an iterative process that cost more money to find the right documents and if AI is already doing that you know at a better than that level of human capability then well I mean it's really good for justice it's really good for efficiency and makes your point the human in the loop suddenly becomes the negative not the positive.

SPEAKER_01 32:38

I think it was your point not mine you came up with that.

Philip 32:42

Our joint point well there you go you see that that's me also being the human in the loop and getting it wrong. So it was in fact my point or too modest. Did you have any other bugbears?

SPEAKER_01 32:53

The other one is this discussion around confidentiality so I am a bit surprised that we are still in a market where lawyers constantly talk about losing privilege because you're using a non-confidential AI. And I would have thought that by now lawyers would know that actually if you're using it for work and providing legal services, you need to make sure you're using an enterprise version of whatever it is so which has the right confidentiality protections. So that is something that I'm just wondering how that's come about that that's not not well known. But maybe it's just um here's AI 101 here's a quick course that everybody can take and it's actually quite hard to come by the basic explanations.

Philip 33:35

I don't know what do you think Philip why that is I actually don't think there's much of an excuse for that anymore because um the topic has been ventilated so much in the the legal press and there've been plenty of know-how and articles written about it. And also even if you're a firm or a lawyer without much resource getting a co-pilot subscription or a a clawed enterprise subscription is actually pretty cheap. One of the other topics that sort of that connects to that perhaps we won't touch upon today is sort of the topic that you see ventilated a lot in LinkedIn where lawyers are saying oh I could buy Clawed Enterprise and that gives me 95% of some of the other tools on the market that are sort of aimed at lawyers but cost a lot more money. That's interesting as well to see where that debate ends up. Just moving on have you seen much in terms of the client side of this for big law firms you know a client sort of asking what AI tools lawyers are using and is there sort of Oh yes. It's just I bet they are yeah I assume so very actively yes very actively yeah yeah and is a collaboration did clients sort of say can you build tools for us or can can we integrate this tool with you? Is that those the sort of discussions that are happening?

Human In The Loop As New Risk

SPEAKER_01 34:41

Clients just want to know what we use and how we use it. And I think the main driver is usually we just want to make sure that you're using this in a responsible way and confidentiality is maintained and just please just confirm which obviously we're happy to confirm because we have a very rigorous approach to that. And then the other thing I'm seeing and this is not baker specific this is just generally in the market is obviously clients will run something through AI, get legal advice from AI and then ask the lawyers to check it. So that's something that has gone from zero to everybody's talking about it. I'm sure you've heard people discuss it as well which also from um from a privileged perspective that you might have to have a think about what you you know at least which AI you're shoving that in as a client to make sure you don't lose your your protection I'd have to think about that.

Philip 35:27

Listening to law firms talking about this, I've felt for them because there's a lot of risk there and not much reward because what clients say is oh I've got this advice out of an AI could you just do me a a really quick review and and not charge me very much money basically but then the it comes back actually to a point we're talking about earlier that the lawyer hasn't sort of necessarily gone deeply into the topic hasn't looked at all the relevant documents is just being asked to do a review of something else's product and may not even be able to charge a fee that reflects the risk that they're taking on if it transpires to be wrong. So I I sort of feel quite a bit of sympathy for law firms when they're asked to do that. But I mean it's a world we live in so you're law firms are going to have to tackle this in some way.

SPEAKER_01 36:12

But to be fair sometimes it works the other way around because sometimes what you get isn't very good. So you actually end up you know having to charge the client quite a bit because you're checking it and correcting it which was actually more expensive than if you'd just given the advice by itself just using your own expertise.

Philip 36:30

That reminds me of that famous anecdote about a Victorian era probate judge who um wrote into his judgment I've forgotten his name now he wrote into his judgment that the the client in question obviously now sadly deceased had uh gone and basically got some sort of uh will off the shelf must have gone into whatever was a Victorian equivalent of WH Smiths and bought bought uh sort of a will and and then the judge went on to say I thought he'd done a good day's work because he'd saved lots of money on lawyers. And as it transpired for the legal profession he had indeed done a good day's job because he'd written a will that was defective and to years of litigation between all the the the the beneficiaries of the estate.

SPEAKER_01 37:07

Yeah yeah yeah yeah that's the AI version of of that I guess yeah as we know we wouldn't be doing this podcast if you could just take AI, ask your legal question and it came out with the perfect answer. There's a bit of an art to it at the moment I mean who knows what it's going to be like in the future but at the moment it is complex and there's a bit of an art and you have to really think about how you do it. Otherwise I wouldn't have a job I guess.

Philip 37:28

I don't think we any of us would have a job yes there's a lot more to lawyering than just what is the right answer. But often there isn't a right answer. There's a range of answers and and sometimes part of it is knowing your client you know knowing what your client's risk appetite is and what their objectives are and are there particular things that wouldn't be obvious that that they would want to you know either achieve or avoid maybe a final topic actually that we can just talk about is um and without in any way wanting to sort of trespass onto your powers of prophecy as they are obviously very much your sort of uh special power um the future of of of large language models and and how it might impact on legal services what what do you broadly think is the trajectory do you think we're all going to be like I think Elon Musk keeps going around and suggesting all going to be living a life of luxury while robots do everything powered by LMs and we don't have to lift a finger and just sort of sit around watch TV and read books or do you do you think um the future is some other sort of trajectory?

Client Expectations And AI‑Checked Advice

SPEAKER_01 38:30

That's a good question. I can tell you what I'm looking at and what I keep tabs on to try and answer that question. So I spend quite a lot of time looking at the papers that come out and the research that tell me how the models reason and there's some really interesting stuff that Anthropic do on tracing thoughts of large language models. I don't know if you've read it oh yes yeah yeah that's really interesting. Yeah yeah the as the last year this paper they had on poetry I don't know if you saw that one you know where they said basically they could see that the model if you ask it to write poetry and it was thinking about a rhyme it was thinking ahead of what it was writing so it was the first time they could trace that things like that don't seem relevant to law but I think they really really really really are the more we understand about that reasoning capability the more we will know whether there is any limit to that. If we know that it can just keep developing and developing and developing then I guess we might assume yep it'll it it'll get there it'll get to the point where it can at least argue and reason at least in terms of what it can access and what it can know. A bit like a lawyer it might not do everything else that we do so there might be a few things left but that's that's the sort of thing I'm looking at and also the context windows. So I've been looking at context windows since early 2023 because I thought that was a really really really important bit of trying to understand how far you can push the models again for reasoning and and understanding and I remember when Gemini had the first one where you could actually put a whole book in in it. So I think they did a they were trying to work out whether you could teach it a rare language and they managed to put the whole book inside the context window to see if it would speak or understand and it sort of did. And again a bit like poetry it doesn't sound like very relevant to to law but if you think about language as a system of grammar and grammar is kind of rules and um but a bit like law they're not always rules exactly if this then that but they're sort of a bit vague sometimes and have to do with meaning that is kind of a lot like law. The other thing I'm noticing is the architecture is sort of since sort of the mixture of expert models come out with all this sort of you have a generalist model at the top and it dishes out parts of the question to specialist submodels and then they all come back together and put it like oh that's a bit like working in a team you know that's a bit like that. That seems to get really good results and certainly I mean for when I play around on my own laptop not my firm laptop on with Deep Seek and various models and so on I just try out various things. I mean I certainly can see that reflected in it answering legal questions without access to any specialist legal knowledge other than the internet. It's a very long way of saying I don't see a limit at the moment it doesn't mean we might not find one but at the moment I don't yeah I don't see it.

Philip 41:20

Well there we go I mean it it sounds a bit like I'm slightly exaggerating here but it sounds a bit like you're saying at some point we should say hello to our AI super overlords.

SPEAKER_01 41:31

Yeah I don't I don't know we'll have to we'll have to have another podcast about that's a whole different topic.

Philip 41:37

Exactly who knows I mean I I think we we tend to use models in quite a discrete way because that's the way they're built. So you might have a language model you might have a model that's trained on images you might have a model that's trained on music those sorts of things and of course we're heading in a direction where more and more models will become multimodal and I think the interesting thing about that for me is seeing how that changes things because in my head at least if you think about a text-based model it still contains a representation of the world albeit something it's taken from its corpus of text it's been trained on so uh the way I like to explain it to lawyers is to say okay if you remember your sort of your GCSE maths and you remember vectors and and sort of drawing lines on charts and things and now think it's not just two dimensions or three dimensions, think millions of dimensions. And there's all these sort of points in a enormously multi-dimensional space that are connected to a concept and then they'll be connected to other concepts that are connected to the concept. So for example if you have cat it'll be connected to feline it'll be connected to legs it'll be connected to forelegs or connected to tail and so you end up with a world model which I think you can say is vaguely analogous to the way uh any other neural network works like a human brain only vaguely analogous though because of course they're two very different physical structures I think as the the world model gets richer and more complicated because you've put in their images and and sound and things then you're going to end up with something that I think inevitably is going to be able to provide more useful outputs because it will have a better a better world model better understanding of the of the world that we humans interact with. And then there's a whole separate debate actually which um I think I've mentioned to you Olivia um between Dan and myself about whether that is going to be close to what they call AGI or whether in fact in order really to achieve that we need to have models that can learn in real time like we do, which is presently human superpower and also animal superpower in that we learn as we go through life and deal with new situations and challenges and have new experiences. So yes well it would be really interesting what the the future holds.

SPEAKER_01 43:49

I think the point you make is really fascinating about what the difference would be for moving from the text based models to the other ones and what that would mean for law. I think it's such and such an interesting idea and I was You know, because I read what you sent me the other day, and I was like, oh wow, this is a really, really new novel way of thinking about it. And I'd never thought about it at all. Um, but I agree, I think you would be probably getting more meaning to concepts, even like concepts like justice, you know, and what that means and how we think about it, because clearly I think humans, I don't think there's any human who could just write you definition of justice in one sentence, you know. But we sort of know what we mean by that, and there's you know whole philosophies attached to that. But I agree, if you had a model that was able to not just do the text-based stuff, but also the looking at the world in a more multimodal way, I suppose. Yeah, I agree. That would actually enrich the meaning and the, dare I say, comprehension by the model of what these concepts actually mean to us. Do you think you could have those models like watching a lawyer do their work, for example? So not only would they know things about the world because they've seen lots of footage of what happens in the world, but also you could watch some lawyer working and it would understand much better what needs to be done.

Future Trajectories: Reasoning And MoE

Philip 45:02

Yeah, I mean, I think that's a possibility. And funny enough, I was listening to a podcast the other day where uh Winston, uh one of the founders of Harvey, was talking, and he was talking about Harvey's Moat following the anthropic plug-in and the hit or the share prices of some of the big legal information providers. And he was saying that he felt that Harvey's Moat wasn't just their distribution, the fact that lots of law firms are already using them, but also they captured what he called decision traces. So they sort of captured how lawyers make decisions and how they choose between I know putting different clauses in contracts and how they will amend clauses or make decisions in litigation and things. And I I do think that was quite an interesting concept. And I think that if how good lawyers collectively thought could be captured, then yes, you could see why then you could end up with a product that was able to sort of mirror their approaches. The challenge there, I think, though, is that in a lot of legal situations it is just a question of judgment, and you could have two equally good lawyers who reach out entirely opposite views on things. And I I can remember when I was in practice, there were some decisions I took. I mean, there was um sort of descend into an anecdote again here, but um it's the way I operate. But my own neural network likes to sort of prioritize anecdotes. So it was, it was, it was a substantial. I was being appealed. We we'd um we'd had a hearing in the commercial court and we'd applied to leave to amend to to plead uh a claim in deceit on top of a claim of um negligent misrepresentation, and we'd been granted permission, and the other side weren't satisfied with that, and they appealed to the Court of Appeal and were granted permission to appeal. In between the time of the original first incident decision and um the appeal, they'd had to do disclosure. And as part of disclosure, lots of documents had come out that obviously were very material to the decision and actually supported why we had a claim in deceit. In fact, it was a treasure trove of smoking guns. There were so many smoking guns I felt I was in a smoking armory. And um I we therefore made a Ladd and Marshall application. So if you're if you're a technologist listening to this, not a lawyer, Ladd and Marshall is the authority that governs when you can adduce new evidence into the appellate process that wasn't before the judge below. We made a Ladd and Marshall application on the basis we wanted to bring all of this material in to show the Court of Appeal to say why it was to further support the judge's decision below. And um uh we approached a Silk to advise on the application, and the Silk was adamant that the application shouldn't be pursued. Absolutely adamant. And he gave very good reasons, very good, compelling reasons why that wasn't the case. And and maybe because I'm a little bit difficult sometimes, I decided he was utterly wrong. So I that the reasoning was logical, it was it was reasonable, it was put compellingly, and I was absolutely of the view the conclusion was wrong. So we didn't instruct that silk, we instructed a different silk. And here's a punchline. The only reason I tell this anecdote was, of course, I was proven right. If I was proven wrong, I wouldn't tell the anecdote. But um, no, the other side conceded the application before the the hearing. They conceded it because they realized that the the subtext of why we were doing it was not just to support the judge below, but also to show the Court of Appeal just how appallingly the other side had acted. Suddenly it wasn't a speculative case of fraud, it was a very, very solidly founded case of fraud. So that's part of the reason the other side conceded it. They didn't want the judge's noses rubbed all over these documents for half a day in a contested application. And um actually during the hearing, one of the Lord Justices uh said openly that that was a very sensible application to make, and the documents are very useful to the court. And so I came away from that thinking, you know, sometimes you know you can have two lawyers who um very reasonably the best one in the world and for very sensible reasons reach utterly diametrically opposed conclusions. Now, how you build that into our model, I don't know. Um, because I think fundamentally that might be just a question of judgment. But um I'm sure that there'll be thinking people thinking about that.

Multimodal Models And Legal Meaning

SPEAKER_01 48:48

Yeah, and I'm wondering, Philip, if you can you explain yourself how you came to that conclusion? Because I sometimes can't, you know. I I sometimes I'm like, oh, I think it's gonna be this, and then hopefully it is. But you know, you can't necessarily explain it and put a finger on it. But that may not mean that a model can't. Because I'm just thinking, sorry, I'm just thinking very laterally, but I'm thinking about the medical sphere where I've seen things about scans, medical imaging. I think it was for breast cancer actually, and where they've had really, really good results with AI, and the AI can find cancers that the human radiologist cannot see. So even once you check it, once the AI said yes, this is a positive, and then the radiologist goes back to check, they still don't know what's there. So that the AI can see something we can't see. And I'm just wondering if that could be something. Maybe we have a blind spot, you know, in our uh litigator's judgment or whatever it may be. And we we're sort of doing something we don't even know what we're doing. I mean, it's perfectly possible, I think. As a matter of human psychology, I think it's possible.

Philip 49:50

Yeah, definitely. I mean, uh, there's been many occasions in my life where I've been on the other side of this where I have been convinced that something is correct or true or the right approach, and um and I was just wrong. Um, I think no, never. Yeah, no, no, no. Very, very common. I think that's part of the human condition, really, uh, and why why it's you know important to try to retain humility actually as you go through life to the extent you can, which is very difficult often. Yes, it's it's just an interesting topic. I mean, we humans are fallible, and I sometimes can't explain why we reach the conclusions I do. Sort of I think you can sort of backfill your reasoning. You can say, Oh, this must be the reason why, but it's very possible that wasn't the reason that your brain reached the conclusion you did. And um, often the process is sort of feels intuitive, but it can't be and there's got to be a physical process underlying it, um, which is where we get into my co-founder's Dan's, one of his really interesting topic areas, because he's fascinated by neuroscience. And um, so he he if he was on this call, he would be um saying quite a few things now about um how the human brain works. But unfortunately, we don't have time. We've come to the end of um of today's podcast. So I think all that remains to say is thank you very much indeed, Olivia, for joining me today and for such a very fascinating discussion. And uh, like you, I'm looking forward to seeing what the future holds, and I'm also looking forward to you continuing accurately to predict it.

SPEAKER_00 51:17

Thank you. I'll do my best. And but thank you so much for having me. It's been super, super interesting.