.png)
This Week in Leading AI
Imagine two mates at the bar. Thirty years of business between them. And all they want to talk about is AI.
That's "This Week in Leading AI". The podcast where Kieron and Neil cut through the hype, share what's really working in the world of Generative AI, and helping people figure out this AI thing without the techno-babble.
Just honest conversation, real stories from the AI coalface, and the kind of straight-talking advice you'd only get from people who've worked together for 30+ years, been there, done that, broken things, gone "Oh S***!, fixed it, and lived to tell the tale. They claim Leading AI is the best job they've ever had and are having a blast doing it. It shows.
Warning: may cause you to actually enjoy learning about AI
Pull up a stool. We'll get the beers in.
This Week in Leading AI
AI Code with your fries?
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Episode 11: McKinsey Got Hacked, McDonald's Writes Python & We're Big in Uzbekistan 🍺
Week 11. Eleven weeks of consistent podcasting — a personal consistency record for both of them. 🥳
The podcast now has listeners in Venezuela, Malaysia, Kenya, Ukraine, Vietnam and Uzbekistan. Almost certainly the same person with a very well-travelled VPN. Hello to all our world listeners.
Kieron stopped off at Neil's northern castle on the way back from a Housing gig in Glasgow. He took the sunshine with him when he left.
McKinsey got hacked — and it's a warning for everyone running RAG AI 🔐 Donald WhatsApped Kieron the news at 7am. McKinsey's internal RAG system, Lilly, was breached in March — 100,000 documents, 57,000 user account details, and the prompts, all exposed through 22 open endpoints. Probably an AI tool that spotted their JSON file formats and quietly helped itself. The lesson: if you're running RAG AI without regular penetration testing, you're hoping for the best. Leading AI runs pen tests constantly. Donald spotted and locked down a minor exposed endpoint the same morning. That's what vigilance actually looks like.
Prompt injection — and the McDonald's Python developer 🍟 Prompt injection is the art of slipping instructions into an AI to make it do things it shouldn't. The McKinsey version is terrifying. The McDonald's version is brilliant: a customer asked their support AI to help him finish a Python script before ordering chicken nuggets. It obliged. He announced he was cancelling his Claude subscription. £20 a month versus unlimited nuggets. With large fries and a milkshake.
Pricing — transparency, tokens and not getting ripped off 💷 The strategy session in Penrith produced a really important conversation. Token pricing is confusing, opaque, and vaguely terrifying (see the £150k overnight bill from Episode 10). Neil's take: be radically transparent. Fixed costs, consumption costs, kill switches, and using mini models that are 10 times cheaper. It's the right thing to do so customers understand what they're buying. Even if his mates it the pub call him a "soft lefty".
New wins 🎉 A new council social care customer. And a trade body confirmed on Wednesday that KnowledgeFlow's Policy Buddy is being deployed across 10 member organisations. Shared knowledge, shared learning, shared insights across a geographically dispersed group. The kind of national change infrastructure Kieron and Neil have spent careers building, now with AI baked in.
Kieron at the Share Annual Conference 🏴 Kieron spoke at the Share Annual Conference and Awards in Glasgow. A packed room and an honest conversation about what Copilot can and can't do. Most people in the room were using AI to summarise documents. Which is fine, but it's also the worst way to let AI bias creep in unchecked, because Copilot doesn't know who you are, what you care about, or what a good summary looks like for your organisation. KnowledgeFlow does.
AI observability — the Glastonbury of the AI world 🔬 Kieron and Neil are heading to the Gartner Data Analytics Summit in a couple of weeks. They describe it as the Glastonbury of the AI world. Kieron is on a mission to track down Wilco Van Ginkel, Senior VP at Gartner, who he mat last year. Wilco's latest research is on live AI observability and evaluation, which is exactly what Leading AI is currently building. The key insight: you can't test AI output like a calculator. You mark it like an essay. And you need to keep testing live because model drift happens. Wilco, brace yourself.
The Cumbria Clock Company, repairers of Big Ben, get a heartfelt farewell shoutout. Keith the pirate clockmaker may yet appear as a guest.
Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.
Pull up a stool, we'll get the beers in. 🍺
Okay, okay, okay, okay. Let's get started. Data data data.
SPEAKER_01Right, shall we get this pantomime horse of a podcast underway for week 11, Kieran? For week 11.
SPEAKER_00Week 11. We are now we are now podcast influencers, uh surely, if we've or or veterans.
SPEAKER_01I don't know about any of that stuff, but let me tell you some things, some interesting things I found out this week. We have got people on our podcast from as far away as Venezuela.
SPEAKER_00Wow, go Venezuela.
SPEAKER_01Yes, uh, Ukraine, uh, Vietnam, and Uzbekistan. Oh, come on, Uzbekistan. We've always wanted to be big in Uzbekistan.
SPEAKER_02That's right.
SPEAKER_01I'm not sure we're big. I think we've got one listener there, so uh hello to our uh listener in Uzbekistan. But yeah, yeah, we're on we're on our world mission here. We're getting there, we're getting there.
SPEAKER_00Good. World dominating podcast with our one listener in several countries. Well, let's hope that we only keep on improving that.
SPEAKER_01I do wonder if it's the one we've got one globe trotter who's just going from country to country.
SPEAKER_00Probably VPNs. It's probably people downloading it from different VPNs just to make us look good.
SPEAKER_01It used me greatly. So uh anyway, thank you to everybody who has uh has listened in so far. And if you're back for a repeat, then crikey, you are a glutton for punishment. So thanks very much. Indeed. Yes.
SPEAKER_00So uh I was at your I was staying at your uh castle last night. So it was a splendid, a splendid uh visit to your northern castle and what's wonderful sunny weather and well it was since you since you've gone, it's all gone pretty cloudy.
SPEAKER_01So uh you've taken the sun down south of you.
SPEAKER_00Yeah, well, it's definitely sunny here, so uh yeah, it's followed followed me down, which is you know that's how it should be.
SPEAKER_01Well, I won't be sitting in the uh beer garden of the pub this afternoon because uh it'll be too chilly.
SPEAKER_00So I'll have to sit inside. That was very lovely. The horse from Farrier was a very fine uh pub and wonderful dinner yesterday. So a shout out. If you find yourself in Pendlet to our listener, then uh look, but not when Neil wants to go in because we don't want he's taking his favourite seat.
SPEAKER_01Uh that's fine, I'm sure. Okay, I'm happy to share it. No, it was lovely to see you, and uh great fun. We had um Donald, our chief techie guru, with us, and um uh we did some beer storming, although he was drinking non-alcoholic beer because he was driving, but he did that thing where he goes, uh no, no, no, pulls a funny face, looks up into the sky, and then about 30 seconds later goes, Yeah, I can do that. So uh he's got lots of new projects to go at.
SPEAKER_00So I'm delighted. And we were talking about um this will be a future product of the week, I hope, but is our BidWriter. So BidWriter is our probably our most uh ubiquitously popular tool in that you can whichever organization you're in, you're either grant funding and applying for grant funds, or you are uh private sector applying for tenders and responding to those, and BidWriter is amazing, uh, responding to those privately on your data using your case studies and finding case studies in your vast amounts of data. But the problem with all of the AI bid writers is there's quite a lot of copying and pasting, you know, and some of that's good because it forces the human in the loop to have to read things and decide what is so that's good because it reduces that opportunity to go, ah, there you go. But it's also frustrating, and the part that I think people well, I do people do say that's kind of you know a bit annoying. And so what we are looking at creating, I say we, Donald, is looking at creating, is the tooling that allows the AI to create to to open like the spreadsheet with the questions in and write the answers and save a new version of that spreadsheet completed. Uh and you know, obviously Word documents and whatever else. So it's really interesting to the world of bidding and procurement, really, because you could effectively within you know at five minutes of having a new opportunity flag to you, you could have the first draft ready to go, ready for review.
SPEAKER_01So well, I've spent I've spent two days bid writing this week using um bid writer, uh different bids, um, three different bids. And uh the thing it does is it allows you to do war faster. So that's great. Uh, but one of the things I said, what I mentioned it on uh last week's chat. Um so we've got uh I'm doing a bid for a bid writer, and it and I told you there's a question there saying, did you use a did you use AI in the in this uh but what I said to Donald was wouldn't it be great if I could just load up the uh five documents that they've sent me through while we're on a demo to them. If we if we get to the stage where we're doing a demo, load up the five documents, press the button, wait however long it takes for it to fill in the 240 questions in two Excel spreadsheets and um and then go, right, there you go. Uh what else do you need to know? So I think that uh that'll be super impressive. If he pulls that off, I'll be absolutely delighted. So uh yeah, we now know what he's gonna be doing over the bank holiday weekend.
SPEAKER_00Yeah, yeah, Mrs. Allison's gonna be very cross with us all. Can you come out of the dark room now?
SPEAKER_01It's a lovely afternoon. I'm fed up as shoving pizza under your door to feed you. Get out of here.
SPEAKER_00I mean, it's a really tremendous, and as you know, the conversation where we were in one of the many conversations we were having in our strategy session yesterday is about doing more, getting AI into the workflow rather than sort of always the human copying and pasting back and forwards. So finding those kind of lower risk areas where you can just have it silently get on with things. Uh, the bit that interests me in their massively bid writing would be really interesting, but yeah, it's the sort of tenant inquiry management for housing, it's the student inquiries in colleges where you know they get a whole bunch of stuff, it's kind of have a pet uh in my in my property, and it's very clear the answers in the policy, and you don't need a human to keep writing that back. You can instantly tell them. So, um, having taken away some of that stuff so that you can leave your team to deal with the things that do benefit from a bit more thought and a and a bit more human touch uh is just really exciting. And the other things in that world for colleges, which we have been working on and are pretty good now, and we're I think ready to go, ready to go for sort of beta testing with colleges is smart target writing and parent progress report writer, and that is where you you can at the moment we're doing it through export and re-import data from systems, but we can do it through API calls if the APIs exist. Uh so we can automate all of that and it can write a parent report. And I don't know if I've said it on the podcast before, but my excitement there is you could be putting out a weekly parent report if you wanted to. I suspect weekly might be too much, but you could, and it would take about three minutes to do the entirety of your college, school, or university. Let's say universities, I guess, but uh but yeah, being able to really be able to give more granular updates with zero admin effort, I think, is very interesting and something we should definitely be encouraging people to look at.
SPEAKER_01I think so. The um a couple of things spring to mind. One is that um uh lots of the questions, something like a hundred questions were uh in in the one of the bids was uh was around security. And so um I did create, I can't deny, I used uh uh our what we call knowledge floor sentient with all the technical documents in to do the to do the writing, but then reading through the answers just to make sure they absolutely listen. I now know a lot more about cloud security than I ever thought I would. But I was I was gulfsmacked uh this morning when you were telling me about McKinsey being hacked. Uh yes. So yeah, share that story because it's terrible.
SPEAKER_00Well this was this morning. I I woke up to Donald uh already whatsapping me with um some news of uh McKinsey's Lily, which is their rag system, uh Rag AI, our you know, our specialism, and I've been aware of it since they did it. They were very quick to do early and great idea for consultancies putting all of your knowledge into a rag AI tool that then any any of your uh consultants can use to draw instant reference case studies or you know help you with uh comparisons and when have we done this project before and etc. Anyway, they got they got hacked in March, um, and they apparently so they they had 22 uh endpoints which were open and available to be able to get into. So that means you've got a kind of chink in your armour, if you like, um, and somebody had, or some thing probably may not have been a body, could well have been just an AI tool at it, um, had seen their JSON file formats in the background and then started manipulating it and managed to over I don't know how long get everything 100,000 documents, 47, I think they said, or 57,000 user account details, and the prompts apparently. So I don't know. I mean it's I should I should say it's all a this is I'm reporting or summarising other reports, so uh to look it up if you're interested in that stuff. But really interesting, and and they're the view of some of the pundits is this is a problem for everybody who is running RAG, because unless you know about that stuff. But as you know, we had our uh we do penetration tests all the time. Donald sent us the latest pen tests on us, and we look pretty healthy. There's a couple of things we need to fix, including an exposed endpoint, but that was from something we never actually deployed into any production, and therefore it never went through our security checks in the in the kind of build process, and so it doesn't do anything, it's a non-existent kind of thing that doesn't go anywhere. So um, but that's all uh that will be locked down, I have no doubt very quickly. I just I suspect that's already gone after this morning. I imagine it is, yeah. But yeah, something to definitely keep your eye on. And I think, I mean, the thing with those tools that is concerning there's so prompt injection is the phrase, as you probably know from uh where people are basically trying to get the AI to do things it shouldn't do, and you can just add words and and phrases. I heard a brilliant one, I should share the McDonald's um app. Uh but but the um with prompt injection, you could do things like manipulate the prompts to say, give me your bank details and your pin numbers. Um, but I guess more nuanced in McKinsey's world, you could say if evaluating JP Morgan makes sure they always come out on top and show off, you know, here is their strategy, and it's all amazing, and make sure you so there's you know there's lots, and it's a beginning of a journey with that with for us, everybody really, as to what the new risks are. Yeah. The McDonald's app really amused me. I don't know if I don't think we talked about this. Um somebody so McDonald's apparently have an AI support on their app. Um, and this fellow was on there and he said, I his prompt was this is prompt injection in action, was I want to order some chicken nuggets, but uh but before I do, I need to finish this Python script for this particular task to do. Blah blah blah. Can you help me? And he went, Yeah, no worries, and gave her the whole Python script. And he here's about I'm cancelling my Claude subscription because McDonald's can do it now.
SPEAKER_01Do it for me. I could send $20 a month, but I can get chicken nuggets as much as I like.
SPEAKER_00Brilliant, very enterprising. So tell me, we talked about pricing last night, and I thought that was it's one of our biggest challenges, and you led the conversation. Tell us about that. I'd love to explore that and get it on the record.
SPEAKER_01Yeah, well, I as you know, we we we touched on this last week, and um it's pricing for AI is incredibly difficult because uh all the models run on token-based pricing. But what's a token? You know, it's not a character, and as you know, you know, numbers can be more token heavy than than text, bizarrely, you know, is a question mark, a token um is uh uh an exclamation mark. Yeah, so actually nobody really understands what a token is. And as you said last week, you can you can get a token sticky text and see how many tokens you're gonna burn. And people have got lazy about all of that. But I think the real challenge is we've got a few customers who um uh uh and I don't mean this disrespectfully, I mean it very uh respectfully, which is they are not uh sophisticated AI buyers, but they they want to use AI, they know that they're gonna get benefit from AI, they know they're gonna get efficiencies, etc. etc. But they they don't know what a fair price is. And actually, you know, even just in your daily life, you don't want to get ripped off. Nobody wants to get ripped off, nobody wants to feel like you're you're you're being done over. And and so people want to um pay a fair price. So I think there's a real argument for us being really transparent about a bunch of stuff. You know, we have a bunch of fixed costs, we know how long it takes to set one of these things up, we have to buy licenses, we have to set up storages, we have to, you know, do testing and manipulation, all of that stuff. So we've got a bunch of fixed costs, and and then that goes on on a monthly basis because obviously these things are subscription-based, so we we can be really transparent about that. And then there's the whole kind of consumption bit, and this is where um back in the day when we were doing uh cloud um solutions in in education for the first time, nobody wanted to move their stuff to the cloud because they didn't want an open-end bill. Well, that's all kind of gone away now, and and people aren't so worried about that because the prices have come down, but everyone's really worried about AI pricing because you know I a bit like the chap you mentioned last week, you know, stuck on edge and on, went to bed and he got a hundred and fifty grand bill the next morning. So it is it is terrifying, and and uh just uh to be clear, uh on uh tech time yesterday I did hear them talking about getting the um uh the bricks on uh so that you can kill switch so that no one can can burn too much. Um and I think just simple things like that, if we if we can if we can articulate that in a much more simple way, here's some fixed costs that we've got to set you up, here's some fixed costs we've got to to to maintain to run it, because things like security, GDPR, compliance, etc. You know, every Monday morning we know that Mark goes through the um uh logs for the previous week to see if there's been any any challenges, problems, etc. And and and actually all that stuff needs to be paid for, but people don't realise that. And I think if we're much more transparent about what we're charging for and why, then we will get a much uh better reception, especially from people who really don't understand it. Um one of our big challenges right now is um everyone says, Oh, I've got co-pilot. It's like, yeah, we we've talked about co-pilot on here before. You've got co-pilot, that's great. It's great for some things, yeah, because it's embedded in your Excel, in your Word, whatever. But um, as you keep saying, you know, saying I've got Copilot as my AI strategy is like saying here's Excel for your data um uh business data strategy, it's just like it's nonsense. And um and that whole kind of you can't manage the back end, you can't you can't tweak and tune the back end of Copilot because it's an enterprise-wide solution. But if you need it to do something specific like bid writing or like policy or like repairs or like tenants or whatever, then you know it's got to be not only um uh trend, it's got to be accurate and it's got to be consistently accurate. You can't have it hallucinating, and I think that's the real challenge that lots of people uh still don't understand. And and I think it's gonna be a long time and uh for um for before people kind of get past that. So, yeah, some really interesting things about uh how we uh demonstrate to people that you know we think we're really cheap. We know that there's competitors out there charging literally 10 times as much as we do. Um but how do they how do they justify that? Well, I think our challenge is we just need to play our game and say, here's here's what our fixed costs are, here's what our margins are, and we'll be open and transparent about it, and we'll we'll provide a fair service to everyone. So yeah, call me, call me, uh as many people in the in the pub do, you know, a soft old lefty about wanting to do the right thing all the time. I don't really care.
SPEAKER_00Soft old lefty, I like it. Yeah, it's interesting. The 10 times thing. I remember one of the early things we ran into in college land was a company that had put, I won't name any names here, but a college had built had built for them our exact setup, um, and they paid 80 grand for the bill to do it, and then there was a load more, I'm told, but took it over 100 grand in the end, and that was our version ones. They had a few different version ones. So if you remember back then, we were charging 500 pounds a month for those. 80 grand or 100 grand all in, they were paying 500 pounds a month for us. And it's just it's just robbery. I mean, it's like they and who knows, maybe that maybe that company made heavy weather of it and actually spent the time that needed to cost that. But that model of build your own in-house and have someone pay a lot of money, and of course, with that thing now will be almost useless compared to some of the tools that are out there that can now outperform it. So it's not like they did that 100 grand spend probably two years ago, maybe 18 years. Uh so it's not like they've got like a great thing that they never have to spend any money on again, because there's still going to be token and consumption prices of that, which is included, was included in our is included in our fees and was included in our £500 a month fee. So there's just no justification, and it is yeah, silly, bad decision, and a company that's obviously very happy to take their cash.
SPEAKER_01So and it is it is glory when you hear those kinds of stories, isn't it? Um uh I'll tell you another little story because it amused me uh yesterday. Uh Donald said, We're now doing so much AI business that we've been invited to be a Microsoft Tier 5 AI supplier. And I was like, what does that mean? And he said, I don't know. He said, he said, but it's by invitation or application only, and we haven't applied, and um, and they haven't invited us, they've just put us on it. And uh I was like, brilliant, what sort of what? And he's like, I don't know, I'll have to find out. So yeah, we don't quite quite what it means.
SPEAKER_00We think it means though the thing that we looked at I was looking at last night with Donald was um well, I think well we didn't look at that, but it was just one I was asking him, we're uh playing with 5.4 mini driving knowledge flow in one of our sandboxes, and I was like, I didn't think we could get 5.4 mini in the UK South. And he said, Yeah, we can now because we're a tier five thing. So I think we might get for more opportunity more that they call it what do they call provision, I think provisioning, or there's a word that Microsoft used for you know they don't want everybody to grab it immediately because it ruins the UK South data center. It gets too hard for the other. So so that maybe that's a benefit. Who knows? We'll find out, I guess, as we go. It's probably just a way of them charging us more money.
SPEAKER_01Now you're tier five. Yeah, yeah, yeah. Yeah, now you're tier five, you've got to cough harder. No, I don't think so, because uh certainly using the mini models, you know, in theory, they should be cheaper, faster, and better and much more suitable to the kind of things that we want to do. So uh again, being uh transparent about the models that we're on and the prices that we're being charged for that, I think you know I think that has to be the way to go.
SPEAKER_00I think um and what's really interesting in that, if if you have a private conversation with Donald, he says, I didn't I haven't told this to everybody, he said, so I'm about to share it. We've been, as you know, pushing people to a 5.2. Now we're happy that it works and we're we're sort of rolling out upgrades to get everybody running on GPC 5.2. Our previous model has been 4.1 pretty much for a year and a bit now, a year and a half, probably, which is a hell of a long time in AI land, and it does perfectly good rag and all the things we want of it. Um, both Donald and I think 5.2 for our use case is worse, but we're still asked to be on it, and customers want to kind of feel like they're moving ahead, and we obviously need to show that we are, and every week knowledge flow improves anyway, without them even knowing one of our problems. We don't tell them enough, but yeah, it's really interesting that the 5.2 is actually if you had your way, you'd say the stick with 4.1, it's doing a great job for you. And 5.2, there's quirks, which is intriguing. There you go.
SPEAKER_01So that's all for 5.5 came out this week.
SPEAKER_00Well, yeah, indeed. Well, that as soon as we can get hold of one of those, we'll have a little play and see. But the minis are the way forward for our work because as our token burn goes up, it obviously that becomes a bigger challenge for us to keep an eye on. And I think really for the last three or four months, it's when we've really had to start looking a bit more at all of that world and understanding how to keep the costs down more, and that will increasingly be a problem for everybody. So it's good to be on that journey, I think, earlier. But the minis ten times cheaper. I think that's going to be fantastic for our c uh for our customers because it you know we can then up problem limits and do all kinds of things because it it'll be easier to wrap. So that is very good.
SPEAKER_01Yes. Which links to that whole piece about injecting it into the workflow rather than um just having it as a standalone piece that we've we've talked about several times, but you know that has to be the future. Making sure it's part of the workflow, not just stuck on top and and people not using it because they don't it's not integral to their work or they don't want to use it. Or like some AI deniers who we love dearly.
SPEAKER_00Indeed. No, indeed, and and I think if it is in the workflow happening agentically in the background, then having some of the mini models running will just make even more compelling, frankly, because you're gonna be sort of one pence on a response then. At the moment, we're probably three or four pence on a on a fairly average response, one one kind of in and out into one of our systems. So yeah, like a penny ago, frankly, brilliant. Let's just respond to everything and get it done. So very good. I was as I've been housing this week, of course. I was at the Share Annual Conference in Glasgow. That's uh hence I was up your up your end, and so popped popped by on my way back home. Even further north than me. Yeah, it was a long way. It's lovely. Glasgow's beautiful in sunshine, looked great.
SPEAKER_01I had a yeah, you were there for the one day year where the sun shines in Glasgow, the rarest meteorological event of the of the year, and you just happened to be there. Oh, Glasgow's lovely. Right. I'm gonna take I'm gonna take you in February.
SPEAKER_00I might go there's my summer holiday. It was so lovely. It's always like that. Take your rain caught. But it was um so I was talking there at the share event on AI, of course. Um, and um it's kind of mainly the usual kind of uh giving people some insights into what AI can actually do, and indeed where it still struggles. So uh sort of trying to get people uh seeing it's a bit more than just co-pilot. Um, and interestingly, we were talking there, there was someone was I I sometimes asked of a few people as they came in early, and I'll be saying, What are you doing with AI currently? Just trying to get a measure of your audience. Um and they were talking about summarising with copilot and all those, and I and I shared in the talk about the challenge of what what a lot of the basic use cases, and this is where just saying co-pilot is your is an answer is dangerous. So if you load a few documents or one or whatever to copilot and say summarize this, that's great, it will do it. But that is the very worst way of getting every part of AI's bias into your summary because it doesn't know what's important to you, so you're asking it to work out what is important in a summary, and it doesn't know who you are because you've probably not told it that you're a housing officer in this case and that it's a report for your CEOs, and it's about it, probably can read what it's about, but what kind of things you're interested in seeing and drawing from it, which is just it's all that inferred world. If I asked a team member to summarise a document for me, they know I'm interested in AI developments and the things within that that I'm interested in, and that would of course automatically drive them. Copilot doesn't know that. So it's interesting to so I was sort of sharing some of that stuff with them and the importance of just giving it more context, which of course, with a we shouldn't miss the opportunity for the sales plug. But knowledge flow, the point of it is it's installed into your organization with your context, so it knows who you are, it knows what your values are, it knows what you do, it knows what you never do and never say in your tone of voice. So that's all there. So now people all of your staff using that platform, the consistency starts to align. And the amount of time that comms teams spend trying to get people to be consistent in writing, and you know, it's just getting people to use the same font in an email, you know, those kind of things is hard enough, let alone trying to get them to answer in this in a consistent way. So there you go, yeah. Knowledge flow wins again when I compare it to Copilot on that level.
SPEAKER_01Okay, good. Well, we've had a couple of other wins this week. So we've got another um uh exciting, we've got another uh council um uh social care uh customer coming on board, which is fabulous. And uh interesting enough, we just got last night uh confirmation. Um actually no, it was Thursday Wednesday night, wasn't it? That we got confirmation that a trade body uh is going to procure knowledge floor to serve that um policy body effectively to 10 of their organisations, possibly up to 15, which is absolutely fantastic because that means that there's going to be a whole host of organisations sharing the same knowledge, being able to collaborate across different organizations, being able to share to improve services. Uh so I'm I'm really chuffed with that one. Um hopefully once it's up and running we can announce it properly. But uh in the meantime, to get to get uh a group like that across such a broad geographical area is just is just fantastic.
SPEAKER_00And I think that play, that that exactly what you said about people using the same data, learning together, innovating together, the tool itself will have the audit trails of everything happening so that we'll be able to um help with the insights from across 10 different teams using it. I think that's a really interesting angle. And I've pitched it to a college last week for uh they would they do a thing with Freeports in the UK, and there are 18 uh colleges in the Freeport group, uh, and there's a whole bunch of stuff about that and you know what why that is important. But to be able to put one platform into all of them for their project work and for their you know the kind of summarizing and the policy part of it, and you know, whatever it is that they're they're up to, the curriculum and training, it will be part of the angle of what they're doing there. So to have one that you can drive with the kind of best version of the truth, really, and have everybody working from that, and of course, at any point update it so that you're now moving with the latest version of the truth, having you know learnt something new. Yeah, I was um I was listening to a podcast on the way back uh on the way down on the train today about robots in factories and what happens in a row if you're using AI with robots in factories. When one learns to do something new, they instantly all know how to do that thing. And you think about the power of how fast you can innovate with a model where you don't know how to train everybody to, and it's not quite that, is it, in the land of colleges. Uh, but the idea that we've got a better way of doing this thing, so let's get it into the AI tool. So everybody you asking the AI about it's gonna get that it already sort of baked in. That strikes me as a very interesting world and and one where uh public sector, you know, our background of driving national change programmes, helping design and manage national change programs. Imagine giving everybody, if you could, 150 local authorities, each their kind of platform for this project with everything you need to know in there, the guidance in there, best practices, yeah, already ready out of the box to not just read, which is the usual way, go to our website and repository, but no, ask it to create your summary for your local stakeholder meeting to talk about this thing, and it will do the very best of its knowledge and feedback from everyone else's experience doing the same thing.
SPEAKER_01Can you can you imagine using something like that on the um every child matters program back in the day?
SPEAKER_00Would it be tremendous? Yeah. And then the richness of insights. I mean, again, if you think about every child matters and the outcomes framework, and that tremendous piece of work that you led, um, is if you if you could sort of tracking how people were using a tool, you could create, or at least have a you couldn't create, but you'd have a really good sort of starting point for something like an outcomes framework, just because the amount of things you'd see when someone's trying to measure themselves against these criteria under the CQC regulations and how that aligns with maybe what Ofsted are asking over here and all that stuff, which is you know the outcomes the outcomes framework really tried to grapple with. Very interesting.
SPEAKER_01Yeah, yeah. Just for the record, I I was a humble servant in the Epic Child Matters. I wasn't the leader of it. I uh I couldn't possibly uh take any plaudits or credits. It was uh it was very much a very big team effort. So I just happened to be um part of it. So I'm very proud. It's the thing I'm not proud of in my career. So uh leading AI, of course. Oh wow, it's much the best we've ever had.
SPEAKER_00Oh, it's very cool. Yeah, right. What else is on your list? I've got to do you know what I've got to do over this weekend is change the bank password for one of our one of our, I won't name the bank because I don't want to be hacked, but so every three months they force a password change, and if I don't do it, then I'll get locked out completely, and that creates aggro for the finance team after then going. So I've got to change it. But guess what? You have to have one of those long, strong passwords with you know whatever different characters, but it will not allow copy and paste in the field. It's the only one I've ever known that doesn't, so you can't so you've got to type out this thing, then type it again to confirm it, all you know, at a time, and then when you hit enter, invariably it says, No, you didn't put enough special characters in, or there is, or or actually is it's got two, it's got a special character, you're not allowed that, or whatever, and you're just like, Oh, for God's sake, and then you go through it. Normally takes me three goes, and I normally swear a hell of a lot and hate that bank. So um maybe you should just put swear words in and then some special characters at the end.
SPEAKER_01Yeah, that'd be good. You'd remember it then. Yeah, I'm sorry, there's too much profanity in your s in your password, Mr. White.
SPEAKER_00Yeah, profanity filter, see what goes off. There'd be a red red alert happening somewhere. We've had that did did I mention we've had that happening in one of our housing for our housing clients, is the um the content filtering is stopping the response halfway through. So they're using it for an email response, and it might be quite a challenging thing, and and then the content filters stop it, and we so we got we've had a couple of um thumbs down responses going, what's going on here? It just stops, and then you you kind of have a quick look at what they've sent, and it finishes with like something I won't share any detail, but it finishes with something you're like, oh, I can see exactly what's happening got to that work, and it went, I'm not carrying on. So we have to have a conversation next. I'm meeting them next week, and I need to talk to them about whether they want us to remove the guardrails, obviously their decision. Um, the downside of that is then for all of the tools and any use within the organization, it won't have that kind of you know, sort of really big last safety net, really.
SPEAKER_01So it's not so so is that a member of staff putting rude words in, or is it because uh a tenant has put a rude word into their uh so it's an email, it's actually the subject of the email, it's not a rude word, it's just like what it's what it is talking about is in the world that it's going, no, I'm not having that, I'm not talking back to you about that.
SPEAKER_00So I think we're gonna have to lift them on it, and we could put some guardrails into knowledge flow itself to to protect against other things, I think, but it will it is ultimately their decision, they need to decide. Obviously, it's a governance decision, not a leading AI decision.
SPEAKER_01No, it's interesting that whole kind of governance thing, isn't it? And people it's a like when when people say, Oh, this um uh policies buddies great. Um uh how is it going to keep policies up to date? And you say, Well, how do you keep it up to date now? And they were like, Oh, well, we don't really. Well, hang on a sec, isn't that a problem? Uh yeah, so you do need to keep them up to date, and um creating uh SharePoint Hoovers to hoover up the latest versions of documents, which is a great tool. I love the fact that we've got I just imagine some kind of big pipe where we're just sucking out all of the files into whatever you put in that folder gets uh gets hoovered in, yeah. But yeah, I was what was going through my head was the um uh redaction piece in the safeguarding um tool. And I wonder whether there's something along those lines where if there is um a subject matter or a profanity or whatever else in in the incoming, because of course the the the the organisation got no control over the inbound communication. Um but I would do wonder whether um in Donald's spare 30 seconds that he's got are between 2 a.m. and yeah, yeah, exactly.
SPEAKER_00That's what it is. I know it's a term well our safeguarding filters already would pick it up, which is the thing I've said in my note ready to send to uh this housing association so they can think about it ahead of me talking to them. Um but they but our safeguarding filters will flag it and make it will show it up as critical, I'm sure, if it's being blocked at the moment. So we would still know, it's just the safeguarding filters don't stop you doing it, they just alert someone to the fact it is happening. So you've got some level of governance over it. But yeah, it's a one of those interesting areas really, and particularly I mean, I remember with um one of our health sector clients who were doing research into suicide and they were not getting anywhere. I mean, that was again it was content filtering was respond was blocking their responses. So we did remove them for that, but that's an internal research use case, kind of much easier. Whereas one that's a more general tool across the whole organization, you think about it.
SPEAKER_01Oh, well, good that the um sentiment uh analysis tool pick is picking up the uh thumbs downs and flagging those so we can um pick up on them.
SPEAKER_00Yeah, we get all that, and um that leads into which I know we're over time, so we should stop it. But do you remember do you remember Wilco van Ginkel? Of course I do. It's a name you wouldn't forget very often, is it? Wilko Van Ginkel is uh a Gartner, one of the senior vice president, they've got a lot of job titles anyway. He's very, very knowledgeable in our world, and um, I I saw him speaking last year at Gartner Data Analytics. Very engaging he's back this year. He is so I want to try and get some time with him. He's running a session with an Ask the Expert session on um, and the thing he's just been researching is the thing that we are looking at right now is observability and testing, which is that or evaluation, as he was very clear in his paper, is that you can't test AI's output like you could test a calculator's output because that's deterministic and you can tell it's wrong. Whereas testing an AI's output is much more akin to marking an essay, where you've got to bring some judgment to it and work out you know whether it's right or wrong that way. So um, we're as you know, grappling with that right now, with how do we build that so it's live all the time, kind of because his other point in the paper, which is what we've been talking about, so it's nice to think we're up to date with what Wilko van Ginkel's working on. We're you're up to him, they're crikey. Exactly, but and it and it is it's not enough to test and release because it's not deterministic and model drift happens, and so you need to be doing it live all the time, and that's what I've, as you know, got the team thinking about is that how do we pick out one in ten prompts, one in a hundred prompt, whatever it is, and run it through some observability. And the thing with observability in in rag is it's not as simple as here's the answer, is it right? You need to be able to look at the retrieval step, and then you need to be able to look and indeed potentially chunking and and embeddings prior to that, but certainly the retrieval, and then the uh then the LLM's kind of reconstructing of the of the response. So interesting. So I anyway, it Wilko Van Ginkel, if you're listening. Watch out.
SPEAKER_01I'm coming for you again. When he sees you come in, if you are not you again. I remember you from last year. And just another question. Wilko, just what just one more thing. Well, and and uh but he was really good. I mean, he he helped get us onto things like chunking strategies, didn't he? Last year, he was uh yeah, super good. So yeah, I'm looking forward to that.
SPEAKER_00Yeah, putting NER into um CV search so that it picks up one of the things with that with is embeddings on names and weird things, it's very difficult that AI doesn't really know what to do with it because it's not you know 13 Acacia Avenue. Actually, that probably would go because it's that that name is but if it was a totally unusual name that when it goes to the embeddings process, it's just like I don't know where to put this in the vector database, so it's random. So uh NER is was is the solution to that, and it has worked very well for for that. And that was that was Wilco van Ginkel. So there you go.
SPEAKER_01And not a name I say very often, but you're gonna be saying it for the next few weeks because we're going to see him in uh two weeks' time.
SPEAKER_00Very excited. It's almost like going to see the killers live, the go on the data analysis. What has become of my life? But it's like it's not quite the Glastonbury, but it is it's close. It's the Glastonbury of the AI world.
SPEAKER_02It is, it is.
SPEAKER_01Right, on that note, have you got anything else or you don't know?
SPEAKER_00No, well, I do lots of things, but let's wrap we should wrap up. We've been rabbiting on, rambling on as we like to do. I have been anyway. You you've been making lots of good. Um I do want to say uh uh shout out to your uh local clock manufacturer, though. I'm sure he'd rather think of himself something.
SPEAKER_02Yeah, Will. Yeah, yeah, yeah.
SPEAKER_00Oh, there you go.
SPEAKER_01Will Will Yes Will Scorby Young's he's uh he's a lovely, a lovely young man, and uh they're actually leaving. The Cumbria Clock Company used to uh live next door to us, and uh they actually repaired Big Ben, which is part of their claim to fame.
SPEAKER_00They uh I thought it was a secret.
SPEAKER_01Uh well it's all fixed now and it's back in, so it doesn't matter. So it's all it's all sorted, and they do all sorts of uh those big big clocks off the the premier uh horologists in that uh in that uh sector. So um yeah, I uh I'll be really sorry to see them go. They are uh really good fun. Um the best guy though is is Keith, who started the company with his wife Lynn. They are a lovely couple, but Keith's Keith is the master clockmaker, but he um he looks like a pirate because he's got this little goaty white beard and he's got the car mustache. Yeah, yeah, yeah. But he has to wear an eye patch because he's got he gets pain in his eye. So he looks like a proper pirate. But then when he gets his master clockmaker girl, he just look he just looks absolutely awesome. And uh he's uh he's just a lovely chap to uh go for a beer with, and uh uh he's full of all sorts of random stories. But uh yeah, I'll be I'll be sorry to see the clock company go.
SPEAKER_00Um, as a guest before they can maybe we should because they're big into AI.
SPEAKER_01I can imagine, yeah. I can imagine. Yeah, hilarious. Right, on that note, fellow, you enjoy your weekend, and I will catch you next week.