$500m reasons you need an AI kill switch

Episode 16: $500 Million in Tokens, AI Slop at Interview & Trigger's Broom 🍺

Week 16. The pantomime horse rolls on — and the transcript called it pantyhose again. Neil is blaming his Northern accent. Kieron is firmly in the back end. Nothing has changed.

Someone burned $500 million of Claude tokens in a month 💸 

An unnamed organisation gave all staff unlimited access to Claude Enterprise with no budget limits, no kill switches, no guardrails. By the end of the month, they'd burned through $500 million of tokens. Presumably the CTO is available for “new opportunities”. KnowledgeFlow builds trigger warnings and kill switches into every deployment as standard. This is precisely why.

Kieron's big talk 🎤 

Next Wednesday, Kieron is on stage in front of 60 senior leaders — MDs, CFOs and more — at a large private sector organisation in an industry he doesn't know well. The brief: inspire them to do something with AI on Monday morning. Last year's effort was apparently too technical and too salesy. Kieron's approach: Nokia, Blockbuster, Kodak — the companies with brilliant moats that simply didn't see what was coming. Then: how hard would it be for a new entrant with zero capital and great AI to disrupt your supplier and customer relationships tomorrow? And a use case matrix: 22 different applications mapped against feasibility and business impact. The challenge, as always, is getting people to hear a use case from a different sector and think "that applies to us" rather than "we don't do that." Tune in next week for the verdict.

BidWriter in bid writing season 📋 

It's peak public sector tender season. The one where the commissioners are releasing everything before summer so they can sit on a deckchair while everyone else frantically responds. Neil's been deep in BidWriter and the Agentic version with Donald. Key discovery: GPT 5.4 mini writes answers that are too short and too succinct. Asked for 2,500 words, it produces 900 and tells you it was being efficient. GPT 5.2 works better for bid writing. The latest model isn't always the right model for the job. Next week: a live demo for a large organisation using their own 200-question tender as the test.

The "build it themselves" customer 🔧 

Kieron's been in a five-month scoping process with a charity that wants a public-facing RAG assistant. Five months of free consultancy, steering group sessions, demos, statements of work, and every time Leading AI provides guidance, the customer takes it to other people for more opinions. They also want no vendor lock-in and the ability to lift and shift the whole platform at contract end. Kieron's honest response: I don't know how you'd do that. If you want something that bespoke you’d have to set up your own company. And if you want cheap (because you’re a charity with no money) then you need to accept an out-of-the-box solution. The lesson: we need to price in the complexity upfront, or both sides end up miserable.

Measuring AI by numbers of prompts is nonsense 📊 

Kieron was at a university AI champion session this week where the IT director showed Copilot usage stats: 785 users. Of course the numbers look high. That’s because everyone's automatically logged in whether they like it or not. A single prompt in a KnowledgeFlow policy assistant probably saves 20-30 minutes. A single Copilot prompt probably saves 3. Usage numbers tell you nothing. Impact is the only measure that matters.

Leading AI is now a Claude Partner 🤝 

Donald announced it. He just did it and told them afterwards. What does it mean? Still working that out. But model switching between OpenAI and Anthropic inside KnowledgeFlow is becoming increasingly important. And, as Nate B. Jones (their AI guru) says the frontier model race is now a two-horse race: Anthropic vs OpenAI. Google and Meta have dropped off his radar entirely.

Trigger's Broom and company culture 🧹 

Kieron muses on Only Fools and Horses: the ancient broom that's had four new heads and three new handles… but is still the same old trusty broom. So, when does a company stop being the company it was? What holds culture together when almost nobody original is left? Media and advertising agencies see almost 100% staff turnover every few years, and yet the culture somehow persists. Something to muse over at the weekend.

AI slop at interview 🤦 

Neil interviewed someone for a finance role this week. First question out of the candidate: "How has remote working impacted your revenue and culture?" Neil's reaction: as a finance professional, you should probably know that remote working impacts cost, but is unlikely to impact revenue. He asked if they'd used ChatGPT to write the question. They denied it. Neil's verdict: if they didn't use AI, they shouldn't be in a finance role. If they did use AI and didn't check it, they still shouldn't be. AI slop at work… in both senses of the phrase.

Neil ends the episode by blowing smoke up Kieron's backside. The Audient is presumably still drooling on his pillow somewhere.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Transcript

SPEAKER_02 0:00

Okay then, well let's start, shall we? Let's get this pantomime horse of a podcast underwear for week 16. Here we are again.

SPEAKER_00 0:08

Here I am, still in the back of the pantomime horse.

SPEAKER_02 0:15

But it seems fine because as I mentioned previously, sometimes I think it's my accent actually that um the transcript can't work out. It it it called it pantyhose again last week, which is like, oh crikey, it's really bad. So um anyway, moving on swiftly, the pantomime horseboard podcast for week 16. Um, I've got a bunch of things I would like to talk about this week. I've had some really interesting customer conversations. I've um been doing some more work with BidWriter and the latest versions of that, which are pretty spectacular. Um I've uh got a uh another customer story where basically they want to do things themselves, and I've been trying to point out to them the challenges around that, and then um I've got a uh a little a little funny insight to uh to finish off. So that those are things on my list for today. Kim, what have you got for you?

SPEAKER_00 1:10

I have so so on my my big thing that is on my mind a lot at the moment is a talk, uh stage conference session I'm running next week with a large private sector business with all of their senior leaders in the room. So that's 60 people, and they want to talk about AI and how it helps them. It's in an industry I don't know very well. Um, but um so I'd love to talk a little bit about what I've been thinking around that and how to pitch it. Um it's difficult. I also would love to talk about uh how we judge which customers we should be taking or not. So we've had, as you know, a challenge through the last couple five months, in fact, of um of scoping something with an organization. So I I want to talk about that and the the challenges with that. Um you've just prompted me that um I've spent quite a lot of time this week on embeddings, uh strategies and um sort of metadata that goes with those. I've been talking with Ibby in our uh dev team about that a little bit, so I'd love to uh explore some of that stuff um and a couple of other bits and bobs along the way. So uh buckle up and uh I shall get firmly in the back of the plant mine horse.

SPEAKER_02 2:23

Well, maybe you should move up front and start with your uh private sector AI presentation. So that that sounds like a challenge. Um so yeah, tell tell us a little bit more about it without naming the customer.

SPEAKER_00 2:35

Yeah, indeed, and I will keep the them. So the the challenge is 60 people, I am told, uh I don't know them. I've met now four, five of them, five different individuals, which is great, and it's lovely that they've spent some time with me to kind of help me understand. But obviously, you're always very aware that you're getting their perspective on the rest of the group, and that may or may not be actually where everyone else is, and within where they are in AI. So for me to know where to sort of start really and and what to cover is really ch challenging because I'm trying to get that message, and I'm told there's there's certainly the the ones I've spoken to are all pretty adept and are using some good tools and kind of have a good enough knowledge, I think, of what it can do and help help with. Um I'm told others will have none at all and and barely used it, might have just played with Copilot a little bit. And so kind of trying to work that out as a it's a 90-minute session, and I've pitched it to them as a working session, so that they will leave with some ideas they're gonna go away and work work on with their teams on Monday morning, that kind of style. And that's what the um the CEO of the group wants. They had a crack at it last year, AI, and I'm told it was too uh technical um and a bit too um advanced the the the sort of positioning and a bit of a sales pitch, so obviously I'll be keeping well off of that. So I really want to add value to them. And I think so. What I'm here's what I'm what I'm planning to do is do a little bit of the why AI, why now, why do you need to do it? And I've got a slide at the moment in my deck with the Nokia Blockbuster and Kodak logos, um which I quite like because they're kind of all of these had good motes, they had customers, they had products, and they just didn't see what was happening, didn't believe it, didn't or whatever. Um, and even in their businesses that with the the organization that I'm speaking with, they have customers, they have physical product, and they've got supplier relationships, all of those really important. But my I've got a slide talking about how hard or easy is it for a new organization to arrive tomorrow with no capital, with amazing AI systems running everything. How much is your moat enough? And you know, initially you think, well, yeah, we have to ship millions and millions of pounds worth of product around the place. Well, okay. But your supplier relationships and customer relationships, they could be presumably quite easily disrupted by someone that had a lot more data, a lot faster data, a lot more personalized approaches. So I'm gonna do that, then I'm going to spend a bunch of time on use cases in their sector, which I've done with some AI research around what's going on right now. Um, I'm gonna show them a couple of things that we can do on knowledge flow that will I think play to what they have to struggle with every day. Um and I and that that's the bit I don't normally do in any of my talks, is sort of going out there on and it's 22 different use cases on a matrix, so I've got that kind of feasibility versus business impact. So you've got some really advanced stuff, uh, but really some really basic things as well, like meeting summaries and and that sort of world. So I want to kind of get across some of that, but I I'm kind of worried about that section as well because I want them to use it to stimulate their thoughts, not see it as a list of things they've got to choose from. And I'm hoping that that I can get that across because my experience of that is people aren't very good in the space of kind of like hearing a story and thinking, ah, hang on, if you applied that to this bit of my business, that would be amazing. They people generally, in my experience, tend to hear the thing and go, Well, we don't do that, so why would I care?

SPEAKER_02 6:24

But I think one of the benefits of this group, I don't know, I don't know them, but I understand there's lots of uh managing director and uh CFO types, so you should have a bunch of bright people in the room, and um you know, in every cohort there's gonna be some who get it and some who don't, there'll be a bell curve and and you know, appealing to those. I I think I think it is a difficult pitch when you've got something like that, especially in a uh how should say more traditional industry, which uh isn't necessarily used to that kind of disruption. But if some of the guys are already using uh tools and and doing things, then you know you're gonna have some bright people. I think you're also gonna have the challenge that some of them will think they know best and they'll want to they'll want to be pushing their own solutions in order to promote themselves. So you've got a really tricky uh balancing act with all of with all of that stuff. But uh uh you've dealt with more tricky situations than mine.

SPEAKER_00 7:18

It'd be fine, it'd be fine. I have. I have definitely dealt with more tricky situations. I'm just really keen for it to add value to them. That's the thing. I want them to leave feeling inspired to do something, and I'm giving them some more of the out there examples as well for those that are already well on the journey and have got their data organized and they've got a load of automation in place already. I've got a couple of like use cases which are really kind of out there that will make a big impact if they can get to that. So we'll see. But yeah, so next this time next week I'll be able to give you the uh feedback, or or maybe I'll just be dead somewhere in the ditch. I hope not. That would be a very bad outcome.

SPEAKER_02 7:56

That would be a very bad outcome. I don't think it's gonna be that bad. No, and I I well, I think there'll be a lot of learnings from it. I and actually it's wrestling with the tricky characters, isn't it, that always you get the most out of people who challenge you and make you think and and actually push the boundaries. That's where that's where most of the a wise man once said to me, Kieran, if something's easy or something that is hard, do the thing that's hard because uh you'll get more out of it, or something along that. I think that's what you said, isn't it?

SPEAKER_00 8:22

If there's two courses of action and one's frightens you, do that one. Yeah, exactly. Exactly. No, you're right. You're right, it will be good for me. And it is good to actually dust off the slides literally from start again, because you know, you fall into a bit of a trap speaking at conferences. There's easy to you know, I I know I'm comfortable with my slides, I don't even really need to think about it before they come up on stage and I know what I'm gonna say, whereas this is different, so I'm gonna have to get right down and prep again. So I think that'll be a good thing in the long run. Just a bit alarming for next Wednesday.

SPEAKER_02 8:52

Well, it's always good to have some butterflies. I think the trouble with all the confidences it's always uh it always comes just before you trip yourself up.

SPEAKER_00 9:02

Pride and a fall and all that stuff. So you've been uh wrangling with customers this week, or at least um on uh uh a little bit of wrangling.

SPEAKER_02 9:10

Um uh I've been having lots of really interesting conversation. Let me I'll I'll share, I'll share a couple with you. One um uh is is around bid writer, and uh I've got a couple of different bid writer stories because um it that seems to be the most popular thing right now. I don't know if it's just bid writing season. Uh certainly in the public sector, it's common for a whole load of bids to come out before the summer so that whoever's um it's putting out the tender can go off on holiday and um sit on a deck chair somewhere with a uh with a with a cold drink while everybody else is slaving to get their bloody responses in um for the for the for the end of the summer holidays. Um so it but the um all joking aside, so we've got a we've got a presentation next week for uh um it was a bid that we uh we used bid writer to bid for. And I've mentioned it on before on the podcast. Have you used have you used AI in the writer this bit? Of course I have, it'll be rubbish otherwise, wouldn't it? So anyway, we we're funny, no surprises we've we're through to the to the demo stage. And what what they said was we want an hour and a half of your time, okay? We want an hour's demo and half an hour questions. I was like, hang on, you want what? An hour's demo on bid writer, it's like 10 minutes. The whole point, the whole point is that it shortens the bid writing process and you press a button and it gets you 60 to 80 percent of the way there within an hour and instead of two weeks. And so, you know, we're gonna we're literally gonna load up some, I'm gonna show you some files that are they want they wanted um a demo. So I'm gonna I'm gonna use their their exact forms that they set with the uh with the with the uh nearly 200 questions that they posed to us. So I'm gonna use that as an example and then just press the button and put the short that cut down version, it'll only take a few minutes just to prove that it works, and then and then show them the rest of knowledge flow and all the other good stuff that it can do and the you know perpetual storage and all of that good stuff. But it's like the the bid writer piece is like, right, well, there's your first there's your first version. Okay. Now set now that's now send that around the team to to sprinkle their their knowledge and and all the bits that they haven't written down, like you know, who's the latest customer or the case study that they've got on their laptop or you know, the latest financial numbers, whatever you want to put in. But here's here within the first hour is your uh but we're just not it's doing it's working away. You don't I mean we can have a cup of tea while you're doing that, uh or the beverages are available. So it's really odd to me that they want an hour's demo. It's like who's gonna do that? Bid writer. Anyway, good luck to them. Well, good fun. But the thing that's been really interesting is playing with the agentic uh bid writer with Donald, who uh, as you know, is is a bit of a genius in this area, but we've been we've been playing with different models and um there's a couple of pieces around this that are super important, I think. So we we we switched up to 5.4 mini uh a little while ago. And um the responses we were getting weren't just weren't right, just kind of too short, too succinct, didn't uh expand on the um uh didn't expand on the uh the answers that we needed. And um so we we switched it down to 5.2. And actually we got better responses, but we were still having problems. And turns out the more you kind of dig into this, the more you understand the way that the different models work. They are driven by writing very short, very succinct answers. And so, for example, uh there was a question for two and a half thousand words, and the first answer I got back was was like less than 900. So I'd use less than half of the uh the word count. And I was like, why didn't you go? I I specifically told you to go up to the word count. Why didn't you go up to the word count? You said, Because uh long answers can be waffly, and uh I was writing it to be succinct and clear. Yes, but you haven't included all these examples and things, and why not? Um I can include those if you want me to, yes, please. Why not? So you have to fight with the thing in order to get it to do the thing you actually want it to do. But the models, the the the difference in the models was was was really interesting. And if I compare um uh that to the stuff that we've been doing in um Claude and the differences between 4.7 and 4.8, some of the 4.8 stuff is brilliant. Absolutely, I mean, it only came out what last week or was it this week? I don't know which week it came out. They change all the time, as you know. Uh but it's really, I mean, really, really good. But it's only good for some things, it's not good for everything. And the word on the on the why on the AI wires is that it's not as good as as some of the other things. And it was it was a release brought out to line up with their their IPO announcement that because they wanted to say, look, we're moving things forward. So so there's a whole piece, I think, for people about um the latest number isn't always the best model. It's about using the right tool for the job. And thinking about how we help customers do that because their natural reaction is just to want the latest number. They just want the latest thing because it's the latest. And actually, that's not necessarily true. Um, which links to uh another customer I spoke to earlier in the week who who basically said, uh, love your bid writer, it could really help us. Uh, but we've just got a new uh CTO and they've um they want to do it in um copilot. And so, you know, that's not an unusual story for us. I so I say that quite right. I'll I'll jot a few things down for you to take to the uh to the uh uh uh CIO or CTO, whatever the job title is. Uh just some things to think about in our experience of doing this for the last three years. And I I talked a lot about um some of the things that we've discussed in the past on here around things like chunking um uh and the importance of understanding uh that and and how you how you structure the model so it pulls the information back correctly, um, how you ingest, where you ingest from what you're doing. Are you doing um uh do you need OCR for optical character recognition? For those of you who don't know what that means, um uh for our audience in case he's still awake. Um and things like um uh which uh cloud uh hosting solution you've got. So, for example, just be aware that if you're using AWS, it can be 10 times more expensive than um Azure. So be really careful. You don't you make these choices uh and then you repent at leisure afterwards. So there's a whole stuff around that. Then I got into some security things, which are obvious and uh and and any good security chap should know about uh about them. Um but what I mergered on was the ongoing maintenance piece. And we've had this with so many customers, like you know, how do I keep my documents up to date? Uh, how do you do it now? You know, what are you doing now to keep your policies up to date, etc.? Uh what about quality and evaluation? We talked last week and and we will come back to many times, I think that whole and how do you evaluate? How have you got a standard set of questions that you can test responses against and indeed test models against, which is what we were doing with 5.2 and 5.4 mini? That whole kind of um things like um keeping um permissions up to date. What happens if somebody leaves? Can they still get in? You know, what happens if you get you know a poison code or a poison document injection, which you know tells you to do daft things or actually passes information out? How do you do that? And then the other thing which I've I I did finish on was the whole uh cost piece, because we've seen prices starting to rise. And and last week we talked about Microsoft shutting down their cloud um uh options because of the um the rising cost and it becoming unsustainable. So cost is a massive issue. But the the my summary to to this chat was yeah, it might take just a few weeks of engineering, but it's the kind of if you don't get it right in the first place, it will go wrong, it will down a vine, it'll waste a lot of time and money. But you will also, if you don't keep it up to date, if it's not maintained, if there isn't a commitment to maintenance, then um you're gonna you're gonna be doomed as well. So uh really really interesting that whole kind of corporate view.

SPEAKER_00 17:07

And it is interesting because it really feels to me like it's coming from a place of massive ignorance, and it's that whole I mean, this new technology that's around. We're all in the sort of um what's it unconsciously incompetent, isn't it? That's the first stage of the learning cycle, and we're all there, but then you have a bit of a play, and then you think, oh, that's good. Oh, look at that, that's magic. And I think I mean this particularly co-pilot studio is the thing that mostly alarms me because that's the rag model of co-pilot where people are you know putting their document sets in and then running that and deploying that to staff and have no idea how it works, or importantly, as we've said before, what to do when it's wrong. Yes. And I mean I just imagine I can't imagine what an IT, even a really good geeky IT fella or lady, uh, I can't imagine what they're gonna do when they're told this is incorrect. I don't know where you'd go with that. I mean you obviously start running it through AI and ask them what what you should do. But in Copilot Studio, you then can't get at the settings that you need to to, unless you're developer mode, you can't get into the bits where you need to be. So I do feel it's all yeah, it's kind of I don't know, maybe it happens every time there's a new tech and people kind of get overconfident. Maybe it's that whole we'd rather we always want to build and own it and have an enterpr and then have my empire rather than uh outsource to people with specialists. Who knows? It is crazy. But I've had um on on a similar conversation at one of our university clients I was with this week in their AI champion session. So they've established a lot of AI champions, which is great. Um, and we've got knowledge flow in there, um, and they also use copilot, which is perfectly fine because you know copilot lives in all of your documents now with Microsoft busy making it uh well what should we say, easily accessible is one way of putting it, isn't it? Um like bloody clippy everywhere you turn, you get a bloody anyway. They so they're using Copilot a lot for sort of general productivity. Um, and their IT director introduced he did a little update on how they copilot usage was and and that kind of thing, and then talked about the difference with knowledge flow. And I thought he did a really good job of um in a very short introduction of you know, copilot is general productivity. Knowledge flow is the specialist agency tool that is there with a bunch of agents which know all about the right data that they need and the process, whether that's creating a new degree program, whether that's policy kind of answers for students or staff, or their annual monitoring review, or their APP process, or a whole range of other tools that they have. It each one of those agents knows all about that, know has got access to the data it needs and only the data it needs, not everything anywhere that means you pick up all kinds of dirty old data. Um, and so yeah, I thought it was a nice intro that he kind of did to to talk about the difference, which is great. And and then the big challenge, of course, is when you measure use, Copilot, his copilot had a load of use. Of course it did, 785 users. That's because everybody's got is already logged in, whether you like it or not, you're logged into it. So you're a user. Secondly, you it's all over everything, so you only got to touch it by mistake or whatever.

SPEAKER_02 20:23

Which I do all the time. I don't think if you try and write an email or something and you click slightly the wrong place, it opens up co-pilots. Go away. I just want it to bloody type number.

SPEAKER_00 20:32

Yeah, exactly. So all of those probably counting, and I think but the real point is uh measuring just purely on numbers, because a single, as I said to them in the conversation, a single prompt in knowledge flow into say a policy agent or assistant as we would call them, and not a gentic, by the way, those things. We do have a gentic stuff, but they aren't, just to be clear. Um, but a single prompt in there has probably saved you 20 minutes, half an hour of looking around or asking people, or you know, having a map waiting for your next week's meeting or forgetting altogether. Yeah. Um, whereas a single prompt in Copilot probably saved you about three minutes. So I think just the simple measuring of the numbers of use is really not sensible. And I think the interesting thing, I think I read something about this this morning, is we need to get to the point which CFOs are already at now, I think, or getting there, of measuring the outcomes and the impact. It isn't about how much use have we got and how many tokens have we burnt, and isn't that great? It's are our staff staying and then liking their job more, are they more productive in their job, and are they are we delivering better outcomes for citizens and customers? That's the real measure. And I think if if and when people get to that point, that's when knowledge flow is gonna show itself as the platform you need to be on. So I really think so. But on your pricing, just quickly, um I so I think I touched on this last week. About token costs in Claude, and this is a real problem for Claude. Is there is an organization and they have remained anonymous, and you'll see why in a second. They burnt $500 million of AI tokens in Claude in a month because they gave all staff access to Claude and they had no restrictions and they did not set a limit on the on their account. 500 million. That is unbelievable.

SPEAKER_02 22:28

That is the ex um the ex CTO looking for a new job.

SPEAKER_00 22:36

That is incredible. Can you imagine? And our blog our blog post this week, this weekend, is on the challenge of if you give people unlimited access, then guess what happens to people are asking what the weather's like and uh getting it to do all kinds of things it needn't bother with.

SPEAKER_02 22:56

Yeah.

SPEAKER_00 22:56

But uh yeah, really. A really good use.

SPEAKER_02 23:03

Yeah, yeah, yeah. But you you yeah, whoever's in charge of the uh uh the uh payment doesn't want to be paying for that, do they? That's just an incredible amount of money. At what point did the alarm bells not start going? Because actually when the invoice arrived. Well, but but but well, we set we set we get trigger warnings, don't we? When we go when we go above a certain amount.

SPEAKER_00 23:26

Trigger warnings and we have a kill uh a will stop uh a certain burn on our clients ones.

SPEAKER_02 23:32

Did you just I tell you what, let's just put it we'll put it at a billion dollars, shall we? Yeah, maybe that'll get that.

SPEAKER_00 23:38

Who does that?

SPEAKER_02 23:39

It is crazy. That won't that warn remains secret for a long way.

SPEAKER_00 23:43

We're back into the world of ignorance where someone has said we can do this, we can get a clawed license, and then clawed enterprise license, and I'll have it in. Look, it's magic. And not and just you know, the first things we do is like data retention, for example. How they're legally required to define how long you're going to keep your audit logs for. No one asks that, no one's thinking about that. What are the budget limits that we need to do? Is it appropriate for everybody? Do we want role-based access? Loads of questions you need to really think about before you suddenly smash something in. So well, they go, they've learned a very valuable lesson. I imagine that'd be very good if you ever wanted advice on it. That's right. There's someone that there's a book and a film about that somewhere. Indeed, yeah. So you um what you uh have uh you well, you know, you share the news of our wonderful um Claude announcement.

SPEAKER_02 24:37

Well, it's all Donald, really, isn't it? He just does these things and he doesn't bloody tell us, and then he goes, Ta-da, here we are. We're now Claude partners. That's fabulous, Donald. A, when did you decide to do that? Uh and B, what does it mean? Uh we don't know. Yeah, he's still he's still going through the training programme. And he sent the great little image of somebody flicking through pages really quickly trying to read the to catch up. So uh yeah, we're not quite sure what it means yet, but I I I do think it is super important for us to be able to do that model switching within knowledge flow. I I get asked about it on a weekly basis, uh, probably you do as well, but that whole kind of, you know, what's the best tool for the job? And actually, if we could have a governing agent sitting above and then choosing between OpenAI or Claude or whatever else. Interesting enough. I don't know whether you heard any of this stuff on the on the AI wires this week, but uh our guru Nitty Jones said uh it's a two-horse race, and it's really interesting. It's not so long he was talking about uh Google and Meta. He dropped Meta quite a while, stopped talking about Meta quite a while ago, but he loved the whole kind of nano banana and the Gemini stuff when it first came out, but he stopped talking about it and he and he and he used the words two-horse race uh this last week. And it made me just sit up and think, if that is the case, that's really interesting. It's just between anthropic and open AI. And um loads of I mean, as much stuff as you can read uh as you ever want to read on this uh topic um uh on on the internet, but um talk of um you know mythos already being used in cyber warfare, even though the US government's declared them a threat to national security. Is that right? Yeah, so they're kind of gosh, double standards, uh uh double counting, I don't know, whatever you want it, whatever you want to call it.

SPEAKER_00 26:30

But the mythos thing is um one of the the the rumors are the reason it's not released is there's they've got the security concerns, but is that there's not enough compute to run it. Yeah. And so they're trying to limit who can have it for now. Interesting. I expect there's a lot of truth in that.

SPEAKER_02 26:46

But people will be seeing that already. Um I'm sure that is true. I mean, the compute to run these things is massive. And you know, I do get kicked out of Claude on a regular basis. It says you know you've run out of tokens for today, come back tomorrow. And um uh or upgrade to a a much more expensive enterprise version. Yeah, yeah. And actually, I think that is the bit that people have just you know, they've taken the whole um it's kind of free or it's nearly free, too literally, and uh the prices are going to go up massively. I've said that before on this this podcast, but I think it's just something that we and our customers need to be super aware of is that you know this stuff isn't it isn't free and it's it's going to get more expensive. And I think the real the real challenge for lots of organisations, especially those um customers, who should we say are a bit more challenging sometimes and you know the ones that are uh uh AI deniers or even AI curious, you know, they're they're gonna get left behind by those who invest in in AI and the whole speed thing. You mentioned it already. Um but I do think you know the speed kills, it definitely does, and and keeping up to keeping up to date is so important, and and people just it just ignoring it is absolutely the wrong thing to do. But then I would say that, wouldn't I? But um I'm I'm believing.

SPEAKER_00 28:06

I think you've got to have the strategy now. That's what I mean. The point for the talk I'm doing next week is that you've got to have a strategy, and your strategy might be move slowly, we're not gonna do anything this year, or that's fine if you want to say that. I don't think it's fine, but it's at least a strategy. I think you've got to now decide where what is the part AI can play, and how are you going to go about looking at your business in a way that frankly a disruptor can turn up and go, right, thank you very much. We can do anybody shipping electronic stuff, anything that could be shipped by easily by you know, either because it's software or or just knowledge or whatever. I mean, all of that stuff is just hard to see how that survives. You just have someone turn up with a better version of it with more personalization, probably. That's you know, that whole thing we've talked about before about being able to instead of here's the training, you can have here's the training as it applies to everything you do. And that's like we can do that now, right this second, and it would be amazing. And you know, so what you know, I don't know what the future of that kind of world of trainers that are just given blanket training, run this thing and it will know all about your business, and the training will be exactly personalized to what how you the things you need to know.

SPEAKER_02 29:20

Yeah. I think if we can crack that um knowledge layer, the um the knowledge graph. The context layer we talked about. Yeah, the knowledge con the on top logical knowledge graph semantic taxonomy.

SPEAKER_00 29:35

I got into trouble today at stand-up on that to interview, as you all know. I was um well, Ibby, I I was talking to Ibby about it. Yes, yes, no, you did, yeah, yeah. I know it's settings. I flooded I flooded Donald with all kinds of things, and uh I I was chatting to Ibby about it and just said, Look, just have a little think. I you know, don't you don't need to talk about it yet, and I'm not asking you to do it, but and then in stand-up he said his number one priority today is working on the knowledge graph. And then Donald's nice, what I'm gonna get to no doubt Donald will shout at me later, but um and deservedly I deserve that too, I'm sure.

SPEAKER_02 30:12

Don't don't worry, he's been shouting at me all day about the uh the the agentic bid writer. I keep going, and another thing, there's a is this can he just do actually and and that's a problem that's a problem for us, and it's a problem for our customers, isn't it? The customers who keep going, and can it do this other thing? And can it do this other thing?

SPEAKER_00 30:28

Let me tell you about my frustrating customer, though um I'd obviously have to keep this very um uh what's great.

SPEAKER_01 30:35

Yes, indeed.

SPEAKER_00 30:36

But so back in January, I think it was when I met a customer that was wanting to do something that involved a rag AI assistant um being publicly available. And I was like, Great, yeah, no worries, that all sounds pretty straightforward. Um, and they were like, Well, can you give me an idea on costs? I was like, Well, how many people will use it? Well, I don't know how many people will use it. Well, and then I can't really tell you about how much it's gonna cost you, can I? And they're like, Well, can't you just give me an idea? And I was like, Well, I can give you an idea, so I've given them an idea. Anyway, five months later, and I've been to various meetings with them, I've given them uh we've created a demo for them, I have explained to them how they should run a steering group session and then joined that steering group session on how they could best gather the information they need for this thing. I've like, I mean, I've days and days of work as part of a kind of sales cycle, and it seems to me every time I give them something else, they then go away and talk to other people about that thing. So they're kind of using free consultancy advice from me a lot. I don't mind because I I'm happy to help, but ultimately we're obviously here running a commercial business, we can't afford to spend days and days with people, loads of questions back and forth, and then eventually it was like, right, can you quote against this here's our statement of work? Um, I always am frustrated at that point because it's like, well, I've showed you what to do, talked about you've now written it back clumsily with some other things which are really difficult, and you now want me instead of sending you a standard proposal, which would be quite simple for me to do, and now I'm gonna rewrite a proposal in a way that answers your question. So I'm always frustrated in that. Um, and then anyway, so um, yeah, I have taken the view that we will need to charge them more, and that's their fault, really, because I know they're gonna be challenging all the way along. I know from the conversations we've had, there will be things that they say, well, can it just do this? Can that button just be there? Well, we just want this little thing, we just want that, and they all the way along are saying, you know, we're a charity, we don't have much money, we're very budget conscious, and then they're like trying to dream up. I said, I as you said to them in the last meeting that um you know, you you if you if you want to be low cost, you need to take off the shelf, and knowledge flow off the shelf is amazing and will do everything you're talking about. But the fact that you want it slightly branded like this, or a button over there that says that, or an ability for this thing to happen, is all stuff that adds complexity, and anyway, so we'll see where they go with it. But um, yeah, I'd be nice to work with them, I'd like to help them with what they're doing, but we need to price in the extra time they're gonna cost us because otherwise it's gonna be a horrific experience for both sides, and uh, we'll both feel unhid, yeah, we'll both have a bad experience. So let's see, let's see what they do, and if they choose not to, well, that's fine too. And uh at one point I advised them to get a contractor and build it themselves because they wanted no vendor lock-in, that's in their statement of work. So they wanted to be able to build it and then at the end of whatever period of contract lift it and take it somewhere else. And I said, I I don't I genuinely don't know how you would do that. Yeah, because it's like you're we're gonna have put all the embedding stuff across it in the way that we do all of that, our settings.

SPEAKER_02 33:45

How do you just take that and what you need to do is you need to create your own leading AI company and then go and find the developers and the CTO and and get them to do it is um for it for a charity that claims to have no money, I suggest that's not possible. Back to the ore ignorance thing, you know, it's kind of yeah, I guess they are trying to educate themselves, but I also I I am 100% with you on the kind of we need to be we need to be um careful about the customers that we take on board because if we if we deliver a bad experience, it's the old adage, you know, it takes takes ages to to create a good reputation and then it's to create a bad one, and and when we we can't afford for for anyone to be saying bad things about us. So we you know we I think we do need to pick carefully and and you know lots of the advice you get is around uh picking your customers carefully and and not just taking on anyone. I think I think because AI is such a tricky topic, and by that I mean it's potentially it's potentially um organisation changing and threatening for lots of people. I think it's you know I still hear the kind of is the AI gonna take my job question on a regular basis. And uh and people genuinely do feel threatened because they don't really understand it. And um so yeah, I think it is tricky. I do think you're right to um um uh I'm not sure the right set of words, price in um some buggering about factor, but actually, you know, if we could get them into a good place and it's you know a worthy cause and it's all doing good stuff, then then great. But you know, we've got to be super careful. We can't it it's kind of the long tail thing, isn't it, about you know, how much effort uh is is is uh is involved in in somebody who um doesn't just want um I know it's a toaster, but can it do eggs as well? Can you put the coffee on?

SPEAKER_00 35:40

Yeah, it's no use to me a toaster and makes toast. No, I like eggs on my toast. It's a nightmare. So yeah, so that's been my grappling this week. Um, and then a couple of other bits, and then you should talk about the um what did you call it? Large scale comms, I think is the way you put it. Yes, really, yeah, that's come up a few times, so I'll come on to that in a second. You go first. Yeah, um so I did it. Here was an interesting thing I did today, which is pretty basic AI use, but um, I just come out of a meeting which we didn't record, it was just dumb, um, as usual, forgot not to put to put the recording on. Um, and then so I had notes and I thought I can't we bother to write those up. So I used knowledge flow in voice mode and just read the actions to knowledge flow and said make that into a meeting note, and it did, and then I sent it to the what would have been 20 minutes of typing was about four minutes of speaking, which is quite a nice way. And and there's a there is an app that does it whisper flow, they're called, which are very good apparently at uh reorganising your thoughts. You can ramble at it and it will write an amazing blog post or something, and it learns with you. So when you correct it with your when it thinks you're saying pantyhose or whatever, instead of pantomime horse, um, you can correct it, and it's apparently very good at that. So um but knowledge flow was pretty decent considering we're not really playing in voice, it did a good job. Definitely picked all the words out. Um because I obviously I speak the Queen's English, so the King's English now. The King's English, that's right, yeah.

SPEAKER_02 37:10

And you're not northern. Go on, say it. Yeah, exactly. I'm not I'm not no, I'm not northern. Very funny. Right. Well, let me talk about uh one of the other topics I wanted to discuss, which was the large scale cons piece. And we've had another organization come to us uh and they were doing um multiple uh parties in multiple parts of the country, um, all wanting to get on board with the same set of staff, same set of training, same set of information, same quality of uh communication, style, tone, all of that good stuff. And it just strikes me that um uh it's just becoming much, much more um uh obvious to people that it's it's a great way to go. So I'm genuinely excited about that. And we and we were chatting before about you know, if we'd have had this tool when you know I was doing every child matters or something, or you were doing academies back in the DFE days, yeah, it would have been a great way to get you know 150 local authorities on board and uh share the same information, allow them to do the natural language careering, allow them to create localized plans which you know reference their staff but also uh took the stuff from the centre, you know, it would be a great way of uh uh of running policy. And I'm and I'm sure there are lots of very clever and bright people in um in and around Whitehall doing that kind of thing. But um I think for us, finding organizations that want to do that is um uh is super exciting. And um, so yeah, I'm I'm I'm looking forward to the next one and and and hopefully being able to share some some information about the real customers in there soon because it I think it will make a difference to them.

SPEAKER_00 38:51

Yeah, it's really interesting, isn't it? And that's when I think exactly back to those those days at working with the Department for Education trying to drive national change programs. Yeah, you want people to have the same kind of guidance but be able to work with it locally. You want to share the best practices. There's a PMO tool you can put in there. So, how are we doing? What are the updates? What are the comms messages? I need to write a comms message for my teachers, staff, whatever. Here you go, use this one because it's already knows the context of the it's an amazing potential that Knowledge Flow could could offer in that place because you can have a single organization at the centre kind of driving the data that's inside it, that becomes the single version of the truth for everybody that needs it, and as you say, localized plans drawn from the national kind of requirements or central requirements, but taking on board your local context as well. I think it's just a really, really interesting angle on the world we used to live in and AI bring crash together. So on the world we used to live in, here's a reflection for you, right? I was musing over this. You remember old um uh only fools and horses with uh uh what's his name, Triggs Broom. Yeah, the oldest broom, the oldest broom is I've had this broom for so long, and it's only had four new heads and three new handles in all that time. That's right, yeah. I was thinking about company culture and that, and and that same analogy. It's like and and I was thinking back to my media and advertising days, consulting there, and they and the reason for that is a lot of their staff turn over quite rapidly, they generally are two years around and then move on. So you get I was like, when is a company no longer the previous company? When is it triggs broom? And I was like and and and what does that tell you about culture, really? Because you know, that whole kind of change the culture thing everyone wants to do, the hardest thing to do, and yet somehow it grips an organization because I I don't want to name any organizations, it doesn't really matter, but the culture is very similar ten years later than it was, even though you've probably only got three people that are still there. So, what's going on? Anyway, there you go. So something to muse over over the weekend.

SPEAKER_02 41:04

Okay, I will. Speaking of culture, I I've got a final final little story for you, uh uh which amused me uh a little yesterday. So I interviewed um someone for a finance job. And um uh and I'll let you into a little secret to our audience. If anyone's ever on this podcast going to be interviewed me, I I always I always start with um the same line of uh uh hello, nice to meet you. I've got some questions for you, but it's only polite to let you go first. So, what questions have you got for me? And if if they haven't got any questions for me, that tells me all I need to know. Um, but anyway, this this person did have some questions for me. And the very first question off the bat was, How has remote working impacted your revenue and culture? And I was like, Well, um those two things don't aren't natural bed for those revenue and culture, are they? As it has it really, and I was like, hang on a sec, as a finance person, you should probably know that uh remote working is probably more linked to cost because you don't have an office and people aren't commuting, and so you could you could argue very clearly that it brings the cost up, but no, the rev the revenue is really interesting. And I said, I don't really understand where you've got that from. Did you use Chat GPT to write that? And the person went, oh no, no, no, I uh I I made it up all on my own. And then I thought, if you really did make it up on your own, you it's not the right, this isn't the right place for you. And if you did use ChatGPT, this definitely isn't the right place for you. But yeah, just uh uh they did have some other questions which were clearly they had thought about, but there was two in there which I just thought you've just you've just made these up and you've just taken the first things out, and you've just you've just thought, yeah, that's that's good. That'll get him. I it did get me, it got me in the wrong way, and it uh didn't get you a job. So thanks very much.

SPEAKER_00 42:58

Yeah, be careful what you wish for. Yeah, that is interesting. That is AI slop at work, isn't it? It's like AI slopping. That sounds important.

SPEAKER_02 43:05

Correct, yeah, yeah. This will bamboozle, yes, it will, but in the wrong kind of way. So uh interesting.

SPEAKER_00 43:12

Well, I will watch out then if I'm ever being interviewed by you. Um hopefully that won't happen.

SPEAKER_02 43:20

Yeah, I don't think it but I think you'll come with better questions. You always have very good questions, you always put effort into thinking about things beforehand, and you're very good at that. So there you go. On that blowing smoke up your backside comment. I'll let you go. Yeah. Have you got anything else for the weekend, or is that is that us done for this week?

SPEAKER_00 43:40

I think that's that's done. I've got a couple of things that I'll release next week. Um, and uh we'll be able to update our audience on I know what we'll be on the edge of his or her seat uh to find out how my talk. Or bed, depending on where we sleep.

SPEAKER_02 43:56

Drooling onto his pillow like Homer Simpson. There you go. Cheers, fella. Alright, take care, see you like