Fable got banned. The moral isn't good.

By the time Kieron listened to last week's episode, Fable had already been released and banned. That's the pace of AI right now.

This week: why the US export ban on Anthropic's frontier models should worry every business that's quietly become dependent on one provider, the bid that went brilliantly until a customer who didn't believe a 100/100 score was real, and the KPMG AI report where 40 of 45 case studies turned out to be entirely made up. Plus a school AI alliance doing something genuinely smart with output grading, and a qualification-design tool that turns weeks of regulatory mapping into minutes.

Two mates. A bar. Thirty years of business between them. And all they want to talk about is AI.

Pull up a stool — we'll get the beers in. 🍺

Transcript

SPEAKER_04 0:00

Right, are we doing the pantomime horsing again or are we done with pantomime horses? I feel like it's part of the uh it's part of the shtick now. We have to stick with it. So should we get this pantomime horse of a podcast underway? Although although I noticed that you're wearing the same shirt as me. So are we just gonna are we gonna coordinate outfits from now on? Is that kicked?

SPEAKER_01 0:25

Pink tie cheese days. Do you remember that back in the day? That was a that was just by that was a random thing that happened by accident. And then that was just you and Nigel, to be honest.

SPEAKER_04 0:33

I think that was just you and the schools team in sure young people and families who did other things instead.

SPEAKER_01 0:38

Schools, team, schools, team. Yes, that's why we were in school working, work consulting with schools director in the Department of Education. That's right. Yeah, good stuff.

SPEAKER_04 0:47

Anyway, that's ancient history. Let's talk about more modern stuff. Morning, right?

SPEAKER_01 0:50

What's on your list this week, Kieran? Uh well, I've got a couple of things to talk about. I um I want to talk about the uh E-ling Learning Partnership AI Alliance. So I was like, I saw the picture. I saw the picture. Yeah, LinkedIn picture. Yeah, they had a great big picture of me. It was quite uh very um yeah, overwhelming. Um uh that was good. So I talked a bit about that. Um, and then I've got a few stories from the news and things. But here's the big thing that I wanted to say. I listened to last uh week's podcast, and we uh talked about Fable and how it was released yesterday. And I'd like to talk about the pace of AI. So by the time I listened to it, it had been released and already pulled.

SPEAKER_00 1:28

Yeah, yeah, it'd be banned.

SPEAKER_01 1:29

So I do want to talk a little bit about Fable and uh and it's being pulled from the world.

SPEAKER_04 1:35

Yeah, that's on my list too. Should we just start there and then crack on? So yeah, I'd do that. I I only got to use it once, and um, the ones that the once that I used it, I thought this is brilliant. And then the next thing I get a message from you saying Trump's pulled it, and um, and then there was all sorts of nonsense circulating about it's because of a it found a jailbreak, which just seems like nonsense to me. It seems like politics is just like we're gonna mess with anthropic because they won't do the things that we want them to do. So I don't know what your take is.

SPEAKER_01 2:06

Well, I read a thing that was quite interesting was uh someone saying, Did anthropic's marketing about uh blow up in their face? So their marketing saying that this is so powerful and it can hack every system. Um, and also apparently the CEO of Anthropic has been publicly saying we need to regulate AI. Um, and so suddenly uh the Americans have decided they have regulated it in the space of three seconds and it's banned. But I think I mean who knows exactly what's going on, and it's and it's poor for everybody, I believe, because what the US have actually said is non-US citizens are not allowed to use it, including non-US citizens within living in the US. Um, and that means Anthropic is unable to have a way of testing of show of knowing that, uh validating that. So they've had to say they've just got to turn it off for everybody. So that's what they've done so far. But I think, I mean, for me, the the bigger thing is the implications of you know sovereign data is important, everyone knows that stuff. We've long since uh said and under GDPR you have to process in the EU or UK unless you've got a particular contract and clarity and agreement with a particular US companies. Um but the so now I think that just comes right into the fore in such a bigger way. If if literally your model could be pulled from underneath it, do you think if they did that with ChatGPT and just said no one else, that's I mean, our business ends. We run our I mean we we're model agnostic in that we can switch to a different model, but not securely yet. As soon as we can do it securely, we will be testing others. But at the moment, we are locked into what Microsoft make available within Azure OpenAI, and that is only ChatGPT models that are secure enough. So I think it is very concerning. And everybody should be I mean they could do that with with Excel or you know, the whole Microsoft suite. Actually, no one's allowed it anymore.

SPEAKER_04 4:04

Yeah, I I read something similar, which was along the lines of what happens if they switch off Microsoft productivity suites across the world because they want everyone. I mean, other things will rise, you know, there's no doubt about it. And obviously, there's lots of talk about the Chinese models um uh coming through, etc. But again, it comes back to uh security and that whole piece about where's my data, who can see it, who's got access to it, how do I how do I protect it? And um, and uh you will probably if you re-listen to our last week's podcast, you will you will know we we will uh we were talking about um uh doing on-prem um secure models uh testing pods. And I think that that you know that that that was pertinent, wasn't it? Uh prescience is that the correct word? Um so yeah, we need to uh we need to talk to Big D about that and make sure that uh we're we're pressing ahead. I'm sure he I'm sure he's got his big new toy in the basement. Uh yeah, yeah, we should be loading it up as we speak. So yeah, the security thing's really interesting because um I got into a conversation. I I messed up this week, I had a big error this week. Um yeah, indeed. I haven't told you about this, so this is me fessing up to you um live uh as it were. So yeah, I did um I was asked to do a uh demo of knowledge flow, uh bid writer, and so yeah, of course did and da da da. And I did all the things that I would kind of normally do. I asked the customer or potential customer about you know what the problems they were facing, you know, what you know, how they were doing bids, and you know, really interesting uh person. Uh they run two organizations, one Spain, one in the UK, um, and they've got two different bid teams, one for public sector, one for private sector. And um uh they are using uh ChatGPT uh and Claude for uh but the the free models and they they claim they're not putting any confidential information there. And I was like, oh yeah, are you sure about that? Um teams are doing that. Uh but anyway, they they um uh they they were uh something went wrong. You know, when you're chatting to someone and something changes, and you just don't know what it is, and um uh there was a comment along the lines of I I I asked if if this person understood what rag AI was, and they said they did. So I didn't want to explain it, don't want to patronize people. And um, in BidWriter, you know, when it goes through and it checks the questions and then it answers the questions and then it produces a file which says we couldn't answer this question because the information wasn't the data, it needs human intervention. And that's in a that's in what's called a markdown file. So I I show the markdown file and I explained it's a markdown file, and they just snapped and sort of said, Um, I know what a markdown file is. I was like, okay, uh uh and as it's like it was really odd. And it's like I wasn't trying to patronize them. I was it I've reflected on it quite a lot because it's um I I I like to think I'm pretty good at at uh having holding a conversation with someone and and chatting to them, but something and I don't know what it was, I don't know whether they felt I was patronizing them or or something, or they just didn't like it, and they just that was just their way of like shutting the conversation down and saying, right, well, we don't I don't like the product and we're gonna shut it down. So I swang from that to another customer who loves Bidwriter, and um we uh we got into a very different conversation, uh, which was along the lines of uh and they kind of went through a cycle of crikey, this is brilliant, brilliant. Actually, it doesn't, you know, and and it's got us to 60%, it's got us to 80%. Actually, it hasn't done 100%. Why hasn't it done 100%? It's like that's because you need to add in the other bits. You know, getting to 60% is the easy bit. That's the bar that you take from your other data and your other information. But actually, to get it to 90% or to get it to 95%, that's not impossible. It's just hard work, and you've got to put the hard work in. I can't make the stuff up for you. You've got to bring that level of expertise and knowledge to the to the bid. And and and they were kind of disappointed. It was really, really a weird conversation. It's like, hang on a second, you've already told me in the last 10 minutes that you're really grateful because it would have taken you two weeks to get to this point, and now you're complaining that it's not 100% ready. So, really odd how people's kind of thinking about AI tools and how they work. Uh uh go what goes through their head is is really interesting. So, yeah, there's a couple of bid writer stories.

SPEAKER_01 8:31

Well, that's familiar, all of those things are familiar, frankly, because you do sometimes have those difficult sales meetings which just doesn't work for some reason. And it's yeah, as you say, it's very difficult to know why, particularly on teams when you can't sort of see people really and sort of know what they're doing and how engaged they are. But I think on the um 60% thing, I mean, yeah, that is the mad stuff, in my opinion, because you think you you know getting to even 60% in an answer is taken days away from you having to do it, yeah. Which is which should and and it's the hard bit, the 60% as a human, thinking about how do I answer this question, what what case studies should I bring to it. That's the bit most people find the most difficult of starting. So the fact that it's done that, I think, is is part of it. And I agree entirely that if you want to get to 95%, and it can get to 95%, it really can, but it takes effort. And that effort is is, I think, what the future of work looks like. And I really do think I genuinely think that you know when you look what's going to happen to jobs, I think there's gonna be a load and loads of senior people who will need to coordinate an AI agent, uh, give it in giving it access to the right data, keeping that data up to date, making sure it's up to date and it's right stuff and tweaking. And and then the agents, effectively, what what was a managerial task, probably managing a team of five people and checking their work and making sure you're delegating well, I think is going to be very much like you with an agent or many agents, uh, with a different uh different approach. So I think people need to firstly understand it's worth a bit of effort because once you've got the thing doing 95% answers, it will keep doing those for at least a while till model drift turns up and your data needs updating, but it's going to keep doing those consistently all the time. But you're not doing that, it's not going to happen without people engaging. And you know, we get so much feedback from people along the lines of well, the tone isn't quite right, or you know, it didn't quite answer it properly. And you're like, Well, we can't do anything with that information. You need to tell us more specifically, and then we can, we and we can get it to 95%, but it does require that iteration and feedback loop. And there is still that, as we've said before, laughingly, is the the the thing I meet more often than not in sales calls will be I as I as a the analogy I've used is I I like you know, look at this amazing thing that makes toast for you and butters it, and you're like, Yeah, but I like eggs on my toast, so uh unless it can do the eggs as well, it's completely useless to me. Yeah, you're like, Well, not really, you could just do your eggs and this would be the best of me.

SPEAKER_04 11:09

I tell you the other thing that I you as you were talking, uh, I kind of realised there was one other point in the conversation where uh uh uh things didn't go so well. And that was the when I said uh a customer had used it and they got a hundred out of a hundred for the first part of a bid, and they said uh that's not possible. I was like, Well, it is I can prove it to you. I've got receipts, I can I can show you. And they were just they and I think that was probably where it went wrong was that kind of incredible incredible increduking speaking. Thanks very much for helping me out. Your incredulity, yeah.

SPEAKER_01 11:44

You just move your lips and I'll do I'll do the I'll do lip syncing with you. No, that is uh it's a tough one. Well, yeah, they I've had those several times, very similar kind of things, and you never really know whether you're in a sparring competition with someone that has told their team they can do it. That sometimes happened, particularly if technical colleagues come. Yes, and there is a little bit of like, let's try and show you I know what I'm talking about, and how dare you talk down to me, which is a real challenge. So uh yeah, well, never mind. As you always say, you've got to kiss a lot of frogs. So uh there you go. You've another frog kissed.

SPEAKER_04 12:24

Right. Well, uh, let me tell you very briefly about another uh frog kissing episode. So uh at the weekend I uh was trapped in the car for six and a half hours with Mrs. Watkins. And I say trapped because uh uh she whenever I'm in the car with her, she's like, right, you can't escape, you're gonna talk to me now for uh forever.

SPEAKER_01 12:44

You're going straight to the doghouse if she let me know.

SPEAKER_04 12:46

She's not listening, she's not there, she's not there. It's fine. Anyway, um, we got on to talking about AI, and she was talking about well, of course we got on to talk about AI, don't talk about anything else. And um and she was saying, I'm not really an AI denier, I just don't see how it's relevant to my job, and I'm you know, I'm politely thinking about whether I think it might be pertinent. And she said, the other thing that you don't realise, because you you're steeped in this stuff, is there's a lot of fear out there about AI. And I was like, What? She said, Well, in the news, there's lots of talk about uh people losing their jobs, and you know, uh amongst my colleagues, there's a fear that you you're going to come along and and get AI to replace us, replace us all. I was like, that's really interesting because that's not how I had seen it. I could definitely see the whole um piece about the improvement. What's that? I said, can you hear the dog barking?

SPEAKER_01 13:40

Oh I heard you uh you got you got um muted slightly, that'll be what that is. Let's put a timestamp on this and you can edit it out.

SPEAKER_04 13:49

Yeah, I've got it 20 2847. Um so yeah, um she said, Oh, there's a lot of my colleagues are worried, you're just gonna use AI to replace all of our jobs. I was like, this is about augmenting the jobs, making people do better, be better, taking all the drudgery away, taking all the really boring stuff away. And she said, that's not how it comes across when you talk about it. I was like, I was I was really quite I was really quite both shocked and surprised. And I've been thinking about this quite a lot. And um, and we keep up with all the stuff that's going on on the wires, and you know, we keep up with the news. So and and we look, we're looking for the positive stuff, not the negative stuff. Lots of people in the mainstream media are also talking about some of the negative stuff. But um, our guru, Nate B. Johns, I uh come back to him, and he talked about trying to keep up with uh AI news is is bonkers. It's like it's like you don't you don't try and keep up with all the news in the world all of the time, it's it's more like the weather. You don't you don't worry about the weather in countries that you're never going to visit. You look out the window or you get the weather forecast for later on and you use the bits that are useful to you, and you take the bits that are useful to you. So um I thought it was a really interesting analogy about rather than um rather than kind of doom scrolling news and from around because you could do that for for all of your life. You could do that with AI, but actually just take the bits that are useful to you.

SPEAKER_01 15:07

And I thought I agree with that as well, but um, I find the job the job losses thing is quite interesting because I do I'm quite dismissive of people fearful of AI because it might take their job. I'm pretty dismissive of it, and I think I'm dismissive of it because I just don't think it's true. I think it's going to change work, and I think it's going to change work for the better because who wants to do the stuff that is, you know, some of the agentic email uh inquiry handlers we've got. You know, your your job is to come in, look at an inbox, and deal with all of the inquiries in there, which most of which are just low-level nonsense, and some of which you really need to get involved with. Surely you'd rather do the stuff that's the interesting, I've got to get involved with this stuff, and leave AI to do the other stuff. And that, you know, I'm I'm using that as an example for finance. I mean, the world of finance half of the time is just looking looking down to find out who needs chasing and then drafting a chasing email. You know, I mean, why not just do that? Have that happen for you. So it's kind of I thought, yeah, I'm quite dismissive, but equally, yeah, it's interesting to hear that even uh Mrs. Watkins so close to what we're doing here and uh and is still feels that. So maybe I should be less dismissive of it.

SPEAKER_04 16:21

Or I should talk to her more often, which is possibly a possibly that's what you should do.

SPEAKER_01 16:26

There's some interesting things though, in talking about the news of AI. Um, there's some interesting things I picked up this week, which I think is interesting. So having said that word three times. Um uh firstly, a Munich court has judged that the Google AI summary that it makes now, when you search anything on Google and you get that text at the beginning, is Google's and they have to stand behind it. And it is they can't say that's AI and it might be wrong. That is their choice to put it there, and they've argued the Munich court judgment is that it is they're liable for it, which I think is very interesting in the kind of where do you have liability landing when things don't work out, things go wrong. We talked about um insurance and AI, didn't we? A few quite quite a few episodes ago, and that's kind of brings all that back to my mind. Um, and then I read, which always amuses me when the consultancies do this, KPMG again, um, who uh had a they pulled a report about AI, about the use of future of AI and that, because 40 of the 45 case studies in it were all hallucinated and made up.

SPEAKER_03 17:36

So somebody has we should get onto them and say we could create a rag for them so that they actually just pull correct data from their rugs.

SPEAKER_01 17:45

Sounds like KPMG needs some uh needs some education in AI. And I mean it's interesting your point though, being clear about it, is yes, if you build in a rag, you don't get hallucinated data, like much, much less likely to. The um interesting thing about that is I've I've had several conversations over the last week or two about website uh you just you just gathering stuff from websites to feed, you know, to feed the answer. Can't we just get the data? It is on websites, so let's go get it. And you know, we've always shied away from it because it's just so untrustworthy. And even the really good tools and perplexity I use quite a lot instead of Google, really, very rarely on Google. I can't really remember the last time I Googled something. Um, but on perplexity, it is often wrong in its sources, often, in fact, more often wrong than right. It will tag the source, like our tools do, where we say this came from this document and it's this section, and ours are bang on accurate. In perplexity, the world leader, arguably, in this space. It nearly every time, if you look at the source attributed to a paragraph, it is nearly always incorrect and takes you, it's a clickable link and it takes you off to somewhere like what the hell am I doing here?

SPEAKER_04 18:58

Yeah, no, I found that. I I found that even just yesterday when I was using it for something, and I clicked on the button and I was like, Why am I why is it taking me to this pitch? It's got nothing to do with the thing I just looked at.

SPEAKER_01 19:09

Because I think the the important test in AI, we're going to talk a little bit about how uh this in a moment, aren't we? Of of AI accuracy. One of the important tests if you're using any of the search tools is to check the sources. Yeah. To make sure that you're yeah, you're getting trustworthy sources and not just looking at some Reddit thread or or some some rant on some weird blog post.

SPEAKER_04 19:29

Did this um come up in your um when you're at your event yesterday where you had a massive picture of your head on a screen? You you mentioned something about um what a good segue. Yeah, well, uh uh you mentioned something uh before about output judgments. Yes. Yeah. So is that is that linked to what they were talking about as well?

SPEAKER_01 19:52

Indeed. Well, good segue. We um so I was at the E-Ling Learning Partnership AI Alliance. Um this is chaired by a headteacher, Hannah Widdison, and she is really excellent. And um, in fact, she's just uh is getting she's in the short list, I think, for a silver head uh award from Pearson for Head Teacher of the Year. So uh she's really impressive, really and she's a lovely, lovely lady as well, but just impressive. And then I I was uh asked to uh invited to come along and asked if I'd present an award, which is why I have that big picture with me on it. Um, and I sat through and listened to five presentations before of what they've been doing over the last year in their AI research groups. And what they've done, I think, is really sensible, is that they've got a whole bunch of the E-ling schools together and they've agreed together, you know, what things will you look at, but not just look at it in the way most of our AI is judged, of like, yeah, I liked it, or I didn't like it, or yeah, it's half the job, whatever. They did a proper kind of research back to, you know, let's look at our research hypotheses, let's then gather some baseline data, let's then do the thing and then recheck the data. So they've got some real evidence about you know pupil outcomes and send learners, and they Ealing has a lot of um EAL learners, that's English as an additional language. Um, so obviously AIs can be really helpful there because it translates, which really helps, but also help to teach them uh you know English in a in ways that that can be more supportive. Um, so what I think what they've done really sensibly is that collaborative effort, but also with that research back research methodology means that this their work is really useful for others to look at. And I hope that uh others will look at it. I really do hope it will. But one of the things that uh I really liked in there was the output assurance grid. Like uh that was the phrase that they use, and one of the other groups used an acronym of BRAIN, which I can't rem tell you what the acronym stands for, unfortunately, because but it was the same concept of how do you judge an AI's output? Because it's difficult. To judge is written persuasively, always, isn't it? Um, and can be hallucinated and entirely wrong, or indeed, quite often, just doesn't really say anything, it's kind of impressive sounding paragraph, and you get to the end of the thing, well, I learned nothing. Um, so they've they've created this output assurance grid which looks at different models because they're using a lot of the free tools and some of the uh sort of like co-pilots and that kind of stuff. Um, and then across a whole bunch of things that you need to be thinking about when you're reading the response, and they've got the grid matrix for uh sort of matching up in Google Gemini under hallucination, these are some of the known challenges, and you need to be looking at this stuff. So I think I think that's really sensible, as I said to them, because I it is the problem we are all going to face as humans more and more. And I I mean that Google story of well, there's no you can't just say AI content can't might be wrong. That's just not an acceptable thing, is what Munich have said to Google. So um, and I think that's where we're going to head more and more because you know I I've seen hallucinations or or um incorrect parts in some of our tools occasionally, and you would only know if you were like a deep expert, and any other user. This was I remember it was one of our universities, and um they had a a number of days to make a make a an extenuating circumstance claim, and it's five. There's a thing that follows in the very next paragraph in that chunk of data that talks about 10 days before uh an appeal after after a judgment, um, and it mixed that up. And I can see exactly how it did that, and very difficult, but you'd only know that if you knew that policy inside out, yeah. Otherwise, you busy send that off to a student, they then put in an extenuating circumstance claim at day eight, it's rejected, they fail their degree, their you know, whatever. It's uh suddenly becomes a big problem. And I don't think it's acceptable to say, well, you should have checked. Yeah, I don't see how you yeah, I just don't see how you can stand behind that. You can't check, unless you read the policy, in which case, what's the point of views in the AI?

SPEAKER_04 24:10

Maybe we could um contact your head teacher friend and uh or or the E-ling Learning Alliance partnership or whatever they were called, I beg your pardon, I can't remember, uh, and see if we could share some of that stuff. Uh because I think that's really interesting, and I think lots of people would be um uh interested in it. Uh interesting, I'm using too many interesting things in this conversation. Uh I got contacted by another head teacher this week and uh wanting to know about what we're doing with uh um AI. So I think up uh upping the information that we're putting out there uh in the community or uh for schools uh is probably a useful thing. We talked about this you know two years ago. Why is nobody doing this stuff? It seems to the stuff seems to be moving. So um yeah, we we should we should try and help people find good sources where wherever they can and and help them make the right decisions for them.

SPEAKER_01 25:03

Well, I will certainly put to Hannah and ask her if she if she wants uh our audience to be aware of her work. It will be you know probably a bit too much for her to be put in front of such a large audience. That's right.

SPEAKER_03 25:18

Especially if he's still snoring.

SPEAKER_01 25:21

That's right, yeah, we'd have to wake him up. Matt, we've got a guest on. Wake up.

SPEAKER_03 25:27

Stop drooling again, honestly, it's rubbish.

SPEAKER_01 25:31

Very good. So, what else have you been up to? Anything exciting this week? Have you been uh battling? No, nothing else. You've just been looking to be bid writer for getting to zero.

SPEAKER_04 25:41

Getting that stuff is important, isn't it? I mean, because so much depends on it, and and it's it's the lifeblood of of lots of organizations winning new business. And um, so getting it right for people is is hard, uh, but it's necessary. And so, yeah, I have spent a lot of time with it. I've spent a lot of time with Donald, and we've been tweaking things and you know, getting the output pass. The thing that's actually consumed quite a lot of time is um getting sensible answers out of the users, getting them to just rather than just go, uh, I don't like it. It's like, well, no, no, explain to me what it is you don't like, or why is this answer wrong? Or and then just going, This is AI word salad slop. I get that. Okay, so what you know, what could we do instead? You know, what where is it not pulling the relevant case? So we can fix all of those things, but actually getting people to put the hard work in uh and then on the other side, I've got another, I've got two customers who are or potential customers, they're not actually customers, uh, who are super enthusiastic. I mean, a crikey, it's like having puppy dogs running around, it's uh hilarious. And I yeah, could we put all of the all of the data in the world into a rag and then we could just say, oh hang on a minute, that's just yeah, yeah. It's got the internet, yeah, yeah, but yeah, but it'd be secure, and then we could control it, and then we could then we could interrogate it, and then we could get really good answers out of it. And um, yes, you could, but yeah, actually, how you do that, it's not as just simple as you you put 10,000 documents into a rag and then let it go. It's how does it I mean I uh it probably bears uh repeating, but your analogy of the kind of the library, you know, uh the this uh uh it's you know, top P, top K, I'll let you, I'll let you tell this expert because you do it better than I, but that you know, how many books do you bring back, and then which parts of those, which chapters in those books do you look at, and then how do you reference them? You know, people uh you know, it's we're back into the magic bullets and and and unicorns of uh so uh yeah, it it it's interesting. So it's been it's been quite a frustrating week in that kind of juggling those two ends of the spectrum of kind of and uh everything in between. So yeah, it's been nuts. What else have you been doing apart from bloody uh having your big, big old ugly mugs on big screens?

SPEAKER_01 28:06

I have been hackathoning, which is um always fun. So uh we we put on, as you know, a uh AI hackathon for our tip for our customers, um, which is for them to choose what they would like to work on, them to bring the people um that sort of involved in that process, and then we kind of work through workflow with them and what's the where are the real uh laborious tasks and what data is involved, and then we can normally create very quickly something that can at least start that process. So we're doing that with um uh qualification development, which is pretty challenging stuff. So that is um creating a new qualification in response to either a demand in the market that you perceive if you're a university or if you're an awarding body, equally a demand in the market. Um, an exam board is an awarding body for anyone else for our listener. Um, and um you or the government say, you know, we're launching this new thing and we want some qualifications behind it. So really quite obviously quite a complex process to sort of think through how do you design what are the learning outcomes that will deliver a person at the end of the qualification with the skills and knowledge and experience that they need in order to do the thing, whatever the thing is. So, really interestingly, so um, with one of the teams, we built out this long workflow with them. I think it's the first time they've ever had that conversation for a start. And do you remember back in consulting days? I I've always said I think probably the highest value thing that I've done many, many times in my career is either workflow business process mapping or project planning, which is effectively the same thing, only looking to the future and sort of working through the questions and and and thinking through the steps needed to deliver a project. So I think they've I think they actually they got a hell of a lot out of that without really any AI being involved yet. Um, and I've then been working with the team here on uh building out the prompt and tools that would be needed to deliver their thing, and really interestingly, um, very much human in the loop. I mean, it's really important human in the loop on this stuff, because you're designing a qualification and you can't just press a button and go, There it is. Secondly, it's regulated because of uh uh qualifications are regulated, you need a really good audit trail. Um, so you need to capture kind of everything that really happens along the process. So we've built a version in knowledge flow of a kind of what we would call a step-by-step function. So you effectively work with the with knowledge flow. Um, it asks you so you load the kind of whatever you've got, which is either your business case, because I think there's a gap in the market, or the government document that says here's the qualification, or anything else. It then takes you on the journey towards completing the qualification. Um, and takes a while. It's really interesting. You don't have to do it in one go, of course. You can come back to it as often as you want. But in the RAG index, we've got all of the documents that they might need to draw information from. So that might be sector skills, things, national occupational standards, the regulatory rules on designing regulations are in there, um, lots of skills strategies, all the kind of stuff that you might need to draw on is already there. So you literally it will start off and take you through the process, and then it says to you, I think, you know, the here's the quote based on what you've uploaded, here's who it's for, am I right? Um, okay, these are the skills I can see in there. And by the way, I've had a look in the subject skills and the national occupational standards. So there's some more skills that might want to be in there. Do you agree? Do you want to change anything? Then it's like, let's theme those into groups so that we can start to design the kind of modules and units of the qualification and so on. It's really interesting. And of course, and it can do all the mapping to skills, it can do the kind of you know, picking out all the kind of regulatory pieces you need to cover. And it does that, you know, in seconds, all the bit that must be really frustrating if you were having to do this, you know, go and just wading through and checking off mapping, they call it. All that stuff happens instantly, leaving you to do the bit you want to do, which is like, oh, how would I which order would I put these units in to deliver those skills? And maybe I should assess like this. It will still suggest all those things to you if you want it to, or you tell it, and then it just writes it up and adds the bits in the or the admin. Brilliant. And it's all an audit trail at the end of that. You get your your output, and you've got a full history of all the discussion you had, when you said no, when you said yes. So you've got the regulatory part captured as well, should you need it later. So I'm very impressed with it. I think it's uh an interesting um approach, which is what much more human-involved, human-in-the-I would say that is, which is the garden review, isn't it? So, yeah, I'm really I had a great time doing all of that stuff, and um I've got to do a few more of those over the next couple of days because there's other angles we want to take that into.

SPEAKER_04 33:02

Excellent. The uh switching on the reasoning thing was one of the things that helped with my bid writing piece with my colleagues, just switching on the reasoning so they could see the kind of right, and this is what I'm gonna do, and this is what I'm looking at. That really helped them think through uh some of that stuff. Um, because actually that's looking at the wrong thing, or it's not making the right kind of I can understand why it's now making those choices. So that observability, as we as we has been pointed out to us a lot, that observability piece and that audit piece, that who do you trust? Um, and how can you can you show your receipts? Yeah, it's just it's just super important, and it's not gonna that's not going away anytime soon. It's just gonna it's gonna be a topic we come back to time and time again, I'm absolutely sure.

SPEAKER_01 33:42

Yeah, indeed.

SPEAKER_04 33:43

Yeah, yeah. But the hackathon things are really interesting. Did you um you didn't get out pieces of brown paper and post-it notes after us from your post?

SPEAKER_01 33:50

I did. I did I used um a mural board for it. So the electronic version of brown paper. Exactly. It's the po it's the better version because it's got a limitless end and you can suddenly move everything up one when the problem project planning was always a problem because you always end up going, oh no, I haven't got enough space left and I've cut it everything here because the next the next three weeks are yeah, funny, funny. But they all asked for it afterwards, which was the giveaway that that was the bit that they they said, can you send us that workflow? And because it was like, you know, step by step through what happens, what decisions are needed, where's the data come from? You know, it's a really useful thing for any team to do because you know rarely do teams sit and actually work through that stuff. It it's kind of taught to you almost tacitly when you join an organization. You don't rarely does someone set a workflow out and say this is how we do it, and be like, right, can you do that thing for me? And eventually you work out what that's part of a workflow. I think that's it. So actually, having that uh that session is really good. And I it's great for us.

SPEAKER_04 34:51

Two things fashion through my head were one is um interesting that it takes time, you know, and and people think uh especially when AI is running, they like expect instant answers. It's like it's doing a lot of processing in the background, you've got to understand this. Is this challenging? It would take you weeks, if not months, to do this. So, you know, the fact that it's taking three minutes, stop complaining. Um, and the second thing is I can just see that being super useful for personalized learning. You know, that whole kind of, you know, I want to, I want to do. Um, you've you've undoubtedly already thought of this with some of the people you're working with, but that whole kind of you know, I want to do a degree in underwater basket weaving with uh pottery and um uh uh something else random. You just like here it is, here's the modules, here's how they all put together, here's how you create a qualification, here's here's here you need to teach, this is just everything from kind of which specialist room do you need to be in. I remember that was always the time-tabling challenges with schools. How do you get but I I could just see that from a qualification, but a personalised learning perspective. There was something on the radio this morning about um you know, should university places be limited uh to only to people who've got a GCSE in in English? It's like it's kind of and there was somebody on saying I didn't have it and I've benefited from it, and you know, I'm now 30 odd and and I've got a really good job and I'm doing some important work with lots of different interesting people, but actually that whole kind of personalized learning around I want to be able to do this skill. So absolutely.

SPEAKER_01 36:24

Well, do you remember the so we did a long time ago? So um personally, I completely agree. We've pitched exactly that model to uh an executive business school, uh, which didn't go anywhere. Uh yeah. But but the idea is, and the university that we're working with um could do the one I'm working with on Qual Qual's um uh it could could do this because they've got a module um rag index within Knowledge Flow. So all of their modules are in there. Okay. So they could like literally say, I want to weave together the modules to create this new thing, and it would know all the modules, it already has all the details on them, could do it instantly. But the the executive school uh was they have a whole bunch of different things going on, some of which they know about, but no individual knows all of it because it's so much leading edge stuff and research. The idea that you fed that into a rag index and then could create your own, particularly as an executive thing, which which might be, you know, I want to be a I'm currently a director, I want to be a CEO, uh, and I want to get the finance stuff and public speaking, whatever it is. There you go. There's your personalized set of things to work through. And I've had to think about the right order to do them in. Here it is. I think that's it's really interesting to be able to do. We can do that right now, um, but yeah, they didn't they didn't go forward with it yet. Maybe one day. Maybe maybe. Yeah, and I think and I think that that I mean I I remember working in DFE. I say this on stage quite a lot to schools, audiences is um, I remember being right at the beginning of DFE, and personalised learning was in this transforming secondary education for the white paper in 2001. Um, and um I remember sort of even then thinking, what a load of toss. You've you can't you've got to remember all your students individually and give them all a personalised experience. Oh, by the way, you've got 30 30 pupils in the class, so uh just do that with you. Um and by the way, there's another different 30 coming in after that, by the way. So um but just you just do the personalized stuff. That'd be kind of nonsense. But AI does make it possible, which is what I think is very interesting because it also can lower the reading age for various things. The one of the things that came up yesterday was um one of the one of the people speaking was saying that they are uh teaching year six, year, I can't remember what yeah, year six, at the end of primary school boys phonics that they should have known in year one, but they haven't. And but he said, I've been using AI to write stories for them on whatever we do it on what they're what what do you want to write now? And I've trained he's got like a I think it's a Gemini model, so he's got a gem trained on the phonics stuff, and then he basically says, Right, let's do it on Minecraft, let's do a 120-word story. There it is, done, let's read that. And you know, so something that really presses their buttons, of course, engages them, and of course, is giving them some of that stuff which you know they're very behind in their reading. And he's the results are very impressive. They've got they hears they have tracked because they're in the AI alliance, they have got the benchmark data, the baseline data, sorry, and know how it's improved.

SPEAKER_04 39:26

Well, you did you did personalized learning for your boy with his geography GCSC, where it got it answered questions in the style of Drake, and nobody none of us knew who Drake was. I thought he was a I thought he was a buccaneering pirate type person from the uh middle ages.

SPEAKER_03 39:41

Uh he was uh I did beat the Spanish Armada after playing bulls. Turns out that was it was a different Drick entirely.

SPEAKER_01 39:49

Different one, not the tobacco, not the tobacco potato discovering version.

SPEAKER_03 39:54

Yeah, yeah, that's right. I think that was Sir Walter Raleigh. You you need you need a you need a history rag.

SPEAKER_01 40:00

Yeah, right. I haven't got room in my brain for all those things. I'm busy filling my brain with AI stuff. Yeah, no, I did create that for him exactly. And that idea, and we've also we had a um for one of the colleges we did a some experimentation around, so colleges have a particular problem that there is a regulation, a law, is it a law, regulation, don't know, uh, on the you if you have not got a a grade four at GCSE English and math, then you need to resit. And colleges, given the kind of audience they tend to deal with, uh, have thousands of students like that. And guess what happens is they sit back through the teaching of English in exactly the same way they failed last time. And guess what happens at the end of it? No big surprise, you fail again, and that goes round and round and round, and presumably it doesn't make you feel particularly confident and as awful for you as a human thought. And so they were looking at can they personalize the content more, make it more relevant, you know, make it about rugby if that's your thing, and make it about the if you're if you're doing a psychology course or bricklaying course, let's make the examples relevant to that rather than just the regular stuff that everybody else reads. So um, yeah, that I mean interesting and and possible at the press of a button. That's the stuff once you've designed it, it's you know, you can do that quite quite easily. So I hope there'll be more of that because I think it will help out in all those spaces. On that GCSE thing, here's the weird thing, right? Because the way GCSE grading is done, they are always the same proportion of unless they unless awarding bodies make a um uh submission and ask for a change, they're always the same sort of A nines, nine, seven, and eights, uh seven, eight, nines even. Um, uh, and and then go down. So so it means that you always have the same number of people that are gonna fail. And that made me think, hang on a minute, what's going on there then? And how does that work? That those ones that are then gonna resit, do they displace some others that are definitely gonna fail? How does that really? I was thinking that can't be there's something odd in the maths there, I think.

SPEAKER_04 42:02

Yeah, maybe uh somebody uh maybe our audience could uh enlighten us onto how these things work when he wakes up.

SPEAKER_01 42:18

Do you remember mail? Do you remember post that?

SPEAKER_04 42:21

I still get posts, it's mainly junk mail, is the honesty. Although I've started getting things for you, dear Neil, you're getting very old. Would you like to go on uh a bus trip to uh the outer Hebrides? Or uh you should be you should be have you planned your will? I was like, hang on a minute.

SPEAKER_01 42:38

Bus trip to the outer you're going on holiday. Where are you uh where are you off to? Anyway?

SPEAKER_04 42:44

Uh we are going to Scotland actually, just to be just to be absolutely clear, but we're not going on a coach trip. Uh yeah, no, no, no. Mrs. Watkins has a very significant birthday coming up. Oh, I saw Saturday. So, yes, we're going away. Her sister is over from Australia. There are a whole lot of friends and family going. It's going to be awful. Um, and I will have to uh medicate heavily with the usual in the usual fashion uh in order to get through it. But uh yeah, I uh I'll I'll I will be on my best behaviour to avoid dog uh the dog house again. Um so yeah, that's that's that's next week. So you're gonna need to find um uh a guest. Um you're not doing that to me again, are you?

SPEAKER_00 43:26

You're not doing that.

SPEAKER_04 43:26

You're gonna well wait, what are you gonna do? You're gonna be able to do that.

SPEAKER_01 43:29

I'm gonna I'm gonna dial you up in Scotland and you can join with your uh with your well no.

SPEAKER_04 43:34

We might have to change the day. No, we'd have to change the day. Um uh because the our regular day is a travel day, so I couldn't I couldn't do that day. But yeah, all right. In effect, that's a really good excuse we get out of doing family activities. Kieran, you're a genius. I always knew you were a good friend of mine. You said you just said, oh now I'm gonna talk to Kim for the next hour. Uh three hours. No, the next I'm on all day. Yeah, yeah.

SPEAKER_00 43:57

Yeah, yeah, it's an all-day hour. We're doing an all-day podcast. God audience really would be asleep. Yeah, he definitely would. He definitely would.

SPEAKER_01 44:06

On that note, shall we wrap up?

SPEAKER_03 44:08

And um probably should.

SPEAKER_04 44:11

Uh so so yes, I'm gonna go and pack my bag. I will um make sure that uh I have uh all of the goodies that I need to get me through next week. Uh I've also make sure I've packed Mrs. Watkins' uh presents or some of them. There's some of them are too big to get in the car. So uh this one's first. Yeah, yeah, yeah. So I uh I'll report back next week from from Bonnie Scotland. So yeah, all right, let's do uh let's do that. Uh we'll arrange a time with you. In the meantime, you have a good day, and I'll thank you there.

SPEAKER_02 44:41

Cheers, yeah.