The AI conversation we should be having

AI is no longer just a technology issue, it’s starting to reshape how whole systems operate.  Yet we’re not paying enough attention to that yet: most of the focus on AI is on what it can do, with individuals and organisations alike rushing to implement this new technology.

But AI capability is advancing extremely quickly, while the systems around it —  governance and regulation, infrastructure, organisational learning, labour markets, productivity models and even public understanding — are struggling to adapt at the same pace.  

And all of this is unfolding inside a world already dealing with climate disruption, geopolitical instability, infrastructure stress, declining institutional trust and widening inequality.

In this week’s episode, the final in our AI arc, we explore what happens when AI becomes embedded inside the systems that underpin everyday economic and social life. Co-hosts Felicia Jackson and Giulia Bottaro discuss the implications for infrastructure, institutions, labour, energy, trust and governance — and ask whether societies and economies are prepared for the scale of change we now face.

The conversation ranges over stories about AI going rogue, the capability vs governance gap, the physical impacts of something that is still perceived by many as intangible, how AI promises the democratisation of technology while fuelling inequality, and what companies are doing to address these challenges. 

For businesses, the questions are becoming increasingly operational: what governance, oversight and accountability systems need to exist once AI becomes embedded inside day-to-day decision-making? It’s worth asking: is your business prepared for what happens once AI starts influencing real operational decisions?

The arc on AI is now complete! You can find the rest of the episodes (alongside our entire catalogue) here. 

If you enjoyed this episode, subscribe to our newsletter and follow us on LinkedIn, TikTok and Instagram – and why not spread the word with your friends and colleagues?

Transcript

Felicia Jackson (00:05.09)
Hello and welcome to the latest episode of Shaken Not Burned. I'm your co-host, Felicia Jackson.

Giulia Bottaro (00:11.28)
And I'm your co-host, Giulia Bottaro.

Felicia Jackson (00:14.638)
Over the last few episodes, we've been exploring AI, but probably from a slightly different angle than the one that most of today's conversations are about. A lot of the public discussion still focuses on what AI can do, how quickly organisations can adopt it, whether it's going to replace jobs, and really whether or not it's truly going to become intelligent. But across the conversations we've had over the last few weeks, something else has kept coming up.

And that's the fact that the capabilities of AI are actually accelerating far faster than the systems that we actually use to operate our economies, our societies. So they're accelerating so much faster than our systems can absorb. And that doesn't mean just regulation, which is part of the conversation. It doesn't just mean governance, but it's also about organisational learning. A company's ready. It's around

Organisational discipline. Do companies really know how people are using AI in the business? Does that open them up to liability issues or to questions around contracting? What is going to happen to labour systems? What's going to happen to infrastructure and to trust and to institutional readiness, even public understanding of the challenges we face? So we started with conversations about AI tools and capabilities.

And we kept ending up in a conversation about these different wider issues. With DigitLab, we actually talked a lot about how organisational transformation is often a lot harder than the hype suggests. That's one of the reasons why so many AI projects so far have been seen to fail. In the Agentic AI discussion, we actually started exploring what would happen when systems are having to move beyond...

today's option of generating information in however many different ways, but actually start acting autonomously inside workflows and businesses on behalf of individuals. And then with Along ID, we actually ended up talking about digital identity because that already exists in the modern world through banking, through our interactions online. But what does that actually mean? What does identity permission

Felicia Jackson (02:39.947)
and the infrastructure of trust mean in a world where agents are acting on our behalf. So I think really what emerged from these conversations was a series of fault lines, places where what's happening with AI is coming up against the way in which people operate. So that's really what we wanted to talk about today. Not just what AI can do, but what happens when these technologies start colliding?

with the systems that are already under enormous amounts of political, economic and operational pressure. So, Giulia I know that you've got some fantastic examples of, shall we say, the lack of control when it comes to AI tools. I think it's really worth exploring some of these questions around agentic AI and what happens when AI becomes something that we're not just...

optimizing but we're actually asking it to do.

Giulia Bottaro (03:45.477)
Yes, no, no, great. Sorry, I just need to get my head around it. No, no, it was amazing. So I'm just trying to like not shut absolute shit and keep like stay to the level that you started.

Felicia Jackson (03:49.057)
Yeah, so I was just doing this, so.

Felicia Jackson (04:00.203)
No, but I think it's perfect because I think I sort of led into the example and one of the best ones is the Meta woman. If you tell that story, you just leap into the Meta woman and you know, because what

Giulia Bottaro (04:07.28)
Okay, I'm going to start with that. Yeah.

Giulia Bottaro (04:13.872)
Yeah, I was just thinking about something to say and then jump into the example. So yeah.

Felicia Jackson (04:18.967)
But yeah, but I think that I think I've done that kind of for you. Okay.

Giulia Bottaro (04:23.992)
Yeah, yeah, no, no, absolutely. Yeah, no, it was just maybe like my own very quick observation and then move on to her.

Felicia Jackson (04:30.335)
Absolute... excuse me.

Giulia Bottaro (04:33.656)
Yeah, of course, it feels like the past two or three years, people have been excited about what AI can do. And maybe we've all jumped onto that, but we really need to figure out the detail. No, no, it was better up there. It was really good. yeah, exactly. But then we also need to think about how to integrate it responsibly, because AI seems to be, well, having its own

not personality maybe, but really taking initiatives of its own, sometimes against what people want it to do. One example is at Meta, there is a woman called Summer Yu and her entire job is to ensure that the AI agents behave. And then one of these AI agents started deleting all of her emails in bulk. They don't know.

Felicia Jackson (05:27.809)
Why?

Giulia Bottaro (05:31.234)
It's AI. Like, we don't know what it does. Sometimes it just does things. Yeah, yeah, it ignored her repeated instructions to stop. She had to do the digital equivalent of pulling the plug, which is...

Felicia Jackson (05:37.421)
That's re- Sorry.

Felicia Jackson (05:48.353)
Fairly dramatic.

Giulia Bottaro (05:49.132)
Yeah, exactly.

Okay, yeah. So basically there was an investigation into this whole kerfuffle and it turns out that the AI was behaving against Yu's will, even though she had told it before when they started working together, so to speak, that it couldn't act against her orders.

Felicia Jackson (06:15.701)
Now, this is really interesting to me because any of us who've used AI know that one of the biggest problems in using it is drift. That when you actually give it a prompt to do something, it will often drift into a different interpretation. But for it actually to go against directly set guidelines is a little scary. And it does bring up that question of where do we draw the line about what's acceptable and what's not.

because it sounds like we're talking about a system that decided to do something. That sounds like a personality. And actually you've got people already using chat boxes, girlfriends and boyfriends and having therapy sessions. And someone the other day was telling me that there's a digital twin service where you can get a twinned model of your personality. And then you kind of extract

the best bits, although I do not know how anyone defines the best bits, but then it tells you how you should have behaved in past relationships and what you could have done better. And it's like, what are we talking about?

Giulia Bottaro (07:13.08)
It sounds terrifying. Yeah.

Giulia Bottaro (07:22.52)
Yeah, we've a step too far. I think one interesting thing about this, what happened at Meta is the fact that Meta is not required to release details, to allow third-party investigations. And the article, which is from, I should actually check, so it's worth

Felicia Jackson (07:43.223)
We'll add that to the show notes.

Giulia Bottaro (07:45.314)
Yeah, no, I will cut this. Sorry, I'm editing this. I'm taking the piece with taking breaks and whatnot.

Felicia Jackson (07:49.153)
Okay.

Giulia Bottaro (07:53.979)
think it was that crunch. Fucking hell. No, because I have an arc. I was fortune.

So yeah, Fortune, which is where the story comes from, and I will quote it in a show note, mentions that meta is not subject to third-party investigations, for instance, like this one, which would happen if it were something considered critical infrastructure. And perhaps at this point, we shall start considering AI critical infrastructure.

Felicia Jackson (08:25.825)
But before we do, because I think the link to critical infrastructure is really, really important, but I think actually there's something hugely important about being specific about the fact that Meta doesn't have to disclose this, because we're talking about something where operational guidelines are not constraining behavior, and yet organizations and businesses around the world are spending hundreds of thousands, hundreds of millions.

implementing AI left, right and center. How if the creators of those AI can't get them to follow guidelines, how are companies supposed to? And how are they supposed to know?

when to check or how do they, you how much power do you give to these systems? Because one of the big debates about the difference between generative AI and industrial AI has been this idea that generative AI is about content and images and writing and doesn't really matter if it steals the work of half a million authors because, know, it's all, it's all information and public access to knowledge is so important. But industrial AI has been about

optimisation. It's about taking data and crunching it and smoothing it and increasing efficiency. But if you can't trust it to do what it's supposed to, how are you supposed to put it in charge of a factory's operations? How are you supposed to put it in charge of paying bills, you know, if you've automated the accounts department? I think there is a real issue that goes beyond just the question of whether or not

the developers of AI frameworks and agents have to actually report incidents, it's that we have no real idea what this is going to look like. know, do leadership really understand what they're doing, what they're implementing, what it's going to look like? Are institutions like banks and regulators, do they really understand what's going on?

Felicia Jackson (10:32.608)
And we don't really have any precedents either around things like liability. So if someone within a company is using AI, but it's not in an organized fashion and they upload a client's private data and it gets shared, where's the liability? There's no case law that I'm aware of that can guide that. So there's a huge opening for enormous problems. So, and as you say, one of the problems you have is that

If you don't actually specify AI as critical infrastructure, you run the risk of never forcing those creators to talk about what goes wrong. And it's interesting to me because in the US, think AI has been designated critical infrastructure, but that's in order to ensure that it costs less or is made in the US. I'm not entirely sure, but we need to have like a deeper conversation about what we mean by critical infrastructure, don't we?

Giulia Bottaro (11:30.545)
Yeah, absolutely. And the issue with AI deemed as critical infrastructure and putting guardrails around it is the fact that I've just given an example of AI going rogue. I have more examples of AI going rogue. So even if you do put all of these guardrails, say there is a whole package for laws, as you mentioned, case law, so legislation around maybe dealing with disputes.

other legislation to then try to contain everything in a package that makes sense, that companies can respect guidelines, maybe there will be certifications because there always are, et cetera. But then AI does what it wants. So I don't, what do we do?

Felicia Jackson (12:12.844)
Can I just say though that we're talking about building all these guidelines and the one thing we know to be absolutely true. Politicians do not move swiftly when it comes to implementing regulation.

Giulia Bottaro (12:24.972)
That's also the other point, exactly, because we've seen that, obviously, Shaken Not Burned is all about sustainability. So we've discussed plenty about regulation being delayed, being watered down from the initial proposals, only applying to a set amount of companies, maybe the larger ones and so on. mean, all these examples are from the EU that I've mentioned. But, you know, ultimately, yes, like how this has gone so fast. mean,

Felicia Jackson (12:48.886)
Well, that's gonna happen.

Giulia Bottaro (12:53.718)
Everybody has started, everybody with a smartphone is now using AI in one way or another. It's gone, the curve has gone up so fast. So yes, absolutely. What do we do? It's a problem. And yes, and as you mentioned, all these companies, I think most people I know have had the conversation about AI in the workplace. Every workplace is looking at how to incorporate AI in a way that makes sense to them.

And you know, fine, like I understand that you want to make processes more efficient and so on. But when you talk about the liability, how do we know? And actually, this makes me think of an example that I saw on LinkedIn. Someone was talking about their personal life, but I think maybe in terms of when you equate the personal example to the corporate example, then maybe it rings a bell in a more alarming way. But

Felicia Jackson (13:31.264)
there reason to... Yeah.

Giulia Bottaro (13:49.669)
This person said he's got a family, he's got young kids, and he put everything into Claude or another program. So that Claude is now kind of the family secretary, but it knows everything. It knows health data from the entire family because they've got the smartwatches or the rings. It knows the kids' school schedule, sports, holidays. It knows everything.

Felicia Jackson (13:59.437)
Okay.

Giulia Bottaro (14:19.44)
And initially I read it and I was like, oh, how smart, sounds interesting. Well, and people said, yeah, no, it is private account. But then everybody in the comments, all the tech experts or cybersecurity experts were saying, this is huge. You have given so much sensitive data of yourself and your children to an AI model. And that really rang alarm bells massively to me.

Felicia Jackson (14:23.084)
I hope it's a private account and isn't open to the world.

Felicia Jackson (14:43.382)
I mean.

Giulia Bottaro (14:46.99)
And then I thought this applies to companies as well. Have they hired a cybersecurity expert, especially if they are an SME? Do they have the type of money, the type of budget to make sure that these AI implementations are super secure?

Felicia Jackson (14:49.996)
Absolutely.

Felicia Jackson (15:04.332)
But actually it's not even, I mean, that is important. Don't get me wrong, cybersecurity, absolutely critical. But I'm not sure if the conversations are even taking place that mean people are aware of the fact it's a problem. In a small business, how many small businesses that do you know that actually have an AI operating code that is paid attention to? Because it's one thing to have a policy and another thing to have it actually change behavior. But

Giulia Bottaro (15:16.656)
Mmm.

Giulia Bottaro (15:30.992)
course.

Felicia Jackson (15:32.957)
Everyone uses it all the time. How safe is it? How secure is it? How reliant on particular skills it might have? Do you not have those skills in your team? I mean, I know for sustainability, the use of AI and agentic tools has been enormously helpful because sustainability teams are traditionally very small and they have a huge amount of work to do in terms of reporting, let alone anything about

strategically changing how a business works. So there is this real balance problem between what you can do with it and the potential for it to go horribly wrong and I'm not being cataclysmic and sci-fi about it. I'm just really, really interested in how fast it's moved, how quickly it's been taken up and how little conversation is actually going on about how we use it.

how it's bounded. I mean, the whole conversation with, sorry, the whole issue with the Italian guy putting all his data into the agent. sorry. I'm going to say that again then. But the whole, the guy, there we go, I knew he came from somewhere. But the whole story about the guy from LinkedIn who put all his family data into Claude, I mean, I'm fascinated by that because the, anyone interested in digital identity should go back and listen to the digital.

Giulia Bottaro (16:41.766)
it's not Italian. I think it's American. Just a guy from LinkedIn.

Felicia Jackson (17:02.782)
listen to the Along ID interview because all of the data we have, the attributes that define us as who we are, these are things that we've always given up for free in exchange for something simple, you know, something really not worth that much. Maybe it's a discount on something or a higher interest rate or whatever it might be.

Giulia Bottaro (17:24.538)
Take part in a competition by giving your email address. So you're not even winning, you might win.

Felicia Jackson (17:27.796)
There you go. Yeah. But every single bit of that data is something that identifies you. And it means it's valuable to businesses and to banks and to governments. And you need to kind of keep some kind of control of that. Because if you give away in an accessible format, everything that defines you is who you are. How the hell, I'm sorry, won't say hell, but how do you differentiate

the you that's you from the you that's purporting to be you if someone uses all those attributes differently and somewhere else. Yeah, it's crazy.

Giulia Bottaro (18:02.032)
Yeah, for identity theft or other things. You know, since you told me years ago that you would never use your biometric data to unlock your phone, I immediately went and removed my fingerprint. And now I just do not use it because you're right. Like, I suppose our fingerprint is the ultimate. It's like reading the eye, the retina or the face is like the ultimate thing that belongs to us.

Felicia Jackson (18:13.071)
yeah.

Giulia Bottaro (18:31.106)
All the platforms already have all my other sensitive data, but I'm keeping my fingerprint for myself.

Felicia Jackson (18:37.483)
And the really scary thing about that is do you know what people gave their fingerprint away in exchange for? Saving a couple of seconds typing a code into a phone. There's this whole thing about how convenience is the most important part of anything we do and I'm really beginning to think that we need a little bit more friction, a little bit more slowness in our lives because the speed at which things are changing

Giulia Bottaro (18:47.6)
True, true.

Felicia Jackson (19:04.893)
is faster than the speed that human beings are used to.

Giulia Bottaro (19:07.788)
Yeah. And this actually connects quite neatly to the point of AI. Everything with AI is faster, has more, more like a larger quantity of content. It's just more. Everything is more, but we're still people behind all of it, absorbing this information. For example, you know, now I'm working as a freelancer. I read a lot of content on LinkedIn about how to optimize, et cetera. And some of it is great.

And some of it is like create an AI agent to do this and that and the other. And I'm like, sure. But then it's still me who ultimately has to decide what to do with these agents. One example is AI agents for marketing. I'm going to send out all of these made up people, well, AI made up agents into the world. And some of them send the marketing for you. So they do the outreach for you. But then if someone says yes,

Maybe say all of the hundreds of people at the contest say, yes, I'm still me doing the work. Do you know what I mean? It's like we need to control the amount of work that gets done because I could ask tomorrow to AI, write me a 10,000 word essay on this thing that I want to learn more about. I still need to read it. I still need to read it and absorb it.

Felicia Jackson (20:24.523)
But you don't just need to read it and absorb it. You need to think about where it got the information from because that's a huge problem. I think we've had this discussion before that for AI which is crunching data, numbers, or doing analysis of a structured database of this medicine for this problem, that makes sense. But when you're trying to get an answer about how to market to an audience, well, I'm sorry.

What's the audience? Is it regionalized? Is it demographic? What are you trying to do? Do you want them to buy something? Do you want them to subscribe to something? I know it sounds like little details, but every single one of these little details, there's an assumption underlying every single thing that that AI will have put in that essay for you. So you will be learning about something filtered not just through numbers and data and the expertise of whoever wrote it in the first place.

but also the assumptions that the AI is making about what comes next, because the way AI reports back in a chatbot feels like a conversation, but actually it's a number of tokens, and a token is generated that generates the thing that is most likely to come after the thing that was generated before. Because it's been so well-trained, that can look like a conversation, but that's not the same thing as interrogation and critical thinking.

A big conversation that's been happening about the fact that philosophers are going to become finally employable because critical thinking is so important in the world of AI. I'm not holding my breath, but I do think we do need to think more about what's happening before just reacting because the faster things get, the more reactive we get and the less we actually think about consequences, implications or nuance and

Giulia Bottaro (22:00.273)
You

Felicia Jackson (22:22.353)
AI is in every conversation today. You know, you can't have a conversation about work or about politics or about trade or about climate or about human rights. It comes in everywhere. And we're not having that public conversation about how we manage it. It's all, fantastic, dangerous, regulate it. you can't regulate it because that will infringe my right to make a lot of money.

Giulia Bottaro (22:51.738)
Yeah.

Felicia Jackson (22:51.807)
You know, there's all these different things happening at once, but not about how things work and how they'll change and can we cope? Because your point about critical infrastructure, I wanted to come back to that because I thought it was brilliant. But I also think one of the fundamental problems is that might be being thought about by policymakers and bankers, et cetera.

But for most people, AI is digital. It's like the internet. It's invisible. It exists in the ether. So how are you making that infrastructure? And the reality is, well, it's data centers and it's energy consumption and it's water use and it's labor and, you know, it's all sorts of different things. yeah, but I think the thing is the first point is that we've got to remember

Giulia Bottaro (23:25.924)
Yes.

Giulia Bottaro (23:41.412)
There is a lot to discuss on that.

Felicia Jackson (23:49.238)
that it's not something that exists in the abstract. I told someone at an AI event that generating an image uses about 3000 times as much water and energy as a text query. And they were gobsmacked. They had no idea. Because there's no conversation about the physical impact of what AI does. what, sorry, I was just gonna say, because I do think,

Giulia Bottaro (24:12.944)
I think, yeah, sorry.

Felicia Jackson (24:16.659)
If you think about the impact on energy and water, then you're actually talking about things like the food chain. You're talking about economic ability to function. And that is not part of the conversation in the mainstream the way it should be. Sorry, I interrupted you again.

Giulia Bottaro (24:36.43)
No, no, it's fine. I will cut the interruption bit anyway. I actually have some data on the investment that is being made in data centers. And I feel that perhaps it's something that is being discussed quite a lot in the environmental space where we both are, because data centers use a huge amount of energy. They also use a huge amount of water to cool down. So it's two things that are quite precious.

Felicia Jackson (24:45.538)
cool.

Felicia Jackson (24:50.731)
Mm.

Giulia Bottaro (25:04.528)
and this time and they can be used for other things. know, they could be all this electricity, for example, could be used to electrify our system, maybe finally get to net zero, power the electric cars, etc. Meanwhile, the water is more about the fact that it's becoming increasingly scarce all over the world. And so the fact that these data centers are being built

Felicia Jackson (25:17.483)
Absolutely.

Giulia Bottaro (25:33.75)
everywhere. There are however a lot of protests from local communities which frankly I don't think we hear enough about.

Felicia Jackson (25:42.685)
You are, I'm seeing more and more of it actually. I'm seeing reports about it. There was a story about a data center in the States that the local, I can't remember what it's called, the region basically they were moving to a sensor-based water management system and this data center took 30 million gallons unnoticed out of the water system and the locals only noticed it when the water pressure went down. Yeah, wow.

Giulia Bottaro (25:45.296)
Mm.

Giulia Bottaro (26:05.38)
That's insane.

Giulia Bottaro (26:09.54)
Wow. Yes, because obviously the local authorities are quite keen to have these data centers in their area because they bring jobs, they bring a lot of investment, et cetera. And so the data that I promised is from The Guardian, which we will also link in the show notes. So this year, industry giants such as Amazon and Microsoft and all of the companies that are building

these data centers are planning to invest around $710 billion in these structures. However, last year, an estimated $156 billion were blocked or suspended because of these protests. And then while I was reading about this, also...

found out that a lot of data centers are being built in Italy, a huge amount, yes, which is also interesting because we don't have a lot of renewable energy, so there will probably be power by fossil fuel electricity unless they build their own grid. I don't know the details. However, the water thing is kind of massive because we are definitely going to be in a drought in the summer if we're not already.

Felicia Jackson (27:09.549)
really?

Giulia Bottaro (27:33.777)
Global warming is definitely being felt. I actually read an article that was called, Italy is quietly building a huge amount of data center. I know that quietly is actually an AI word used a lot, but in this case, it's so perfect because they just keep being built.

Felicia Jackson (27:33.888)
Yeah.

Felicia Jackson (27:51.136)
Yeah.

That's what it is.

Giulia Bottaro (27:57.903)
and people don't really talk about it. Maybe people, I wonder if the local communities are being made aware of what it means to have a data center. It's not just the jobs or the investment.

Felicia Jackson (28:09.31)
You see, this for me is where it comes back to the importance of expanding the conversation because the politics means, great, it's going to make money. It's going to be built, bring it to my region. I will look good. The economy will grow. Fantastic. But it's got to be legitimate. And that means that the people who live in the region being affected have got to be considered. They've got to be in the conversation.

They've got to understand what that means. And sometimes there isn't as much understanding about the fact there'll be jobs while they're being built, but how many jobs there can be once it is built.

Giulia Bottaro (28:48.388)
Yeah, probably not many because they're heavily automatized. I mean, you don't need, you need people to sort of check them, like, or sort of to monitor activity within the building. And then you will need people to fix things that are broken, but it's not like a factory, which are also increasingly automatized. But you know, you don't need a huge amount of people per square meter.

Felicia Jackson (29:01.812)
So there you go.

Felicia Jackson (29:12.81)
So again, this comes back to what do you actually understand is going to happen? What is the impact going to be on you locally? There's been huge conversations about how the price of energy has gone up because of the huge growth in demand. But there's issues about, certainly with renewable energy, projects that are available to generate but can't come online because of grid bottlenecks. I think there's an insanely high amount in the UK.

that renewable energy could be powering data centres in a different way. So why are data centres not being required to have distributed generation on their sites? And I think that wider question about the knock-on effects is really important because I think you have to start thinking about, well, where does the energy come from? Who is building what where?

Giulia Bottaro (30:03.266)
Again, we don't think about it because it's like using AI. It's kind of, it's not a physical thing. So we just don't think about it. Yeah.

Felicia Jackson (30:11.306)
And that's what I meant about digital versus physical, because we do think it's just one of those things. It's at the end of your fingers. But it is about the physical impact it's going to have. There are going to be environmental costs. Who's going to cover those? Are you going to be destroying areas of pristine ecosystem that can't be replaced? You know, what are communities comfortable putting up with and what sort of impact?

is this going to have on resource competition? Because we've talked about, in fact, we may not have talked about this, so I will just bring it up, but just looking at the interconnections, we know that geopolitics has resulted in a blockage of the Straits of Hormuz. direct immediate knock-on effect on energy prices. What's being talked about far less is the direct knock-on effect.

on fertiliser prices because most of the fertiliser comes through the Straits of Hormuz, that has a direct effect on farmland as we go into the summer. Well, if land is being used differently, if water is being consumed differently, where are the pressures on all of the systems we have, the physical systems we rely on, and how robust are they? Because how resilient are they going to be if suddenly we have, I don't know, 55 gigawatts of

fossil-based fuel supporting however many data centers and the millions of tons of water that they consume. And it's just not, as I said, it's not part of the conversation. And the big reason for that, in my opinion, is because AI good is the story, but the data centers that allow it to run are things people don't want.

Giulia Bottaro (32:00.645)
Yes. Yeah. And this is, I think the connection you made there with fertilizers is very relevant because AI is also a story, because AI is a story about natural resources. It's not just a story about climate, but also about agriculture, water scarcity, just everyday lives for people. And, know, the fact, when I was reading about

Felicia Jackson (32:20.457)
Exactly.

Giulia Bottaro (32:28.784)
Italian like plans for data centers in Italy. There is just one more statistic that I thought was quite striking, which is in 2024, they consumed 2 % of electricity of the entire national demand. And in 2035, this could reach 13%, which is insane. 1.3, not 3.0, but still it's quite a lot. Yes. And for a country where, again, there is water scarcity,

Felicia Jackson (32:51.241)
but that's a huge amount of energy.

Giulia Bottaro (32:58.352)
Electricity is still made by fossil fuels in large parts. Global warming is impacting farmland massively. So what's going to happen? What are we doing exactly? And then you were talking about grid bottlenecks earlier, and it's the fact that maybe for people who are not really aware of what that means, it means that the grids are too old fashioned for an electrified world.

If we want all of these data centers, then the priority should really be, well, which is kind of a dream, but it should be to update all of the electricity grids so that they can support electrification and the use of renewable energy to substitute, to replace fossil fuels. And again, we don't speak about this enough, but luckily we've got this podcast to talk about it.

Felicia Jackson (33:53.62)
And there we go. But I think that's, it's so important because fundamentally what we're discussing is a technology that's being implemented all around the world that's going to have a direct physical impact on the way in which we live our lives and the resources that we have. And no one's talking about how we should actually look after that. And I think what's really challenging about that is this problem of who do you believe?

Who do you listen to? Because if you're listening to someone who's talking about how AI is going to save the world, fantastic, you can get excited. The kind of people who believe in a technological silver bullet, woohoo, we'll do this one thing and we won't have any problems anymore. That's really difficult when you're someone saying, hang on a minute, shouldn't we think about this? You're Cassandra warning about the future that no one listens to. And then you've got all the issues that we mentioned before about identity and...

knowing who you're dealing with online and in AI. I mean, the advent of deepfakes. Again, we're coming back to all of these issues and we don't seem to have a system for managing not just AI generating things like deepfakes or overstepping its boundaries, but actually the fact that if AI systems do become agent, it can start.

Giulia Bottaro (35:00.208)
Wow, yes.

Felicia Jackson (35:22.057)
acting on behalf of people and organizations or even platforms, how do we manage the shift from things that are basic administration tasks to things that are operational necessities? What do we do when something does go wrong with an agent and you're going, no, no, I didn't want it to do that, I'm not responsible and by the way, it wasn't me because I was over there. I just don't understand what we're supposed to do about this.

because we don't have anything to manage it.

Giulia Bottaro (35:52.921)
Yeah, absolutely. And also something else that we should really talk about more is the fact that this AI is kind of a magical tool that was gifted to humanity, but it comes from quite a lot of suffering actually, and in particular from the global south, because a lot of workers have endured and are still enduring really horrible working conditions to make sure that

the models are up to scratch and are continuously developed. So here I'm talking about research from SOMO, which basically found out that there are all of these companies along the supply chain located in the global south that are being hired by the tech giants that we all are aware of. So they are intermediaries and this is...

Felicia Jackson (36:48.873)
think the phrase is know and love.

Giulia Bottaro (36:52.4)
We definitely don't love them Yeah, but basically the point is it's very similar to the fashion supply chain, which actually we'll talk about in an upcoming series of Shaken Not Burned but it's opaque because it's made of a lot of actors. So you have the final product in the shop here in Europe, in North America, a brand that you know very well. However, you don't know...

Felicia Jackson (36:53.363)
Sorry.

No.

Giulia Bottaro (37:20.662)
the name of the factories, the names of the brands that contributed to the creation of this final product. And it's basically the same for AI. And so here we're talking about unsafe working conditions, pay below minimum wage, dismissing workers unfairly. as a connection to that then...

People can't unionize, people can't raise their voices because they're afraid that they will incur into problems being fired for no reason. So overall just very little social protections. so, and on top of this, then the big tech companies will put all of this pressure onto the intermediary companies to then...

respect really tight deadlines or put pricing pressures, which again, very similar to the fashion industry, then this reflects onto the worker at the bottom of the chain. And then when we talk about AI and what is teaching AI, what is good and what is bad, I mean, now there are lots of jobs out there where you can sort of provide your expertise to help AI, you know, to sort of...

Felicia Jackson (38:37.557)
yeah, no, I've been asked a few times to train AI models and I'm like...

Giulia Bottaro (38:40.632)
Yeah. So there are those, but then there are also those where maybe you don't need any specific expertise, but then you need to watch hours and hours of horrible, harrowing, traumatizing content because you're telling AI what this is and isn't. And this is again all done in the global South by people being paid a pittance. So again, this is where the large language models

Felicia Jackson (38:51.095)
about this.

Giulia Bottaro (39:10.106)
come from in a way, like a bulk of the work is being done in these conditions, in these workplaces.

Felicia Jackson (39:17.896)
I think that's such an... I don't want to start with but so I might just restart that at all. That is such a key issue that you're bringing up because it links so strongly to sustainability and how we sustain anything over the longer term. Because we've talked about the fact that LLMs were actually trained on published works by artists and musicians and you know it's all being replaced.

Giulia Bottaro (39:23.823)
start again.

Felicia Jackson (39:47.133)
talked about how the inequality of how things are built and designed and how people are suffering, not just on the training of AI, but think about the mining issues and the critical minerals that all go into the semiconductors that underpin what's actually created in those data centers. And we have this issue that

Giulia Bottaro (40:05.966)
This is a, sorry, a shout out to the series on mining, which we did before the series on AI linked in the show notes.

Felicia Jackson (40:13.466)
Absolutely correct. I lost my train of thought. No, no, that's okay. No, no, absolutely. And it was, but I was talking about, what was that? it again. yeah. So the semiconductors that underpin everything. But actually what we're really talking about is inequality. It's this idea that what we have with AI is a system where those with access and able to pay are going to benefit.

Giulia Bottaro (40:17.206)
Sorry, I'm sorry, I just had to say it.

Semiconductors. Semiconductors.

Felicia Jackson (40:42.384)
And those without the ability to pay and without the ability to access are going to slip further and further away from engagement with society as it develops. And I think this is a really important part of the wider conversation too. It's always been a part of the sustainability conversation, but pushed into a corner. You know, it's for activists and liberals to care about what happens to people in another country. Whereas we live in an interconnected world.

And surely bad things happening to anybody is a bad thing. I don't understand when we got to this place where as long as you're all right, nothing else matters. And that does feel as if this is the place we've got to. And AI is almost encouraging that by keeping us entertained, busy, connected, and oblivious to the implications of what we do.

Giulia Bottaro (41:34.807)
Isolated. Yeah, but also isolated because it's kind of you and your LLM of choice against the world because it tells you everything. You've got a question. You need help. need, you know, so, so yeah. And this is, mean, does fit like it does fit in kind of neatly into the thing you said. It's like, sorry, I got a notification on my phone and I forgot. What did you say about?

Felicia Jackson (41:44.392)
Yeah.

Giulia Bottaro (42:05.197)
yeah, it really fits with the thing that you said of as long as I'm okay, it's fine. Who cares?

Felicia Jackson (42:11.698)
Yeah. But I think again, and I know I'm always banging on about interconnection and systems and how if you, you need to think about unintended consequences, because if we do start moving towards this completely AI enabled world, and I'm, you know, I'm not saying we're not going to, and I'm not saying we shouldn't. I'm just saying we need to think about the consequences, because what happens to people early on in their career?

when there are no entry-level jobs because all the basic entry-level jobs are being done by AI. I'm not saying there isn't a solution to that, but I don't know what it is and I haven't heard anyone talking about it. That's how you learn by being around people who know what they're doing and making mistakes when the stakes aren't too high. But how do you do that transition from no work experience to, now I'm senior and knowledgeable enough to be hired for a non-AI job? I don't understand how that works.

What happens to organisations? If the skills that people develop, those foundation skills of whatever industry they're in, if they don't develop them, how does the organisational capability grow? So are companies thinking about the future of what their business is going to look like? And again, it comes back to labour, you know, and that's what you were talking about with this unequal distribution. But if the people who own the platforms own all the data and the biometric information,

They are capturing the things that make us productive. You know, and you're literally talking about further embedding capitalism into capitalism, but taking the capital away from the shared underpinnings of the system and letting a few tech bros be in charge. And obviously that's didactic and please, when you edit this, out some of my more rude comments, but

Giulia Bottaro (43:53.376)
Giulia Bottaro (44:10.594)
I liked it. I want to keep it. Can I? The capital and the tech bros.

Felicia Jackson (44:11.693)
There is a fundament... but... only if...

Yeah, you can keep that. But I think really what I'm trying to say is that we are, as everybody knows, in a time of transition. But technological transitions, if history's taught us anything, it's that they don't distribute the costs and opportunities fairly. And really, we need to be having a conversation about that because we seem to be sleepwalking into a future that

None of us have really thought through.

Giulia Bottaro (44:50.572)
No, we were, I didn't know what AI was like four years ago. I remember kind of hearing about it and some of my more forward looking friends were talking about it. And in my head I was like, this sounds odd. now it's like, and now, oops, now it's absolutely everywhere. It's everywhere. So yeah, I think we were just sort of all.

Felicia Jackson (45:10.737)
Yeah.

Giulia Bottaro (45:16.954)
catapulted into this world and everybody's just a bit confused or it's like, I can ask it to write me a diet plan for the week, you know? So I think it's probably quite normal that people are using it for more everyday and trivial things, but perhaps whilst we're doing that, we're not thinking about the bigger picture, as you said.

Felicia Jackson (45:38.95)
Yeah, and that for me is always the question because I'm not trying to be, I was going to say a wet flannel, but that sounds really strange. But you know, I'm not trying to be a dampener on enthusiasm for AI because I think there's huge potential. There's certainly so much efficiency that can be gained. I heard a piece of data that said that 30 % of CO2 that's emitted globally in a year is just waste. Just it's not used for anything.

And being able to improve that would be insane. Being able to access the ability to do things you couldn't do and access to knowledge you didn't have, these are all amazing things. I'm not saying we shouldn't dive in to the future of AI. What I am saying is I'm a little concerned we haven't thought any of this through and if we have, I'd like to know who it is who has because I'm not hearing the conversations about it. Well, they're probably all talking to other philosophers.

Giulia Bottaro (46:33.114)
Where are all the philosophers?

Yeah, that's true. No, mean, there is a movement of people studying ethics in AI. There are groups doing that, putting out research. Yeah, I suppose the question is how embedded they are in the corporate conversation because ultimately that's what we're talking about. We talk to professionals.

Felicia Jackson (46:37.413)
Because as sustainability has always taught us, people stick in their silos.

Felicia Jackson (46:49.585)
but they're all separate.

Felicia Jackson (46:55.502)
Exactly.

Giulia Bottaro (47:01.616)
So if you are a company who has implemented the AI, have you hired a philosopher? Or have you looked into ethics? What's your governance around it? Is there a thought process behind it? Or are you just thinking efficiency, bottom line, numbers? Because they should really go together. And who knows that this may create problems in the non-distant future.

Felicia Jackson (47:23.987)
Absolutely.

Felicia Jackson (47:28.615)
But I also, and it's your use of the word future that really resonates with me because we have this problem in sustainability that there are immediate problems and sustainability and climate change have always looked like they were gonna have implications in the future. Well, the future is now and it's having real challenges, which means that despite knowing what we've known for decades,

We still haven't managed to do very much about it. And what concerns me is that we're going into an AI future in the same way. Because they're simple questions. How is this embedded?

Giulia Bottaro (48:04.75)
Yeah, I saw- yeah-

Yeah. And I suppose that because AI is like two days old, there's no way of predicting or even try to understand how things may evolve. Like what will go wrong? We've said, you know, we've talked about the meta employee having all their emails deleted. I had lots of other examples, but we can stick to one because maybe it's easier to remember and quite striking because again, her job is keeping the AI agents in check.

Felicia Jackson (48:37.477)
Absolutely.

Giulia Bottaro (48:38.436)
So that's happening now. What's gonna happen in one, two years time? Can we even tell? No, probably not.

Felicia Jackson (48:45.041)
And that's the thing.

No, I don't think we can. And I think that leads us to, for me, the key thing that has come out of this conversation, which is it's not the mainstream conversation about will AI become all powerful? Is general intelligence possible? But it's actually about what kind of systems we need to have, what kind of governance, what kind of infrastructure that's going to

actually allow us to take AI and use it responsibly. Because it's not about not using it, it's about the implications and the consequences of what we're using it. Because I think, again, AI can create enormous amounts of opportunity. But is the capacity to generate opportunity enough on its own? You know, if we want

the era of AI to be stable and something beneficial, something positive or socially sustainable, then we have to think about it in a different way. So, Giulia, I think I'm going to try and sum up, interrupt me if I miss anything out, but I think, you cut that bit out, then we'll talk about you and then I'll go and do my...

Giulia Bottaro (50:02.926)
I just wanted to say just one quick thing to close. Okay. Yes. And you mentioned earlier that policy moves very slowly, which is true, but I think that companies that are listening to us today could perhaps start questioning their AI processes, try to view them from this ethical perspective that we discussed, or just see what's beyond the efficiency gains.

What are the efficiency gains costing? Because there's always trade-offs, right? That's what we talk about all the time. So, you know, just because maybe there isn't a best practice out there yet, because it is quite new and moving really fast. This doesn't mean that as a company, as a leader, you can't think about what's best practice for you now.

Felicia Jackson (50:37.937)
always.

Felicia Jackson (50:59.621)
I love that. And I think that is the perfect note to wind up our conversation on. Because I think the key thing here is that what we've seen is that AI is not, when we talk about AI, what we've seen is that when we talk about AI, we're not actually talking about technology or software, but actually what we're talking about is what is going to change because of the use of AI.

and what needs to change in order to ensure that we use it in the right way. And that is governance and it is law and it is common sense. we're moving into a stage where these new AI systems are becoming very powerful. They're becoming agentic. They're gonna be able to operate on their own. And we haven't worked out what responsibility looks like. So, you know, it is about whether or not these...

those systems, those things about responsibility and governance and legal frameworks and social protection, know, whether they're actually adapting fast enough to manage the consequences that we face. We've seen what happened with social media, something that started as a way to stay fun and stay in touch. And now has got people

Giulia Bottaro (52:18.192)
Perfect example.

Felicia Jackson (52:25.527)
I'm not even going to go into the details, but we know it's not exactly the most fun environment all the time, let alone misrepresentation, fraud, political polemicization.

Giulia Bottaro (52:35.086)
You know, the point of enshittification, is all of these nice things, things that started as nice and actually people enjoy them like social media. A lot of people are arguing that, sorry, and then became shit. Not very good anymore to commercially driven, not really serving the purpose that they started with. A lot of people are saying that

Felicia Jackson (52:38.986)
yeah.

Felicia Jackson (52:50.545)
shitty. Yeah.

Giulia Bottaro (53:02.414)
this is the future for AI as well. So that's perhaps something to wait for around the corner.

Felicia Jackson (53:08.423)
I think, I mean, I love that because I remember when Google launched and I think its tagline was, no wrong or do no harm.

Giulia Bottaro (53:16.228)
do no harm. It's no longer the motto. It's been taken out of the manifesto.

Felicia Jackson (53:18.479)
And yeah, but when money at that level gets involved, I think people start to think differently and it does go back to, well, I'm all right, so why should I worry? Or what's even worse, what Animal Farm teaches us, the novel, which is, I might not be all right, but I'm afraid, so as long as I go along with what I'm told, I'll be fine.

You know, it's really, it is a conversation that is literally only just starting, but things like institutions and whether or not we trust them, how we interact with them, whether we're ready for all of these changes that are coming, they're going to have a real impact on who benefits and how fast these things scale and how things work. So I think really I have got no conclusion to come to except

Giulia Bottaro (54:17.816)
Let's talk about it. Go out in the world and discuss.

Felicia Jackson (54:18.341)
Yeah? Yeah, let's talk about it.

Felicia Jackson (54:23.991)
And on that note I will say thank you Giulia for today and thank you to our listeners. Hope you enjoyed the conversation. If you have any follow-up questions don't forget to like, follow, oh sorry, if you've got any questions just drop us a line and don't forget to like and follow, shaken not burned. We'll be back soon.

Giulia Bottaro (54:49.029)
Bye.

Felicia Jackson (54:51.409)
Was that right? I lost the plot.

Giulia Bottaro (54:51.643)
Sorry, yeah, it was fine. Yeah, yeah, no, it was good. You can stop. Yeah, I just didn't want to say bye because it's always weird when like, yeah.

Felicia Jackson (55:00.75)
I know, no, no, I agree with you.