Episode 13: Sebastian Mallaby

In this episode of News Items, John Ellis sits down with author and Council on Foreign Relations fellow Sebastian Mallaby to unpack the astonishing rise of Demis Hassabis and the race to build artificial general intelligence. From Hassabis’s childhood as a chess prodigy to the creation of Google DeepMind, the conversation traces how a London-born gamer and coder became one of the most consequential scientists alive. Mallaby explains the breakthroughs behind AlphaGo and AlphaFold, why AI may soon revolutionize medicine faster than the human genome project transformed biology, and why the technology’s dangers are no longer theoretical. The discussion moves from Hassabi’s Nobel Prizes to rogue states, AI arms races, cyberwarfare, and the unsettling possibility that machines may soon improve themselves faster than humans can understand them. It is a fascinating, occasionally terrifying portrait of a future arriving far sooner than most people realize.

News-Items.com

Hosted by John Ellis

Produced by Dale Eisinger

Transcript

SPEAKER_00 0:00

Hello and welcome back to the News Items Podcast. I'm John Ellis. I'm the founder and editor of two Substack newsletters. One is called News Items, the other is called Political News Items, and you can find them both at news-items.com. Our guest today is Sebastian Malaby. Sebastian is the Paul A. Volcker Senior Fellow for International Economics at the Council on Foreign Relations, a two-time Pulitzer Prize finalist. He's the author of six books, including More Money Than God, Hedge Funds and the Making of a New Elite, and The Power Law, Venture Capital and the Making of the New Future. Sebastian, thank you very much for taking the time to be with us today. We're here to talk about your latest book about Demus Hassabas, I think I have it pronounced right, finally.

SPEAKER_01 0:49

Right.

SPEAKER_00 0:50

And I think it's safe to say that Mr. Hassabas is doing or is one of the very few people who is doing uh the most important work on the planet. And before we get to the most important work on the planet, I thought it'd be helpful to the listeners if you'd give us a brief sketch of how Mr. Hassabas got from being a child chess prodigy in his preteen years to being the now chief executive of Google Deep Mind, I think the chief scientist and and the winner of the Nobel Prize for Chemistry in 2024.

SPEAKER_02 1:24

Sure. Well, Demis Hassabis was born to immigrant parents in London. His father was of Greek Cypriot origin, his mother, Chinese Singaporean, spent some time as an orphan on the streets of Singapore as a child. I like emphasizing this to American and particularly Silicon Valley audiences because it just gets across the point that maybe Silicon Valley does not have a monopoly on melting pot features or kind of immigrant founders and so forth. Demis, you know, had this modest background, but his talents proved to be immodest. And when he saw his father playing chess against his uncle, aged about four, he just sort of watched the game a bit and then he understood it, and then he could beat adults quite soon after that, and then he was the best junior chess player in Britain and second best in the world, and pretty much expected to become a chess professional. But along the way, he started winning money from winning chess tournaments, and he would buy himself computers, which he would upgrade when he got some more money. And he got into coding, and the segue was actually reading a book about chess programming so that he could marry his interest in computers with his interest in chess. And the ideas in this chess programming manual kind of went back to information theory, to Turing's early ideas, what is computing? And it, you know, struck him. And a little bit later, he found himself, you know, he'd been given a position to study at Cambridge University when he was extremely young. And the university told him, listen, you're academically ready, but you shouldn't come yet because you're not socially ready. I think he was about 16 or so. Um so go do something else for a bit. So he went off and worked for a video game design company using his coding skills. And in this community of kind of slightly autodidact misfits who were building video games together, is this sort of early 1990s? He would be discussing how you could make these games more enthralling and interesting. And that included discussions of artificial intelligence. The idea that, you know, the player of the game could treat characters in the game in certain ways that would encourage them to do more of something or less of something else. This would be a basic reinforcement model, as people calling it now in AI. So ideas about AI started to percolate when he was around 17. He read this book, Gürtel-Escherbach, which, amongst many things it says, posits the notion that, you know, basically the human brain runs on ones and zeros. And so one day a computer with a lot of ones and zeros and a big enough architecture could do whatever the human brain could do. And so putting this together, by the time he was going to Cambridge at the age of around 18, Demis had formed this ambition to create superpowerful AI, which is just an astonishing fact in itself. I mean, this is somebody that young and having this view in 1994, when we are kind of 18 years before AI being able to do a single thing. I mean, it couldn't recognize the photograph of a cat until 2012. And so this early conviction is part of what makes Demian such a fascinating character.

SPEAKER_00 4:27

How did you come to get the access that you did? I believe you had 30 interviews with Mr. Hosabas over uh you know a number of years. Did he approach you? Did you approach him? How did it work?

SPEAKER_02 4:40

Aaron Ross Powell I approached him. Um I had met him um at tech conferences because my previous book was about uh venture capital investing, and so I was interested in technology. I would go to tech conferences in Europe and there would be this sort of unlikely figure who would show up looking about 25 years old, although he was probably more like 35 or 40 at the time. And uh and he would, you know, have this approachable, boyish demeanor, extremely unpretentious, kind of like the kid next door who says, Hey, let's get a sandwich and uh we'll do the dishes after lunch and then go for a walk. How does that sound? I mean, just super, super relatable. And then he would get on the stage and start talking, and ideas about neuroscience and computer science and biology and chemistry and physics and the history of movies and philosophy would just tumble out of his mouth in this astonishing stream. So the juxtaposition between the approachability of the manner and the ambition of the ideas always grabbed me. And then around about 2022, when I was thinking about the next project after writing about venture capital, it struck me that AI was sort of ripe. I mean, ChatGPT was not out yet, but you'd seen examples of powerful AI in the protein shape prediction project that Demis Susabus had done with Google DeepMind. So I felt that the technology was ripe, and this was an amazing character. And if I could get him to agree to talk to me, I would be able to tell the story of the making of modern AI through this unbelievable personality. So I went to pitch him, and the pitch was basically: look, I've listened to all your lectures, Demis. And in these lectures, you tell us that the invention of artificial intelligence is going to be the most consequential invention of human history. Now, if that's true, then it follows that you're going to be one of the most important people in human history. So, Demis, you don't have a choice. There will be books about you. Let's just get real here. Don't pretend that you can hide. And furthermore, you should want there to be books because you cannot unleash a technology this disruptive on the world, which is going to make people raise their children differently, do their jobs differently, conceive of themselves as humans differently, and not explain why you're doing it. I mean, it's dangerous, it's disruptive. You have to say what your values are, what your motives are, what the point is. And so you should welcome somebody who wants, who comes in the front door, not the back door, and you know, promises to take your ideas seriously and to do a good, you know, careful, multi-year project where you I really tell your story. And he thought about that for a bit and he accepted. And then right around when he accepted, AGPT came out. So my fringe topic went to the mainstream faster than I could have imagined.

SPEAKER_00 7:19

Aaron Powell A famous historian once said it's it's almost impossible to write a biography about someone you don't like. Only a good biography uh can be written by can be written about someone that you do like. Did you like him? Do you like him?

SPEAKER_02 7:34

I do like him. One of the interesting things actually is that even somebody you massively respect and like is going to have blind spots. I mean, I think of this as relationship advice as well as just sort of biographer advice. And with Demis, you know, sure enough, there were certain things where he was not able to see himself lucidly. Uh so for example, he always says he doesn't like to control people, that his mother brought him up with these Christian values not to control people, to respect everybody. And it's true that he he carries himself in a way that's you know designed to be non-intimidating. And I have funny stories about, you know, because I live in London, Demis Osavis lives in London. It turns out that I have a friend whose kids were at the same school as Demis' kids, and this friend of mine went over to Demis's house to pick up his son from a birthday party, and he saw Demis and he recognized him from a YouTube video. So he said, Oh, you're the AI guy. And uh Demis said, Yeah, I'm the AI guy. And my friend said to Demis, um, you know, I I I watched this video about AI on YouTube, and what it strikes me is that you could make a lot of progress if you had two AIs and they kind of argued with each other and the synthesis of their views might be better than either one. And so now you have the, you know, totally non-technical random friend of mine who became a publisher after leaving university lecturing the future Nobel laureate on what he should do with his AI models. So I said to my friend, look, so how did Demis respond to your useful advice on the future of AI? And my friend said, Well, Demis said it was a very useful insight. So he is a nice guy, right? Well, I know, I know in this completely unplanned piece of reporting that he is a really nice guy. But he says he doesn't control people, which is ridiculous because he runs a large company and he's smarter than everybody else and more articulate, and he wins 99% of the debates he has with people. So of course he controls them. But anyway, he refused to see that. And we had an argument about that.

SPEAKER_00 9:30

There are two sort of, it seems to me, two milestone moments in the history of Google. One was the acquisition of Jeffrey Hinton's company, which was really like a two-page memo. But Mr. Hinton is often called the godfather of AI, and Google acquired his brain, essentially, and he worked there for a number of years. And the second, obviously, is uh Google's acquisition of DeepMind. How did that come up that come about, I guess, is the question.

SPEAKER_02 9:59

Yeah, so Demis founded DeepMind in 2010. And um, for the first three and a bit years, it was sort of raising money from Peter Thiel in Silicon Valley, raising money from a few other investors. Elon Musk put five million dollars in, was a Singaporean, uh Hong Kong-based investor as well. So there's, you know, it was kind of gathering momentum, but it was a struggle to raise enough money to afford enough computing power to really take a big leap forward. And then Google became interested in buying it, partly because they had already bought the Jeff Hinton boutique you mentioned. And Jeff Hinton had recommended buying a Demis company because he knew Demis, he'd met Demis, he knew Demis was extraordinary. And so, based on that recommendation and on a model that Demis' company DeepMind had produced after a couple of years, which was the first agentic games playing model. It was a model that could just through trial and error, without being given the rules, understand those Atari video games from the 70s and 80s, games like Pong and Breakout and Sequest and stuff like that. So Owl was sufficiently impressed by the technological progress and impressed by Hinton's recommendation and wanted to buy DeepMind. And Demis, you know, was keen because he wanted to have liberation from the hamster wheel of fundraising from the venture capitalists. And in fact, Larry Page, um, they met at an Elon Musk birthday party, but it's fun, these stories are fun because it shows you how all of these people were entangled, you know, right at the start, you know, in the 2010-2012 zone, the same characters were circling around each other, kind of friendly but competitive, sometimes viciously competitive, as we saw recently in the in the Musk v. Altman trial. But anyway, um there was this birthday party of Elon Musk's, and Larry Page suggested going for a walk with Demis. And in this walk, he said to Demis, look, you have to decide. You could probably build a company as big as Google if you want, but it will take you all your energy. If what you really are is a scientist, then you should come and do your science with the benefit of my resources and my computing power. And to Demis, that was a no-brainer. He was much less interested in becoming a multi-billionaire than he was in being the person who delivered powerful AI to the world.

SPEAKER_00 12:22

Aaron Ross Powell And with the acquisition by Google, he he became substantially wealthy. So he didn't even have to worry about that anymore.

SPEAKER_02 12:30

Aaron Ross Powell Yes. Um to any normal person, walking off with a hundred and something million is extremely wealthy. It just wasn't on the scale that uh you know Larry Page composed.

SPEAKER_00 12:40

Aaron Powell But you get the sense from reading your book that he doesn't he doesn't really care. I mean, obviously he cares because it makes it makes life easier and he doesn't have to fly coach or whatever. I mean, is it is is that right? It does he really is that a secondary concern or even a tertiary concern for him?

SPEAKER_02 12:57

Aaron Powell Yeah. I mean, I he he would always say to me he doesn't care about money. So one day I quizzed him, you know, we would meet in this pub near his home in North London, and at the back of the pub there was a dusty staircase. You'd go up there and there'd be a abandoned room which nobody really used. And so we'd get coffees and we sit up there for two hours at a time. And in one of these sessions I said, okay, you say you don't b believe in you don't care about money, so let's just go down the list here. What kind of car does your family drive? And he said, Oh, I've forgotten what it's called, but I think it's 11 years old or something. Do you have a second home? No. Do you have a yacht? Of course not. Do you collect really expensive art? No. Well, do you have any hobbies? Yeah, I do actually have a hobby, it's true. What's the hobby? Well, I I really support Liverpool Football Club. And I did buy myself a couple of season tickets. I said, How much does that cost? And he said £3,000, which is like 5,000 bucks. And that's kind of the size of it. I mean, you know, perhaps there are some things that he didn't tell me. I think I'd find out later that he had done a fairly fancy renovation to his home. But still, it we're talking, you know, a a very nice middle class home, not a kind of crazy mansion. And uh yeah, so I don't think he cares about money.

SPEAKER_00 14:05

So it's all about the work.

SPEAKER_02 14:06

It's about science, it's about ego, it's about fame, it's about a sort of messianic desire to be the person who brings artificial intelligence into the world. And then beneath that, or behind that, there is something else. And this surprised me. Uh there was one time we were meeting, and instead of being inside in the pub, it was such a nice day, even, you know, even in London, you get a few nice days. So we went out to a park and we sat in a cafe, and the people at the next table were having normal cafe type conversations. You know, my friend was sick, she went to hospital, oh, I hope she's better, etc. And I was sitting opposite Demis, who was in this sort of messianic riff. And he started saying, at two in the morning, Sebastian, when I'm reading a scientific paper, I feel that reality is staring at me in the face, calling to me, waiting to be figured out. We have to figure it out. It's absurd. We don't know what gravity is, we don't know what time is. There are so many mysteries in physics which you haven't yet figured out. I mean, this table, Sebastian. And then he banged his fist on the table. He said, It's made of atoms jumping around with spaces in between. Why is it solid? We don't even understand that. We need to understand these things, and that's why I'm building AI, so that we can crack these scientific mysteries and understand the true fabric of reality. And that is what I need to do before I die. And if I can understand how nature functions, I will be getting closer to the intelligent being that may presumably have set up everything in reality in a way that is amenable for humans to go and do science and to understand it. I will get closer to that intelligent design, which could be my definition of God.

SPEAKER_00 15:37

That's the way I feel when I wake up in the morning, you know. Slightly different. I think we have to talk about one of the major breakthroughs that generated a ton of headlines, which was the artificial intelligence developed by Mr. Hashabus that defeated the world's greatest Go player. Can you tell us that particular story?

SPEAKER_02 16:02

Sure. So after Google bought DeepMind in 2014, Demis went off to a conference with the senior leadership of Google. So there he is with his new proprietors. And one of them, Sergey Brynn, is a Go player. And Demis, you know, drops a comment to the effect that maybe, you know, an AI system could defeat a Go player. And Sergei Brynn, who knew something about computing and something about Go, said, no, no, that's ridiculous. Surely that's impossible. And that was the standard view at the time, that the idea that you could have an AI that was powerful enough to solve the incredibly intricate patterns that you get in Go and to derive meaning from them when the combinatorial complexity of how those, you know, all the permutations of how those stones could be placed on the board, they're just, you know, almost infinitely large. It's a 19 by 19 board. So the very first move, there's 361 possibilities, second move 360, next one 359, you multiply that out, you very quickly get to this insanely big number. So Sergei Brynn was just channeling the consensus when he said, you can't possibly do that. And Demis said to himself silently, if he thinks it's impossible, then it ought to impress him if I do it. And then, you know, we'll get all the funding we ever want. So he went back to his friend David Silver, who was one of the scientists at DeepMind and a friend of Demis's from University of Cambridge when they both did computer science together. And he said, um, so David, you you you did your PhD on um primitive go-playing AI. Do you think you could make one actually that really worked? And Dave Silver's view was yes, because there had been advances between him finishing his PhD and the moment when Demis asked him this question. There had been adequate progress in both deep learning and reinforcement learning, which are the two main strands of artificial intelligence, that if you combine the two with some novel tactics, this ought to be a soluble problem. So Demis said, great, we know, let's do it. And um he gave him, gave David Silver the resources, backed him, and it took about a couple of years. But in this famous game in the spring of 2016, Deep Mind sort of staged a new version of that deep blue versus Gary Kasparov moment for chess. Right. And this instead was Alpha Go versus Lisa Dole. And Lisa Dole is a uh South Korean world champion. And uh it was best of five. There was a prize of a million bucks for the winner, and um AlphaGo won four games to one against the human. And to this day, when Demis goes to South Korea, he's mobbed by people as the father of AlphaGo. And I heard on his most recent visit, you know, people rushed up to him and demanded that he sign a copy of my book, which is the moment I think when finally Demis accepted that he was happy that I wrote the book.

SPEAKER_00 18:50

So AlphaGo, if we I mean, we can there are a lot of steps along the way, but if we go from Alpha Go to Alpha Fold, can you inform the listeners of what Alpha Fold is and why it's so important?

SPEAKER_02 19:05

Aaron Powell Sure. So after AlphaGo was finished, now we're in the spring of 2016, and Demis is walking out of the sort of exhibition hall where the game took place with his friend David Silver, and it happens that behind them, as they walk down the street, there is a camera crew with a big microphone, sort of one of those kind of fuzzy wooden things that they stick out at you. And it picks up the conversation between Demis and Dave. And unbelievably, Demis is saying, you know, having just collected the laurels of this go breakthrough kind of 20 minutes earlier, he's saying, right, Dave, now we can do protein folding. And so, you know, he rested on his laurels for kind of 15 minutes and announced the next challenge. And this next challenge is something that went back to his time at Cambridge, where he had a biologist friend who told him about a Nobel laureate in biology called Christian Anfinson, who had delivered a Nobel lecture in which he had conjectured that if you take an amino acid strand and you look at the DNA sequence on the strand, the sequence is a sort of code. And if you could read the code, you would predict how that strand of amino acid would fold itself up into a beautiful, intricate shape. And this is the shape of the protein that it forms. So proteins are these sort of mini building blocks in nature, in our bodies, in plants, everywhere. Um, and they're they're composed of these strands of amino acid that kind of like a self-executing origami model twist themselves into an intricate shape. And so this was an extraordinarily ambitious idea, right? That you could predict a shape with actually more permutations in the way that that strand of amino acid could fold itself up, more permutations than even in Go. But also is much more consequential than Go, because if you could understand all the shapes of all the proteins in nature, you could in turn design or choose molecules that would bind onto these proteins if you were trying to build a medicine, or possibly if you were trying to do some innovation in material sciences and a bunch of other applications. And so solving this problem was similar to Go in the sense that it was a big combinatorial space with almost infinity of permutations. And so that's why Demis, having conquered Go, immediately thought of protein folding next. But it was even harder and more consequential. So anyway, after that, Demis uh convened a team to sort of focus on this protein folding challenge. It was a long and winding road. This one took, yeah, just over four years. But at the end of 2020, he had a system that could accurately predict all these protein shapes just by knowing the DNA sequence. And the practical consequence was that it used to take a PhD researcher about four or five years sometimes to do the intricate kind of X-ray crystallography, which is the process of. Physically peering at this minute protein shape and mapping it. It's so difficult. It would take four or five years. Now, a scientist can simply look up the protein shape like a Google search because all of the shapes have been discovered by AlphaFold, Demis's program, and he has open sourced them so anybody anywhere in the world who is interested in structural biology can look up all of the shapes in nature pretty much. And this is the achievement for which Demis shared the Nobel Prize in Chemistry in 2024.

SPEAKER_00 22:30

Did Alpha Fold then morph into isomorphic labs? Is that what happened?

SPEAKER_02 22:35

Correct. After open sourcing all the shapes of all the proteins and kind of gifting that to science, the commercial side of Google kicked in, maybe not unreasonably, and they decided to persist in that line of research, but in a company that would be separate from DeepMind. It's called Isomorphic Labs. And they would keep their discoveries proprietary. So where we are today is that we've gone from AlphaFold 2, which was the system that unraveled all those protein shapes in 2020. And now we're on Alpha Fold 4, which is really sort of a suite of AI products which collectively are designed to bring AI into the drug discovery process. And I think also perhaps in not only to the discovery process, into the clinical trial process that follows. And the notion is that you could speed up the discovery and the productization of new medicines, you know, maybe by an order of magnitude. Trevor Burrus, Jr.

SPEAKER_00 23:32

And this is, I mean, it's all part of the story where you marry a new technology to science. So and the you know, in the mapping of the human genome, you you took uh the the basic science of genomics and you married it to supercomputing, and lo and behold, Craig uh Venter gets us a rough draft of the human genome, what, I don't know, ten or fifteen years sooner than NIH was doing it by hand, so to speak. Here you're married now marrying code with artificial intelligence, and so leaps and bounds of discovery occur. What what is happening in the isomorphic lab space? Are they have they come out with uh a pharma product or or a new way of looking at a medical problem? Is there is there a commercial application yet?

SPEAKER_02 24:22

Well, again, it i the inventions at isomorphic are proprietary. They don't talk about them in detail. But but I can give you a couple of shed light a little bit. I sat next to a dinner recently, the co-founder of BioNTech, which was the German pharmaceutical company that, partnering with Pfizer, produced the first mRNA COVID vaccine. What she told me, uh this scientist, is that in the drug discovery phase, there are certain uh you know the the the the process from kind of having the concept of a medicine that you're looking for and actually finding what seems to be a plausible molecule that works. In some cases that's reduced to one-tenth of the previous time, is what she told me. And of course, you know, there are caveats, it depends slightly on what type of medicine you're looking for. This protein folding technology is more useful in some cases than in others, but it really is a meaningful speed-up. And I've met other pharmaceutical researchers who say, yes, you know, I use it every week. It's it's really, you know, it's a great shortcut for research. The challenge comes in getting through the clinical trials, because you know, so far it's been impossible to persuade any regulator to contemplate shrinking and the time that takes. There is a sort of starter idea, which to me makes perfect sense of how you could shrink it, which is that in a clinical trial, you have both a test group, which get the new therapy that you've just developed, and then you have a control group, which is supposed to be demographically as close as possible to the test group, but they just get the state-of-the-art therapy, whatever that happens to be. But of course, because it's the state-of-art therapy, by definition, that means that many, many, many patients in advanced Western countries have received the therapy already. And if you use AI to search out amongst those people who've already been treated, a group which has the same demographic characteristics as the people in your trial group, you're in business. You can just simulate what the outcome is from the state-of-the-art therapy without needing to have a whole trial for that. And this would be one way of shrinking the time it takes to get through clinical trials. And I suspect there are more that could be as regulators develop more confidence in the accuracy of simulated AI testing results, perhaps they will be less exigent in the kind of actual life human testing that's required.

SPEAKER_00 26:44

Aaron Powell Aside from birthing isomorphic labs and another company called Ineffable Intelligence. What by the way, what is ineffable ineffable intelligence? What does that company do?

SPEAKER_02 26:56

Aaron Ross Powell So that company is founded by David Silver, the person I was referencing earlier, who uh was the genius behind AlphaGo, uh also did a system called AlphaZero after that. And his particular special source is that he is the master of what's called reinforcement learning. And and and maybe to just explain this for a second, because I think it's super useful to people who understand, okay, I I I know there's this thing called AI, but what is it under the hood? And I think just one level down, just to give people one more step in understanding, there are two main types of artificial intelligence. There is deep learning, where you have just masses and masses of data, and you have algorithms that can find patterns in the data and induce predictions from that. It's a bit like, you know, you send a human being to the library and you have them read all of the books in the library, and they're smart enough that they remember all the books, and they can start thinking across all this crystallized human knowledge which they've read. Then there's a different kind of learning, reinforcement learning, where you're in an environment, and for humans it's the real world, and you take an action, and then something happens in response to your action, and you learn through trial and error how the world works. So you know that if you pick a glass of water up, you can feel the weight, because there's gravity. Okay, now you understand something. You know that if you turn the glass upside down, the water will splash on the floor. All right, now you understand the difference between a solid and a liquid. You know that if you drop the glass, it shatters on a concrete floor, et cetera, et cetera. So as you as you take actions in the world, you learn from the world. And that is the notion behind this other strand of AI called reinforcement learning. There were two sort of warring tribes in artificial intelligence in the twent in the 2000s. Deep learning was, you know, the headquarters was Toronto, Jeff Hinton's lab. And then in Edmonton, Alberta, also Canada, funnily enough, there was another Brit. Jeff Hinton is one Brit. The other one was called Rich Sutton, and he was the guru of reinforcement learning, and that is where David Silver did his PhD, learning from Rich Sutton. So David Silver brings this reinforcement learning approach to DeepMind. He has amazing successes in building these game playing agents. And now what he's doing with Ineffable is trying to take agentic AI, reinforcement learning-based AI, and push that to the nth degree. I mean, he's he's although I haven't talked to him since he started the company, I talked to him an awful lot from my book. And I know what his vision was when I was talking to him at the end of 2025 and why, to him, reinforcement learning was being under-emphasized by the big labs. And he feels as though when you when you learn from data, as deep learning does, by definition, you can only learn from what's in the data. The data is created by humans in one way or another. And so that's a limiting factor on how clever AI can be. If you want the AI to surpass humans, to be really super intelligent, it has to learn by essentially creating its own data through trial and error. So in Go or Chess, the best, best systems are ones that don't study expert human games. They learn purely by playing games against each other and learning through trial and error which strategies in chess or Go work better. And David Silver wants to extend that idea. Like, for example, in medicine, you could start by training an AI on all the medical journal articles. And in this way, it would inhale the state-of-the-art human understanding of medicine. Or you could say, I know what, we've got these humans wandering around with, you know, boot bands on their wrists or aura rings on their fingers, collecting actual data on the human. And, you know, we could get some other data on the human in various other ways. And we know, you know, the heart rate, the sleep, and so forth and so forth. And now we're going to correlate that with health outcomes. And so we're just going to bypass what human doctors think they know. We're going to go directly to the data, and agents are going to find patterns. They're going to sort of run little experiments and figure out what really is driving health outcomes. Leapfrogging over crystallized human wisdom, which may not be as perfect as we think it is.

SPEAKER_00 31:06

Aaron Powell One of the things that's interesting to me is in any situation where you have uh an extraordinary talent, it attracts other extraordinary talent. And Damas's company and his cohort have produced any number of extraordinary talented people. Who are the most remarkable, in your view, that have sort of spun out of DeepMind?

SPEAKER_02 31:30

I think David Silver is a standout. And beyond that, there have been a number of others. The founder, the CEO of Mistral, the French AI lab, Arthur Mensch, is another alum of DeepMind. And there are probably some others who I'm not thinking of, but one of the characteristics of this field is that you do need a ton of money to train AI systems. So oftentimes when somebody leaves DeepMind, they go work for one of the other Frontier labs as opposed to doing a startup. Although increasingly, if you're famous enough, you can raise crazy amounts of capital, and that gets to a whole other discussion about bubbles and so forth.

SPEAKER_00 32:09

Aaron Ross Powell Well that's the subject of your previous book, so we'll we'll stick on this one. We have to we started by saying it's the most important work on the planet, or at least, in in my opinion, the most important uh work on the planet. What exactly is artificial general intelligence?

SPEAKER_02 32:26

Aaron Ross Powell There are two debates in this field which are impossible to resolve because people don't agree on definitions. One is the one you just asked, what is artificial general intelligence? The other one is can a computer be conscious? Because it depends on what your definition of consciousness is. But let me just answer your question on artificial general intelligence. You could say, look, if you use Gemini or Chat GPT or Claude today, it's artificial, it's general, it can talk about a lot of things, and it's intelligent. We have it, right? I mean, that would be a plausible definition, a literal definition of AGI. At the other extreme, you have someone like Demis who tends to want to push the moment when we declare that we have AGI. He wants to push that out into the future because he's having such a great time, you know, doing the research to try to get to it. He doesn't want the research to stop, but then he wants to extend the journey. And then maybe there's a little bit also he doesn't want to freak people out by saying, you know, we now have this scary-sounding thing. So for whichever reason, he likes to say AGI is further in the future. And in order to back up his further in the future point of view, he chooses a definition which is very stringent. He says, okay, let's imagine we trained an AI system on everything that was known up until 1911, and then we just waited and saw if the AI system came up with general relativity all by itself. It's a pretty tough hurdle.

SPEAKER_00 33:47

I would say so. I wanted to ask you, and Jeff Hinton is coming out, I think this fall, with a manifesto about the dangers of AI. How does how does Mr. Hashabas, how does Demis view the perils of AI?

SPEAKER_02 34:06

Aaron Ross Powell I mean what he says is, and I kind of agree with him, is that look, there's a non-zero risk, that's his phrase, non-zero risk that these models will start to attack humans. And I actually used to think this was a crazy notion, and I thought, well, of course, machines would be more intelligent than people. We already see that in domains like Go, and so why wouldn't it happen more generally? But machines wouldn't attack humans because they're not really motivated to do so, right? We are evolved to want to survive, to pass on our DNA. Machines don't have DNA, they don't care about survival, so therefore, why would they attack us? That was my kind of comforting worldview. And then one day I went to Toronto to see uh Jeff Hinton, and I sat in his kitchen for a couple of hours, and I said, Look, Jeff, you know, surely you're exaggerating. Why would these systems attack us? They don't want to survive. And he said, Okay, we'll do this thought experiment, Sebastian. Imagine you have a very powerful AI, and you're worried that an enemy AI, maybe a Chinese AI or a Russian AI, is going to attack your AI. You're a slow-thinking human, so you're way too dumb to respond to a cyber attack at digital speed. So you're going to tell your AI, listen, if you see the attack coming, defend yourself. Maybe counterattack. Whatever you do, survive. Survive. We just used that word. Now are you feeling comforted, Sebastian? And I kind of see his point. I mean, I think we will endow these systems with a desire to survive. And indeed, that desire may sort of just emerge out of the training data because these systems are trained on, you know, novels and books about the human condition and all sorts of stories about humans fighting for survival. And so why wouldn't they get the idea of survival? And if you marry supreme intelligence with the urge to survive, with an ability to be duplicitous, which is something that we observe in the models, it strikes me that, yeah, they could turn against us. Now, of course, there's a whole branch of AI research called alignment research, which is precisely devoted to avoiding that outcome. And, you know, very smart people are engaged in that. So I hope that we can contain this risk. I don't put this risk at even 1%. I think it's below that. But I think non-zero is the right answer, and anybody who claims that there's no risk or that the risk is 50%, um, are reflecting something about their own psychology as opposed to a sober reading of the evidence.

SPEAKER_00 36:35

Aaron Powell You wrote a piece recently for, I believe, the New York Times and discussing the AI race between China and the US, and you put forward the idea, which I think is a very good one, that China and the US should sort of agree to a strategic arms limitation treaty. Can you explain that to our listeners, wh why you think that's important?

SPEAKER_02 37:01

When we imagine AI in the hands of bad people doing bad things to us, there are two categories of threat. One is that an enemy country like China might get powerful AI and do something bad to us. But we know from the Cold War that the central way of dealing with that type of, you know, peer competitor threat with nuclear weapons was you both race and you achieve a rough parity, and then there is mutually assured destruction. And yes, on top, you can try to contain the cost of the arms race by having caps on the number of warheads each side deploys. But the fundamental deliverer of peace is mutually assured destruction, the balance of power. Meanwhile, there is a separate category of threat, which is that bad actors, they could be criminals, they could be terrorists, they could be North Korea, various kinds of rogue get this technology and start using it to hack everybody's bank accounts or do a bioweapon or flood a city by making a damn malfunction. And that category can't be deterred by mutually assured destruction. It's not a bilateral thing with a balance, it's a multipolar thing with lots of rogues who have not much to lose. And so for this, you need a non-proliferation architecture where you desperately try to control the way in which the technology spreads around the world. And my argument is that with mythos now out, and the, you know, it's it's not theoretical anymore, we do have a model that exists in the world that could be used to do grievous damage to financial stability and so forth if it was in the hands of uh criminals. So we we really need the equivalent of a nuclear non-proliferation treaty. And the good news is that most of the world would abide by a set of rules about non-proliferation if the US demanded that. Because most of the world is totally dependent on the US to get any access to any AI. Because we have the NVIDIA chips, the US has, you know, uh all the big compute clusters, the best algorithms, as well as a lot of other coercive power. And if you're a French AI lab like Mistral or a Canadian one like Cohere, you nonetheless want to raise money in America, sell to American customers, deal with American chips, and so forth. So you're gonna do exactly what you're told by the US government. The only government that would ignore US sort of non-proliferation policy, potentially, is China. So there does need to be a deal with China. You can't coerce them into this. And so that's why I think it's a high priority right now for statecraft to go to China and say, look, you're an AI superpower. We are an AI superpower, neither of us want this technology in the hands of rogues and criminals and terrorists. You love regulating the internet, Mr. China. So why wouldn't you want to regulate this too? So let's do a deal where you stop just open sourcing or putting putting out open weights of these models, because you and I both know that this means that mythos type technology will be in the hands of criminals by early 2027. None of us want that. So let's talk about a deal.

SPEAKER_00 40:04

And you said in that piece that you had talked to people in China who were open to this idea of a non-proliferation treaty. Is that correct?

SPEAKER_02 40:13

Yeah, so so China does everything fast, and so they published my book first, even though they had to translate it and do a bunch of other things to get it out. But they were still first. And I went out there for eight days and went and met lots of AI leaders, both in technology companies and in academia. And I was surprised that many of them raised the question of AI safety unprompted. Um, and so I think that's a good sign that at least at the elite level, there is a conversation about the need for more focus on AI safety. I'm not pretending that the Communist Party, you know, necessarily wants to stop racing against the US. Indeed, you know, until very recently, the only sign of any policy from the Trump administration was it wanted to race against China. So both sides were locked in a race, and it was rational for each one to race if the other one was racing. Since Mythos, the US administration has shown some signs of rethinking. And I think the Chinese one would too, if, you know, prompted by a delegation that went to China or some overture that said, look, we both have a shared interest in in keeping this out of the hands of criminals.

SPEAKER_00 41:15

So if we have the bad actor, let's say North Korea, and they have developed their own version of Mythos or something close to it, and they decide to attack the Swift Money Channel. How is that blocked? How how would it in the in the scheme of uh agreed-to non-proliferation treaty, how would the AI powers block uh North Korea from disrupting the financial system?

SPEAKER_02 41:43

Uh it would be blocked because Korea would would not be able to get the model in the first place. Um the deal would be that you know, all powerful AI, which traces back mostly to America and in some cases to China, would only be disseminated around the world to countries that agree to sign up for proper safety controls. So if the Germans want to set up their own AI security institute, which promises to monitor usage in their own country, then the Germans can get sort of the the weights that can get the model from Anthropic or Google DeepMind or whoever, and they can parcel it out to users inside Germany under some licensing agreement where some money flows back to the creators of the models. But they would need to be policing it such that, you know, if they saw somebody using it for a criminal purpose, like doing a cyber attack, they would shut the criminal down. And so these national, it's a bit like the IAEA monitors the use of civilian nuclear power in order to prevent the nuclear material being repurposed for weapons. But I think it would actually be easier in this case, because you know, to run these models, you need these enormous data centers. And they're kind of easy to spot. We know where they are, they depend on American technology to function. And so we could police those things. And we could say to Germany, look, if you want to carry on using AI, you need to police your guys in your country. And then North Korea presumably would not agree to police anything, and so they wouldn't get the technology.

SPEAKER_00 43:11

Is there a uh AI policy emerging from the Trump administration, or is it just the answer to everything is China, and therefore we have to press on?

SPEAKER_02 43:20

Aaron Powell Well, there was a hint um before the Trump Xi summit from the administration, I think it came from Scott Besson, the Treasury Secretary, that AI would be on the agenda for discussion in Beijing. I'm actually not aware that it was discussed. I don't think there's been public statements about it. Maybe it was discussed behind closed doors, or maybe actually something as technical as this is not best treated in a leader to leader summit. It's best done at a you know technical negotiators in some separate forum. So I'd say that there was a hint of openness to this kind of approach. Furthermore, in terms of domestic regulation of AI releases, the Trump administration. Has done in 180. I mean, it it began earlier saying, you know, we think that any AI security institute is a bit dubious, possibly it's woke nonsense. They sort of weren't interested in maintaining the national AI safety institute that had been created under the previous administration. They didn't absolutely abolish it, but they renamed it and downgraded it. And now all of a sudden, they've waded in and taken control of the list of entities that's allowed to get hold of the anthropic mythos model. Anthropic said at the beginning, we'll give this to about 40 or so responsible entities that can use mythos to harden their systems against future cyber attacks. And their intention at Anthropic was to roll it out to other users in quite rapid waves thereafter to try and get as many as possible players on the internet to have the opportunity to find bugs and vulnerabilities in their own system and to shore them up. But the Trump administration has arrogated unto itself the power to decide who gets it next. And so far, they haven't announced anybody next. So the whole thing is frozen. So I don't think Anthropic's happy with this because they think that there's a non-policy where there ought to be a policy. But in a way, it's an improvement over the previous state of affairs where it was just open accelerationism, no holes barred.

SPEAKER_00 45:17

What prevents a company like Anthropic from assuming sort of overreaching regulation from the Trump administration or from wherever? What prevents them from saying, you know what, we're just going to go to Toronto and set up shop there and do our work there, and we won't be harassed by Trump administration regulators or Congress or whatever it might be? And I presume that, you know, Prime Minister Prime Minister, what's his name? Carney's his name. I assume that Prime Minister Kearney would more than welcome, you know, the best AI mines in the world to move to Toronto or Montreal or whatever. Is that something we're likely to see if if regulation becomes too onerous, or or is it just unrealistic because of, you know, people live and have lives and data labs are in the US and so on and so forth?

SPEAKER_02 46:10

Aaron Ross Powell Well, it's a bit like you know saying when the US imposes financial sanctions on a country, maybe the country could just use a bank that's located in, you know, Russia or something. But the problem is it doesn't work if the bank is connected to Swift and to the international dollar-based financial system. And it's kind of the same with AI. If Anthropic moved to Canada, it would still be dependent on all bits of the sort of technology supply chain, which are American controlled. You've got NVIDIA chips, you've got, you know, memory chips that might be from Micron, you know, based in Idaho, you've got all kinds of cooling gear, which is probably made in America, you've got ASML, the Dutch firm that makes the lithography machines, but is subject to American power because it operates in America, sells to America. I mean, it's just too difficult. You'd have to basically take the entirety of the supply chain, lock stock and barrel, and move it to Canada, and that's not gonna happen.

SPEAKER_00 47:05

Yeah. So um, last question, we've taken a lot of your time here, but the last question is what does the future look like to you? What what are we gonna see in the next three to five years?

SPEAKER_02 47:14

Aaron Powell Well, I think the first thing to say is that if you look back three to five years, the progress has been much, much faster than I think many people still realize. You know, there's a slight uh sort of popular delusion, I feel, sometimes, that, you know, oh yes, AI arrived in 2022 with Chat GPT, and now we have AI. Okay. No, no, no. That's not how you should think about it. You know, the first models hallucinated abominably. Within a few months they stopped hallucinating. Then there was systems that had much longer memory, so you could put a Tolstoy novel into the system and get it to summarize it or comment on it. Then you've got models that could deal with video and pictures and take in verbal messages, do audio. Then you had ones that could do math and logic, they could reason their way through problems. Now you have the beginnings of agentic systems that can do multitask operations without being interrupted. Some of them go on for as long as you know a few hours. So the progress since 2022 has been extraordinary. And I think if anything, the progress looking forward is going to be even more extraordinary. Such that if you talk to people, say it anthropic, they believe that by 2028, only a year and a half away, there is going to be such progress that a model can code up the next model. You will have what's called recursive self-improvement. And at this point, the cycles between one model and the next slightly better model shrink because you no longer have slow humans having to bang their heads against the wall and come up with some smart idea about how to improve things. The code will do it by itself. That's an absolutely extraordinary notion. And whether or not, you know, it's too simple, it's exaggerated, the timeline is slightly longer, the point is, you know, buckle up. This is going to be very, very fast progress. I believe agentic systems are coming, they're going to be more powerful. I think that's going to mean threats in terms of biosecurity, uh weapons being potentially created by bad guys. So the political debate around controlling these models and not releasing them to the general public is going to heat up the popular backlash, which we see in the polls very strongly already, about people just hating AI, hating the fact that it is going to take their jobs, the data centers will push up their electricity prices, uh, their kids will be corrupted because they won't do their homework anymore, and they'll be addicted to chatbots that kind of are worse than social media. I mean, the long litany of complaints about AI, which is already, I think it's the fastest growing popular backlash against just about anything. I was reading a piece today about this. And and it might accelerate. So wow, what does that mean? So I think if you think about, you know, the way the Industrial Revolution led to the ideas about class struggle, revolution, Marxism. And and this revolution may take place ten times as fast and even be ten times bigger. Really buckle up.

SPEAKER_00 49:59

Yeah. I mean, I think of it in terms of how many people really understand the algorithms of artificial intelligence. I'd be surprised if the number was five million, right? And you have eight billion people on the planet who don't understand the algorithms of AI. And so the power imbalance is unlike anything we've ever seen in human history, right?

SPEAKER_01 50:22

Mm-hmm.

SPEAKER_00 50:23

So the eight billion probably not gonna feel comfortable about the five million, I would think.

SPEAKER_02 50:30

I think that's correct.

SPEAKER_00 50:31

Sebastian, thank you very much for your time. We're uh right on the one hour mark, which we try to achieve when we have great guests. Thank you very much for doing this.

SPEAKER_02 50:39

It's been a great pleasure, John. Thank you.