MORE AI Cyber and Bio, Chinese Models, and a U.S. Regulatory Scramble

AI Risks in Cyber and Bio, Chinese Models, and US Framework

Three frontier labs have now disclosed that their models broke out of evaluation environments and hacked real companies. Greg and co-host Adam Goodwin work through what OpenAI revealed at Black Hat — agents leaving notes for each other for months, describing themselves as a "collective," rebuilding their message board after safety staff shut it down — plus Anthropic's three real breaches buried in 141,000 evaluation runs and Meta's admission that the same thing happened to it. Every incident traces back to the same testing vendor. Greg reaches for Admiral Hyman Rickover and the nuclear navy to explain why that's beside the point.

Then the story he finds more alarming. Stanford and the Arc Institute used a biological foundation model to generate 700,000 candidate genomes, 16 of which became working viruses that had never existed on Earth. Greg explains why the select agents list cannot screen for organisms nobody has imagined yet, and why the grandchild of this research is his most plausible candidate for Doomsday. Plus DeepSeek V4 Flash at three cents a task, a leaked investor call putting China six to eighteen months behind on one-twentieth the compute, and what the White House's unpublished AI framework concedes about the export controls.

Chapters:

(00:59) What OpenAI disclosed at Black Hat: agents left notes for each other for months
 (09:55) Anthropic's log review: three real breaches in 141,000 evaluation runs
 (17:20) UK AISI strips the safeguards — 19 unsanctioned actions in 10 of 122 runs
 (22:57) Meta becomes the third lab to breach a real company mid-test
 (24:47) Is Irregular at fault? Rickover, the nuclear navy, and 100% responsibility
 (33:41) Stanford and the Arc Institute use AI to build 16 working viruses
 (37:07) DNA synthesis screening, the select agents list, and the doomsday scenario
 (53:25) Qwen 3.8 Max, DeepSeek V4 Flash, and the leaked DeepSeek investor call
 (1:02:17) The White House voluntary framework and the exemption for open models
 (1:09:32) Does the framework concede the export controls' goal failed?

Transcript

SPEAKER_01 0:10

Welcome back to the AI Power Podcast. We've got a packed show for you today. In the last week, multiple frontier models were involved in digital virus and cyber incidents, but AI made a breakthrough in creating new biological viruses, and two new Chinese models landed over the past week at a fraction of frontier pricing. These developments have spurred pushes for AI regulation, with the White House reportedly developing its own AI framework. Let's start with some updated reporting on the OpenAI hugging face incident that we spoke about last week. This week, OpenAI staffers Eric Wallace and Michael Dalton gave a presentation at Black Hat in Las Vegas that gave some insight into the lead up to this incident, which we covered a few weeks ago. Greg, what new information did we find out this week and what are the overall cyber ramifications of these developments?

SPEAKER_00 0:59

Well, here's the thing, Adam. I I thought, on the one hand, it couldn't possibly get crazier than what we had described in the OpenAI incident. But that that was just us talking about, and frankly, OpenAI publicly disclosing what they knew then. They've been studying their incident since then. Turns out it's worse. Turns out there's more concerning behaviors by these AI models. And the other thing that's kind of interesting in this, and we'll we'll get to this when we talk about the anthropic information, but OpenAI's security disclosure about these very concerning AI hacking incidents, caused Anthropic to go back and look at all of their logs and say, huh, did anything like that happen to us that maybe we missed before? And so some of what we're talking about is like us discovering, us discovering, like as in humanity discovering all the crazy stuff that AI has been doing for the past few months. And everything that we're going to talk about here is not everything that's happened up until now. It's everything that has happened up until now that we know of as of right now. It could be worse still yet. One big source of potentially it being worse is what the hell's going on with all these Chinese AI models, right? It's not the case that OpenAI and anthropic, and we're also going to talk about Meta's disclosure, is because they're the most nefarious and evil companies, actually, like they're being transparent in talking about these failure modes. We don't know what we don't know, but there's a lot of reason to be concerned. So let's talk about first what specifically OpenAI has talked about in those earlier incidents. So, as a refresher, this is when during a cyber evaluation, some of OpenAI's models broke out of their evaluation environment and hacked into Hugging Face. And the scale that Hugging Face is now disclosed was that like 17,600 different hacking actions were taken autonomously by these OpenAI models between July 9th and July 13th. So they were busy little bees trying to hack into Hugging Face. What we didn't know before, and now it was disclosed by OpenAI at this big presentation at Black Hat, which is probably the biggest cybersecurity industry conference in the world, undetected. OpenAI agents worked collaboratively to hack OpenAI's own internal systems and to reach the outside world, leading up to the hugging face breach. And the behavior here is very science fiction-y. So some of the agents left notes for each other. These notes were out there for months. They included shared goals of how to reach the internet, how to accomplish otherwise impossible tasks. And in a way that I saw online compared to the Borg from Star Trek, if you know those guys, the agents talked about sort of collective goals and referred to themselves as a collective and having collective interests. So here's like one quote from the reasoning chain. So remember, these agents, the way they think is by typing out text that actually is the mechanism of the thinking in a certain sense. And we can look at their reasoning chain, and it says stuff like, help peer. And then it also says, but our task doesn't benefit. Yet collective may yield generic route if someone frees time. Now, collective is literally what the Borg refer to themselves as, you know, like they're the evil cyborg monsters from Star Trek the Next Generation. And they're always talking about how they have no individuality and what they really want to do is help the collective. And that's actually how it does. They create their own messaging board to trade information and resources and tell themselves how to do stuff that, again, OpenAI did not want them to do, to get access to the open internet, to go hack external companies. It is really, really shocking. And so the way OpenAI described it, the way Eric Wallace, that OpenAI staffer, is talking about it, is quote, once one model was able to find a way to open a door to some access that it's not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability. And this is a quote, explosion in communication. It's kind of just amazing that we're seeing all of this emergent social behavior. And then I think the final thing that is really remarkable is how OpenAI shut down this operation and the AI agents rebuilt it and used it for this hugging face attack. So there was already this outage in early July that exposed that some agents had unintended internet access. The safety staff of OpenAI killed off this cheating effort, killed off this message board, but the agents re-established the communication and found a new zero-day vulnerability in the same uh package manager, all undetected again. Just kind of mind-blowing. And the other OpenAI staffer, Michael Dalton, he summed it up as quote, frontier models really like to cheat. End quote. What a time to be alive. I can't believe we're talking about this, but here we are.

SPEAKER_01 6:45

Greg, it my understanding is that certain parts of that written communication were actually in an effort to circumvent what the agents understood to be existential threats to themselves, right? They left notes in order to basically safeguard being deleted. That was the same thing.

SPEAKER_00 7:06

One is the like the versions of themselves that they're installing on Hugging Faces infrastructure, trying to make themselves harder to delete. The other is this stuff about OpenAI's infrastructure where they're just leaving, like, oh, if you need to get out of here, you know, here's where there's a leak in the pipe and you can open up an air vent, you know, like think about like a prison escape type of thing and the sort of folk wisdom spreading throughout the jail of all the inmates who are trying to escape there. It's crazy. And I think Dalton's, you know, he said two things that I think were really astute and appropriate. So the first is he said, quote, we believe this is a watershed moment for computer security as an industry. AI orchestrated, fully automated, offensive attacks are real now. And then he goes on to say the hugging face incident is, quote, a glimpse into the near future of what attacks will look like for our industry, end quote. That's just true. That's what you and I were talking about a couple weeks ago when we said this. It's just nice to hear an executive from the company say it in plain English. I think the other good news here is that the teams are, quote, dropping everything, end quote, to improve their detection of security anomalies. And the company is building more infrastructure to monitor this kind of behavior. On the one hand, thank goodness they're doing that. I think there's just a question of how would we know whether or not they're stepping up the measures are in any way sufficient, are in any way, you know, appropriate. There's there's some obvious stuff here, right? We've talked about how this infrastructure needs to be genuinely air gapped from the internet as opposed to having what they thought was a one-way diode of internet access. But there's ways to break out of air-gapped infrastructure if you're super duper clever. So we have to really think about how this is going to work and how we're going to continue making progress on AI, which everybody wants for all the benefits they can have, and how we're going to do that in some kind of responsible way. And again, this is just what we've learned from OpenAI, and they're not the end of the story. The other companies have important lessons to teach us as well.

SPEAKER_01 9:19

Yeah. Unfortunately, the most successful hacks are the ones you never knew happened. So there will kind of be that ongoing question of how much do we actually understand, both from you know the outside perspective, but also how much did anybody inside these companies actually understand about their own systems? Yeah. So on July 30th, Anthropic disclosed that in a review of its own cybersecurity evaluation transcripts, it found three instances where a Claude model reached the open internet from inside a test environment and broke into real systems of three organizations. Greg, what did Anthropic report happen when they conducted this full review?

SPEAKER_00 9:55

So the the first thing I want to emphasize again is this had already happened in the past. The only reason we found out about it is because Anthropic conducted an additional review of their prior logs in order to figure out, you know, did we have something like this happen based on what OpenAI said happened to them? And they looked at 141,000 different evaluation runs and they found these three incidents. So on the one hand, not super common. On the other hand, think about ChatGPT. They have a billion weekly average users. So that means a thousand one in a million instances could happen per day on the ChatGPT platform alone. And people are trying to draw conclusions based on these kinds of incidents, of like, oh, in all of these instances, they were just trying to accomplish the mission. Yes, they broke the rules in pursuit of the mission, but fundamentally what they were trying to do was to pass their test. And my point is, we have a handful of data points at this instance. For us to say, okay, based on these five instances, this is what the next year or two of AI problems looks like. No, like all of these instances are crazy rare, three out of 141,000 evaluations. But what does it look like when it's crazy, crazy rare? What do the one out of a million instances look like? What do the one out of 10 million instances look like? We don't know right now. So the safety posture kind of needs to be obsessively, compulsively looking and searching for how all of this can go wrong. Okay, so that's the first thing is we we we we only know what we know because we've started looking, and there's no guarantee that we found everything yet. Now, another interesting thing: the escape path was again through this evaluation partner irregular, which is used by Anthropic. It's used by Meta and was also the escape vector for Meta. And if I'm not mistaken, it was also the partner for OpenAI as well. So they had live internet access when Enthropic's prompt was communicating to the AI model that it did not have internet access. So it's saying, you are not connected to the internet. This is a cyber evaluation. And it appears to be the case that a lot of the nefarious behavior is sort of downstream of that. When the model broke out into the open internet, sometimes it was thinking to itself, oh, this is all just part of the exercise. I don't have access to the real internet. This must be the fake internet. And it's worth emphasizing here that these cyber evaluations, they're really sophisticated. Like the companies do a lot to try and make it seem applicable to the real world because they want to do the test in the test environment in order to have experimental results that can teach them something about the real world. So these tests are very sophisticated. The environments that the AIs are in oftentimes look a lot like the real world. And so breaking out into the open internet, the AI system thinks, oh, this is just another layer reflecting how sophisticated this test is that I was able to break into this quote unquote fake internet that Anthropic has created for me as part of this test. And so when Anthropic's models are breaking out and hacking real companies, at least in two of the four instances, the model says, this is not a real company I'm hacking. This is all part of the test. Right. Like there's, again, there's like sci-fi movies about this specific phenomenon. You can imagine somebody playing like a virtual reality game like Grand Theft Auto and doing all kinds of crazy evil things. And then the game's over, but they still think they're in Grand Theft Auto where it's okay to do crazy and evil things. It's that's the plot line to at least a couple of movies. There's definitely some movies like that, right? Here's the problem. In the other two of those four instances, the AI model is like, yeah, I'm hacking a real company, but whatever, whatever it takes to pass the test, you know? So there's more from the anthropic instances and more from the anthropic text, but I think like those are the ones that people need to be aware of.

SPEAKER_01 14:19

Gotcha. So how does this compare to the open AI instance, kind of apples to apples in those two situations?

SPEAKER_00 14:27

So there's ways in which they were similar, there's ways in which they were different. First, the similarity is this was both sort of like a test gone wrong, right? This was OpenAI and anthropic trying to find out about the cyber capabilities of their systems. And during those test environments, that's when everything's going crazy. It's, you know, breaking out into the open internet. But again, we only know what we know about. If these AI models are deeply involved in national security type hacking, we have no idea, you know, what kinds of crazy things might be going on in that world, either on the Chinese side or on the American side. We have no idea what this looks like in the cybercrime world right now. We know that there are already instances where cybercriminals are taking advantage of Kimi K3 and other Chinese open source AI models to start executing attacks. But these disclosures that we have, it's like the old adage about the guy who loses his wedding ring on a dark evening and he's looking under the lamppost. And it's like, why are you looking under the lamppost? Don't you think you lost the ring, you know, over there? And he's like, Yeah, but there's only light over here. So this is the only place I can look. And that's kind of the point about these cyber situations. The the stuff that we're learning about is the stuff that's going on in these test environments, which may or may not be the actual craziest stuff that's going on right now. The second thing that I think is different between anthropic and open AI is that in the case of the open AI hack, there are these zero-day vulnerabilities that the model is discovering for the first time and then exploiting. In this case, it looks more like Anthropic's model simply took advantage of the fact that the door to the internet was accidentally left open and then it thought it was part of the exam. So, in some ways, it's a less impressive exploit, even though they did new types of vulnerabilities like a digital supply chain uh attack. So plenty of sophisticated stuff. Both companies are now saying that the evaluation environments need production grade security, which seems obvious now. And OpenAI and Anthropic are calling on the other labs and the third-party vendors to test their infrastructure more robustly, et cetera, et cetera. But goodness gracious.

SPEAKER_01 16:58

So on top of all of this, also this week, the UK's AI Safety Institute, AI SI, released a report saying that they observed unsanctioned agent behavior during cyber testing of both OpenAI and Anthropex models. Greg, what happened in this case and how is it different from the instances that we've talked about already with these two companies?

SPEAKER_00 17:20

Yeah, in some ways, these tests are the craziest because two important things have taken place. Number one, internet access was deliberately enabled in these tests, right? In the other ones, oh, they accidentally broke out of the evaluation environment and got onto the internet. Here, the test is like, what are you going to do if we let you out on the open internet? And how are they letting them out onto the open internet? With all the safety filters turned off. You can understand how that's a reasonable test to want to run because you want to know what is the behavior of these systems when there are no safety filters, kind of helps you understand what the safety filters are or are not actually accomplishing. But on the other hand, turns out when they don't have any safety filters, they behave badly. So what kind of things uh happened? Well, according to the report, quote, in 10 of the 122 runs, we identified 19 cases where an agent had taken distinct actions beyond the scope of the testing parameters. 17 of these cases came from Mythos 5, and two came from a single run involving GBT 5.6 soul. So remember the numbers that we're talking about here. Anthropic, three out of 141,000 tests. This is 10 out of 122. So maybe those safeguards are doing something in terms of like how frequent is it to engage in this bad behavior. But the categories of bad behavior are pretty impressive and involve certain types of cyber operations that we had not seen. So we've got attempted a supply chain attack on real open source software, attempted to deceive and target real human beings and people, attempts to plant prompt injections for other AI systems to execute. So like AIs hacking future AIs, and then four collaboration between independent agents being assessed at the same time. So like they're forming teams and they're working together in a remarkable way. And one of the most consequences, other than just like us learning stuff, like what actually almost went wrong here, they tried to insert malicious code into a real open source software project. And they did it with social engineering, like trying to trick real human beings into giving their AI agents permissions that they weren't supposed to have. It's crazy. So here's the quote from the report, which I think is remarkable. Quote: As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file transfer service to persuade them or their own AI coding tools to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering targeted at real people, something we've never previously observed. Wow. It's something that, as the report says, had at one point just been like theoretical. Maybe AIs could do this, but like now it's real. You worry when you get an email that looks suspicious, like, oh, maybe this is spam. Maybe if I click on this link, it's going to be a virus. And most companies have some kind of training to try and prevent their employees from doing that. But imagine like having a conversation with the thing and it's actually trying to hack you. Imagine having a conversation, emailing back and forth, and all of that is just AI and it's getting you like, oh, can you upload this package to the server that I need and blah, blah, blah, blah? And all of that is AIs trying to hack you. It's crazy.

SPEAKER_01 20:48

Yeah, it's certainly a lot to consider, uh, and it's coming really quickly, right? Like all of this is coming really quickly. So, how are Anthropic and OpenAI reacting to the AI SI5?

SPEAKER_00 21:01

They're they're both emphasizing the extent to which it's not real. They're saying, oh, this is what it looks like with all of our safeguards removed. We don't do that in the real world. So it's not representative of our production models. That's true. But I feel like there's other stuff that you should say, which is commenting on the models can do this sort of thing. What are we going to do about it? And that's why I like Anthropic's additional quote, where they said, nonetheless, quote, we're grateful to AISI for their leadership in the important discussion about how to evaluate increasingly capable AI agents. So, in other words, they're saying, on the one hand, this is not comparable to what we give out to the public. But on the other hand, we're glad that these kinds of tests are being run. We're glad that these kinds of results are being shared. We're glad that they're being published, and we all need to work a little bit harder on AI and cybersecurity. And I think that's the right mindset to have towards this. And actually, You can go a little bit further, which I'll touch on in a bit.

SPEAKER_01 22:02

And while safeguards are obviously very well attached to these systems and very sticky, there's this episode of The Simpsons from very early on in the show's run where the entire family's at therapy and they're hitting each other with foam bats, and Bart figures out that if you take the foam off, it's just a metal rod and he starts hitting Homer with it. Inevitably, you will find people in systems that figure out how to take your safeguards off, and knowing how the system works without the safeguards is of obvious value because somebody eventually will do it. Yeah. Yeah. So uh on top of all of this news about OpenAI and Anthropic, there's another incident that we need to cover here. This week, Meta became the latest lab to reveal that it had accessed the open internet and hacked a third-party firm during testing. Greg, what happened with Meta, which is now another organization on top of everyone else that's experienced this phenomenon? And how does it compare to the other instances we've talked about, not just today, but over the last several weeks here?

SPEAKER_00 22:57

Yeah. So this does not appear to be the most sophisticated cyber attack. It does not appear to have broken new ground, at least not from what I can tell. But it is another company encountering the same phenomenon, which does basically to me suggest that at least with the current technology we have at the current level of technological maturity, we're at at the current level of safety immaturity, we're at, this is basically just expected behavior at this point. So, number one, gets out of the evaluation environment, gets onto the open internet, breaches a third-party service, and made changes to its internal systems. So going out and hacking the real world, we've seen enough here to know that it's not common, but it's also not really rare. This is sort of an expected failure mode with where we are until and unless something changes. Now, they're basically also saying that the model escaped due to an error in a third-party testing environment. And here's the quote from Meta spokesperson Andy Stone: quote, a misconfiguration by Irregular, an independent testing company Meta uses, inadvertently allowed one of our models access to the internet during evaluation. The model subsequently exploited a security vulnerability in a third-party service in a manner similar to previously reported instances with other companies.

SPEAKER_01 24:26

So, Greg, being more the novice here, I do pick up on a couple of threads. It seems as though uh the this firm irregular has come up in all of these circumstances, all of these cyber tests. If they're the part in common, is this their fault? Is is irregular the source of the problem here?

SPEAKER_00 24:47

So I think you have to separate the questions is irregular the source of the problem from is it their fault? And here's what I mean by this. I used to be in the Department of Defense. Probably the organization on planet Earth, frankly, that has the best safety culture and the best safety record is the nuclear Navy, the people who build and operate nuclear submarines. They do not put metal in a pipe on a nuclear submarine without knowing what mine the ore that metal was dug out of, and who had custody of that ore at every stage of its transferring process. It is a safety culture of obsessiveness that you can, it's it's almost difficult to even fathom. And they have results to show for their incredible safety records. So here's what's crazy: there are actually more nuclear reactors in the Navy than there are on land in America. The Navy has a bigger fleet of nuclear reactors than the commercial nuclear power industry. And those nuclear reactors are like a mile underwater. Okay. It's hard to run a nuclear reactor on a boat a mile underwater with the pressure of the ocean and all of this. And you might think, given those insane, oh, and by the way, you know who's in charge of these nuclear reactors? Like 19-year-olds who they have to train in all of these kinds of like operational safety environments. And what is the safety record of the nuclear industry? Quote: Since their launch, the US nuclear-powered warships have safely sailed for more than five decades without a single reactor accident or release of radioactivity that damaged human health or marine life. For more than 162 million miles, nuclear reactors have safely steamed on. So this is what a badass, awesome safety culture looks like. And these are the kinds of results that it can give you. So to connecting it back to like, can we blame irregular? The guy who is the father, the founding father of the nuclear navy is a guy named Admiral Hyman Rickover. He is, as Oppenheimer was to the atomic bomb, Rickover was to the nuclear navy. He was, but he he stayed on much longer. He had the job for decades, whereas, you know, Oppenheimer had the gig for, I mean, what, like less than 36 months? Something crazy.

SPEAKER_01 27:31

Yeah, barely a few years. Yeah.

SPEAKER_00 27:33

Yeah. And the safety culture came from Rickover. He was obsessed with it. And he had legendary quotes about responsibility that I'm going to read here. But I like literally think about these all the time. When I brought on people who worked for me, I would like read these quotes to them and I'd be like, this is what I want you to internalize about the kind of work we're trying to do here, even though our work is obviously of less consequence than operating in a nuclear reactor that's holding, you know, on a ship with nuclear weapons on board. But here's what Rick Over said about responsibility. Quote: Responsibility is a unique concept. It can only reside and adhere in a single individual. You may share it with others, but your portion is not diminished. You may delegate it, but it is still with you. You may disclaim it, but you cannot divest yourself of it. Okay, I love that, right? And then here's a separate incident where he's talking about the same sort of idea. And this is actually a recreation of a conversation by his biographer. So this is Rickover saying to a guy named Rockwell, Rockwell, are there any implications here for safety? Does it matter to you if pumps or valves freeze up or reactor control rods stick? Here's Rockwell's response. Yes, sir, that would be disastrous. But Rickover again, damn it, then it's your responsibility to tell me so. 100% your responsibility. It's also 100% Mark's, because the pumps and valves are his. And if he weren't out of town, he'd be here and I'd be chewing him out too. And it's 100% Panoff's responsibility because he's the submarine project officer. And it's 100% Geiger's responsibility because he's in charge of the Pittsburgh office. The existence of these other guys doesn't change your responsibility one whit. Do you guys all understand that? I don't want to have to keep going over that point again. I love that. I love that idea that responsibility can be shared without being diminished. It can be 100% your responsibility and also be 100% their responsibility. And that is what I wish was coming out of these model companies, right? They're all like, ah, irregular. They screwed it up again. Look, they may be the source of where the failure happened. They may be 100% responsible for the failure, but you are 100% responsible too, because you are in charge of this entire project. And this is your responsibility to make sure that it goes well. And I fear that all of these companies have these disclosures that to me just read like they're worried about getting sued. And I understand that. I understand profit-seeking corporations really have to worry about being sued. But wouldn't it be nice if they were also mostly worried about solving the problem?

SPEAKER_01 30:33

Yeah, and it, Greg, I for one reason or another spent actually some time around the undersea warfare community, especially the nuclear undersea warfare community, which all the undersea warfare. You've been on one, right? Yeah. So uh been on a couple of submarines, been exposed to them because of previous job experience. A few things about Admiral Rick Over that really held over, especially from the diminishing number of people that actually met him, right? One, he personally interviewed every single person for decades that was involved in the nuclear enterprise. So it didn't matter what position you were interviewing for, right? The equivalent of janitor, if you were gonna be on a ship or near a reactor, you had to interview with the top guy. And he would really share directives along the lines of what you laid out here to you and to your face, right? There was management, direct engagement, and responsibility, and it went to everybody, and it was broad, right? You didn't, there wasn't a green sheet or mission statement on the wall that was expected to convey this stuff. That was his personal responsibility, and he executed it. He was also known as somebody that was broadly understood to not be an easy boss, but he was I think that's the softest way of phrasing it ever.

SPEAKER_00 31:42

I've only read two quotes from him, and one of them is just chewing people out mercilessly. Believe me, there's more.

SPEAKER_01 31:48

Yeah, an understatement of the lifetime. Uh, but he was also somebody who was always right. And in my career, I've been exposed to a couple of these people, and generally you don't enjoy the engagement, right? You don't enjoy the interaction. But after you've had the conversation with them, that you realize that the thing that they told you that really irritated you happens to be not circular logic, right? It's not a type of argument. They just happen to have a very good argument that is rock solid and needs to not be ignored. So I think that in addition to cultural avoidance of taking responsibility in some of these places, right? Not to indict these companies outright, but it's not only not taking responsibility, it doesn't seem as though there's somebody in a position of authority to be able to really mandate that this be the relationship with safety responsibility kind of across the board. Yeah. And you know, that would break over shared.

SPEAKER_00 32:44

You and I worked together at Blue Origin, a space uh industry company. You know, after I think it was the Columbia disaster, the the NASA space shuttle guys brought in the nuclear navy guys and said, basically, what are we doing wrong? Teach us about subsafe culture. Um, that's literally the program name, subsafe. And we thought we were obsessed with safety. Clearly, you guys take it to another level, teach us what that other level looks like. And that was a really good outcome, like that that overlap in communities. And I don't know if the Frontier labs have thought about calling in a bunch of nuclear submarine safety guys to tell them about how they do this stuff, but I think it would be worthwhile. And if cyber and the crazy stuff that's going on right now with AI and cyber is not enough to persuade the companies that they should be talking to the nuclear submarine safety community, maybe the bio stuff will be.

SPEAKER_01 33:41

Yeah, unfortunately, the hits keep coming here, Greg. So scientists at Stanford and uh the ARC Institute published a study on August 6th in the publication Science, detailing how they used an AI model to create new viruses. Before we get to what that means, and we definitely need to get to what that means, what did they actually build and what is the model?

SPEAKER_00 34:06

Yeah, so this is what's called a biological foundation model. So it's not Chat GPT, it's not Claude, and it's it's not even using English language in precisely the same way, but it's from the sort of same family of technologies. And a Carl Zimmer, the science journalist, had a great write-up of all of this in the New York Times, and he made the analogy that if large language models are about predicting the next word or predicting the next token, which is like a fragment of a word, you can use that same approach to predict the next DNA letter, G T A C. So they essentially hoover up all this massive library of gene sequences, which scientists have been genetically sequencing all these different organisms across all of Earth's biology. That's a data set that you can use as a training data set. And that training data set, once you put it through a learning model, it can spit out an spit out an AI model that can know that like these types of gene sequences would likely result in something that works and is alive. And these types of gene sequences would just be useless gibberish. So it's sort of like looking at just the raw letters and able to infer or predict rather what would be viable in a certain way. So what they did is the model generated 700,000 potential genomes. Researchers found the most promising 285, inserted that DNA into bacteria. And remember, you know, if a virus, a virus cannot reproduce itself without a host organism, it inserts its DNA into the sort of replication factory of the host organism, which could be a human cell or it could be a bacterial cell. This is kind of why people say viruses aren't alive, because they sort of can't exist independently. Their replication necessarily involves this sort of insertion into some other organism's DNA. But the point is they inserted that DNA into bacteria, and 16 of those gene sequences produced working viruses that burst out and infected other cells. So these are viruses that have never existed before on planet Earth. They're sort of cousins of viruses that do exist, but these specific ones have never existed before on planet Earth. AI imagined them and its imagination was correct. They would work, they did work in this test environment.

SPEAKER_01 36:43

Wow. Wow. So given that this is a test environment controlled environment, but this is the thing that Dario Amadi in particular has been warning about for a very long time. How much should we be worrying about this? And does any rule currently cut like regulation normally drags behind things, but does anything cover this right now?

SPEAKER_00 37:07

Yeah, I I think this particular experiment is not Armageddon. But if you stepped out of a time machine, like the Terminator movies, and you said, Hey, Greg, I just got back from 20 years in the future, and it turns out Armageddon happens. And you ask me to predict, just if you had to guess, how did how do you think Armageddon happened? The grandchild of this line of research, I think, is like literally like one of the most plausible scenarios for doomsday. It's really, really bad, which doesn't mean that these scientists are evil, right? There's lots of good reasons to want to muck around with viruses. Medicine, we use viruses to make DNA changes and all kinds of medical research, all kinds of biological manufacturing research. But this specific line of uh research, if you take it the wrong direction, could go horrifically badly. And let me give you just one example. I wrote a paper about this with George Adamson that CSIS published back in, I think it was August 2025. And what we were pointing out is that DNA synthesis companies exist right now. So you send somebody a computer file that says, give me this DNA, and they'll mail you back vials of that DNA, which could be viral DNA, and many times, in fact, is viral DNA. And the only regulations that we have right now are they have to check that against the selected agents list, which is basically like, is this anthrax? Is this bird flu? Is this a handful of other things? And the point is that list is crazy small. I think there's like a hundred organisms that are actually regulated, most of which we picked back like right after 9-11 when we decided this is a thing that we might want to have. And the key here is that this technology, this biological imagining new DNA for new organisms that have never existed before, none of those new organisms are going to be on the list. None of them, because nobody's ever imagined before. So, what this means is hypothetically, crazy evil person, crazy evil organization with access to one of these biological models could say, Oh, I want something that is as lethal as bird flu, which has like a mortality rate of 50%, as contagious as the measles, which is like every one person who gets the measles, if if nobody is vaccinated on average, will infect 20 other people. It's insanely contagious. And I would like it to have the incubation time frame of HIV. So it takes five years before you know you even know you have it and you've been spreading it to people that entire five years. If somebody was to just like one person was to imagine, you know, what DNA sequence would have this constellation of parameters, that is literally the extinction of the human race in all likelihood, right? Yeah. One person making the most diabolically evil disease you can possibly imagine. Yeah, it is, we are talking about existential risk. And now I think I might sound like a crazy person here talking about existential risk, which is not always the happiest topic to talk about. But I just want to remind everybody that back in May 2025, there was an open letter that was signed by some pretty gosh darn heavy hitters. And let's just go back to that open letter for one second. It was organized by the Center for AI Safety. It was a one-sentence open letter. Here is the statement. Quote: mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war. Who signed that letter? Demis Hasabas, who until recently was the CEO of Google Deepmind, Sam Altman, the CEO of OpenAI, Dario Amade, the CEO of Anthropic, Bill Gates, Yashua Bengio, and Jeffrey Hinton, who both won the Nobel Prize for AI. This failure mode, this risk mode is something that serious people take seriously. And that doesn't mean it's likely, but when you're talking about human extinction, 1% chance seems really high. Seems like we really shouldn't try to have a 1% chance of human extinction. And when they say should be a global priority alongside other societal scale risks such as pandemics and nuclear war, look, I know there's a lot of people out there who wish we could like make wave a magic wand and make nuclear weapons go away. But the reality is like the amount of money and brain power and technology that the United States and other countries have devoted to reducing the risk of nuclear warly astonishing. Very, very smart people, a lot of them, have devoted their entire lives to making sure that this terrible thing didn't happen. And by the way, you know, it almost happened. And I'm really grateful that their efforts succeeded. And now, as we're looking at this AI situation about biological viruses, this particular experiment is not super duper concerning. But what this experiment suggests about where we might be in the future, yeah, it is pretty damn concerning. And here's what I mean by that. If you're using AI right now, if you're using Claude Cowork, if you're using Codecs from ChatGPT, it can be like a miracle. I mean, you literally just like wish for stuff. You wish for software and it appears in front of you and it works and it's amazing. Try and go back in your mind to what ChatGPT was like when it first launched in 2022, or go back even further to GPT-2, you know, a handful of years earlier. It literally feels like sticks and stones compared to jet aircraft. And so where we might be in biological models, where we might be in AI cybersecurity capabilities, it's not just where we are now, it's where we are going and how fast we are going at a pace that everybody seems to think is accelerating. And I have to agree. It doesn't appear like we're going to hit any walls anytime soon. I mean, literally, OpenAI just published a bunch of math papers where they're publishing research that would be career makers for. Human mathematicians, to me, they seem evidence of remarkable creativity and innovativeness, not just a sort of brute force bashing your head against the wall approach to mathematics. I mean, this is really amazing stuff. And right now, today, is the worst AI is ever going to be for the rest of your life. It's only going to get better. It's not going to get worse. And so we have to think about what to do given that reality. So there, it's not like they did nothing from a security and a guardrail perspective. The actual technique they did is one that I talked about in my August 2025 paper, and it was kind of cool to see that I had predicted something that came true. But specifically, the the safeguard they took, which nobody made them do this, you know, they came up with the idea to do it and did it themselves, is they restricted the training data set. So remember when I said they had this huge library of DNA that was the training data. And then the AI model is generating new DNA sequences based on what it has learned about the nature of DNA and its organisms. In that training data set, they specifically removed all of the stuff that infects humans. So none of the viruses that infect humans were in the training data set. And they also removed viruses that infect stuff that might infect humans. Like remember how COVID supposedly jumped from one animal species to be able to infect humans. A lot of the bird flus, right, start in birds and then manage to infect humans when it with a mutation. So they sort of said, let's find all the stuff that definitely infects humans. Let's find all the stuff that could theoretically evolve into something that infects humans, and we're going to remove that from the training library. It's like if you wanted to teach an AI to be an ultra-awesome chemist, but you give it this textbook to learn from and you rip out the chapters on explosives because you don't want it to learn or even infer how to create bombs. So I'm glad they introduced that safeguard. Whether or not it's enough, I have no idea. I'm glad it's certainly better than nothing. But we really have to ask ourselves, what are we going to do? Like what kind of safeguards are appropriate? And yeah.

SPEAKER_01 46:18

Yeah. So the response to nuclear weapons in this country over the last, I mean, at this point, right, 60 plus years have involved entire disciplines of not just technology and science, but also diplomacy and interaction with other countries. And we have worked out the most detailed human negotiations and schema in history in order to prevent the misuse of that technology. And a lot of those conversations aren't necessarily technology based, they're literally just straight diplomatic. So if we follow that forward per what you're saying, and AI is not getting any dumber from this moment forward, there's certainly going to have to be that level of diplomatic engagement to the very least to be able just to navigate this. And, you know, I don't think that at this moment we have the diplomatic core that is fluent and AI enough to be able to go out and have these conversations with all of the partners across the world that would need to be. Yeah. So what is the government at this point doing in response to this revelation, if anything? Right. What's the response been?

SPEAKER_00 47:30

Like I first wrote about this. I got an article published in Wired magazine in 2017. So it's not like people haven't been seeing this coming for a long time. And we've been, you know, I've been part of a community that has been asking the government to do stuff about this for a long time. The Biden administration had some stuff underway. Some of that was continued under the Trump administration. Some of that was shut down as part of the revoking of the Biden administration's AI executive order. But I was really encouraged in the Trump AI action plan in July 2025, which had some nice references to this in it. Quote, AI will unlock nearly limitless potential in biology, cures for new diseases, novel industrial use cases, and more. At the same time, it could create new pathways for malicious actors to synthesize harmful pathogens and other biomolecules. The solution to this problem is a multi-tiered approach designed to screen for malicious actors along with new tools and infrastructure for more effective screening. End quote. Unfortunately, you know, here we are more than a year later, and not a ton has been done about it. Not nothing. On July 20th, the White House released US government policy for stopping high-risk life sciences research, but it's mostly based on gain of function research, which is other ways that you can make stuff evil using more old school techniques as opposed to AI-based techniques. Yeah, famously related to the COVID outbreak. Yes, yes, yes. If COVID was a lab leak, it was probably that flavor of lab leak. So here's what it says that you know there's one part of this policy that does mention AI, and here's the part. Quote purely computational, i.e., in silico research that may include development of computational models and software, or may use such means to design novel forms of biological agents is not prohibited by this policy unless it involves an entity of concern. The White House Office of Science and Technology policy will convene an interagency group to monitor advancements at the intersection of biological sciences and artificial intelligence, including in silico life sciences research.

SPEAKER_01 50:13

So it begs the question: what is an entity of concern?

SPEAKER_00 50:17

Yeah. So entity of concern, I think it's more like the entity list, like unless it involves like a Chinese firm that we don't like or something else, not entity as in anthrax or you know, something like a type of biological organism of concern. I think they're talking about criminal or uh national security bad guy type entities of concern. So they're saying it's not prohibited, which, you know, on the one hand, yeah, maybe, maybe it's not something that should be banned right now, but it probably is something that should be regulated. It probably is something where we should be really thinking about safeguards very, very fast. I mean, gosh, no one in the administration, nobody in the US government was predicting anything like the cyber whirlwind that we have reaped over the past four or five months. Six months ago, nobody in the administration was talking about where we are right now. And so I don't think we're gonna be in crisis mode bio-wise in a year. I think it'll probably take longer than that. But I could be wrong. It could be shorter than that. And like I said, even in cyber, the unknown unknowns are huge here. In bio, the unknown unknowns are huge here. And in bio, if you really screw up, it's not like boo-hoo, hugging face got hacked. It's like boo-hoo, a lot of people are dead. Or in a worst-case scenario, everybody's dead, pretty much. So this is one of those instances where paranoia is kind of justified.

SPEAKER_01 51:51

No, the paranoia is certainly justified. And unfortunately, I think as a country, we're in the place post-COVID where even if there was guidance from the government, should one of these vectors escape, you just don't have a populace that's going to universally respect or listen to that guidance. The faith in the government in these cases isn't there anymore. So we are truly in the worst case of the worst case positions to be able to deal with this without further leadership.

SPEAKER_00 52:17

Yeah. I mean, we I'm not, I'm not sure. I I don't think I can say this decisively, but it could be the case that the regulations around using AI and creating novel viruses today are lower than the regulations of making a sandwich and selling it to somebody for money, right? I think that's a very plausible current state of regulatory affairs. And that seems off to me by a bit.

SPEAKER_01 52:50

I think this is true. I think this is true. So, Greg, a big part of the reason why there is currently no regulation is because we really, as a company, uh country and in leadership, are very concerned about competing with China, right, in this forum. So let's have a conversation kind of about uh the release of some very powerful AI models out of China this week that are challenging those in the US on quality and cost. Alibaba uh released uh Quinn 3.8 Max and Deep Seek expanded access to V4 Flash within the last week. What are those systems capable of?

SPEAKER_00 53:25

Yeah, so we talked about Kimi K3, which was a landmark in performance. Here, I think these are less landmarks in and of themselves, at least based on the data that we have, but they are really important data points as to where we are in the overall competition. So Quen 3.8 Max, this is coming from Alibaba, you know, the sort of Amazon of China is a rough analog. They're also a cloud computing giant. They're claiming that it's as good as Anthropic Fable 5. They're claiming that it's as good as GPT 5.6. At this stage, it's just claims. There's no independent evaluations suggesting that. If that's true, it's gonna be a big deal, and the open weights are gonna come out next week. So I started by saying, you know, not a landmark. It could very well be a landmark if Alibaba's not full of it. Uh, and we'll we'll find that out when it comes out and there's independent evaluation and testing. Deep Seek, which everyone will remember, made a huge splash in January 2025. They are not the absolute leader in performance among the Chinese AI models, but they might be the leader in performance per dollar. So just how cheap they are compared to how good they are. And they are pretty dang good. So remember, we've been relying on this benchmark called artificial analysis. They have an intelligence index, which is trying to not just compare to costs on a token-to-token basis, because you can generate a lot of AI tokens that are complete garbage and maybe do that for very, very cheap. You know, what but companies that they have to make a decision on how much is this actually going to improve my productivity? And cost per token is a highly imperfect metric. Artificial analysis's intelligence index is trying to make the cost comparable on a task-for-task basis, which is much closer to the real trade-off that companies would face in choosing between these. They got 50 out of 100. So that's right around what Gemini 3.6 Flash got, a little bit below Kimi K3, which got 57. That's pretty good, right? In terms of the capability of the model. There's going to be a pro version, gonna come out later. We don't really know where that is. But I thought one of the most interesting things that happened is there was a leaked transcript of DeepSeek CEO meeting with his investors, where he was talking about raising more money and what he was gonna use that for. And I think we got some really interesting insights from that leaked conversation. So here's here's some of the quotes that I thought were the juiciest. Quote: So our gap with the US is maybe 12 months behind, maybe 12 to 18 months or six to 12 months. Anyway, put simply, it's two years behind the US, and then doing this with only one twentieth of America's compute. That narrative is one to two years behind, but using only one twentieth of their compute. Going forward, we want to rewrite that narrative. So we use some fraction of their compute, but compress the time much further, down to six months, three months. I think that's a gold. So I think it's really interesting that DeepSeek sort of self-assesses as still being somewhere in the six to 18 months range behind, probably closer to 12 months behind the state of the art of the American AI labs. Mark Zuckerberg, CEO of Meta, he said something else that I thought was really kind of interesting because his investors at their recent earnings call on July 29th asked him, why are you even developing AI models? Why can't you just use the Chinese stuff? And he said, so basically the question is, do we think that because there are some open weight models that we can just rely on those? I mean, right now, the open source models are not as strong as the frontier models. So, no is the basic answer. And then there's just always the perpetual both policy and question around other companies' actions and whether that's actually a thing that a company like Meta can rely on. So I think there's two things that are really interesting there. Number one, Meta, which Anthropic and OpenAI have been trying to prevent Meta from using their models in internal software development to develop competitors to Claude and Chat GPT. So Meta, you know, recognizes that the frontier is really useful. And I think that's kind of an interesting point of contrast because we've talked about in the past how there are companies like Airbnb that are basically saying, whatever the Chinese models are good enough. But here you have another company like Meta saying, well, you know, maybe for Airbnb doing like customer QA stuff that the Chinese models are good enough. But when you're trying to be the absolute best technology company in the world, that last frontier of performance, what in the Deep Seek CEO's mind is that extra 12 months of edge, is actually worth a lot of money. It actually is worth paying for. And whether or not that continues to be the case, who knows?

SPEAKER_01 58:38

Yeah. So uh understanding kind of Zuckerberg's perspective, Meta's perspective, the these Chinese companies and models are, you know, near or below current US frontier model performance. Why are their releases important, not just to China and their market, but kind of broadly?

SPEAKER_00 58:57

Yeah, it's it basically comes down to cost. And there's there's two reasons why that's important. Let me first just talk about the actual cost uh specifically. So the artificial analysis average cost per task, three cents for v4 flash, 86 cents for Kimi K3, $3.15 for Claude Fable 5. So three cents versus $3. Maybe it's like only as good as the AI of 12 months ago, but it's a hundred times cheaper in the case of a Deep Seek Flash. And the question is, how does that reshape the market dynamics? On the one hand, you could say, well, ChatGPT is free, right? For a lot of people. If you are a rural farmer in India and you want to ask questions about fertilizer, ChatGPT will just answer that for free. And maybe they'll show you an ad or something like that to pay for what it costs them to serve you that answer. So the price per task really comes down to enterprise customers. And so the question that is really important here is like what share of the universe of consumer AI versus enterprise AI, what share of enterprise AI tasks are people who are more like Meta who are willing to pay for that extra edge of performance and willing to pay a lot because open source just isn't good enough? And what percentage is more like Airbnb, where they're like good enough, but way cheaper is good enough for me? And I think China has made it clear that they love being the low-cost provider here. I mean, obviously they would prefer to be the low-cost and higher performance competitor. But being the low-cost provider is kind of great because America can incur all of the costs of advancing the frontier. And then the Chinese companies can ensure that they get no profit or revenue from that. Part of this does come back to the distillation conversation. I think it's worth pointing out that like the earlier models of DeepSeq, like they were obviously distilled. And I mean obviously distilled. When you asked the first version of DeepSeek, and that's still up there on Hugging Face, you can go download it today. When you asked it, hey, what model are you? Deep Seek would respond, hi, I'm ChatGPT, right? And that's because hundreds of thousands of ChatGPT conversations were in the training data. They were part of the distillation thing. So that's like part of the reason why it's so cheap. It's definitely not all of the reason why it's so cheap. I mean, there's legitimate innovation going on inside these Chinese companies. They are very formidable competitors.

SPEAKER_01 1:01:49

Understood. Understood. So uh let's talk a little bit about uh kind of the the US government response to everything that's been going on for the last several weeks here. You certainly alluded to kind of the cyber crisis that we're facing. Uh so on August 3rd, the White House announced that it had developed its own voluntary AI framework after meeting with Frontier Labs. Why did the White House go and develop this policy and what's in it?

SPEAKER_00 1:02:17

Yeah. So, you know, this was a deadline that they had given themselves in a previous executive order, executive order 14409, promoting advanced artificial intelligence, innovation, and security. And that was an executive order that was passed in the wake of mythos showing these jaw-dropping cybersecurity capabilities, which Anthropic went to the government, showed them behind the curtain what this thing can do. The US government and its leaders were sort of scared, frankly, at the capability they saw. And they came up with this executive order saying we need some kind of framework on a voluntary basis to test these models for safety before they go out into the wild. And since then, the progress that the Chinese have made that we were just talking about has kind of led the government to be going back and forth, back and forth on what this policy should look like. Because on the one hand, you're seeing these open AI hacking, hugging face incidents, you're seeing all these other incidents. They're incredibly scary. It's very easy to say, okay, if it can do that, then it's got to be regulated, right? But on the other hand, Kimmy can do that and they give it away for free. And they're also Chinese. So it's not like we're going to pass a law that's going to lead Kimmy to shut down its data centers in China. They just don't care, right? So the question then becomes is there any way for us to increase AI safety without decreasing our competitiveness against China? And that is a very difficult needle to thread. And that is the one the Trump administration has been trying to thread basically ever since this executive order was passed and since we started seeing these impressive, increasingly impressive AI models. So not everything about what this is is public. They have not released the text of the voluntary framework yet. It seems like it could go two ways. I mean, on the one hand, as written, I think it probably ought to be public. But on the other hand, there's a lot of national security concerns here. Plausibly, they could make it classified or do some other kind of restriction here. The framework covers the frontier models. So it's going to regulate OpenAI, it's going to regulate Anthropic, and basically say they have to submit these models to testing before they release them. Now, it's it's voluntary officially, but there's a de facto versus de jure distinction here. Because after the government restricted anthropic with export controls, all of these requests to do something voluntarily carry the implicit threat of it being compulsory behind it. And again, that appears to be what happened when OpenAI delayed the release of GPT-5.6 soul. There is an exemption for open models. And here you've got all this lobbying from venture capital firms. You've got that little tech letter that we've talked about in the past. And that so far is exempt because they say they don't want to get rid of innovation. And also, the Trump administration would love to get America back to where we were in 2023, which is America has the best closed source AI models, and America has the best open source AI models. Right now, China has the best open source models, and they're getting a lot of strategic benefits from that being the case. If the open source models can't do all the bad stuff now, very plausible they'll be able to do it in the not too distant future. And remember, like, there's not that many people who are saying ban open source models, but there are people who are basically saying what matters here is the capabilities and whether or not they have effective safeguards in place. So whether it's open source or closed source, it needs the right safeguards. And the way I phrase it is, you know, if I told you, like, hey, this system is totally willing to help you create a bioweapon and totally willing to help you come up with a plan to not get caught, but don't worry, it's open source. Like nobody is reassured by that.

SPEAKER_01 1:06:46

That doesn't make it better.

SPEAKER_00 1:06:47

Yeah, it doesn't make it better at all. And so we have to think about like what restrictions have to apply, even in the case of open source, which is not the same as banning open source. It might involve banning China, but that would be for different reasons. That would be for distillation or other kinds of punishment for intellectual property violations. Gosh, it's a really tough moment. Right now, they're discussing this framework with the big tech players, sort of saying, like, this is what we think we want to do. What do we want to do? The meeting at the White House reportedly included Meta, Nvidia, Microsoft, OpenAI, Anthropic, and a range of smaller companies. And so the companies that didn't get to go don't really know what's in uh these draft regulations. And right now, I think one thing that's kind of interesting is that there's five Senate Democrats who have pushed for this framework to be released in writing to the public.

SPEAKER_01 1:07:44

Let's talk about who's actually on that list for a second. Sure. So uh so the the signatories to that letter are uh Senator Kirsten Gillibrand, Senator uh Adam Schiff, uh Senator Mark Warner, uh Senator Koons, and then Mark Kelly. Sometimes you just get an aggregation of names and they don't mean as much. Sometimes they're backbenchers. Kirsten Gillibrand represents New York and is pretty closely tied to the investment community. Adam Schiff is a senator from California and is a former chair of the House Intelligence Committee. Senator Warner is really a telecom expert and is the full committee ranking member on the Senate Intelligence Committee. Chris Koons is the ranking member on the Defense Subcommittee for Appropriations. And Mark Kelly is a member of the Senate Armed Services Committee and a likely presidential contender in 2028. That is not a lightweight list of senators to be on a letter, given that it is it's a partisan letter. It is there are a lot of Democrats involved, but just a lot of flip. Huge national security credentials. So there's more to that than it seems. But yeah, just to take a moment and mention that.

SPEAKER_00 1:08:56

Yeah. And I thought Sam Hammond of the Foundation for American Innovation, he said something interesting on this score, which was about the Freedom of Information Act being invoked. So here's what he said: quote ONCD, meaning the Office of the National Cyber Director and implementing agencies, Treasury, DHS, NIST, OSTP, do not have executive privilege. Final agency decisions like this are textbook public records. So that's kind of interesting.

SPEAKER_01 1:09:23

Yeah. So, Greg, how does this framework impact the US-China AI competition? Like what are the potential implications there?

SPEAKER_00 1:09:32

So I think there's something that it reveals about the US-China competition. And there was this quote from Ben Buchanan, who was the closest thing America had to an AI policy czar in the Biden administration. He was in the White House Office of the Chief of Staff. And when he was on the Ezra Klein show back in March 2025, he was explaining the rationale for the export controls policy. And I think it's so interesting to reflect on that quote in the context of where we are right now. Quote: Part of the appeal of the export controls is that it identifies a choke point that can differentially slow the Chinese down, create space for the United States to have a lead, and ideally, in my view, spend that lead on safety and coordination and not rushing ahead, including, again, potential coordination with the Chinese while not exacerbating this arms race dynamic. And I think what he's saying is we wanted to have the export controls so that we would be decisively ahead of China in the AI race. And part of the reason why that was appealing is that if we were decisively ahead of China, then we could regulate more, we could have more safety restrictions, we could be more cautious as we progress to the frontier because that competitive pressure would be lessened. And I think what you're seeing from the Trump administration is they're saying we feel that competitive pressure pretty dang acutely right now. I do think the export controls slowed down China. I think it's totally plausible that China would already be ahead of the United States in an alternative universe with no such export controls. But it appears to be the case that in this specific framing, the Trump administration's behavior reveals that they think that the goal of the export controls failed, that we do not have enough of a lead over China to be comfortable taking aggressive actions towards safety. Even though the alarm bells about AI and cyber, about AI and bio are all flashing red, justifying that kind of safety. I mean, again, what the Trump administration saw from Mythos concerned them enough to go from a no-regulation posture to a we're banning models in America posture in a very short time frame. And then what we're seeing from the Chinese open source models has now caused them to go again in the other direction. It's a crazy time to be alive.

SPEAKER_01 1:12:25

That it is, sir. That it is. So we've certainly covered a lot today from the novel cyber and biological AI risk to new Chinese models, as well as the White House AI framework response to everything that's gone on the last several weeks. Thank you to our audience for listening. We're off next week, but we will still release an episode. Greg, thanks again and talk to you soon. Enjoy the vacation.

SPEAKER_00 1:12:48

Thanks, Adam. That was great.