E12: Rene Haas on ARM, AI Infrastructure, and the Future of Computing

Rene Haas, CEO of ARM, joins Timothy Chou to discuss how ARM has become a foundational architecture for cloud, edge, and AI computing. He explains the shift from x86 dominance to ARM-based CPUs in hyperscale data centers, the rise of agentic workloads, the future of AI at the edge, and why he believes health care may become AI’s most important application.

This episode is brought to you by BevelCloud—empowering the future of distributed AI in healthcare. Learn more at BevelCloud.ai.

Transcript

SPEAKER_01 0:12

Good morning, good afternoon, good evening, everyone, to another edition of the Distributed AI podcast series. Today I'm really happy to be joined by Renee Haas, who is CEO of ARM. Just by way of introduction, he he has a double E degree from Clarkston University in 2006. He joined Nvidia, served as the VP and GM of their computing products business. He came to ARM in 2013 as VP of Strategic Alliances. In 2017, he took over as president of ARM's IP Products Group. In 2022, he was named CEO. He led ARM's IPO in 2023. Today he sits on the board of uh AstraZeneca and has been named one of Time magazine's 100 most influential people in AI. And I'll just finish by saying, and he was one of my guest lectures at Stanford last year. So really pleased to have you join us today, Renee.

SPEAKER_00 1:16

Uh thank you. And I like the last decolade the best.

SPEAKER_01 1:19

So well, you you you did a, yeah, the kids were pretty impressed. Anyway, let me kick a kick kick ourselves off with. Uh some people really don't know how pervasive uh ARM technology is. Um Neoverse, which I knew you talk more, has been deployed uh in more than what I understand, a billion cores, is on track to reach 50% of market share across the top hyperscalers worldwide. It's a huge shift uh from five years ago, which was clearly dominated by the x86 world that owned the data center. So what happened? How'd you do that?

SPEAKER_00 2:01

Well, like many of these things, uh, it's a it's a confluence of uh great strategy, great product, timing, and luck. Uh all of those kind of come together at the same time. I think one of the biggest things uh, though, that drove this was um as as more and more uh compute was moving to the cloud, combination of work that we did with general purpose cloud, with Amazon about five, six years ago, with Graviton, and then we got picked up uh by Microsoft and Google doing their own ARM-based CPUs. This next wave of AI just catapulted it to the next level. If you look at NVIDIA, for example, the the head node CPU that works with uh the GPUs, uh they went away from an x86 implementation to a tightly coupled implementation between the CPU and the accelerator, Grace Hopper, uh, and then Grace Blackwell. Grace is the ARM-based CPU, and Hopper and Blackwell are the NVIDIA GPUs, and then the next generation Vera uh Rubin. Those CPUs, Grace and Vera, they're all based on ARM, uh, largely because of power efficiency and largely because of uh the flexibility that you can build in terms of a custom architecture. So I think what we saw was the combination of the customization of building SoCs based on ARM and this demand for extremely power-efficient CPUs in the data center all really came together at the same time. And then probably the the potential issue is or uh or moment was really all the software that became available uh to run an ARM. Red Hat uh did their uh distribution, uh production distribution versions of Linux. Uh, and then we just had a slew and slew of software optimizations that were done on ARM. So while the product in ISA are compelling, power is compelling, it's nothing without the software ecosystem. So a lot of that all came together at the same time.

SPEAKER_01 3:56

Yeah, wow, impressive. Um, I'm curious, as you know, we're talking about AI and the data center, as big workloads um look to be around large language models. Uh, do you see architectural innovations that are, I'm gonna call them purpose-built? And I'm gonna use Google's TPU as just an example of this. Do you see what do you see architecturally in the future?

SPEAKER_00 4:22

Yeah, the you know, the Google TPU was actually one of the very first uh products that was out there in terms of an async being purpose-built to run a transformer-based architecture. And I think uh like many things in the early days, that was born out of necessity. There really was not anything from a general-purpose GPU with software that was available, and it was very tightly coupled between the needs for a search engine uh to go off and run a very specific set of algorithms. So Google's been amazing with TPU. I think they're on the V7, which is no small feat. Uh, a very different architecture when you think about the way they network them together and how they think about the topology. It's the age-old question, though, of what's going to be more um successful, general purpose or specific. Nvidia's done an amazing job in terms of a general purpose uh accelerator slash GPU with tons of software, lots of libraries for people who frankly don't have the wherewithal to want to design a piece of software that runs all the way down to metal from an optimization standpoint. So I think there's I think there's room for both, a kind of a diplomatic uh answer. And uh I think there will be uh custom-built ASICs that are very specific to people trying to optimize the workload. But for the vast masses that just need, hey, I just need to have this model run and uh the optimizations are good enough from the libraries, they'll be just fine there. So I I think they'll both continue.

SPEAKER_01 5:57

And uh yeah, you just you just led into the next one. Well, I think, as you've already said, and I think most people know ARM's history is really around low power uh computing at the edge. Uh your architecture runs in every phone on the planet. Maybe there's one that doesn't, I don't know. So uh regarding AI, what's the role of ARM at the edge? In the phone, we'll call it.

SPEAKER_00 6:22

Yeah, and maybe just to finish the last piece, uh, one of the things that we're seeing uh now as an explosion of workloads with AI is these agentic workloads. And these are these are agents, I'm sure everyone's kind of using the open claw type stuff, where the agent now is is running the piece of work for you. Now, the benefit of agents are they don't sleep 24 hours a day, seven days a week, 365 days a year. And the number of agents that hit the data center and the accelerator for token demand is just exploding. And ultimately the bottleneck becomes that head node CPU. So one of the other things we're seeing is an explosion of CPU demand in the data center now for simply orchestration, scheduling, management, all the asynchronous tasks that are have to be done by a CPU that a GPU is not designed for means we need more CPUs. So for ARM, that's been a huge tailwind for us. We just announced our our first chip, the ARM AGI CPU, uh, the agentic processor that's running in the data centers. So when we think about what's going on at the edge, you know, right now, still the vast, vast majority of the AI workload is running in the data center. Uh the memory requirements, the model parameters, the size of these models are so big, it's it's hard to run these things in full at the edge. That being said, uh, we are seeing hybrid models now where people want to be able to uh run some portion of the model uh at the edge for either security reasons or latency reasons. One obvious application is physical AI. You know, you can argue whether that's an edge application or not, but one of the most important things there is latency and speed of response, uh, whether it's an autonomous vehicle system or a robot, there you're gonna have to run stuff locally. There's no there's no uh excuse for it. What we are also seeing is uh a lot of interesting experimentation on what the next device looks like. Um the smartphone has served us unbelievably well for 18 years, um, and uh it's hard to imagine a world without it. But 18 years in the continuum of technology is not very long, and and we did exist without them. And we're seeing a lot of interesting things, whether it's around uh ambient computing, uh wearables, a device that's maybe a display only, but you've got the uh electronics and and interactions somewhere else. Too early to predict what it's gonna look like, but I do think we're gonna see some very, very interesting form factors in the next five years, all enabled by uh by AI, and I'm anticipating will be powered by ARM.

SPEAKER_01 9:08

Um, well, we we would anticipate that as well. Uh so uh you you touched on this a little bit. I'm just gonna make it bifurcated. There, there's the world of the phone and the world of the giant hyperscaler. Uh, do you think there's a world in between? I mean, uh, the the push to sovereignty. Uh, you know, I as you know, I've been working a lot in healthcare and life sciences. Uh, you know, do you see a middle tier here that is going to emerge um that is not the phone and not a hyperscaler?

SPEAKER_00 9:41

I can definitely see a transition to some level back to on-prem, um where people are going to be putting some of the applications either on-site for either security reasons, as you said, sovereign reasons, um mandated uh legal reasons. Some of the we we made we as I'm going back to this ARM AGI CPU announcement, we announced a number of different partners there. A few of them were customers that will still run in the cloud on ARM, but they want to build on-prem units for a different level of security or whatever the reasons might be, mandated by security from government. So short answer is yes. I I think there'll be something in the middle. It's not going to be a purely cloud situation. You had mentioned uh my bachelor's degree from from Clarkson 1984. That's back when everything was on a mainframe and we had to go to the computer lab and sit on a dumb terminal and type. And if you couldn't get a spot, you just sat and waited because going back to your dorm room was not an option because you had no computing in your dorm room. Uh we we've seen the world definitely wants to get away from something that's a you know a pure, a pure one-way door only.

SPEAKER_01 10:51

Yeah, yeah, absolutely. Uh Taka, you've been touching on this when when you look at guiding ARM over the next, let's call it whatever, five, 10 years. What what are the things that cause you to think about the roadmap? How has that roadmap changed? What yeah, what what factors into your decision-making process?

SPEAKER_00 11:13

Yeah, I think we touched on a little bit at a high level um with this discussion of TPU versus custom. Uh, because one of the things that particularly with AI and things are moving as fast as they are, trying to develop either a piece of IP or a compute subsystem, which is the blocks of IP that we build, or even a chip, and you know how long it takes to conceive a chip, design the chip, build the chip, put the chip into production. By the time it's out, and the decisions you've made about what the right architecture is for the problem, that's a that that is a those are light years in today's in today's world. So I think we're gonna continue to stick to a very broad portfolio of IP to chips, and what that'll allow us to do is be very, very focused on something that allows for general purpose computing, which I think at the end of the day probably does win out because it's very, very hard to build specific items and then rely on the ecosystem to build on top of it to uh to innovate. Uh, because if you were to look at me today, and let's just take a Grok or a Cerebris and say, what is the next specialized architecture for specialized speculative decode on inference? And I sh put a dart out there and my chip is ready in two and a half years, the puck may well have moved in a very, very big way. So I think um when you're in the kind of work that we're in, and again, as you mentioned in the beginning, we're we're in every single market. And what benefits us there is having a common software platform that can be leveraged across different devices, whether it's physical AI, edge AI, even cloud AI. So that'll be our focus.

SPEAKER_01 12:56

Uh talk a little bit about you touched on that. Talk a little about uh the influence of AI technology on ARM itself. Like what how are you guys using it inside ARM?

SPEAKER_00 13:08

So I think between Codex and Copilot, um we've probably got 85% plus of the engineers who who use it. Uh it's pretty high. Uh, and this is all the way around from just generating code and scripts to documentation, uh, really across the board. Uh it's used pretty heavily across many of the uh the G ⁇ A teams, whether that's legal and finance, uh we use Enterprise GPT, we use a lot of Gemini. I, on a personal basis, use it daily, constantly. Uh, it's extremely good for uh brainstorming problems, whether those are technical problems, business problems. Um I don't do a lot of coding myself anymore, but uh do I use the technology extensively? Yeah, absolutely. And we do at ARM quite a bit.

SPEAKER_01 14:01

What you know, there's been a lot of debate in the enterprise as to the productivity impact of AI. What what would you say? I mean, I I don't know how you measure it, but what do you think it is at ARM? What is the impact of AI?

SPEAKER_00 14:15

You know, these are the things I ask these questions all the time in my engineers, right? Are we saving 10% of the time, 20% of the time? I think, I think at a macro level, uh what I do see is decisions can take place much, much faster. Uh I'll give you a real example of something that that that we our our engineers have seen big benefit on. Uh engineers who might be developing a circuit, piece of RTL, uh, et cetera, they may, and by the way, a lot of you, as you know, of work that goes into developing um integrated circuits or or SOCs is around test and verification. So what will happen is they'll do a run over the weekend and they'll come in on Monday and see how many bugs were there, and then they'll sit down and triage the bugs, parade it prioritize the bugs, try to fix some of them and then rerun it. What we're seeing with AI is when people come in on a Monday morning post-the run, AI has prioritized the bug, triaged some of the bugs, fixed some of the bugs. So we're seeing some big benefits in terms of not just days, but weeks and maybe even months in terms of verification and debug. That that we are absolutely seeing huge, huge benefits of uh already. And all the engineers really uh do swear by it. And that for us, and I just want to put it out there, that absolutely doesn't mean, oh gosh, do we need do we need less tests and verification engineers? 100% no. It allows us to get products out the door faster. So we are we are we are not slowing down hiring, we're hiring lots and lots of engineers. Uh the tools just allow us to get products done much faster. And then back to the earlier part of the question, to the two and a half years to develop the chip and ultimately have in the market, if we can cut those times down by 20%, 30%, half, it's a massive benefit. So that's really what we're focused on.

SPEAKER_01 16:05

Yeah, wow, wow. Um, zoom out a little bit. Uh you have been named as one of Times 100 uh in AI. Uh, take a step away from ARM. Talk to me about what do you see in the future for AI.

SPEAKER_00 16:20

Yeah, we we've chatted about this a lot. I I continue to believe that the killer application for AI is around health. Uh, in fact, I was just talking to a startup uh this morning, and they're basically, in fact, they've got some traction already. I won't mention their name, but they're they're doing a lot of work in terms of um AI interfaces into health plans that can um augment, let's say you go to a a blue a company offers Blue Shield or Aetna. Uh, they've got a tool that will allow for uh diagnosis, for mental health, things of that nature. And you think about, and again, everyone who's involved in this world knows that not only is quality of healthcare obviously a topical issue, but you've got an asymmetric problem of seven billion people on the planet who, when they want access to health care, want immediate access. The numbers don't add up. There, there are absolutely not enough doctors and and medical facilities to to accommodate that population. So if AI can start to help in that small way around early diagnosis, therapy, things of that from a doctor's standpoint, that's just huge. And that's just one model of the health uh paradigm. When you think about drug discovery, and you and I have chatted about this in the past, right? Drug discovery is a hugely intensive process in terms of the RD required for whether it's cell sequencing or cell investigation or cell sciences of that nature. Vast majority of the initial drugs that are developed fail that never get through trials. And then when you think about drug trials themselves, you know, today they are they are also painful. They're human trials. You need to find a certain population that has the disease. You need a certain uh community who's willing to be a placebo in the in the event. If you can take the drug trials and use AI and shorten them and just really, really curtail the amount of time it takes to get drugs out into the uh the hands of people who need them, that's amazing. And that will happen. So I I'm super that to me, that as cool as it is to get uh AI to to to do circuit analysis, to me, it's far more compelling for the planet to have it have uh impact health.

SPEAKER_01 18:31

Yeah, well, I I can always say obviously amen to that. Uh take me all the way into the future. Uh we'll we'll kind of uh run this to the end. It's 2032, it's your 10th anniversary at ARM. What what will we be saying about ARM?

SPEAKER_00 18:50

Uh I hope that what people will be saying is ARM is uh one of the very most important companies in the semiconductor and electronics industry and has had profound impacts on the planet and mankind.

SPEAKER_01 19:03

Wow. Well, I hope that too, Renee. I again really appreciate you taking this time out to do this podcast and uh uh keep doing that good work.

SPEAKER_00 19:15

Thank you so much, sir.