Mark Saroufim

Earlier this year, basically, sort of the the the great, you know, the great lock-in of Christmas, we came back from this kernel competition on NVFP4, which was like one of our most popular competition yet that we worked on with Nvidia. And like the fourth place winner was like, Oh hey, like I've never written a kernel day in my life, but I have the fourth fastest kernel in the world for gems. And I'm like, What? And then like a few days later, someone's like, Oh, like I'm a high school teacher, I don't even know what a kernel is, but like I have the fifth fastest kernel for like a dual gem problem. Like, thanks, Mark. And I'm like, okay, like this is bizarre to me because it and generally it felt like systems was this very gatekeepy thing. People are like, No, you're incorrect. People are just like very grumpy, you know, sort of traditional vibe of the C programmer. Sorry for the sort of casual insult, but but but you know, it was like not the most like friendly bunch because like there's like sort of a lot of rigor and a lot of ways you can shoot yourself in the foot.

Conor

Welcome to ADSP the podcast, episode 303, recorded on July 20th, 2026. My name is Connor, and today with my co-host Bryce, we continue part two of our three-part chat with Mark Serafim. In this episode, we talk about open versus closed models, core auto, GPU mode, auto research, and more.

Bryce

I have two controversial questions for you, Mark. Question number one What do you think about you you mentioned sort of the the first like open models though with a good license? What do you think about the state of open models versus closed models today?

Mark Saroufim

Yeah, it's it's a it's it's a good question. Let me see. Like it is just remarkable, like how quickly some of the best like Chinese models have gotten. Like basically, it's sort of there was I I think initially this hypothesis that what they're doing is at best distilling. And then you're like, okay, maybe not. Like maybe they're actually pretty good. I think there's sort of like a lot of open questions, like where are the flops coming from? Where is the data coming from? I think those questions don't tend to get very clearly answered in the technical reports, but at least like as far as like on architecture research innovation, like I think I'm seeing sort of fairly world-class work. I will say I've never myself worked at a like a Frontier lab, like OpenAI or Anthropics, so I don't know like what they have and how much better it is. But like based on the evals, like it can't be like that much better. And so I don't know. I will say though that like one thing that's I found like a bit disappointing though is that like you know, like you might say, well, like you should just use an open model, but like the definition of local LM has changed a lot in recent times, where like it's basically hey, you have a GB200 node for your local LLM, like, yeah, okay, bro. Like it's a six million dollar machine for a year, you know. Like, like what are you talking about? Like, who's actually gonna rent this out? Like, maybe, but it to me that this always seems like it's gonna be a niche thing. I I do think there's something quite remarkable about like being able to sort of share like basically all requests in a batch with your friends, basically, or your company, and then getting economies of scale out of that. But as far as I can tell, like managed inference providers would rather give you a managed deployment because it's just cheaper for them to do so, as opposed to make you pay for usage, like which is kind of like closer to what modal does. Yeah, and generally, I think the pay for usage model to me is more economically aligned with your customers because you'll just get users and then you'll figure out like how do I route them to all to the same model, to the same priority, to the same batch. And if you're doing managed deployments, you're probably not gonna do that, and there's gonna be inherent waste and high margins because of it. But I think a lot of this is like I think downstream of GPUs being scarce. So, like, if GPUs were not scarce, we wouldn't need these business models. Like I think paying for usage would be fine. So, yeah, like in some sense, I think people, my sense is they overly focus on the like the weights question, but I also think there's like this whole ecosystem around it that like needs to like work out.

Bryce

I I I heard an interesting theory based partially on first hand experience, and the theory was that one of the reasons why there's a perception of a quality gap between the closed models and the open models is because the closed models have one inference provider and they can ensure the quality of that inference setup. Whereas for the open models, it can be served in many different ways. And the there can be, and I I did not realize the degree to which this can have an like who's serving your inference for your model can have an impact on the performance, especially around things like tool calls, etc. I mean, I didn't realize that. And I I've actually heard that some labs have changed their licensing a little bit, not to make them more closed, but to prevent them from being served in a way that would create this perception of, oh, this model's so bad because I tried it on this one provider that did not have a proper setup.

Mark Saroufim

I mean, it's a good question. I mean, like, I like as I sort of think through this, I think there's like a bit of an like iceberg here, which is like if you're like, you know, one way you could be an inference provider is like you just have GPUs and then you just like you know, VLANSG Line and then you charge people for tokens. You know, like slightly you know deeper would be you're like, okay, I'll call a compiler and I'll like make sure CUDA graphs are applied. You know, slightly deeper is you know, maybe you write a few custom kernels here and there, you know, a bit deeper is you provide the fine-tuning service for your customers and then let them redeploy those. And so you have to write some backwards kernels for them, which typically the Chinese labs do not provide. And so, like, and then sort of the but but but what I sort of like and then and then what's interesting, even at this sort of and then you have quantization, I think it's just very controversial because like people maybe don't mind being quantized, but they'd like to know when they're being quantized, which is very bad because if you have some sort of like let's say priority cue thing going on, like okay, so so let's do not match the points. Let's say you're just like a normal user and you just hit a request to an inference uh user an endpoint, you may not even know if you're running on NVIDIA or AMD GPU, for instance. And then the numerics might diverge slightly, and then so it's not like the you're being nerfed, it's just like the numerics diverge slightly, and maybe they were only like thoroughly tested on Nvidia, and then you get like an AMD inference and you're like upset, basically. Another one is like, you know, like AMD has a bit more VRAM, and so now like the the batch size might be like a bit bigger if you're serving multiple requests, and the kernels might not be batch invariant, and so now again your numerics change, and you're like, oh, why am I getting nerfed? And then finally, you could actually be getting quantized. Like, like this is also like very problematic that like it's not sort of crystal clear. And I think why there's sort of sometimes on Twitter, like some conspiracy theories around it. People sort of say, like, oh, like at 1 a.m. Claude is quantizing us. Like, who knows? Like, it could be like, is it running on training? Is it running on like an H100? Is it running on a GB300? Like, is it like running, you know, on a CPU? Is it running with CPU offloading because it's running on an older generation A100 that happens to be the data center? And so some of the layers are offloaded and computed on CPU. So there's sort of like a lot of unknowns, but I think a lot of it in my mind is like numerics suck. And as a vendor tries to make these straight-offs to save costs, they might be creating numerics problems, and as a result, results won't be perfectly reproducible, and people might believe that they're they're getting nerfed.

Bryce

But yeah, I mean it's it's fascinating to me that well, and one of the things that I think is interesting here is the non-deterministic nature of LLMs means that it's very hard to tell whether something changes. And it's fascinating to me how you know, like we call it like Opus 4.8, but like the Opus 4.8 that you got at one point in time might be different from what you get at a different point in time. And you know, like in any other part of software of the software industry, like a version would have like a concrete meaning. And if you just suddenly changed the version that you were serving to your customers, like that wouldn't be cool. But in this, in this industry, it happens all the time.

Mark Saroufim

Yeah.

Bryce

And I like I I've noticed I've noticed like all sorts of just even switching like same model between like different like in for in like endpoints and providers with an Nvidia's like like a hub, like sometimes there's like a weird behavior, and like, could this be could this be in like how I'm getting routed to this model? And yeah, I I I didn't realize before I got more into this, like the degree of complexity that goes into serving inference for a particular model and how it can impact the quality that you get out of it.

Mark Saroufim

Yeah, I mean what I will say though, like I am sort of seeing some light at the end of the tunnel these days because so we recently had like James Demel, like the linear algebra legend from like Lapac, come give us a talk. And like one of the things he mentioned, like I was like asking him a question in private, like about like formal verification, like, can't we just like have better guarantees of like ways of testing these kernels? Because just as a baseline for your viewers, the way a lot of kernels get tested today is basically using like absolute and relative tolerances, which basically means like let's say your output is a tensor, you iterate over all the elements of a tensor, and then you sort of compare them to like a reference to like your let's say your faster implementation, and then you're like, are they far apart from each other? And typically, if like I don't know, they're within one to the e minus three of each other, people go like ah it looks fine. But like, what do you mean it looks fine? Like basically one to e minus three is like very big, and if it's like for float 64, like that's a more major issue than if it's it's for int4. And so, like, what ends up happening is that like these values only ever increase. Like, if you look at PyTorch, for example, or most open source repos, a tolls and r tolls only ever increase because you usually get flakes to avoid the flakes, people just increase the tolerances. And we just don't have like a good way of knowing how to fix these problems because we go, like, oh, like is it a CUDA driver thing? Is it like a library thing? Is it like a new bug in the kernel? And then, like, maybe the bug is okay because the performance speed up is okay, but maybe some customers are not okay with an 8% speed up if it changes the numeric slightly, and maybe some are. Then, you know, there's like a long list of PyTorch issues that get opened up of like, well, we're trying to like I have this CPU program that doesn't match my GPU program, and then they're like, How come it's a bug? And you're like, Well, it's not a bug because turns out like it's not like Nvidia follows the entire IEEE format for like tensor cores, and like that's just like something like we have to like understand and be okay with. But I think it's very counterintuitive, like when you what when you first come across it. So so so just to bring it back to what stuff James was saying, like what James's point was that like you cannot have element-wise comparisons, like you you it's much superior to have like property-based testing. So, for instance, we're doing a QR kernel, like we just look at how orthogonal is Q and how triangular is R. This is a much better way of doing like testing. And I think Lapac, it's like again, it's like a shoulder of giants sort of thing. They've been thinking about correctness for decades. Whereas in deep learning, maybe we've been thinking about it for like nine years and we're not very good at it. And so, this is like to me one area where we kind of need to look at what our predecessors did and then like understand, like, you know, in some sense, we know how far apart the floats are from each other, and so so so there's no like there's no secret there. And so, like, we should be able to just more rigorously think of how to set tolerances and what kinds of properties we expect our programs to have. And I just don't think we're we're there yet.

Bryce

Yeah, it's interesting. When I I used to work on these astrophysics codes simulating uh like binary white dwarfs that'd be rotating around each other. And one of the ways that we checked for correctness was because this code was a code where it was rotating, it was very important to conserve angular momentum. So we would just like one of the checks was you sum up with very high precision the angular momentum across the whole grid. And like if you lose any angular momentum, then like your simulation isn't respecting the most important like law of physics that you're trying to simulate here, which is like conservation. And that was just like a simple, like just from the physics like test, the property of the system that you wanted to enforce. And like, you know, yeah, like we looked at like residuals and like other errors and tolerances, but like that was like the key thing. Like as long as we're not, as long as we don't have outflows of this one property. And the interesting thing too was when you were designing that simulation, there's a bunch of different quantities that were evolving throughout the simulation, like radiation and energy. And it was very challenging, like you have to kind of pick and choose, like, which of these are we going to preserve quality for? And like we knew that this one, this was the variable that like was most important. That like if if we didn't conserve angular momentum, then the whole simulation would become numerically unstable. But there were other things where it was like, eh, you know, if a little bit of this leaks off the grid, it's okay.

Mark Saroufim

Yeah. I mean, like, I like to compare this to how I remember us doing in PyTorch in the Torch AO project, how we do correctness testing for diffusion model quantization. And it would literally be, okay, well, you know, before the model gave a cat, okay, we quantize, okay, still a cat. You know, like this was sort of like so this this was the way correctness testing was done. And then you're like, well, is it like it's a bit yeah, it's it's it's a bit question mark, but but like I think that was sort of the state of the maybe it maybe it's better now. I don't want my diffusion friends to flame me, but like, but at least a year ago, this is what it was like.

Bryce

So there was another tough question I was gonna have for you. But I'll I'll ask I'll ask a different question. So, okay, you you um you said you've never worked at a frontier lab, but you're now working at Core Auto, which I don't know a lot about, but I've like very I'm from what I do know, your mission is like so appealing, so appealing to me right now as I am in search of automating all the things. So like are are you not working you say you haven't worked at a frontier lab, but do you hope that the lab that you're at will eventually become a frontier lab? Like, what's the is the mandate of Core Auto to like develop core technology that other labs would use, or like is your goal like we're gonna build a frontier model?

Mark Saroufim

Yeah, I I mean I I I think these things evolve quite a bit, but yeah, I mean like I I I I I think the the the lab's like mission is to build the world's most automated lab and then figure out how to automate the rest of the world. Yeah, and I think like basically there's a few good proxies for this, which is like, of course, like you know, how quickly can we speed run towards like building like a frontier model? I think is just like a really key component. And because we're making a lot of bets on architecture research, it was based on like when you look at like the talent we have on the team, it's just really critical that this set of researchers feels productive with the kinds of work they can run. And this is kind of where I come in, which is like on kernel alum generation. And so this I think is like a very key and underrated like pre-pre-req, basically, I think of a lot of like auto research work, which is this like, hey, if you want to iterate on whether a research idea is good, but it's like four times slower than what it should be, like that's bad. You know, one notable example that you know we worked on, Rice, was like you know, the the QR kernel. Like the the the context there is that, like, actually, maybe maybe maybe maybe I should take a step back. Let's take a step back. So, the step back is when I first started working on GPU mode, my impression was that like the bottleneck for progress was like an educational bottleneck. Like basically people just don't know how GPUs work. And if you we taught people how GPUs worked, and then we gave them sort of good problems along the way, and then made them more and more ambitious over time that people would figure out like how to use GPUs. And the interesting thing is that like basically near the end of our like and and and halfway through like GPU mode's journey, basically was about a year ago, AI systems weren't very good at writing kernels, like they're they actually they sucked, and the code they would generate would be incorrect, it would be slow, it would be sort of this hallucination between like PyTorch and CUDA and Triton. It just like was wasn't like super coherent. And then, like, earlier this year, basically, sort of the the the great, you know, the like the great lock-in of Christmas. We came back from this kernel competition on NVFP4, which was like one of our most popular competition yet that we worked on with Nvidia, and like the fourth place winner was like, Oh hey, like I've never written a kernel day in my life, but I have the fourth fastest kernel in the world for gems, and I'm like, What? And then like a few days later, someone's like, Oh, like I'm a high school teacher, I don't even know what a kernel is, but like I have the fifth fastest kernel for like a dual gem problem. Like, thanks, Mark. And I'm like, okay, like this is bizarre to me because it and generally it felt like systems was this very gatekeepy thing. People are like, No, you're incorrect. People are just like very grumpy, you know, the sort of traditional vibe of the C programmer. Sorry for the sort of casual insult, but but but you know, it was like not the most like friendly bunch because like there's like sort of a lot of rigor and a lot of ways you can shoot yourself in the foot. And so it makes sense that the community would be sort of more inaccessible and gate, like a lot of things. So here we are, we're now in a world where like turns out like AIs are better at teaching how GPUs work than I was as an educator over two years in GPU mode. Yeah, and so like to me, it was like, okay, well, what like what what do we do now, right? And so looking at it, I'm like, okay, well, it seems that people just maybe aren't sure which problems are worth accelerating. And so this is kind of like now, like my duty, I feel as an educator, has become like telling people what are important problems to accelerate. And so the linear algebra kernels came about because we needed those for our research on like optimizers, specifically like a shampoo optimizer that like Ron, one of our one of our researchers, like very passionate about. And so when I look at this, I'm like, okay, well, as long as we can sort of like very clearly formulate the problem and have like a lot of facilities for reward hacking, like we can make fairly consistent breakthroughs and systems. And once we do that, now like my research buddies can sort of like come up with all sorts of like cursed ideas, and instead of having to like recruit me, which traditionally they would need to do, like, I would need to go look at their code, understand the math, like write the kernel. Now, like I can just basically give them tools for anti-word hacking, and then they can just like go cook, and then they're very productive and they never need to talk to me. And this is great because then I can think of like more general problems. And I and I'm very passionate about this space for a few reasons. Like, I think it applies to like not just kernels, but like you know, inference engines, systems code, you know, what will repos like PyTorch look like in the future.

Bryce

There's so many applications. Like, you know, you can use auto research to optimize for performance. You can also use like if you have the same setup, the same guardrails, the same report reward hack preventions, you can use it to simplify code, you know, like minimize, minimize the diff without regressing performance. You can use it to increase code coverage. Like anything that's a metric that you can like measure and quantify, you can then like automate. And all you need is like to have the guardrails and the correctness checks in place. And yeah, I think like when I talk to people about auto research, I think they think I'm solely talking about just like performance optimization. They're not realizing just like how many anything that can be an iterative, measurable process, you can automate. And then like you can build, you know, these processes, you can chain these different loops together and and build these workflows. And yeah, it's so it feels so incredibly powerful.

Mark Saroufim

I mean, yeah, I mean, at least like my observation here has been like I I I just find the competitive leaderboard format to be a very powerful framing because like so, first off, like you are like the way I think of it, like very broadly speaking, is that like you set an eval, and the eval is still expensive because the eval ultimately needs to run on real GPUs, so it's like a real cost, but it's actually negligible relative to the token cost that people end up paying to like do really well in these things. And and so actually, yeah, Brad, I don't know if you're comfortable sharing like your most recent token expenses.

Bryce

Well, I I I've I've spent 300 billion tokens, I think, since the start of the year, and uh there's various ways of accounting for the cost. But the the thing that I when I first got into this, I too was worried about the cost of running on the GPUs, of renting the GPUs, and I d definitely it is the case that I no longer worry about the cost of running on the GPUs because relative to the token cost, it is it is trivial. But I if I had to guess, I would say that I think today for for most problems, if you want to write like a speed of light kernel, you it's you know, in the thousands of dollars of inference at least, I think to do it to really like if you just want like a decent one, you know, maybe you can get away with less. But but the thing that's the thing that I'm starting to see changing, and one thing that was an interesting thing from this weekend, is I'm starting to see people getting results from the open models that are much cheaper. Like one of the the the person from our group at the EuroPython Sprint who has the best score on the Chelesky problem, it was actually one of the the competitors. Uh there were two me and one other person who I think had the most experience. Um the other fellow I think didn't have as much experience with CUDA kernels, but he was using a variety of models, including some open models. And I haven't checked which of which of the models he's been using is the uh is is the the winner, but I know I've started to see been able to get meaningful results with open models, which could maybe lower the cost of this a lot. I think one of the problems is you know, like how do we how do we if it if you need to have a ridiculous amount of inference to do well in these competitions, how do we like seed this out to researchers and people so that they can learn, so that we can build up this skill set? It seems tough with the current costs.

Mark Saroufim

Yeah, I mean, I'm a bit more optimistic than you are. Like here's how I would think through this. Like basically, like let's say you're trying to serve like uh an MOE on a node. Basically, if you're doing like very small batch size inference, then like only a few of your experts are activated, and so the inference ends up being like quite quite fast. But if you're like medium batch size, then like probably most of your experts will be activated, and so but then if you have a lot of like users and a lot, like a lot of requests, it might be as efficient to do that than to do the medium request scale, because anyway all your experts are activated. And so it's like one of those things where like it seems that like these things do benefit from like economies of scale, but economies of scale not in terms of like I have like a 10,000 GPU cluster and I'm intelligently writing things to them because that's just like not what I think a good chunk of inference providers are actually doing. Like, I think they're doing much more managed deployments, I think it's like simpler to price and finance, like you just pay them some margin on the price of a B200. And then you get like tokens back. So again, like when I when I look at this to me, it's sort of it screams like, well, as long as the emphasis is on one node, and if there's like a few tricks that matter, like basically coupographs, a few kernels, you know, quantization. Like if if that's it and there's no more secret sauce, then like why not just have like a node per small pod or team? Like to me, that seems like pretty economically feasible. If the models get to three trillion and stuff, and this is what we're running, like maybe not, then you're basically because you know, one thing I like to mention to people is like, you know, distributed computing never became popular at home. You know, like we have very powerful like laptops and phones, but we don't have like it's very rare for people to have a cluster at home that communicates. So that to me feels like it'll always be like a niche thing. But yeah, I mean, I'm more optimistic, and and I think like there's this nice virtuous loop where like if the task is make the inference engine faster, you could presumably imagine having something really fast and custom made. It's like one thing I think would be pretty cool as a GPU mode working group. So if anyone's interested in tag teaming on that, like I'll probably I'll probably work with them on it.

Bryce

Uh a working group on what specifically?

Mark Saroufim

Basically, having a very like basically like a starter kit for a very cheap inference service that you can deploy at a company with like auto research that profiles the requests. So like something like that I think would be quite interesting to look at.

Bryce

My my dream is my my dream actually is a world in which when you get your next NVIDIA system, like and you install like you know, CUDA, there's like a little tool there that's just like, hey, like give me your existing CUDA code and I'll just like port it to this GPU and make it fast. So I I I won't say the numbers on air, but but but I can show you guys. Here's my uh here's my spend. I thought it was gonna go up a bit, but it's not it it's in the six figures. Can say that at least. But so this is the amount, and then direct from Claude is direct from the anthropic subscription is 50 billion tokens, and then direct from the open air subscription is 63 billion tokens, and then from our internal inference hub is 162 billion tokens. So that's like maybe it's not 300 billion yet, but I I don't know that this is entirely uh let's see, that's like one one one three and okay, yeah, it's close to it's close to 300 billion. That's the that's through the start of the year. But also I think I think one of the other things is that the the token cost has gone down dramatically over time, and that's just been as I've learned the things that are wasteful that are being done during the process. And I I've I've started to see like increases in token efficiency. Yeah. I don't know. Connor, what what are you at? Yeah, Connor at one point was number seven on the internal like uh uh codex leaderboard.

Conor

I don't know. They don't they don't show us the leaderboards anymore, and uh I don't really I don't really track. I mean I do a lot of auto research stuff too, but um you should tell Mark about you should tell Mark about the I I I think most people now have like stopped talking about the limitations of models, but like you you hear like oh it's not good for X, it's not good for Y, and it's like no, like it's good for everything now. It just you haven't set up the correct, you know, I I I don't know if you want to call it harness or agentic loop system. Well, yeah. Mark left.

Bryce

I don't I assume he's listening, he's gotta say. Well, you know, it's interesting. I so I I I think most people have seeded that for these sorts of tasks, for a lot of tasks that it that, you know, yes, AI can do it. The thing that I still hear is like, yeah, but design or style, or like somebody give me an example of like, well, you know, you know, a model, an agent wrote a new function instead of using an existing function. Like it should, there were, it should have like tried to leverage some existing function and like lift it. And and it's like an and like how do we prevent that without human review? And and I was talking with some other people about it, and somebody was like, well, you know, you could like it's not easy to build like a machine checkable way to check for style for that, but for that particular case, you know, we probably could. You could build a tool that looks at like that like parses the AST of the language and then like compares the similarity of different like entities in the code. And if your diff introduces a new thing that's very similar to some other thing, then you could build a deterministic tool that says, like, hey, you should have used this other tool. Like, maybe we just haven't been trying hard enough to turn all of the like design sense and code style things into machine checkable rules. Maybe we can.

Mark Saroufim

This is kind of what makes working at a like a a Frontier lab interesting, is that like indeed, I think as a user of these models, like probably your best bet is coming up with like linters on top, like basically, and those linters could be other AI systems or they could just be like actual code. But like to me, when I when I hear the problem of like, well, the AI is used too many trick subblocks, or they like reinvent the wheel, a lot of those are RL problems, for instance, like which is basically if the reward is just solved the task, yeah, you know, like code reuse doesn't really matter, but like and defensibility like matters for RL because otherwise you get like syntax errors and those are bad, uh, and they like shoot down your reward. And so, like, I think there's quite a bit of interesting research to do, like, with reward shaping, with like adversarial training. I think that make these things really interesting, but but it's like one of those things where I expect, like, in the same way, like hallucination was like a big problem until like just seems like it's not right now, like post-thinking models and like web search. I think a lot of these problems people are complaining about around code reuse will will be solved by like AI systems even without a harness. Like, they'll just swallow their harness over time. It's just that the harness lets us explore their current capabilities without waiting for like a new model release. Yeah, and similarly, like I mean, you could like come up with a kernel with like thousands of dollars, but eventually, you know, like if you're like a smart human, like presumably you're not going through 30 billion tokens of like KUDA examples before you come up with like a good kernel. Yeah, but you do this much more efficiently. So I think there's also interesting stuff to be done on the search frontier. And it's all like research I'm quite excited about and stuff, you know, I'm working on.

Bryce

Yeah, yeah. I I do think there's ways like to to I I don't feel personally like I have fully cracked the search process, like how to structure the the you know the orchestration and like how to guide a model to actually in particular the thing that I think remains a challenge is like what do you do, how do you prevent a stall, and and how do you you you both want to have depth in a search, but you want to have breadth too, because you maybe you've reached a local maximum, you know, and and the thing that I still run into and and I've now heard lots of other people run into is just sort of sometimes you reach this point where you're on the search. This is really just for things where you're trying to really write the speed of light thing, where you your automated process stalls out eventually. And I think part of it is if you don't shape the the reward and the metric right, then like it can end up writing like spaghetti code. And like once it's written like 10,000 lines a slop and it's got lots of code that's like dead code in there, then it becomes hard for it to like evolve the code. And uh, you know, at some point, like maybe it makes sense. Like if I was a human, at some point I might go and start fresh. And it's like, how do you properly guard guide the model? Like that it's time to go try a new approach.

Mark Saroufim

So so the the analogy I think of is like it's almost like it's gonna be a weird analogy, so bear with me. It's almost like let's say you had a light bulb and you're like it's a lit light bulb, and then you're throwing it in like a dark part of a room. To me, this is how I feel like auto research helps me, which is basically I have like an idea, and I'd like to explore the limits of that idea, whether it's good. But in terms of like idea generation, like what's been more helpful for me honestly has been like reading old books, which is just like you know, I'm sure these are probably already in the corpus, they've been scanned or what have you. But but but but it's like one of those things where like you know, despite so many people doing kernel LM testing, for instance, I wasn't aware of any prior work that does property-based testing in the way like Lapac does. So like there seems to be like a basic read the literature aspect here. Now, yes, you could tell the AI read all the literature and do this, like yes, but but but but but but but I think like it's still quite powerful to have a tool to explore the limits of how good an idea is. And then you know, as you sort of get into reward shape shaping, like it's basically a conversation between you and the AI to figure out like what you actually want and if it's desirable.

Bryce

I guess I I may be I may be like to an extreme here where I like almost all of my work is like completely unsteered. Like I don't want to give the model, I don't want to be the one that comes up with the idea. I want the model to be the idea factory. And I I have both noticed and fear giving it specific specific steering or specific ideas or instructions because if I tell it, if I give it an example of like five things, like if I if you if I put in a skill an example of five things, it's gonna pay a lot of attention to those five things. It's not gonna try five things like those things, it's not gonna necessarily use this as an example, it's really gonna focus on those five things. And if I, you know, if I'm truly searching for like a frontier kernel for something, uh I have a bunch of biases and and I don't have the capacity to generate ideas at the rate that it does. And so I feel like it's very interesting to just let it unprompted search on its own. Let it cook. Yeah.

Mark Saroufim

So I I will say that like I do have some data for this. It's a bit out of date now. This is like February-based data. But at the time I remember when we were working on the Nvidia problems for NVFP4, the we were releasing solutions per problem. Like whenever a problem concludes, we released a solution, and they're all NVFP4 problems. And before our competition, there was just almost no NVFP4 data on the public internet. But then when Gowners released his NVFP4 Gem V, which is a very high quality gem v, in that problem, almost no AI system was in the top 10. Like no AI system in the top 10. But then as soon as they had Gowners' example, which is oh, here's the minimal way of how you can use NVFP4 correctly, then all of a sudden everyone reused that trick and figured out how to already auto-research it for gem for gem, dual gem, group gem. And then the AIs were doing way, way better basically after that. But they needed the like if the token is truly rare in some sense, like a new hardware feature, it hasn't done this work. But but but you're right. Like, I mean, you mentioned earlier, like, why can't you just plop an agent and then have it port all your code? Well, you know, like one extreme version of this is you just like plop the agent, and I don't know, it figures out like the ISA and it figures out that it should append the cutlass and the string of your kernels, and it should figure out all these things. But like I think we all know aspirationally that this is where we'd like to get to. It's basically somewhere in between a performance engineer slash reverse engineer slash like it's sort of like these archetypes are becoming like one and the same. And if we could get like AIs to do that, I feel like that's it. Like basically kernel writing would be solved. Yeah.

Conor

Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

Bryce

Low quality, high quality, that is the tagline of our podcast.

Conor

That's not the tagline. Our tagline is chaos with sprinkles of information. And I actually know while I'm taking a little digression, I am baffled on what's going on with my internet right now. Supposedly, like my Wi-Fi is down. Technically, the there's a green light on the deco, but like I've been trying to like Google stuff while we've been chatting and like the internet doesn't work. And I had a banner at one point on Teams that said like Teams is not able to verify your identity. But like so, I don't actually know. Like somehow I'm getting Wi Fi, but there m there may be there's some like authentication thing that's like failing, which is actually what's causing the Wi Fi to fail. Anyways, I'm here somehow seeing and hearing you guys, so we'll keep going.