Conor

Each harness is a bit different.

Bryce

And like I even asked it, I was like, Yeah, I I I ran I've I've run into this a lot where like harness differences is especially when you're doing like multi-agent orchestration stuff, you become very tied to the particular way that a harness works and that the harness is tool tooling works. And like I ran it I had this weird thing where like initially I couldn't get I couldn't get models other than Opus to like drive clawed code well. And I thought that the models just were not capable of handling this workflow. But it turned out that really there was just like a bug in the structure of my skills that when it was going to launch, you know, a subagent, the way it was describing what to do was just not correct, and that there was a better way to do this. And with Opus, for you know, maybe because they post-trained it to for the harness specifically, or for whatever reason, it just was sort of smoothing over this, you know, this bug in my skills.

Conor

My name is Connor, and today with my co-host Bryce, we chat about auto research, the top LLMs, Opus 4.8, GPT 5.5, Deep Seek V4, Codex, Clod Code, and more.

Bryce

Okay, I'm good.

Conor

We are live. I think I need uh a caffeinated beverage here.

Bryce

We're gonna get the uh the the the bottle.

Conor

Oh yeah, it used to be uh well one six. I mean, it used to be bubbly. I don't buy bubblies anymore now because they're too expensive. And well, I shouldn't say they're too expensive. They aren't the cheapest of the sparkling waters. Currently, right now, I have a president's choice grapefruit sparkling water that I like quite a bit, and I believe it's I don't know, five dollars or five seventy-five Canadian for a box of twelve. Whereas bubbly a lot of these times depending on where you buy it, ma'am, it's like ten dollars for for twelve cans. Anyways, the listeners do not care about the price of the different sparkling waters. How's it going, man? I imagine so we were supposed to have Dwayne Merrill requested guest by our last guest, Marco, and he has agreed to come on, but he's rather busy, and so he asked if we he could come on.

Bryce

We we can we don't need to we don't need to have Dwayne on. We can just have an agent, uh dw uh uh the agent version of Dwayne on and just just ask it.

Conor

That statement is gonna upset people because they probably are not sure if you're joking or not. And he's joking, folks. He's joking. We're not gonna have the I'm joking? Yes, you are. We're not gonna replace our guests with replicated LLM versions of them yet. That's also a joke, folks. Calm down. Everybody's getting upset about AI these days. And so Dwayne's not gonna be on. That means we're not gonna be talking about decoupled look back, cub, and other topics yet, which means we get to talk about whatever we want, which probably if it's just Bryce and me, means we're talking about AI and LLMs, folks. I heard I I heard on another podcast, Oxide and Friends, hosted by Brian Cantrell and Adam Leventhal, I believe. And they were saying that they were getting some feedback that, like, most of their episodes were becoming about AI and LLMs, and they were those that feedback was not happy about that. Listen, folks, it's hard not to be drinking the Kool-Aid and so excited about this stuff. And I I know I asked Bryce how he's doing. Well, let him answer that question in a second. I'm so excited. I I told my wife about this uh while right after it was the wrong time to tell her about it. I was picking her up from the subway. She had a long day at a conference, and she's like six and a half months pregnant, and she did not respond with the enthusiasm that I was hoping for. But let me tell you, buddy, this is let's see if you can tell. What do you see on my this is my Pod God Podcast app. What do you see on the podcast screen here? I'm not sure if you can make it out in the blurriness.

Bryce

Well, I see acquired, I see ADSP, and I see a bunch of purple back people with purple backgrounds.

Conor

Yes. Welcome to Personality Meta Podcasts. You can't make out who these people are. Or actually, yeah, you probably can now. It's it's uh zooming in. We got Andre Carpathy, Boris Cherney, creator of Claude Code, Demis Hasabis, founder of Deep Mine, aka Google Deep Mind, aka G D M. We got Elon Musk, we got Jensen Huang, we got Ilya Setskaver, Mira Marathi, and there's other folks too. These are just it's a starting list of the people that I care about. And look at that, look at that. You hit on Jensen Huang, let's see if I can do it. Boom. And you've got a list. And today was the first time it actually updated. And so now you can subscribe to someone you want to hear from, not a specific podcast, and it it brings me so much joy, so much joy, folks. Because there's a lot of podcasts, for instance, no priors, Sequoia Capital, a bunch of these like venture capital, blah blah blah, talking about taxes, nobody cares. But every once in a while, people that you are interested in go on these podcasts, and sure enough, today it was training data, which is I think the Sequoia Capital podcast. Jensen went on it. It aired on YouTube three hours ago. It showed up in my podcast queue for this personality, and I don't know why no one has done this before. Why is it me? Why is it some guy who's in research doing this? It's fantastic. And you know what the first problem? What do you think the first problem that I encountered now that I have these cues are that I need to make an adjustment to? I I did not expect this at all, but it was rather obvious once I saw it, and I was like, oh yeah, that makes sense. I can't imagine. More than 75% of the podcasts in each of these meta podcast personalities are not in English. They are in Chinese, Japanese, German, like a lot of the Peter Steinberger.

Bryce

Well, then you you just you just use one of NVIDIA's excellent, you know, speech and translation models to automatically translate the uh podcasts.

Conor

I'm not spending money to translate podcasts in different languages. One could do it, but you're just gonna have a filter before you choose the language.

Bryce

They're not they're not very large models. You could run them, you know, run them on a fairly small GPU system.

Conor

I mean, there is the thing where I've heard that like on Twitter now they auto-translate, and so like you might be talking with someone in Japan that's using Japanese, and uh that's created some interesting interactions.

Bryce

But when I was when I was at when I was at the this the Singularity Intelligence Technology Summit in uh Shanghai uh a couple weeks ago, I all the talks were in Chinese, and so I was like, okay, well, you know, I could just use Google Translate on my phone, or I could go spin up a GPU node and like vibe code a deployment of you know one of NVIDIA's new you know translation models, which I I did, and then I learned a lot about that whole space because it's like you know, you have voice to text, you have text to speech, you have you know, speech to speech. There's like all sorts of different like you know, types of models and specialized use cases here for like if you want a transcription versus if you want like simultaneous translation. And then there's this question of like chunking, of like, you know, the longer that you wait, the larger chunks you have, the better translation you'll typically do because you have more context. But the longer chunks you have, the more like visible latency there is. Because like you i i if you if you wait to buffer up a bunch of words that somebody's saying, like a whole sentence, you're gonna be able to translate it better. But if you buffer up the whole sentence and then translate the whole sentence at a time, then there's like a a more of a lag from when the person spoke to when you start hearing the translation. And one of the one of the interesting problems I ran into, I wanted a smaller model that I could deploy on one L40S.

Conor

Tell people what that is, because honestly, I don't even I've seen the Brev instance, but I don't even know what it is. Like I just know it's a GPU.

Bryce

L40S is like a is a is a mid r mid to low range GPU that is inference focused, like inference and data science focused. So it's not like a big like training GPU. So they're very common in like developer clouds, since they're not like a high-end like flagship GPU. If you're just like doing like code development, you need a GPU to test on, there that's a common option to use. I've maybe it has like 80 gigs of memory or 40 gigs of memory, I don't remember. It it had some some some reasonable but not a you know flagship amount of memory, and it's a slightly older chip. But so I picked a model. I was there live at the conference, so there were like two constraints. Like one, I didn't want to, you know, go and get a beefier GPU. I could have, but that was actually wasn't why I picked a smaller model. I I was like, I the conference is happening right now. I need to pick a model that is small enough that I can download it and like deploy it in a reasonable amount of time. And so like I, you know, was chatting with my agent and my agent's like, yeah, this should take like 30 minutes or so to set up. But so the model that I picked was not a multilingual model. It was where I specifically downloaded a Chinese to English variant of this model. There were other models that I could have downloaded, which was just like any language to English or any language to any language, but those are necessarily bigger models. And the mistake that I made is that there was it was a Chinese to English model, so I had to deal with code switching. So during the conference talks, someone would say something, uh hang on, one sec, one second, Connor. Honey, yeah, I'm in the middle of recording the podcast. I understand, honey, but now now Connor's gonna have to edit this. I I picked this this Chinese to English model. And you know, then what would happen is a speaker would be s you know talking, they'd be saying something, something, something, something, KV cash, something, something, something, tensor parallelism, something, something, something, KV cash. Right? Because if you're giving a technical talk, you're gonna often use words that are typically English language words specific to your technical field. And if you have a model that only understands Chinese, then it hears KV cash and it's like, what what Chinese word did you just say? And it gets very confused. And so that completely threw off the uh the little system that I built. I mean, it did like an okay job relative to Google Translate. And I late later talked to some of the the the like people who make the NVIDIA speech models, and they told me uh uh you know the model that you use had picked a slightly older version of our SDK, you should have used this instead. So if I'm ever at a feature conference where all the talks are in a different language, maybe I'll have a chance to to rebuild that thing.

Conor

Anyways. Well, no, I just I feel like it's so it's so maybe we should spend this episode, we should just talk about the different things.

Bryce

You know, you're using this for to be to be clear, I I this was a thing where I like built and stood this up. Like it was a two-day conference, and I like built and stood it up and had like a nice little web app and like a like app on my phone for this in like a matter of hours. It was very fun.

Conor

Yeah.

Bryce

I mean that you can use oh man, it's just but but but honestly, like like doing that of like oh building some little like app like that, like that's that's like so like three months ago. I'm all about I'm all about the auto like so okay, the the the fun part for the listener here is uh so somebody pointed like that's we have this codex leaderboard within NVIDIA. Jesus. And uh how how did you find out about that? I don't know if we should somebody somebody pinged me because I was 31 on there. So Connor is number seven on the Codex Leaderboard. This morning. And you might think that c that you might think that that means that Connor is winning. However, well, this is not a context.

Conor

I don't want to be high on this leaderboard when when when people get fired for spending too many tokens.

Bryce

I had to stop using so so at NVIDIA we have you know a corporate like subscription to codex, a corporate subscription to Claude where we pay at like API billing rates. But then like most big tech companies, we also have our own LLM gateway that goes through things like AWS Bedrock, etc., and also some of our own hosting of models. And I had to stop using our corporate Claude code subscription because I just I just hit the cap too quickly. And uh and likewise for um for codex, well, it's it's less about likewise for codex, all of the like billing and like metering for these things, like it's awful. Like with like I I feel like I could never know, you know, what what how many tokens I was using or like how much it was costing. And so I just moved all of my usage to our NVIDIA inference hub and then centralized it there. So the the the 31 rank, which I'm I'm surprised that I was that high, but I guess I did do a bunch last week before I switched about a bunch last month before I switched. Yeah, you're primarily using Claude Code. I was primarily using Opus. I do primarily use Claude Code as a harness. I do find it to be the strongest harness for the sort of stuff that I'm doing, which is long horizon tasks with complex multi-agent workflows that require multiple concurrent agents in multiple concurrent agents that are running tool calls that need to be uh carefully uh synchronized with each other and uh concurrency safe. And I have found that Clawed Code's model for subagents and subagent completion, I mean waiting on subagents is better than codex's, because in codecs, you I've seen in codecs, like basically there's like a thing that you call the like wait on subagents. And like you have to like explicitly call that, uh I at least my understanding to get like notifications of when stuff finishes. Whereas in clawed code, if you start a background agent, it will eventually send a completion notification. And so I've had more issues in codecs with subagents completing and the manager agent not being aware of that. And uh I also just in general have found GPT 5.5 to uh I don't know, not be as good. The models that I I'm working with these days is Opus 4.8, GPT 5.5, Deep Seek V4 Pro, and uh NemoTron 3 Ultra. And now I'm doing like complex stuff where I have like different like I have like one model as the manager and then different models as the workers and stuff like that.

Conor

Yeah. It's and and well I don't know how much we should tell people about this leaderboard, but I was I was shocked. I when I found out when well, because Bryce pinged me yesterday and he's like, congrats on being number seven. And I I was like because the thing is is I keep waiting. I know that you've you've your manager has received a couple emails about your spend. I have not been notified about any emails of what my spend is. And so I just always assume if I haven't received any emails, like I'm not hitting some threshold. But the fact that I'm number seven on this list out of like 25,000 people using codex is I would expect there to have been an email, you know? Like is uh I really hope this isn't costing the m the company.

Bryce

It's because we're doing the same thing. It's because we're both doing we're both doing like auto research.

Conor

Yeah, yeah, orchestration of multiple agents running at the same time. Yeah. And I mean that's the thing, is like, yeah, no offense. Uh I'm uh you can tell me if I have to cut this or we can keep this in. We had but it was just like there was another guy, I should I should find it, but there's a guy that recently joined NVIDIA from Google that gave an internal talk at a software engineering something. Let me find the name of this guy, and I'm not sure if he's given the name of the talk was or the name of the event is Agent Native Tools and Coding. And if I open it, it hopefully will show me the name of this guy. If I hit the play button, is it Colt Mick Analysis? I probably butchered the pronunciation of that of that name, but he gave like a talk that's like eight stages of agentic coding, and like level one is like you know, ChatGPT or Gemini in like a browser using their web interface. And then like it step two is like a CLI, you know, command line interface, and then you get all the way to step eight, and it's like you're developing a I think what he called it was a knowledge garden, and you have like an orchestration system where you've got anyways, the we'll probably have to trim parts of good chunks of this. Anyways, uh we'll we'll just say that the talk that Colt, who recently in the last year I think he said had joined NVIDIA, that he gave was, in my opinion, a hundred times more valuable because it shows what at the limit you can do with these tools. At the limit, this is what you can aspire to be doing with these tools, because like at the limit, it's amazing how good these tools are.

Bryce

And yeah, so I I'm gonna ex I don't worry about how much the inference I'm I'm using costs. And there's a couple reasons for that, and I'll explain. I would worry if the the thing that I was doing did not seem incredibly valuable, but I I'm very, very confident that the thing that I'm doing, the general area that I'm working in, is incredibly, incredibly valuable. So that's a big part of it. I know there's a bunch of other people doing similar stuff, and a lot of them are more worried about like token efficiency first. My first concern is capability and functionality. Can we build a thing that that does what is, you know, that does this incredibly valuable work? And I think it's a lot easier to first prove that we can build the thing that is incredibly valuable, and then later figure out how to optimize it and reduce its costs. And I just sort of started on that process on figuring out how to do that. But if you go in with a mindset of, oh, I'm not gonna use the frontier model because it's too expensive, then you won't know what is capable. And in particular, if you're doing something like auto research, the real value of auto research is in discovering novel ideas. It's it's you know, almost like basic research, you know, like answering the questions of the universe. The value in auto research is when it can be an idea factory, when it can come up with things that are like net new, novel, that aren't in literature, that that you look at and you're like, hmm, you know, if if I had assigned a, you know, a senior engineer, a researcher, you know, to go work on this for six months, this is the sort of thing of novel revelation that I would expect to come out of that. And I think a lot of people don't have the right philosophy when they think about things like auto research because they're focused on they're focused on determinism, they're focused on like an efficient and clean process, but that's not typically how humans birth like new ideas. Like if you have an open question, you have to have an open search space. And there will be failure. That is part of the process. The other reason why I don't worry so much about my inference usage is because it it I think it's very clear that this is gonna have a huge transformative impact on our industry. And part of my job, my my mandate at NVIDIA is to figure out how people program our platform. And you know, uh I I think of myself often as being a power user of things, right? Like I'm not necessarily a compiler engineer or a a driver engineer, but I am a power user of those things. And so for me to learn what the best practices should be, I gotta go around and cry try crazy stuff, you know? Like, I gotta go and see, like, okay, like what happens if I go and throw, you know, 50 agents at, you know, winning the GPU mode leaderboard? You know, what what can I learn from that process? So, you know, you if you're just going and burning tokens and and your only output of burning those tokens is like the output of the task, then that's maybe more questionable than if your outputs is not only what the task achieved, but also analyzing how it got there and learning from how it got there. Using the data of, okay, I I you know you know, you I spent twenty billion tokens to do this thing. That's a lot of data that you've generated and how can you learn from that about how parts of your stack could be more agent ready, more efficient? You know, how could you improve your APIs, your tools, etc.? So like, yeah, I'm spending a lot of tokens, but I get a ton out of those tokens because now I have all this data and I can go and ask questions about this data. Like the other day, I like there's a particular command line tool in NVIDIA, and like people are always asking me, Bryce, how do we make our thing, our piece of software like agent ready and agent-friendly? And the other day, I was just like, you know what, I'm just gonna go like, hey, like go to cloud, like like here's where all the data is. Just like go like go go look for all the times that this particular tool was used. And like give me, give me information about like every time we called this tool and did it succeed or did it fail? And then like from that I can learn like, oh, you know what, if this this command line tool, maybe we need to add this flag to it or something like that. So I I think you you have to you have to treat your uh your session logs are very valuable, I think. And it's not just like like you know, it's not just the data, the hard data that you get out of it, but it's also like the the learning experience that you the engineer get, right? Like this is a whole new world, a whole new way of working, so you gotta go and kick the tire on it. And we I think we all have to assume that there's a future world where token costs are a thousand times cheaper than they currently are. And if token costs were a thousand times cheaper than they currently are, then you know, boy, like that would that would enable everybody to do this sort of crazy stuff that you and I have been doing, Connor. And uh that is a very, a very interesting world.

Conor

Yeah. I mean, I I I don't want to call people out, but I definitely have listened to folks on certain podcasts that I listen to, and when they whine about the like what these models are capable of, I think in my head like it's user error. Like you haven't spent enough time learning how to like wield these tools. And I think it's a privileged position that we're at NVIDIA and we get access to these tools, and and like you said, I mean, I'm more concerned about my spend uh versus you, but that's just because I think you're a more confident New Yorker and I'm an apologetic Canadian type uh I'm just waiting for someone to send me an email. But I I I do agree that there's like an immense value from like daily driving these things, and even like if I'm using it for some like, you know, random, not even like auto research thing, you learn things from like staring at the trace, realizing like, you know, one time I was doing something with cursor and it had like failed to recognize both the JSON files that were storing some data and then like a Python script. And I was like, without going into details, I was just like, whoa, whoa, whoa, like what's hap what's happening right now? And I started having like a meta conversation being like, Why are you why are you not surfacing the information in this JSON? And then it was like there's no JSONs in these, like this directory. And I was like, What is going on here? And then like 20 minutes later, after going back and forth with it, it realized that like somehow it had become like developed a blind spot to certain files in the repo. And then I was like, Well, how do I make sure that this never happens again? And it was like, oh, well, we have these like dot for cursor, I think it was like MDC, which were like the uh rules, and it was like this is the first thing that I ingest. And like I understand that you had like uh agents.md or some markdown that stored some stuff, that in that markdown it said, you know, blah blah blah. And even without that, I should have been able to look at everything in this repo, but just for whatever reason I missed that. But if you if it is essential that like this data is needs to be at the top of like the context for every conversation you have, just put like a little sentence in your.mdc file, or even just ask me to put that essential information there, and like it'll never happen again. And that was like for something that wasn't like auto-research related. I was just like asking it a question, being confused, and then was like, wait a second, like we're at the point now where it shouldn't be making this kind of mistake. And and then I realized that like, oh, like it still can make mistakes if you put the important information in the wrong spot. And like there are like basically places for whether you're using you know quad code, codecs, cursor, each harness is a bit different.

Bryce

And like I even asked it, I was like, Yeah, I I I ran I've I've run into this a lot where like harness differences is especially when you're doing like multi-agent orchestration stuff, you become very tied to the particular way that a harness works and that the harness is tool tooling works. And like I ran it, I had this weird thing where like initially I couldn't get I couldn't get models other than Opus to like drive clawed code well. And I thought that the models just were not capable of handling this workflow. But it turned out that really there was just like a bug in the structure of my skills that when it was going to launch, you know, a subagent, the way it was describing what to do was just not correct, and that there was a better way to do this. And with Opus, for you know, maybe because they post-trained it to uh for the harness specifically, or for whatever reason, it just was sort of smoothing over this, you know, uh this bug in my skills. And I I think you make a good point there that when something goes wrong with Nell M, you know, you you have two options. One you can just tell it to fix it, or if it's something that, you know, if you want to just put a band-aid over it, you know, you just tell it how to correct itself and move on. But if you're building like a repeatable like workflow or a skill, then you want to be like, no, hang on, like, you know, I'm gonna be like a psychologist now, and like you like you you sit down with me and like explain to me why you did this thing that you did. Like, don't don't try to fix it. Just like explain to me how we got here. And then like from that, you know, you can get insights into into you know what you might need to adjust about about your your process. And you know, part I think part of this, you know, having having like more reliable evals and having ways of like testing and evaluating skills is really important because of this. It is very challenging for auto research to do evals because most of the crazy behavior happens 24 to 48 to 72 hours into an auto research run, right? Most of the interesting weird bugs happen, you know, after there's been some compactions, after it's be, you know, there's some context rot. And uh and so uh I don't know how to I don't know how to reliably create a CI system for stuff that needs to run like 72 hours and currently costs, you know, many tokens. I think at one point over the weekend, so so you suggested to me on like Friday that like, oh, like, you know, we should look at GPU mode. The GPU mode is this uh uh GPU programming uh community and they have these regular coding contests of which my bots currently hold the the the the winning spot for like six of the seven or six of the five of the six problems. But so like I just like spun this up over the weekend and like tried out a bunch of new things. And I think that one point over the weekend it was like five hundred million tokens an hour or something. That's that's you know, uh before caching. So f fewer than that are uncached, but it's still a lot of tokens.

Conor

Yeah.

Bryce

Have you tr have you tried Nematron 3 Ultra yet?

Conor

No.

Bryce

You should try it. It's very fast. It's very fast. And uh yeah, I've been trying it out a lot, kicking the tires on it.

Conor

I'll add it to my list of stuff to uh to do. Yeah.

Bryce

It was funny, we had this meeting on Friday, and um where where Connor and I were just chatting, and halfway through the meeting, yeah, I'm like, hang on, Connor, I gotta go check on my agents. And Connor's like, yeah, I also have to go check on my agents.

Conor

Yeah, and then I re I remarked, what is it what does it say about us that we're we're meeting for like 60 minutes and both of us need to go like gardening our our our agents to you know kick them off. You know, one of mine just runs flawlessly by itself now for as long as I want. But other times when you're steering stuff, you do need to go check and and that that was one of the things we uh you know, we should bring Colt on. Uh the the guy that worked at Google and recently joined that gave that presentation about like the eight levels of agentic coding and because like one of the things that he remarked, which he's not the first person to remark this. You we both have uh made the same statements and and others have, is that like, you know, we're we're busier than we've ever been before because these tools are like a unlock of so many different things, and well, also it feels like it's a race.

Bryce

It feels like it feels like there's a race, you know.

Conor

I guess yeah, a little bit, a little bit it yeah. I mean, there's a lot of people doing this kind of research and uh you know, trying to figure out what's the best way to use these tools, but it's yeah, it's this it's this like you you know, there was a period of time before I hik I had figured out the right way to use these tools that so that you know I could go to sleep and it would just work. And like it is it is intoxicating while this thing is like making progress. And then like whenever I would just like randomly wake up in the middle of the night, now I reach for my phone because like I I serve my like little uh dashboards to my local IP address. So I just like I go to my local IP address colon, you know, whatever the port that I have it hosted on. And that was that was another thing that I realized is I was like, wait a second, I have all these dashboards, I can serve all of this just like locally on my IP address with like uh you know, I don't even know what the command is, but like you know, you usually on your desktop you go localhost, but it's like one extra thing to do just so that you can look at it. And now like I go to watch TV, you know.

Bryce

I I just like well I told you about my thing, my my where I have this mattermost server where I can, you know, where where all of my distributed agents that are running in multiple nodes are connected to, and I can just go check it out. On your phone there.

Conor

That's that's the I I haven't done that yet, you know. It's so that anytime I'm I'm going for a run.

Bryce

You gotta go, you gotta go cloud. You gotta go at the scale that you're at, you gotta go cloud. It's funny. So I I um I I have a dashboard too, and I usually watch the dashboard, but over the weekend I didn't have to check the dashboard, I just like went to the GPU homepage. You know, every couple hours I was like, oh, okay, well, we've we've claimed uh we've claimed prefix um Yeah.

Conor

That is so funny, yeah, because there's a there's uh there's a couple of those as well, and like yeah, you if you're in your house or whatever, you can go and look. But then also too, yeah, I just go check and you you look at the status to see if you have a submission in the last couple hours, and uh sure enough, it's just cooking. And anyways, we should we should kick we should uh we don't need to necessarily stop talking about AI, but we will wrap this as episode one is whatever our our latest updates. Uh be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

Bryce

Low quality, high quality. That is the tagline of our podcast.

Conor

It's not the tagline. Our tagline is chaos with sprinkles of information.