Bryce

We were right. All of our critics are wrong. I'll put my premise out there. The premise is that higher if you want to get to the best performance, or even the like just correctness or performance, whichever one, if you want to get there in the most token efficient way possible, higher level abstractions, higher level languages, and in particular functional languages, I suspect I'm starting to get early data that indicates that they are the superior option. Like I think that the ideal programming language for some class of tasks, maybe a large class of tasks, is probably something that looks a lot more like APL than anything else. And like the APL also, you know, we I always make fun of you, Connor, that it's so like compact and like symbol heavy, but like that's also very token efficient. So so I think I think that I think you may be justified.

Conor

My name is Connor. Today with my co-host Bryce, we chat about AI versus abstractions, the ideal language for LLMs to target, and more.

Bryce

It is, it is, I'll give it to an American. It's 99 degrees out, and it's supposed to feel like 111, with 99 degrees in Canadian. No, not in dollars! It gave me 99 degrees in dollars. 30 so it's 37 Celsius. The C stands for Canada. But it feels like it feels like 43 Celsius outside right now. So I'm not leaving I'm not leaving the uh the apartment for a while.

Conor

Yeah, it's it is it feels the same in Toronto, but it's only 32, which converted to Fahrenheit is 89.6, feels like 109.4. Yesterday I didn't go for a run because it was it felt like 46, and I hadn't gone for a run the day before. Anyways, I think it was like two or three days in a row, but then today I was like, it's too much, I gotta go for a run. And it was pretty bad. I'd like to say that it wasn't that hot, but it was my heart rate. I was just running like jogging five and a half minute pace per kilometer, which is definitely not that fast for me. And my heart rate was at 165, which is way too high for running that slow. So I walked a bit and stay safe, folks, stay hydrated. I think I lost about five pounds worth of sweat on a 13 kilometer run, which is ridiculous.

Bryce

Anyways, weather's hot and I was I was supposed to go for a run today with with with Eddie. Um you know Eddie, right?

Conor

I do not know Eddie. Who's Eddie?

Bryce

Eddie Nolan, he's one of the C people down in New York. Anyways, we were supposed to go for a run today, and then he just texted me and he's like, it's it's it's too hot. Maybe we can take a rain check. And I was like, okay, alright, I guess we can uh we can do that. Oh man, what a week. What a week. I'm back in in the US for like two weeks, then we're going back to Europe for more stuff.

Conor

Did you just get back from Europe?

Bryce

This is just like I feel like the last three months have been like a year. Hang on, my internet's still being really crappy. Yeah, it's much much better. It's crazy that like my I'm getting like one I'm barely getting one megabit per second from my home Wi-Fi right now.

Conor

Holy F, that's so slow.

Bryce

Yeah. I know. There's some something's gotta be. It's the heat. It's the heat. The uh the Verizon can't take the heat.

Conor

Maybe one day I'll tell the story of how I spent a weekend trying to jerry rig with a Giga Hub 2.0 Bell kit a mesh network with like three different nodes and a bunch of stuff. Anyways, I didn't know what I was doing, and it was only like I think it was Claude. It was Opus 3. 3.7 or 4.0. Oh, this is recent. And uh yeah, it was back in September. And it took me a while, but oh boy, oh boy, once I got that, once I got that kit and pern. Whoo! It is and I also I that was the weekend I discovered that Ethernet cables. I mean I I why I didn't think that, I guess I just never had fast enough internet for the cable to make a difference. But like at some point I just had like you know how you have that basket of like HDMI cables and Ethernet cables and a bunch of cables that like 90% you'll never use, but you don't throw stuff, you just throw it in a like a box in case you need it one day. Anyways, I needed a couple different Ethernet cables, and I was just like, How come? Because I think we signed up for three gigabit internet. It wasn't the fastest, but I think it was like the second fastest. Anyways, and like I wasn't getting anywhere close to it, and then AI is like, well, show me your cable. And then I was like, nah, is that's not gonna cut it. You need to like go order a better cable. And I was like, the cable? I was like, I paid for the internet, isn't that it? Well, I'm sure like 80% of our audience is like, wow, this guy doesn't know anything.

Bryce

Yeah, I was gonna I was gonna I was gonna say, like, before you started talking about inner internet, Ethernet cables, I was gonna say, this is gonna be a Connor is solely a software person type of story. I used to put together racks for our lab when I was at LSU. So I've been there, had to deal with that. Yeah, oh God, what was I what was I just about to say?

Conor

What were you just uh We were talking about heat, internet Oh yeah, yeah.

Bryce

I was gonna tell a great price story.

Conor

Perfect.

Bryce

Which is uh when I when I moved to to Berkeley after I was in Louisiana, I moved to Berkeley, and it's really tough to find an apartment in Berkeley. And I'm staying with my step-stepsister. That is my stepfather's ex-wife's new husband's daughter. So her and I share step siblings. So she's my step-step sister.

Conor

Wait, say that again.

Bryce

It's your stepfather's ex-wife's wife's new husband's daughter. So my stepfather's ex-wife's daughter-in-law.

Conor

Well, Bryce is just frozen.

Bryce

So so you you you you you you got the connection, hopefully.

Conor

I think I understand it now. You did freeze in the middle of that, but I'll stitch it together. Yeah.

Bryce

Anyway, so I'm staying with her, I'm trying to find a uh I'm trying to find a place to uh to live. And uh eventually there's just like only really one place that's like in my like price range and that like is still available. And so I just like I go and I see it, and then I sort of sign up without really like looking at it that deeply. And there were a couple things that I didn't think to ask whether they had, because I was moving from Louisiana, and so in Louisiana, everywhere has AC. It's just like you have to have air conditioning in Louisiana. So like I never even thought to ask, does this apartment have air conditioning? The apartment all had no air conditioning, it also had no wash dryer in the unit, and it had no dishwasher, all of which I suffered with for three years, and then I swore I would never be in another apartment without uh all three of those things. And the day I was moving, so I used to stay at Berkeley Lab in the summers until like 8 p.m. because it would be too hot to go home. And uh there was like no cross-ventilation. And the day I was moving out of of that apartment, there was like this epic California heat wave, and it was like 110 degrees in Berkeley or something. And so I I'm here in my apartment completely naked, packing up all my stuff because it's so hot. You can't even put clothes on. Anyways, that's the Bryce story.

Conor

Well, I caught like 90% of that because you kept cutting out. Uh we'll have to see whether we leave in the part where we talk about you being naked, uh, because I'm not sure how many listeners we're gonna lose over that, even if it's just an audio description.

Bryce

Uh the the I'm I'm I'm running speed tests on my phone to see what's going on with the internet. It's starting to come back.

Conor

Alright. And I'll listen to the full story upon editing it, so and I guess the listener won't hear the drops because they'll be listening to the uh locally recorded stuff. Anyways, you got a bright story. The internet is on.

Bryce

I may have to I think we're gonna let's turn off our let's turn off our video.

Conor

Alright, we're turning off the video, talking into the void. And alright, we started off with a bright story. A little bit of weather, a little bit of internet woes. What else is new? For the for the listener, this is a uh if you're wondering why is this episode so chaotic, or why is it so much more chaotic than usual? It is July 2nd. Happy Canada Day and happy early America Day. W what do you guys say? Happy 4th of July. And we are supposed to release an episode tomorrow.

Bryce

And uh It's sir, sir, it's America's 250th anniversary. Show some respect.

Conor

Uh no, that's okay. That's okay. I mean, America's going through it. You guys can't even like clean your ponds or lakes or whatever mirror pools correctly without wasting a bunch of money. And although I will say, I've heard some very heartwarming stories. I listened to the daily this Sunday, or was it this past Sunday? One of the more recent dailies was talking about the World Cup and different like fans from different countries visiting. And apparently, like, there's a city outside of Kansas City, which uh friend of the pod, Jared Hobrock, who was on uh a couple times, talking about Thrust and Friends. He'll know this city. I don't know of it, I only heard about it in the podcast. There's a city called Lawrence outside of Kansas City. I think we've got that right. And apparently it's hosting the country of Algeria. Algeria. And uh anyway, very very heartwarming to see the people of people of America because they apparently they like they got the local college marching band to like welcome them off the plane and play the Algeria national anthem, and like all of the locals, whenever they see the Algeria soccer players, all they're always cheering for them. And apparently they're like rooting for Al obviously they're rooting for USA as well, but apparently they're also rooting for Algeria. Anyways, so you know, there's some positive stuff about America.

Bryce

So I have some irony for you.

Conor

What's that?

Bryce

I was logging in, you're gonna you're gonna just love this. I'm logging into my Verizon account, and it asks me a security question, and the security question is, who's your best buddy? And can you guess what the security answer was? It was you, Connor. You were the security answer. I can't even remember when I would have set this up.

Conor

Well, that's hard. That's heartwarming as well. Maybe they'll call this the heartwarming episode.

Bryce

Uh oh, man. Oh, yes.

Conor

Anyway, let me finish the preamble though. So uh this is all to say. I don't even remember how I started talking about America Day. Oh, right. I was just saying the date. It's July 2nd. You're listening to this, well, to sometime after July 3rd, but this episode is Jul dropping July 3rd. And I only like this morning, because I've it's been such a chaotic last week and a half. I was in Ireland for a wedding and for a few days off, and then when I got back, my sister was visiting me from Calgary with her two kids. One's three and a half, one's one and a half, and they're a handful. Anyway, so it was like we landed, we had a couple hours, then they were staying with us. And oh yeah, this was the weekend where you wanted to crash, but I wasn't in town. And then they only just left. And uh, anyways, and I realized, oh yeah, tomorrow I have to edit a podcast, and then and then like ten seconds later was like, wait a second, I don't have any content to edit.

Bryce

Oh well, I boy, do I have to do that?

Conor

And then I DM'd Bryce and was like, hey, is there a chance you're f and uh Bryce uh both of us are very busy, so the the odds that we have like a free slot for 30 minutes are low. So I thought it was gonna end up being one of these like I talk into the recorder by myself for 10 minutes and recommend a couple podcasts. But Bryce magically was free. That's like 10 minutes out of the way, or it says 16 on my Audacity. So, anyways, now we're gonna talk uh another 20 minutes about probably something AI related. You know, maybe it's gonna be codex. Although you're on Claude Code. I heard I saw you I saw you got a hoodie on Twitter.

Bryce

I don't know where my hoodie is from OpenAI, but I deserve a Hang on, I gotta try connecting back to the Wi-Fi again because now my cell phone has no service. Okay, now that's like fine. Alright, can you hear me now? Yeah, I can hear you. Okay. Anyways, yeah, that's you know, messed up.

Conor

Anyways, you said you've got stories for me, so I'm ready for 'em.

Bryce

I I no, I have a topic, which the topic is essentially I think our entire life's work is vindicated. We were right. All of our critics are wrong. And that is the premise of uh of my story today. That I I've, you know, been doing a lot of this agency stuff for a couple months now, mostly focused on auto research. But the last week or so, I've come upon the question of programming language token economics, specifically like programming language token efficiency. And I'll put my premise out there. The premise is that higher, if you want to get to the best performance or even the like just correctness or performance, whichever one, if you want to get there in the most token-efficient way possible, higher level abstractions, higher level languages, and in particular functional languages, I suspect I'm starting to get early data that indicates that they are the superior option. Like I think that the ideal programming language for some class of tasks, maybe a large class of tasks, is is probably something that looks a lot more like APL than anything else. And like to APL also, you know, we I always make fun of you, Connor, that it's so like compact and like symbol heavy, but like that's also very token efficient. So so I think I think that I think he may be justified. And so so the actual the the the logic behind this argument is twofold. One, a higher level of of abstractions means that more of the control of how the program gets lowered is decided by the uh the compiler, the deterministic compiler, which is the cheaper thing, right, in comparison to spending tokens. So writing something in some high-level scripting language or something, or or you know, something like modern C, where it's gonna be substantially shorter, substantially less code, substantially less time to develop than writing it all yourself in C, that's gonna just be more efficient. And the other piece of this is around guardrails. Um so so there's one, the question of abstraction layer, and then there's the second question of guardrails. And these usually go hand in hand. And the premise here is that languages that have more guardrails, where you catch errors early, or where there's only really one correct way to write things, that that they will be better for agents because agents will spend less time making mistakes and having to learn from those mistakes. So languages like Rust is the premise, are better for agentic work. And I think that if all of this turns out to be true, then that's that just vindigates all of us, all of us who've advocated for abstractions and you know, generic programming, functional programming, all these things that we talk about that the the people of the world have given us a hard time about. Anyways, I wonder what your thoughts are about the potential that we'll live in a world in the future where everything will be written by agents and it'll all be an APL.

Conor

Well, my first thought is like that's almost the not opposite of what I've been seeing lately, but then again, the code that I've been genering generating has been code for authoring GPU kernels, of which no language like APL or J or K or Q or BQM has a good story for. And I'll find I think I linked it before in the show notes, but someone once on LinkedIn DM'd me, or maybe it was Twitter. I don't know, some social platform, they messaged me uh an article that was talking about the most token efficient languages, and this was like half a year or a year ago. And at the time it was Clojure, and someone there was like an update to the article that said a few people have mentioned, did you measure APL? And APL wasn't great because it was Unicode symbols, so it was like multiple code points in a single grapheme. I don't understand Unicode, something like that. But then someone mentioned J, which is an ASCII-based APL. And J actually turned out to be much more token efficient than Clojure, which was the previous top one. Um anyway, so those are my first thoughts, and then I can tell you more about the things I've been seeing. But like my my newest thing is that I I'm not actually sure it depends on what you're trying to do. But the the for the thing that I've been doing, like kernel authoring, it doesn't matter if you're starting with like PyTorch, TensorFlow, Jax, Koupai, you know, or you're doing stuff in CUDA C at the thrust level, the cub level. Every single time I've put it into my like agentic loop research project, it always ends up somehow writing CUDA C ⁇ . If if it's Kupai, it invokes like Koupai.raw kernel. If it's Jack's.

Bryce

But but see, so sometimes, sometimes that's okay at the end. So I I I did some experiments last week of comparing kernel authoring with high-level abstractions like Qtile or Triton versus like where I where I I I run some experiments for this uh there's this GPU mode's this community that's very into GPU programming and they run these programming contests and they had one last week for dense batched QR factorization. And uh so I submitted a bunch of you know agentic solutions to it, but then I also did some like experiments with that problem where I would run, I ran 12 experiments and um, you know, with where each one ran for like eight hours. And in half of the experiments, I I said use higher level abstractions. And in the other half, like write kernels with high-level abstractions like Triton and Qtile, etc. In the other half, I just gave the guidance of write kernels using Q to C ⁇ . And what I found is that it got it got a higher speed up per token by the time it was done with the higher level abstractions. And when doing the analysis afterwards, the the reason was that with the higher level abstractions, you were able to focus more, or the agent was able to focus more on different algorithmic approaches, different techniques, etc., different numerics, etc. Whereas if you write the lower level code from the start, then you're gonna be more consumed with all the low-level details. Now, I suspect if you if you ran this through to completion, if you if you said, I'm gonna let this run forever, that yes, eventually, the, you know, inevitably, coup to C ⁇ would come out ahead. However, maybe what that tells you is that you should start iterating in the high level of abstraction. And then when you reach a point where you can't get any more performance out of the high-level abstraction, that in those, you know, those critical sections, you drop down to a lower level of abstraction. And that's exactly the message that everybody who's been focused on high-level abstractions, on productivity programming paradigms has been saying for years, which is that this lets you iterate faster. And now we have like concrete quantitative evidence of that. So yes, maybe maybe in the end of the day, everything ends up being written in CUDA C or in C if you're just doing CPU programming, or that eventually, you know, maybe it's just it always ends up being handwritten assembly. But there's the question of how do you discover the right algorithm to write in handwritten assembly? And the best languages and paradigms for exploring different implementation strategies are the high-level abstractions, the productivity layers.

Conor

Yeah, I mean, it's a very, very interesting question. And I think the problems that I've like the the kernels that I've been authoring, or having the AI authoring, haven't been tricky enough. And so maybe that's it why it's able to like drop, you know, down to CUDA C. But yeah, I I've I I have been what is it the what is the way to say this? My belief in Higher level abstractions and the need for them has been like shaken significantly. And I guess that's the thing, is right now what you're looking at is token efficiency. What I'm looking at right now is not token efficiency.

Bryce

It's just like you you've gone past the I guess trying to find speed of light to how to find speed of light using the the best token, you know, per but but I will I I will say I will say something in your defense, which is when I originally started doing this work, I provided the guidance that we give to humans, which is I said, you start by using the libraries, use the accelerated libraries, because the libraries are usually going to have better performance than you know than you're gonna be able to write. And I still believe that is true, and I still think like if you if your goal is to land a minimal and maintainable change, right, into like an actual real-world production code base, like a a change that uses some optimized library is a lot easier to land and maintain over time than a change that adds 10,000 lines of bespoke written code. So there is a question of priority. If you're trying to land something in an actual production code base, then I think that the guidance of use libraries is like use libraries first, and then if they don't get enough good enough performance, then start to write low-level code, write, write hand optimized code, or maybe hand optimizes the wrong word now. You know what I mean. But if you're competing in some coding comp contest where your goal is to write the fastest possible algorithm for something, then I don't know if you should start off with libraries. What you should use is high-level abstractions that are performance-oriented. So things like Q tile, things like modern C, things where it's still focused on performance, but it gives you high productivity and high expressivity. And then the other thing is I think the type of library that you might need or prioritize is different. That building blocks are far more useful than fully baked solutions. That, and I this is particularly true for kernel authoring, but I think it's also probably true for a wide variety of domains. That having libraries that give you building blocks, composable building blocks that can be pieced together that are high performance or high correctness, whatever the thing you're you're focusing on and prioritizing in your application, is probably more important in this day and age than having something that's just like a very easy drop-in monolithic replacement that does like a particular task. Because there's these agents that can piece together a custom solution for your particular needs. And then piecing together a custom solution is probably the right thing to do if it can be done in a way that's easy to maintain and uh and easy to understand.

Conor

Yeah. This is all predicated on the belief or the assumption that humans are going to need to be in the loop for these systems.

Bryce

No, no, no, no, no. I don't I don't think I don't think so. I don't think so. One thing I have certainly noticed in my GPU mode competitions, the GPU mode problems tend to start off with a reference problem that is one line of code. And then over time, you know, the models making it more and more complex. It's it's you know, coming up with special cases for different conditions, et cetera. And it's very quickly you end up with like 9,000 lines of code with uh split across Python and C, etc. And the larger the code gets, the more convoluted the code gets, the harder and harder it is for the agent to work on it. That, you know, the the yeah, I'm not gonna look at the code, but I still care about how complex the code is, because even if it's not a human that's gonna look at it, that there is still the question of maintainability, right? Like that still matters. And this is why people keep talking about this idea of direct to binary, that, oh, in the future, programming languages aren't gonna matter because the models are just going to directly write assembly. Guys, girls. Come on, come on here. It lets nobody's gonna ship a software product where it's the only artifact is purely assembly code in this day and age. I know that it used to happen in the past and there's special cases, but nobody's gonna go and ship a desktop app, you know, that's written entirely in assembly. And the reason is simple is sure, maybe that's gonna work great the first time, but then what happens when you need to add a feature to it? When you need to change something? It's gonna be a nightmare. You might you it might pop you out of perfect binary the first time, but as soon as you need to grow and evolve that software, like software evolution, software life cycles will still matter. And all the things that were applicable to software maintenance for humans will be applicable to software maintenance for robots.

Conor

I mean, maybe. I don't the confidence with which you say that statement is like just reminds me of the hubris of humanity. And I think that there's a ton of people out there, every time you mention the word intelligent with respect to LLMs, you know, there's a number of comments and messages that I get either on the GitHub discussions that say, you know, these systems aren't intelligent. And I just like there's this great podcast episode on the big technology podcast where they had a conversation with Jeff Hinton, the interviewer there, Kantrowitz, I think his name is, and basically there's this like Hinton mentions at a certain point in the podcast that there's like these times in history when like the ego of man is challenged at one point. Like, I'm not a student of history, so I'm gonna get this stuff wrong, but like at one point we didn't think or we thought that Earth was the center of the planetary planetary system, and then Galileo or some other guy came along and said, No, it's the sun. But it took like a hundred years, it took like ten decades for people to accept that. And it was like considered heresy. And uh and there was a couple other examples when like this this kind of like either like theocracy was challenged or man was challenged, and it took like it took years for society to accept that like oh maybe maybe this is the case. And like I completely agree with Hinton that we're we're at one of these like points in history where where people think that like, oh, you know, it doesn't think like us, it's just token generation. And I just like whether whether this, you know, whether LLMs are the incarnation of like beginning of like a new sentient, like I don't want to say species because people are gonna freak out, but just like whether it's LLMs or it's gonna be like the sixth, you know, evolution of LLMs that will be called something else because it's not gonna be like the transformer that is the basis of this technology. Anyways, I just like when you say that we're not gonna produce an artifact, I agree. I am I have an ego, and it makes me think that like of course we want something that is understandable by humans, but like why does it need to be?

Bryce

What why what if we just have like a I'm I I'm explicitly saying it doesn't have to be understandable.

Conor

You just said guys, girls, why would you ship an artifact? Like, come on. You said guys, girls, come on.

Bryce

No, no, no, no, but but my my point my point is that nobody's going to ship software that is very difficult and very expensive to maintain.

Conor

And if you if you produce who's who who says it's difficult to maintain if we're not maintaining it? What if we what if like, you know, some company comes along, builds basically like an amorphous like entity that like that is the product, you know, it it monitors crime or something.

Bryce

Directly writing and modifying today's executable binary formats is not the the isn't is never gonna be the right way to maintain durable software artifacts.

Conor

Like what precludes it for being well what you know I was just talking with Michael about this.

Bryce

Like why does it be because executable executable binary formats are just simply not designed for this? Like what do you mean design designed for what? Like for one, if if you need to if you need to change like one aspect of a binary, it might require you to move around a significant amount of data elsewhere in the binary to like like if we're talking about actual like direct to like an elf binary, like if you're adding a new function somewhere, that might require you to rebuild like a whole you know symbol table or something, or like redo all of the relocations. So the you can make a tiny change to a binary that actually requires like a B like or a tiny functional change that requires a huge, massive actual change to the binary code. And so I can't imagine that that will be the level that people will operate at. Will there be more high-level, will there be more programs expect expressed in terms of high-level IRs like MLIR dialects? Probably. I could buy that. But I I don't think that I don't think that we're gonna be free from the deterministic software piece parts of it. And in fact, in fact, the best uses of of AI, I think, are these combinations of deterministic software, traditional software, and intelligent software. And yeah, you mentioned a lot of use a lot of listeners uh question whether these things have intelligence. I do not think that intelligent is the important attribute of these LLM systems. I think there's three key elements that make them uniquely capable pieces of software. The first is the self-healing nature that or maybe to put it another way, the first is that they're flexible, that that they are by their nature flexible to the environment that they're in and the instructions that they've been given. And this flexibility comes in part from the non-determinism of these models. The second is that they're self-healing. So if something goes wrong, they can assemble a way out of it, not always successfully, but they can they can assemble a way out of it beyond the the circumstances that was imagined when the software was designed. And then the third, the third principle is that they're autonomous, that they're not only capable of self-healing, of flexibility, but they do have some decision-making mechanism that can allow them to act without needing a human's input. And I think the question of whether they're truly intelligent, whether they're truly thinking or not, that I don't think is as important as the question of like what can the technology do. And I think those are the three properties that make them very, very useful. And part of the nature of the current AI technology that we have, and probably all future AI technology, is that it is, it is non-deterministic. It's not always reproducible. But that's okay because you can pair it with reproducible deterministic traditional software. You know, like I had a skill to like export and backup all the data on VMs that I'm using. And I could just write the skill, but if I do that, then I'm not the backup's not going to always be in the same format necessarily. It's maybe gonna have slightly different names every time, etc. I could just write a script to do the backup. But if I just write the script, then you know the script could miss something. Or what if the script I'm running the script somewhere and what if there's a a bug in the script? Or what if, you know, I'm out of disk space or it can't reach the place that I'm doing the backup to, then the script just fails. So the thing that's most useful, the thing that I ended up building was a combination of traditional software and and and instant software, the the the AI, you know, just prompting, etc. And that's to have, you know, I have some core script that I use for the export, but then I have a skill that drives that script and that does a scan of its own and looks for anything that was missing and has a way to add you know any missing files to that backup if needed. And so I think those are the things that are most powerful today. And maybe in the future, it will be that we truly won't need any deterministic software, but I'm I'm just I'm not I find that hard to believe because one, the economics of deterministic software are just so much better than of everything being AI, at least today. Token costs would have to go down 10,000 X for that not to be the case. And two, there's something to be said for reproducibility.

Conor

Um I agree with a lot of what you just said. Like I think future systems are gonna boil down to deterministic scripts that are written by non-deterministic scripts, and then like the non-deterministic LLMs, or I shouldn't say non-deterministic scripts. Like you're gonna have your deterministic computing, which is just you know, Python scripts, choose your language, whatever. But then you're gonna have your you know, claude skills, non-deterministic things that are LLM functions. And the more you can push stuff into deterministic scripts, the better. That's what linting is, that's what evals are, you know, you know, you build loops around that stuff, fantastic. But I just when I think about like what an LLM is, you know, it's ultimately like, you know, with a bunch of pre-training, post-training, all this fancy bells and whistles, but it started out as just like a transformer, you know, and token generation. What is to say that like right now we're we're building these LLMs that can generate CUDA C code and Python code, et cetera, et cetera. And there's linting and there's static analysis, and we're building more and more guardrails, which is a part of the reason that all of this stuff is getting so good. If you play around with you know Claude Opus 4.8, you can look in the traces, and they're all constantly, if you're in Python, they're running rough, you know, and and they're they're fixing things that goes like there's so much, so much of the stuff that like if you were using these models a year ago before you know the big jump back in December of 2024, you had to manually like you know add that stuff to your own loop in order to get good results. That's all baked in now. And my point being is what is what what is preventing these models other than the bias of humans to like be like you know, the rule of six that you know our brains are capable of storing like you know six or seven plus or minus concepts in our head at the time, which is why functional programming and array programming, you can get further with it. You know, what's to say that like that's not a limit? Like, you know, sure LLMs have context windows and they have their own limitations, but like what's to say that you couldn't take, you know, assembly code or ptex or sass and like train a bunch of really large language models, build the same kind of linting tools, which doesn't even make sense for us, because like we program in programming languages, not in ones and zeros in a system. But I'm just like, I'm not convinced at all. Like I'm very skeptical. So I'm on actually your side. Like, I don't think that the future systems are gonna be generating binary or executables. But that being said, I'm not like I I also thought that abstractions were gonna be super important, and like I've seen now in some tests that I've run that like they these models just if you ask it for speed, it finds a way to basically invoke Cuda C ⁇ at the end of the day, whether it's like FFI through JAX and a shared object file or like embedded like Q to C in a string in a coup, coupie, like raw kernel, or like, anyways, it always finds a way to do like the fastest thing, which is in Q2 C, and like it just discards all of the abstraction. Anyway, so if you take that to an extreme, you know, where do you end up? Do you end up in Qa C ⁇ ? Do you end up in PTEX? Do you end up like in ones and zeros? I I don't know.

Bryce

But I would I would argue the it depends on what you mean by fastest. A higher level representation that's tunable, something like a Qtile kernel, is gonna be faster across the the world of every possible input that you could test. It's gonna be a faster way to generate the correct, to JIT compile the correct algorithm for the particular input and conditions that that you're currently running in. The a lot of the problems, you know, that you there's a lot of talk about reward hacking, and I think you and I have talked about that in the in the past, and how a lot of our effort should be focused on reward hacking. But the other challenge with a lot of this like auto-optimization work is you can only benchmark for so many inputs. One thing I've been working on has been optimizing, you know, Cubs histogram. And I I did a bunch of auto research with you know cubs histogram benchmarks, which runs for maybe 10 minutes. If you want to test a representative set of inputs for histogram across a representative number, number of bin counts, number of element counts, data types, different input structures, you know, different like like you know, not just completely random, but different like structured input, you know, because that's what you'd see in the real world. It it takes hours. It takes hours and hours and hours. You can't have an iterative loop that is that is writing truly generic performance portable code with this like tight keep, commit, keep and revert process. You I think a lot of what we're seeing right now with people doing this, you know, loop for optimization is writing really good code for a very specific circumstance. The the process of taking that and generalizing it to something that can be deployed in production is a different story. It's still an agentic process. It's just it's not you know these type these type iterative loops. You use those to explore and find techniques, and then you have to do some follow-on process to actually develop the algorithms that you're gonna ship in production. And you have to deal with the trade-offs of of accuracy, of b of you know, code complexity, of the size of the binary that you're shipping, etc.

Conor

Yeah, I mean, like I said, I at the end of the day, I'm I'm on your side. Like I do think I like I I would be very well, I would be surprised to a certain extent if like the best AI systems in the future are generating, you know, executables and binary. That being said, I don't know. I I've I've my wife has started to s say that like I've I've gone a little bit existential over the last like couple weeks because the latest models, they're just so good. And yes, and I I've I've started having like post-AGI thoughts. And I I at some point in this conversation, I was trying to think, in the trailer of a a bad sci-fi movie in the last decade, it showed these like what looked like advanced servers and these like they would look like they were self-healing. And I I can't remember if they represent the like computation that runs society or something like that. At first I thought it was Jupiter Ascending, but thanks to AI, I asked. It's like the instant free tier of uh ChatGPT, and sure enough, it nailed it. Well, it actually gave me like seven different options, but I recognized it as the last one. It's Valerian and the City of a Thousand Planets, which I'm pretty sure is based on a book. I recall this movie being terrible, but I'll I'll share my screen and I'll find a way to tweet this out or something. Let me know if you can see this.

Bryce

Yeah.

Conor

And so if we we play this, it's only like a two-second clip, but this scene right here, you know, we we go back, you know, one second. This is what I think. This is like this is what I have started to think like where we're headed. This is like computing in the future. These are all like algorithms that are running, and these little guys that are like soldering stuff, that's like that's like the self-healing. So when you say like, oh, we need to be able to inspect, I don't think so. I think we're gonna have, you know, algorithms.

Bryce

No, no, no. I'm not saying I'm not saying that I'm not saying that humans need to be able to inspect. I'm saying that other agents need to be able to inspect.

Conor

Yeah, but how do you know that the language that they are most like efficient in is the same language that humans like evolved to understand?

Bryce

I don't, but I but but the current the current technology that we use for AI are large language models that are primarily trained on human language text. There's a lot of interesting little fallouts of this. For example, I think almost everybody who's serious about auto research is running auto research in infinite loops. And one of the reasons you do this is that if you tell it to run for 24 hours, its perception of how long it takes to write code is based on how long it takes humans to write code. And so it is just wildly inaccurate. It wildly overestimates how long it's gonna do something. It'll be like, oh, here's a plan for how we can do something that's gonna take two months. And you're like, no, we're gonna do it tonight. And like, yes, maybe there are maybe there are languages or intermediate representations that'll be better suited for these L and Technologies than the ones we currently use. I could certainly buy that. However, they are primarily trained on data that humans that we created, and they're primarily designed to interface with us humans. So I j just in the same way that like probably the most useful general purpose robot will be a humanoid-form robot, simply not because that's the most efficient form, but simply because that's the world that we live in. We live in a world designed for humanoid-shaped things with machines and equipment that is humanoid-shaped things. That's humanoid-shaped. Yeah, I do think that programming languages and programming models will evolve, you know, very drastically over the next few years, but I don't see a fundamental paradigm shift away from the basic tenets of like software evolution and life cycles. Like I just I find it hard to believe that we're going to be shipping elf binaries as the only software artifact. Maybe we'll come up with new binary representations or new types of computers or entirely new models. I could see that happening. But I think a lot of the things that make a computer system good for AI, that make it agent ready, also tend to be the things that make it good for humans. Like, you know, ha having all of your doc having all your APIs like well documented and like marked down, like that's that's good for agents. It's also it's also pretty good for humans too. You know, like like you if you're a human, you can go read a skill file, you know? I I don't see a huge divergence yet in like away from away away from I don't know, software maintainability.

Conor

Yeah. What are we gonna call this episode? And it's so sad too, because I actually think this is like this is like the most interesting conversation I've had in a while. And uh I feel like half of the people listening to us are just being like, these guys are out to lunch. Like what drugs did they do before?

Bryce

It's okay. It's okay. I mean, we're we're we're we are very much at the epicenter of this, and we're to some degree like in an echo chamber. You know, maybe we'll be wrong, and not every not not everybody has to agree with us, and not everybody has to like it. However, I do think if you're a software engineer, you you do have to be aware of what's going on, because it this is a big change to the way that your industry works. And if you're not at least aware, it it's gonna be very tough for you to remain relevant in this field. And so I think you gotta you gotta stay abreast of what's happening. And uh, and this is one of the best places to do that, this podcast. Yeah, and I mean, yeah, I mean, some people think we're a little kooky. I mean, you know, but but it's also like you think about I think about my mindset on AI a year ago versus two years ago versus three years ago, and I was pretty skeptical too. And I see a lot of people who are skeptical, and I I think they just really haven't. In some cases, it it's not that they haven't tried it. It's it's just that it's sometimes it takes a little while to to pick up and to really drink the Kool-Aid, and not everybody wants to. Not everybody wants to. That's the other piece of it. Not everybody finds it as enjoyable as we do.

Conor

Yeah, I feel like drinking the Kool-Aid is the is the wrong aphorism for that, though, because that alludes to, you know, people that ended up dying. Uh so probably people would find that uh they would be like, yeah, it is like drinking the Kool-Aid, but uh, I feel like this is like it's like Excel coming out as an accountant, except that times like a hundred. And um anyways, we gotta stop the. I would love to continue to chat, but I have to edit this. Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day.

Bryce

Low quality, high quality. That is the tagline of our podcast.

Conor

It's not the tagline. Our tagline is chaos with sprinkles of information.