Fable jailbreak, PRD's are dead, Does anyone know what a Skill actually is suppose to be? | Ep 18

In this episode, industry experts discuss the latest in AI advancements, including the Fable Five jailbreak, SpaceX's acquisition of Cursor, and evolving software development practices. They explore how AI is transforming workflows, the importance of verification, and the future of AI-powered tools for both technical and non-technical users.

Links:
Ray: https://www.youtube.com/@RayFernando1337
Adam: https://www.youtube.com/@GosuCoder

key topics
AI advancements and implications
Fable Five jailbreak and security concerns
SpaceX's acquisition of Cursor and product announcements
The shift from traditional PRDs to prototypes in product management
The role of verification and success criteria in AI development
The concept of skills, harnesses, and agents in AI workflows
The impact of rapid software updates and UI changes on user experience
The importance of focused versus generic AI harnesses
The future of AI tools for both technical and non-technical users


Chapters

00:00 Introduction to the Rate Limited Podcast
01:00 Exploring Fable Five and Its Implications
05:02 The Future of Product Requirements Documents (PRDs)
09:55 SpaceX's Acquisition of Cursor and Its Impact
14:44 The Evolution of Skills and Harnesses in AI
26:24 The Ease of Use in Technology
29:21 Understanding the Software Factory Concept
32:32 The Evolution of Startups and Software Development
37:40 The Impact of Rapid Changes in Software Interfaces
40:33 The Challenge of Automation vs. Human Efficiency
44:49 The Excitement of Innovation and Future Possibilities

Transcript

SPEAKER_02 0:00

This is episode 18 of the Rate Limited Podcast. Ladies and gentlemen, we have a crazy show for you. Fable 5, Death of the PRD, SpaceX, Purchase of Cursor, holy smokes. The Software Factory, what is Skills? What is Harness? A bunch of stuff packed in. With your hosts, Ray Fernando, Gosu Coder, aka Adam, and we also have Nathan Snell. A lot of us have over decades of experiences and have touched millions and millions of devices all across the world, up and down different parts of the stack. So what you'll be able to listen to today is all of our varying opinions in terms of us kind of hanging out, talking as engineers, talking as product people, talking as business people, really trying to figure out where the next thing is happening in AI, what's actually happening right now. And we encourage you to continue the conversation in the comments below. So without further ado, Adam, please go ahead and take this away.

SPEAKER_00 0:54

All right. So Nathan, we only have you for a little bit of time. You're getting pulled into something that you can't miss. So let's start with you. So you had a chance, you had a chance to try Fable Five before it was pulled by the government. I did. Give us a recap on that.

SPEAKER_01 1:09

Yeah, so so my uh I'd say my chagrined response since on the on the last episode, I was like, does it even really matter? Codex 5.5 is great. This was after being heavily pulled on the cloud side. And then I used Fable Five, and honestly, like it was really good. It was like it was definitely above, you know, 5.5. So for those that were like, who is this Joker? Why is he even talking on this stuff? You know, Fable 5 was definitely awesome. Um, you know, any anyway, I guess it was great. I mean, I think the one thing that I was a little bit maybe um not expecting is is uh I feel like a lot of like what made it great is it's like it almost like automatically went through the same kind of like scaffolding that I would put together. And it's like, I don't know if it created its own skills on the fly, but it's like it almost, you know, like where I would go and create like my own QA skill and have that sort of get chained together, it's almost like it did all of that automatically. Um so I was I think I was expecting a little bit different. I don't know why, um, in terms of like the model, I guess, doing more versus seeming to like chain things together more like I would. Uh, but I mean, in either case, you know, with very sort of low effort, um, you know, it did a phenomenal job working through pretty complex tasks, which was awesome.

SPEAKER_00 2:16

All right. So let's uh, Ray, I'm gonna hand this over to you next. But so Fable 5, you know, the the government swooped in and said, hey, this is too dangerous. There's been a jailbreak discovered. Did you guys have a chance to look into what this jailbreak actually was? And if not, I have a funny story to tell you guys. So, Ray, did did you did you were able to look into this?

SPEAKER_02 2:38

I I don't there weren't any very specifics, but I'm just curious to hear what you're saying. Yeah, what's going on?

SPEAKER_00 2:43

Alright, so the news that I heard is that the jailbreak is so they basically if you were to point it at code and say, Hey, find all the security vulnerabilities, fine, it would catch it. It like basically it'd be like, no, no, we can't do that. But if you pointed the code at if you pointed it at code and said, Hey, fix my code, it would then fix your code and find the security vulnerability. So the jailbreak literally was fix my code.

SPEAKER_02 3:10

No way, what you can just like fix my code and just like it could be my oh.

SPEAKER_01 3:20

Guess that wasn't part of the golden prompt set that they were rebelling against.

SPEAKER_02 3:24

So it had all these good. That's how good it is. I mean, that's the marketing trick, right? I mean, uh sales type. I mean, uh, that's uh how good the model is.

SPEAKER_00 3:33

So when I first saw that, I was like, okay, it can't be real. The more I peeled back the layers, that seems to be the case, is that the jailbreak was basically you could prompt around it, you know, not really being specific about security vulnerabilities and just have it find issues with the code, and it would uh it'd find it for you and tell you what it was. So very interesting. Do we think that is something that's worth pulling the model for? I don't know. Like I I it feels odd to me, but um I will say uh I think it is odd to pull the to pull the model for that.

SPEAKER_01 4:09

At the same time, um this has got to be like the best marketing that Enthropic could have the best free marketing that Anthropic could ever have. Like, it's like as soon as that hit, I'm like, I don't like if I'm open AI, I'm like, how do I like how do like how do I cause the same sort of thing, right? Because like, at least from a head-to-head standpoint, it's a situation now of like, well, only one frontier model company has had the government shut down their model. Not like not trying to throchet, but it's like, man, that's a pretty solid marketing situation. Um, I will say too, when I was using Fable 5, so I create I'm a Slay the Spire geek, and uh I created a mod for Slay the Spire. And even though Fable 5 is really good, as Adam can tell you, it still did not do a very good job one-shotting the architecture of the mod because it crashed uh myself and my friend's system because of how inefficient the mod ran. So it might be able to find security vulnerabilities, but it still can't architect a mod particularly well.

SPEAKER_00 5:07

Yeah, I wasn't running said mod, but one of our other friends was when we're playing. And and Nathan and this other guy were the whole time we're like, it's so laggy. I'm like, it's fine for me. And then and then they both like drop from the game.

SPEAKER_02 5:21

We are cooked.

SPEAKER_03 5:22

We are sorry, we're cooked.

SPEAKER_00 5:26

Uh all right. I let's jump over to another topic, which because I think this is actually going to be very pertinent to both of you. So I was thinking about this the other day. You know, back uh traditional software used to have product managers sit with customers and sit with the team, and they'd write like a fully detailed PRD. I think we are at a stage now where we should say there should be no more PRDs. And I would say it's probably like down to we need a summary like problem statement, and then you probably literally just have a prototype that sits with it, and that is your new PRD. I am curious what you guys think. Ray, I'll hand it to you first.

SPEAKER_02 6:03

That's exactly it. I I think just uh yeah, a problem, uh, a north star for the company. Uh, you know, what is it you believe in, and then maybe justifying if it's you know, how is it gonna actually help the end customer whoever you're serving, right? So it's like, okay, we know it's very clear, and then whatever you're making and how it goes with that. And then if you have any dependencies as a company too, that like, okay, I depend on this kit, this kit, this framework, we need these teams, you know, then that's kind of like uh understanding uh because being on the other side, especially for the bug review side and as an organization, you're trying to figure out how risky is this change gonna be, you know, and how many other people we have to buy in. So um that understanding that is probably like the most human part. Uh and the AI tooling is gonna kind of surface that. So um, yeah, the prototype is a good like entry point along with this. And then, you know, from the business side, it's like, okay, what are all the regressions? What other things that we don't know about that we don't know about, right? That's kind of like everyone's fear. So yeah.

SPEAKER_00 7:02

So Nathan, you're interviewing someone and they talk about the three PRDs they wrote last week. What do you think about that?

SPEAKER_01 7:08

Oh man, it's uh yeah, that's that's all that's uh I guess separate separate related, but now I no, I mean I I agree. I mean, I think it's funny on all my teams, um, you know, I I always sort of start off gentle where I'm like, okay, I'm like, I don't want to see massive PRDs anymore. I'm like, I want to see like a hypothesis and I want to see like, you know, like how you actually test against that hypothesis, and then I want to see a prototype. Um, you know, I like and on assuming to me, like that's like that's sort of that's the gold standard. Um, you know, and frankly, like, you know, with we we've even seen the speed, seen the speed accelerate as a result of doing that, where PMs are actually now going directly to customers faster because you know, because then it's like, okay, well, I have this prototype, I'm gonna just go and put it in front of you know five customers and see what they say and make adjustments and so on. Um, you know, it's I think two things that I would sort of layer onto that is, you know, I think that's right of like you really just need like the hypothesis, you know, what you're gonna kind of test against and that sort of criteria, the prototype. Um, I do think that you also need the verifier, uh especially if you're building something with like uh targeted at a particular outcome. So something that's like actually agentic. Um that's one thing that I often see missing at times is it's like we jump into the you know, product folks, we jump into the prototype so quickly that like when it comes to actually building, we miss ensuring that we've got like the right verifier as a whole. Um and I and I do think that becomes a really important loop, you know, to make sure that it's in place because otherwise, like it might look great and it might produce something, but we're not actually sure what the target, you know, what the target is, not just in terms of landing impact of like, hey, make something faster, but like did it actually land like a like did it actually land the impact with quality, right? Did it actually deliver the outcome with quality? So I think to me, you know, for for people that are building systems um that are actually like that are producing an outcome, right? Not just like, hey, you need to add a new you know table to this row, but like, hey, you need to, you know, you need to like create a campaign, like a marketing campaign or something. Uh I think that's another piece that has to be a part of that.

SPEAKER_00 9:01

That's a really good point. So I because the one thing that had gone through my mind is like, if especially if you're doing like ML-based stuff or any sort of like AI things, there probably is some level of evals or like as a verification setup, but that's a little bit different than even what you're saying. So I think it is good. We probably do need success criteria or something that maps with it. So you end up with what is the problem statement, what's the prototype, and what does the like success look like? And then how do you measure that in some way, depending on what you're building there?

SPEAKER_01 9:30

I kind of combined um you know evals with that just because uh I've even found for myself as I like build out different prototypes and kind of work through you know like the you know the PRD equivalent mindset with that. Um actually I actually start now even more so with the evals, like with sort of the eval set and the accuracy set. Um, you know, and even in a simple way, like I'll like I'll use AI to just generate it to generate a simulated set of data uh and then kind of that I can actually have evals run against and everything else. And then like all of my work sort of is constantly hammering against that. Uh and I've actually found that produces higher quality prototypes overall. It also even grounds me in terms of like, oh man, like, you know, does this concept even work? It produced really bad against a synthetic data set. You know, maybe like you know, maybe it needs to be a set, you know, like a harness, you know, with you know, with an agent as opposed to a bunch of skills or that sort of thing. So I'd say it's even kind of allowed me to properly iterate before kind of landing on like what the right prototype even looks like to put in front of somebody. That's super cool.

SPEAKER_00 10:24

Ray, any other thoughts on this one before we move on?

SPEAKER_02 10:27

No, I I think I'm excited for the next topic.

SPEAKER_00 10:31

All right. So the next topic, SpaceX bought Cursor. And Ray, you were fortunate enough to actually be out there at Cursor Compose. You want to give us a recap of what happened and sure.

SPEAKER_02 10:44

Yeah, real quickly, uh uh SpaceX uh had the option to buy Cursor, and if the deal didn't work out, you know, they would just pay them $10 billion. Uh the per option purchase was X you know going through the process and will be finalized by the end of the year, um, because it requires, I guess, some government or regulatory approvals. And so uh that would be for $60 billion. And um there is a really cool thing as far as just Michael talking about three big products that they announced. And so uh they had you know the iOS app, uh the origins thing is has come out, so it's a replacement, not a replacement, but it's their own version of GitHub, but it's meant for agentic work uh and kind of their own um environment for having these things um kind of operate and do that things. Uh and then the last big thing was the big composer three model that just launched too. So, or not launched, but it was just announced that they're working on it. Uh it may come out in you know coming weeks or so. Uh the cursor teams are like now vague posting, which means that it's probably a couple weeks away, uh, that you know the model's gonna be like accessible for maybe some people to test. Uh hopefully I can get some access, but uh, you know, it'd be really interesting. And then uh bigger news too that like Lee Rob uh announced on Twitter that he's gonna be working. So Lee Robb used to be the VP at Purcell, uh, has tons of experience uh in software engineering and um you know has gone from education to now moving into the ML space and uh gonna be working on training the model. Um and so that's gonna be really interesting. Uh so imagine getting some good taste already. I've used a lot of his projects to learn about an X.js and stuff. So um, you know, it's it's it's a really cool, cool time. But the vibe there, I get to, you know, got to build your custom keyboard, got to meet a lot of builders and people. Uh Primogen was there along with some other big like, you know, I call them celebs uh in in the in the AI or like just coding industry and stuff. And like about 300 people taking place in San Francisco. Uh everyone, you know, was was fed and kind of hanging out, get to chat with each other, and um and I got to talk to more of the engineering team a little bit deeper on you know what's what's coming next, and hopefully um, you know, it's very, very promising. It's a lot of cool stuff, yeah.

SPEAKER_00 12:41

Very, very cool.

SPEAKER_01 12:43

I'm curious. Sorry, so I'm I'm curious on that. What uh what was like the one thing, not from like the the what was announced, but like what was like the one like aha moment or takeaway that you had there where you were like, oh my gosh, I need to go home and try that right now.

SPEAKER_02 12:56

Um try that right now. I don't think there was one any of those. It was more about the oh, I guess I bumped into people and we were talking a lot about Fable. And then uh people were building out their own versions of it. And then uh I was just getting more insights because I I came up with a a concept called wave engineering, and so um I was talking about the death of loops, and I I feel like that um you know, loops is just telling people that they can introduce poison into a context because once they go in a loop and it gets something that's bad, it can just loop on the poison and you kind of start looping backwards. And uh I I you know my theory is like that things will happen in waves, and I was kind of talking a little bit more about that. So I got a lot more insight for people, and you know, I think the bigger th takeaway was that verification is the moat in a lot of ways, and how you verify in runtimes, or how you verify in you know, code, or how you verify in all these different states actually make a big difference uh for output and um stuff that you produce. So it's really cool to hear you actually talk about verification as well in your agentic harness. And uh would love to talk with you guys. Like a lot of talk came around harnesses, right? Because that's a big deal too in the industry. And so would be curious to see like what do you guys think is a harness and like what is you know, like all that stuff, skills, agents, and all that like stuff that comes together, you know?

SPEAKER_00 14:15

Yeah, I think that's a great question. And I think that the the terminology of this stuff is like shifting like day over day. What what a s I uh I was talking to some folks on uh from OpenClaw, and one of the things that they mentioned was like they have people that are submitting skills that are like just these massive things, like huge, and then you have skills that are like these really tiny, very focused things. So is a skill the little tiny, like very focused thing, or is the skill the very giant like end-to-end workflow? It's it's honestly a debate. Like I have my own personal opinions on this, and and a harness is I think very similar. Like it because there's like different different pendulums you swing here. Like we you work with like generic uh harnesses, like generic orchestrators, or you could have a like very focused orchestrator. You think about like clawed code, for example, to me is it's more on the fence of like generic. I can use it for things to make code, or I can do it for for uh do it use it for something that is like completely unrelated to writing code. But then there are also like very focused harnesses around very particular workflows like uh Martech, for example. Um some of the stuff that you know we work on with marketing in general. So you can build harnesses that are laser focused around specific jobs. Nathan, I know you've got to hop here in a second. If there's any thoughts you have on this, I'd love to get it before you have to jump.

SPEAKER_01 15:38

Oh, um, I have some thoughts, but I'm also sad to jump. Well, I'll I'll have to sync more and we can, I guess, chat this offline since um I think it's such an important area. Um, you know, I mean to me, like from what I've seen sort of firsthand so far, um, like I do think there's a really I think there's a bigger distinction than that certainly than I realized, uh like on the harness side in terms of like a generic harness versus a focused harness. Um, you know, like I've seen better output quality um when working through more focused harnesses. So, you know, so for you know to your example, Adam, of like, you know, if you've got like if you're using clawed code to accomplish something and you have some skills, I've seen that tend to like continue to like underperform against like a harness, a harness that's sort of say marketing focused or finance focused or what have you, um and like you know, with you know, not necessarily the same, like the same skill, but like a similar sort of concept, right? So and I think just that that sort of scoping in a way, um, you know, just seems to drive you know better results. Like, and as a whole, it makes sense. I mean, I think, you know, at the very least from the standpoint of like, hey, like how do you make sure that your token usage is really efficient? Like that alone from like an optimization standpoint, I think is true. Um, but I mean, I uh one of the things that um, you know, I know you and I were, you know, Adam were chatting about earlier this week too, is like, is even the notion of like, okay, like if you think about the agent, you know, when does it make the most sense to actually use like an agent because it even has sort of its own context window versus you know having a small skill or a large skill, kind of knowing that today there's sort of like that cap, you know, at least if you're on the Cloud Code harness, so that there's that cap of like you don't want more than a hundred skills, you know, uh skills and tools as a whole, because otherwise that rot starts to hit large and that sort of thing. So it's like, I don't know, there's an interesting paradigm right now of like, is it really all about skills? Is it really about skills and agents? And if you look in OpenAI, um, like they're well, they have skills, I'd say they're almost leaning more into the agent side, right? And if you look at the they've got their enterprise tier um like agent uh agent workspace, I think is what it's called. And like, and it's killer, like it does a phenomenal job, but it's again it's a very sort of different mentality to it, which is really interesting.

SPEAKER_00 17:38

Yeah, very cool. Well, Nathan, thank you for joining us. I know you've got to hop off here. Ray, what are you what are your thoughts on this entire topic?

SPEAKER_02 17:47

Um I I I'm trying to like zoom out because uh I I can get really zoomed in and and kind of see like exactly what you know he's talking about. And you know, we kind of live that day-to-day. But I also like it while he was talking about that, I was thinking about environments, right? Because there's just different environments people live in. And I was talking to my friend who does legal stuff, and they're so far ahead of most companies who are in the legal space because he's approaching it from the coding side, right? Bringing in different models, each models have different strengths, and they're sort of tying together these different skills files, and you know, it's like similar to what you guys are doing in marketing, but in the legal side, right? But just, you know, okay, here's case law, here's just procedures, we need to follow them this way, and very programmatic. Um, and so I I think all of us are really I I look at intent, I look at it more from an intent, right? So, like as an intent, what is it that I'm trying to achieve or outcome, right, as a as a human, and what is the right thing to pull in at the right time? Because there's nuance. I I was following Matt Polcock and he was saying yesterday that skills should be really, really thin. And you know, that there's a lot of people I've talked to in Korea who say that same thing and who keep their stuff really thin. But one of the things I remember Eric Brofonche talk about is that with Claude-specific models, um, the system prompts and all the things are just repeated over and over again because they kind of need to beat that back into the model to say, follow these instructions. Are you sure? Doubly sure, and it keeps showing up again. And uh you may need to have a certain uh instruction repeat, you know, later on in the context window because of just the way that the models behave. And you don't really know that until you're actually testing in the API. You you have to also understand, you know, context window lengths too. If you're in the if however happen some event has happened, you know, are you measuring the context window? Are you measuring all these different parts of the environment to track that behavior and to see, you know, whether the what you're trying to achieve? And is the skill the right thing? Is the agent the right thing? You know, are you even calling the right model to do the right work? I mean, there's all these crazy factors, you know. Like I it's kind of a little bit overwhelming as an engineer. Um, and it's like almost takes a full-time job to figure out, you know, how to orchestrate some of this stuff. Um, but once you solve these big problems, then you can have like a nice little working system that um, you know, a lot of people can use and multiply. So I don't know if I actually answered anything. I just I feel like kind of more frustrated as from the engineering side that as a single person, um, you have to like do a lot of diligence to go very deep in all these different aspects.

SPEAKER_00 20:19

Um Yeah, I I think you did answer a lot there, and I think I have some of the same concerns and things that you do. I I honestly have my own personal opinion, but you know how the space is. Like I could change my opinion a week from now when I learn something different, like because of some experiment I run. My current opinion right now is that generic orchestrators, generic harnesses um are awesome for technical tinkerers. So they're they're great for us. Have you ever had that have you ever had to chat to an AI and say, hey, use XYZ skill to go do a particular thing? That's okay for me. That's not gonna be okay for my wife. So when I start thinking about like how do we actually bring this to the broad masses? So the the the question then becomes, is a skill a capability or is it an end-to-end workflow? Or do skills layer on top of one another to create that end-to-end workflow? When you think about like someone like an accounting that's doing book closing, you don't want to say, hey, you know, do XYZ, use XYZ skill, then XYZ skill, then XYZ skill. Something needs to bring that together to do all the steps in like book closing. Same thing for marketing. Like if you want to generate an email campaign, like what are the things that actually need to happen to get the email campaign? So my perspective currently is that there are capabilities. Capabilities are the things that should be thin. So that could be anything from like individual, like fetching data, creating something, maybe even a minor little subagent to do some work. But then the the end to end job to be done is the skill, it is the workflow. Now, one could argue, and I'd Actually, could not oppose this, is that that you could also think about that as an agent, where you have different agents that bring together multiple capabilities to do the end-to-end workflow. And some people could call that a skill. They could call the capabilities a skill. I don't know. It's very fuzzy to me. The mental model I have is like a skill should be an end-to-end workflow to get a valuable job done. A capability, those are the building blocks within a skill, the tools, the context, everything needs to happen to actually make that job to be done. I don't know if you if that resonates at all, Ray, but that's sort of where I've I'm currently at.

SPEAKER_02 22:38

I I I I could see that. And I think what makes a skill maybe if I'm thinking about skills and outputs and what's more valuable, I think um I just kind of zoom out too a little bit. It's like models are like non-deterministic on purpose. And how can you take advantage of that? And I think is probably a better way of saying like um like that because engineering, like coding is very deterministic, you know. And when you have a system that is undeterministic, um what kind of properties can you then leverage? And um from an engineering standpoint, you're like you can sort of start to tailor it in by a verification side. So like you start to create edges around that, like where the model's uh going for it. And uh, you know, I think that's kind of maybe where skill files can provide that verification of edges for for for problems you're pointing to that are a little bit more complex that you don't necessarily want to spend all the time procedurizing every certain thing. So maybe that's the cost that you're weighing out too as as a product builder, like uh it's gonna be cheaper if I just send, you know, a really cheap model at this uh and just give it some verification sections and just go, uh, you know, like two lines of English versus like a thousand lines of code. Uh so it's like, okay, that's that's actually kind of cool. You know, 80 times 80 eight times out of ten, it does this, uh, but then I just write a small piece of code to do verification to make sure that gets rejected and doesn't get poisoned for the next loop or something.

SPEAKER_00 24:07

Yep. And what you're saying there is so important, especially in fields legal, you mentioned finance. There are few there are a few fields where like there's very little room for error. If you're dealing with somebody's money or accounting or transactions or some legal case, like you can't really mess up. So verification becomes like incredibly important there. And then and then I think about it from the perspective of you know, how do we how do we get this stuff mainstream such that uh there are pretty much any non-technical person could come in? I don't know, Ray, are you do you still use OpenClaw or Hermes or any of those tools?

SPEAKER_02 24:47

Yeah, that's funny. I still use OpenClaw and I love it. It's just it's cool, simple text. I take both pictures of my food and it's like you know, it's got the it's basically like Cali I app, everything, like just all in one big ass chat. It's crazy.

SPEAKER_00 25:01

So that I think the market for like when we know that we've been successful is when something like that, personal assistance, can be used by people that are not technical. I we I just went through a fresh setup of it recently. Um I've got it have put it on a new computer, and like it's not a seamless setup. You know, there's a bit of intricacy to it, and so that that gets me to the point of like, I think the future, and I could be totally wrong. I think the future right now we're in is like we're building the platform layer, we're building the things that are like made for tinkerers. What we're gonna get to is we're gonna get to these specialized layers where we're gonna have like very easy-to-use personal assistants that are super easy to set up, that are built on top of things like OpenClaw. Then we're gonna get to a point where we have these agents that are just uh automated. So we set it and forget it. So we we we're tinkering right now. We're gonna get these personalized, like very focused agents, which are starting to pop up now. I we just saw Shopify, you know, do an early access for one for ad optimization, which is a super cool idea. And then we get to a point soon where like now we're gonna have I'm just gonna have objectives. I want to put my company on autopilot. These are the things I need you to do. Flag me when I need to be involved. Like that's where we're gonna head. We're just so early right now. It's super exciting to be part of this.

SPEAKER_02 26:24

Yes. Yeah, actually, it kind of reminds me a little bit of um like I totally agree with all those points, actually. You know, it's just making it easier so that I think the litmus test is can your can your parents use it? Can your grandparents use it? Exactly. Definitely the and I think that's why Chat GPT has been so appealing as a like a term and a verb now, because yeah, grandparents actually are using it now to like chat with it. Um I I showed somebody who's like 85 years old how to use the voice, and he's been like trying to find places and he's like, Oh my god, Ray, look at this research it did. It's like, you know, I I forget that like Google now looks archaic because now you it's like why do you want to click each of those links and then get pop-ups and ads and agree to cookies? Like, no, you just get all that. It's wild, yeah. It's like, oh, okay. Um, so yeah, I think you're you're pulling on a really good golden string there. Um, you know, it's just use of ease of use. And like once you go through the onboarding and set it up and just give it to someone, um, you know, we're Chat GPT out of the box, you know, they it still has its paradigm. Yeah.

SPEAKER_00 27:25

Yep. And I think we'll get there. Uh but I think all of these things are just like moving targets. Like a person, we could go to 10 different people and probably get 10 definitions for what a skill is. Should it be a thin, lightweight thing, or should it be a little bit more end-to-end, like very focused on some outcome? I don't know. It's probably all of the above. They probably could be thin and they probably could be end-to-end. So I don't know if there's a right answer to that. And then we talk about what is a harness. I I don't know. There's a lot of people that are taking like this generic harness approach, which you know brings together skills and tools and everything. And then there's a harness could be a simple agent, like very focused agent loop to do a particular job. So like those also have like very broad meanings to me. It it is uh and what they mean a year from now could be totally different. And is skills actually going to be the thing that we build on forever? I don't know. I'm a little bit skeptical, honestly, but I don't know a better option right now.

SPEAKER_02 28:19

Yeah, I think it's it's um yeah, like uh to me, it feels like we just discovered Excel. Like people just went from writing to like putting it in Excel. I love that. The people who like are going crazy right now are the engineers, they're like, Did you know you can make macros and you can put formulas in Excel? Then like you start doing building these, like they basically started building apps inside of Excel, right? So that's kind of the point where we're at, and people are like, Accountants are like, bro, I'm I'm still good with my calculator, bro, with my thing. It's just easy, it's always reliable, it works. Like, I don't get what you're hanging on to with this new thing.

SPEAKER_00 28:52

That's such a great analogy. I love that. It's great. All right. So next topic, I know we've got um this is an interesting one. So I think you brought this up, Ray. So when you worked at you worked at this fruit company, um I have to imagine, if you really think about it, it really was like working in a software factory. There's a very clear, definitive process for what a task went through from ideation all the way to deployment. So what do you think about this new like coining of this term software factory? Uh is it more, is it just more of the same or like fancy marketing? Or I guess what's your thoughts on software factory overall?

SPEAKER_02 29:33

I like the ambition. I think I appreciate that they see agents doing work, but also like not the means to the end, like being like it's gonna replace the human because you know, when as as humans, when you have tools, uh they definitely enhance a lot of workflows, right? Like, and I think I think it's cool to have a lot of tools that can be built or built for you know on the fly and that can help your teams and so forth. And I think uh yeah, I'm seeing a lot of people just kind of anchor to that because that's probably the best way that people can think about you know, you know, you put something in, you establish uh a whole process, and then you can then output the things over and over and over again, you know, just like okay, just like this person does this, it hands off to this, it hands off to this, and this kind of comes all involved. And what that means is actually different for every single company because they all are trying to, you know, do that, and that's what's confusing a little bit to it. Um and I I don't I think the best I think there's a lot of different uh ways that I'm thinking about this, but I guess the bigger thing for me is just how can the AI tooling help you know us as a company in in any different way? So like just right now I'm just an independent person as an entrepreneur, you know. I'm I'm finding that sometimes I get distracted. Like I'm spending more time, like like you said, like we were just talking about. What's a skill, what's a harness? All this stuff is coming in. Should I adopt one? What's the right fit? You know, and then I talk to my legal friends and they're like, hey, like we're trying to do this and we have much more scrutiny. How can we do it in this way? It's like, well, we gotta have an on-prem device, and then other ones that's not off-prem, and you know, privacy is a big deal, and like there's all these different things. It's like, holy smokes, like I I think some of this stuff is great if you just have like a totally, I don't know who has a startup that's just like maybe if you start today and your startup's completely in the cloud and you don't do anything, like it's just like this cute little small environment. But I think it's gonna evolve, and I think it's getting us to ask a lot of questions. But I don't really have an answer, to be honest. That's kind of why I was just want to understand maybe from your perspective, because you're kind of all in the middle, in the in it right now, like actively adopting AI in your guys' company. And um, not that I want you to speak for them, but I'm just trying to think if you were to step out today, like be, you know, go to coder on YouTube, like now that you've been in the beast, seeing that AI transformed and kind of looking at it from that way. Like, what are your what is your thoughts, you know?

SPEAKER_00 32:05

Man, yeah, I had I have so many thoughts. I think right now is literally the best time in the world to do some sort of startup because you could just move incredibly fast. I remember like a company that I founded in 2012, like how long it would take. You know, this was like early AWS days, like prior to that, even getting a server up and running was a lot of work. You know, so AWS kind of made it easy for like it sped up a ton. But now we're at a point now where we can like get to prototype or MVP so much quicker. And it's if you actually have a product that actually can add real value and has scale potential, you can just do that quicker than ever. Now on the on the software factory side, you know, I think it's just more of the same. Like I think we should we have always created uh treated uh software as sort of a factory, especially at bigger companies. You know, startups, not so much. You know, it's a lot of times it's uh a little bit of vibe in, a little bit of like, what do I think is gonna work? Talk to a customer, you go back and you're like working on immediately. The the process is very small, like there's very little factory element of it. You still have inputs, you still have outputs. But as you get to bigger companies, when you're trying to coordinate between massive amounts of teams, and you need to try to make sure you're doing what's adding value to your customer, you know, it it really is a process. There still is like what are the inputs? How do you get the right input? And then all the way through through like writing the code, testing it, validating it, rolling it out to customers, whatever it all needs to happen. I don't think that's changed at all. And I think we've always thought about software and kind of that factory mindset. What's changed is the pieces we can automate. Like we can automate more and more of that than we've ever been able to do before. And it also introduced a lot of risk, too, because you can see, and you see this honestly on status pages of so many companies. We've got more code going out than ever. And I've always had this mentality that less code is actually better. So, like you actually want you want small, focused commits, and you it so the second commits are or merges start getting larger, you introduce a lot of risk into your into your system. Rollbacks harder, it's harder to catch things, and we've kind of started to move away from uh away from that as an industry. You know, like the teams that I work with, like we're still very focused on like small, like focused commits. We can get to them quicker, we can review quicker, but really just especially when you're in a co in a company where you're managing, you know, if you're working at the fruit company, you know, you're you're dealing with a billion people. If you mess something up, you're impacting a huge amount of the world population. Like you have to you have to take that like super important. So not losing track of like what is important in that factory mindset. It's not about like how quick you can get as much code written and out as possible. It's about how quick you can get the outcome to your customer and how do you do that safely. So it's we just need to make sure we're measuring on the outcome side and not on the lines of code side or the number of PRs or or things like that. That's where my mind's at. I do love the idea of automating everything though. Like I think automate verification, automate as much of the the crap work that you can, like with AI. Let us focus on the cool stuff.

SPEAKER_02 35:29

I'd love that's such a great way to say that. Let's focus on the cool stuff. But that that's very very much true, because it's it really a lot of it's just about risk management, which is why these people are in place, is because they've seen all the scars and they've uh carried a lot of internal knowledge that they can share back with what's happening and saying, well, that that that risk just looks just feels risky. Like what what is it? Have we tested it? How many people have seen it? You know? Um there've been stuff where maybe you don't really know how how much of a delay you need to delay a project, right? Everyone's saying that's delayed, but then you bring someone in who's familiar with the entire stack and they're just like, Well, that's not a delay, just we just need three people to show up, like actually 40 hours, and then we're gonna get this done, you know. Like it's like it's kind of just you know, practical things and and it's it's not automated, you know. The uh this is really what's interesting. The AI agents are really programmed to tell you the right correct timelines, you know. They're gonna just yeah.

SPEAKER_00 36:34

It's like if you think about the velocity of change, um customers typically don't like it when you move things around on a page. If they're used to a button being somewhere or a layout being a certain way. But have you noticed, at least I've noticed, uh in fact, Riverside. Riverside has changed a lot, like constantly. So like the app we're using right now, and it's like there's certain things that were there last week that are not there now that I was expecting to see. And it's interesting to me because like one of the things that I continually see and hear feedback about, you know, if you go surf red at like any of the major like Salesforce or any of the major platforms, you'll hear people uh make complaints like, why do we keep updating stuff so much? Like, why are things moving around so much? I partly wonder if it's because it's a lot quicker to do it. Like, we were always going to do these things, we were always going to make these changes, but we can do it so much faster now that the consumer is now getting inundated with their software changing faster than ever. And it's probably frustrating. In fact, I know some, it's frustrating to at least some subset of customers that are vocal about it. Uh, I just find that interesting. I don't know. Have you thought about that? Like from the velocity of change side.

SPEAKER_02 37:41

Uh my friends, like their phones, like they changed the layout for the how your favorites were saved. It was no longer list view and then mail. Like, I can't tell you how many people walked up to me and say, Hey Ray, how do I change my mail to go back to the new, the older view? I was like, or the phone thing. It's like I get it. Like, I get it. And I'm still kind of mad about the photos UI, you know, the photo UI. I'm still but like it's like, okay, maybe I'm just a boomer now. Like, I think that's kind of no, but I I I think uh yeah, I I I I I agree about these I I was watching this good video, maybe this is a really interesting insight. Because I where this UX designer was describing about like ask an agent to order a coffee at Starbucks, and then you go and launch the Starbucks app, I bet you you'll be faster than most agents. And I was like, damn, that's so true. You know, you go in, boom, like four seconds, right? Like you launch the app, you know where the tap, the da-da, you're in. Like, and you watch an agent struggle through it, you're like, oh my god, uh, you know, if there's a CLI and there's a payment thing, and like you so it makes me in like kind of really question certain things. It's like if you go to do uh book an airline, for example, is another one, right? Like you can tap through, you kind of don't really know what dates, then eventually stuff gets revealed. And then the guy actually pointed out how Brian Chesky from Airbnb tried to have just agents book it and they had to take that feature out. They never even shipped it because it was just taking too long and very confusing, you know, to go back and forth in a chaff to book with an agent. It's like interfaces are are a really interesting thing because they UI does solve some problems really fast, maybe sometimes faster than you know. I know I know a lot of people like to use Vim and they're like so fast with it and stuff. But when you're trying to just give that to customers and they can see very clearly a calendar and they can click it and you know, three clicks or something and they're done. That that actually is actually a good way to solve a problem. You know, how do you design for that? That's you know, that takes years of experience, but then um it makes me think about you know software design and and keeping consistency because you know, like back to your original point, what's your intended outcome? Right, if we're just trying to record a podcast, that record buttons now move to a different place. What the hell? Like that that's the whole point of me using the platform.

SPEAKER_00 40:02

It's like you know, such a good point. So the I love what you were saying about certain things taking longer. So um back in around 2019 or so, this is pre-gen AI and everything, we were building a solution to do automated spreading. So, what that means is you imagine a bunch of tax forms and income statements, you need to extract the numbers from it and put it into basically a software so that uh a banker can go through and be like, yes, we can give this person a loan or not, and they can validate it. And there are there are armies of people that sit in the back office, like just basically taking numbers from PDFs and putting them into the system. So that you know, there are companies that would employ hundreds of people that just sit there and you know they're just typing numbers and they're very fast at it, like incredibly fast at it. So, one of the things we used to do with our platform is we would do one-on-one races with like some of the best spreaders to see if we could beat them. And it was like it was so hard. Like you would think that with software, you should be able to beat them, but it actually took us a very long time to get to the point where we could actually like beat them because they they were just so good at it, you know, they've got their numbers basically on their numpad. And then like, you know, the AI's gotta pull it in, you're doing like a bunch of you know, you're extracting data from it using OCR back then, and then you've got to try to map it to where it needs to go. And then now there's the accuracy side, did you map it correctly? You know, tax forms sometimes change from year to year, or you are looking at the right like kind of layout. It was just a really interesting paradigm because banks at the time, you know, they they were they wanted speed, they wanted to be able to take and like move faster. And if we couldn't prove that we couldn't do faster, it was it was gonna be a hard sell. So we had to go like, okay, if you use our system with a human kind of like reviewing and validating it, does it beat someone just straight up doing it? And it was a it was super fun, but I I think there are probably some cases where we're gonna have challenge with Gen AI actually beating what a human could do.

SPEAKER_02 42:11

That's that's a really good point. I think that's I I I like that thinking in terms of people trying to actually think about the problems they're trying to solve. And I think I think to me also on Twitter it can get a little bit noisy because um people get rewarded for grandstanding and they just say we're cooked or you know, whatever. And you know, I've done those posts just for fun and they it's like they do go viral because you're just I fired my designer as a crazy headline, you know. It's like you just think of yourself as a tabloid writer. And then I think about it, it's like I that wasn't really my intent. I just wanted to you know test out this thing. It's like, oh wow, okay. And so you're like, okay, if the the algorithm in and it's not even the algorithm, I I would just replace it as people. People like drama, people like to follow headlines and share, they get triggered to the comments and you know, there's the whole thing. It's like I don't I don't need that. Like I think that's kind of why I really appreciate this podcast and show and just us getting together, because I just like to talk to it, like there's gotta be another human being like me who kind of wants to like not be around that like clown show. Uh and talk about it pragmatically. And I think that's why we appreciate everyone who's you know leaving comments and trying to be like helpful as well and trying to be very thoughtful about where you're coming from because we figured that you probably have if you even made it this far in the episode, you have way more maturity than a lot of people on it are are probably seeking that deeper knowledge. And we're we're we're on a quest to understand this technology, I feel, and um you know, trying to integrate it in our workflows. And you know, we're burning tokens, you're burning tokens, we're feeling the pain just like you are.

SPEAKER_00 43:52

Yeah, and every day I feel like there's new challenges and problems that I didn't think of before, and the scale keeps getting bigger and bigger. I actually I love it. It's honestly twenty-three years or twenty-four years, however long it's been now that I've been in the industry. I'm the most excited I probably have ever been, like for what we can do, what we can build. It's like starting from ground zero again, because no one really knows how to build with this stuff properly. Yes, we have some good examples out there, but I would even say that those are like very baseline, like from what we can actually do going forward.

SPEAKER_02 44:25

Yeah, my favorite stuff is the tinkering stuff because it's like, okay, now I have to learn this crazy obscure programming language to write firmware to this really obscure device. It's like, okay, that's just a clawed, you know, request and it comes back and the hard part's out of the way, and now the fun part begins. You know, it's like stuff like that. It's like, okay, cool. You know, um, if I do want to go deeper and like figure out the internals, I I'd probably spend more time on it, you know. But the i it's it's been pretty fun. It's also very distracting, too, because like I would rather be doing that than some other actually probably critical work that I should be working on.

SPEAKER_00 44:58

It is very hard. They uh honestly, I was thinking like if I was in college now, I don't even know what I would do. Like, I so when I was in college, I just would sit and build, I would go to class, and whatever new thing I learned, I would come home and I would just start coding. So this was like early internet days, so there was like no Stack Overflow and like stuff like that back then. So literally I'd have books up, I was just writing code constantly, and that and I loved it. Like I was so addicted to just building and seeing stuff visually on the screen. Like I could not imagine what my brain would have been like if I was in college now. Like, I feel like I would just be I'd be all over the place, like trying to build all kinds of things. I'd probably be broke, I'd probably have to try to like get extra jobs just to pay for my tokens.

SPEAKER_02 45:40

Yo, I gotta get those tokens, man. I got 50 million of them running right now, but they're gonna build the next AGI.

SPEAKER_00 45:47

I'd be one of those people that's trying to like min-max like every free thing out there possible. Like I'd just be insane.

SPEAKER_02 45:54

Like you you would probably come up with this like token router where that would like route your own tokens to all the free stuff and have like 20 accounts open, like just oh yeah.

SPEAKER_00 46:03

I would have absolutely done that. Like as a broke college kid, man.

SPEAKER_02 46:06

I would have local models running on like this like 32 gigs of RAM, the stream cutting out, like yeah.

SPEAKER_00 46:13

I I would have like my parents' computer and like everything like running model for me, they not even know that it's happening.

SPEAKER_02 46:20

What if one of your kids is doing that? You don't even know right now, just the background. It's like my dad only gives me $100 or something, but I would be proud of them.

SPEAKER_00 46:28

I'd be very proud.

SPEAKER_02 46:30

Dude, he's like, why you come into the room, you're like, why are the lights off? Why why do you have these neon lights back there? What are you doing? Like on camera and stuff, all scrappy.

SPEAKER_00 46:39

Oh my gosh, that'd be so much fun. Uh well, I think that's a good place to wrap it. It's been fun hanging out with you, man. Is there any closing thoughts that you have before we we end it?

SPEAKER_02 46:48

No, I want to give a big shout-out again to Eric Provence, who um who's graduated and and is having the time of his life, I'm pretty sure, uh, at OpenAI, um, you know, developing and doing really cool stuff. And it was just uh amazing to have him on the show and um, you know, thinking about you know what we're hanging on here with uh Rate Limited Podcasts and uh also thanking all the viewers and everything just for taking their time to hang out with us, you know, every couple of weeks. It's not like every week, it's not whatever. And I I just enjoy catching up. It's one of my favorite things to do every couple of weeks, and uh also enjoy everyone else kind of participating as well. Uh and and keeping it friendly and um light and it really kind of I I I feel like we're on this really great um way of you know, our lives are changing and trying to like not be so fearful about it and just try to really understand and look at it for what it is. I definitely appreciate uh this a lot. So yeah, thanks man.

SPEAKER_00 47:43

Yeah, it's coming whether we like it or not, so we might as well just enjoy it. Like it we're gonna look back on this time and think it was like some of the best times in our life, to be honest with you. That's my thought.

SPEAKER_02 47:52

Yeah, what a what a time to be alive. So, yeah, under the wraps. Congratulations.

SPEAKER_00 47:57

Yep, episode 18. Ray, have a great rest of your day. Take care, everyone. See you guys.