Code with Jason
Code with Jason
274 - Matthew Ford, CEO/CTO at Bit Zesty
In this episode I talk with Matthew Ford about AI-assisted coding at BitZesty. We discuss how AI speeds up development while requiring human oversight, the risks of "vibe coding," why automated testing remains critical, and how AI changes but doesn't replace fundamental software development practices like version control and architecture decisions.
Links:
Hey, it's Jason, host of the Code with Jason podcast. You're a developer. You like to listen to podcasts. You're listening to one right now. Maybe you like to read blogs and subscribe to email newsletters and stuff like that. Keep in touch. Email newsletters are a really nice way to keep on top of what's going on in the programming world. Except they're actually not. I don't know about you, but the last thing that I want to do after a long day of staring at the screen is sit there and stare at the screen some more. That's why I started a different kind of newsletter. It's a snail mail programming newsletter. That's right, I send an actual envelope in the mail containing a paper newsletter that you can hold in your hands. You can read it on your living room couch, at your kitchen table, in your bed, or in someone else's bed. And when they say, What are you doing in my bed? You can say, I'm reading Jason's newsletter. What does it look like? You might wonder what you might find in this snail mail programming newsletter. You can read about all kinds of programming topics like object-oriented programming, testing, DevOps, AI. Most of it's pretty technology agnostic. You can also read about other non-programming topics like philosophy, evolutionary theory, business, marketing, economics, psychology, music, cooking, history, geology, language, culture, robotics, and farming. The name of the newsletter is Nonsense Monthly. Here's what some of my readers are saying about it. Helmut Kobler from Los Angeles says thanks much for sending the newsletter. I got it about a week ago and read it on my sofa. It was a totally different experience than reading it on my computer or iPad. It felt more relaxed, more meaningful, something special and out of the ordinary. I'm sure that's what you were going for, so just wanted to let you know that you succeeded. Looking forward to more. Thank you for this. Can't wait for the next one. Dear listener, if you would like to get letters in the mail from yours truly every month, you can go sign up at nonsense monthly dot com. That's nonsensemonthly dot com. I'll say it one more time nonsense monthly dot com. And now without further ado, here is today's episode.
SPEAKER_02:Hi, how's it going?
SPEAKER_01:So you are the CEO slash CTO of BitZesty. Uh tell us a bit about yourself and BitZesty.
SPEAKER_00:Sure.
SPEAKER_02:Um so BitZesty started in 2009 when I had left a previous agency wanting to start my own agency. And I think um one of the initial ideas was to build our own products as well as run an agency. I quickly realized that that's basically two businesses, and uh starting two at the same time is quite challenging. So um we focused more on the agency in the early days, and now now we're starting to revisit that, and we have a bit more bandwidth uh and capacity uh to see if we can also launch our own products as well as helping clients uh build, maintain, and and support uh you know their their applications.
SPEAKER_01:Yeah, it's a notoriously difficult thing. It seems like every agency has aspirations of building their own product, but very few are successful with it.
SPEAKER_02:Yeah, because I think when you once you've done it once or twice, you realize that the building is not actually the hardest part of launching uh a business. Um building is an important part, obviously, and you have to build something that people need. But um there's everything else that comes after it's the marketing, it's the sales, it's the it's the sort of um making sure that there is enough need in the in the market for what you're building.
SPEAKER_01:Yeah, exactly. You know, coming up with a viable idea is one of the hardest parts. Um but what we're maybe going to talk about today is is on the programming side, um, you know, everybody's talking about AI still. Um as as I think we will be for the foreseeable future. I've been using AI all day, every day, uh since Chat GPT 4 came out. Um I don't see that going away ever. Um sounds like you at BitZesty have have been using AI in your coding.
SPEAKER_02:Same, yeah. We well, I guess I was first well first introduced in AI back at university, but back then, you know, we were talking about predicting one or two words. We weren't talking about predicting entire code bases. Um and uh I think I think it was yeah, one of the early it was actually maybe image generation. One of the early image generation, when mid-journey was released, that got me more interested in in Gen AI and then obviously using AI for for other tasks as as well. Um that sort of that initial sort of, you know, those I don't know if you remember back then what those initial mid-journey images looked like. They were very, very rough. Um, you know, you know, not not nothing, nothing to shout about, really. But then I quickly saw over the period of months, you know, and we're not talking years, in a few months, images started to get better and better. Uh, and yeah, sure they still struggle with maybe the odd hand with five fingers, you know, not five without five fingers, but um it was getting better over time. And I and I could see that that was also going to happen in in the software development space. You know, obviously to start off with, you know, the code was very, very bad. And now we're seeing software development pretty much being flipped on its head with AI um coming out and enabling you to create software at a pace that you know it's this is a fundamental change in in the pace of software delivery and fundamentally the cost as well, because if you can generate something in a week, which might have taken you months, that's that's gonna save you a lot of time and money. So um, yeah, I've been keeping tabs on it and using it either for my own personal projects or internally. And and now we've put I've put in a policy in place for how we're gonna use AI with the team and you know, on clients' uh projects as well. So it's one of those places where I think we're we're now in the stage of AI is able to generate a hand with maybe five or six fingers, but you need to know that it's wrong and you need to be able to detect that and you need to be able to fix it.
SPEAKER_01:So definitely um yeah, and it's interesting the the ways in which AI speeds us up. Um, because it's not like it just takes the same exact software development process and compresses it so that the same work takes half the time, just it's happening faster. Um there are some things that AI doesn't help with at all and never will. And there are some things that AI like almost completely cuts out of the picture. Like, for example, um here's here's something I was messing around with just like 20 minutes ago. Um long story why, but I was interested in making a program that would take a file that represented some three-dimensional object, um and then write another program that would or use an existing program or whatever um to render that three-dimensional object. Um and I asked Claude about that. Um, you know, I was kind of agnostic as to how it's like, okay, first of all, is there like some kind of standard way of representing 3D objects in files already? I imagine there is. And then is there also some standard way of opening them and some kind of program that renders those files? And it gave me a file, a dot obj file. I'd never heard of that, but it gave me one of those, and then it gave me maybe a hundred lines of Python uh that would take that obj file and represent it on the screen. Um and it worked like first try. Um it had some buttons to like rotate the object, and the buttons didn't work at all. Um but it at least it gave me a cube and it showed me the cube on the screen, and so it was like a proof of concept. Um so there was like a whole research phase, which in the past might have taken me a few hours, um, and and that research phase was compressed down to like five minutes, and then I had a working program after that. Um so it's not gonna it's it's not it and to contrast, it's not gonna help me like gather requirements for a web application and then like make the decisions of what to build and stuff like that. But if there's something I don't know how to do, it can really compress that research time.
SPEAKER_02:Yeah, absolutely. I mean, I remember recently I was a client asked us like, what is the difference between all of these AWS services? And you know, I know AWS pretty well, but uh I don't they are constantly launching new services. And you're like, hey, what is the difference between these two services? They look very similar. Um and you know, tools like research, you know, using AI for research with you know OpenAI's deep research or other other research tools is incredible because you know what would typically have taken me an hour, two hours, I can do in minutes. I can tell, you know, as they ask the question, I can almost send them the response, which you know is uh you know, you know, that's that is very powerful. Um, you know, I think if you think you know there's this uh danger that AI will pick up some random reference which is extremely outdated, uh and you're you know you're using uh the wrong information. Like I think I saw some like a lot of frequently people complain that you know tailwinds four doesn't really work very well in in AI at the moment because it's trained on Tailwinds 3. And if you you know keep trying to get to the right syntax for four, it won't do it. So there's I think there'll be edge cases like that where maybe if you need something that's like really current, it may not be great. But if it's an established concept or something that's already out there, I think it'll be okay, yeah, for that kind of research.
SPEAKER_01:Yeah, and I think there's an important point to be made um about hallucinations and incorrectness and stuff like that, because that's something a lot of people have a problem with. And it seems like some people are even going so far as to not use AI at all because they think it's gonna be inaccurate, which is true, but that's totally throwing out the baby with the bathwater. Um, because it's like, okay, AI is not always gonna give you the right answers, but nothing's always gonna give you the right answers. Um, like Matthew, you could ask me what are the differences between these various AWS services, and I could get it wrong. Um, and so you have to have a mechanism in your mind for for telling what's right and and what's not correct. Um and and you can use AI to get at least directionally correct answers. Um like with this program that it wrote for me, rendered a 3D object, the rotation didn't work, blah, blah, blah. But it's like, okay, now I at least have the right direction to go in, and I can research OBJ files, I can look at this Python program it wrote me and figure out how it works, and then extend it myself and fix the rotation and stuff like that. Doesn't have to be correct.
SPEAKER_02:Using AI as a learning tool is probably the most powerful aspects of Gen AI and software development tools at the moment, I think. Because you can so I like say for example, I don't know very much Go. I know how to, you know, I know how to program, I know software development. I can use AI, converse with AI, get it to explain documentation that, you know, I don't have to spend hours trawling through various documentation. It can explain certain syntax. I think using AI to sort of explain complex topics, explain things you're not sure about, can definitely set you on the right path to learning more about various tool systems and the application that you're working with. I think that's one area that I'm really interested in because the work we do is maintaining other people's code. We don't know what that was going through the mind of the original developer when they wrote the code the certain way they did. But you know, with large context windows with tools that can search your code base, you can easily see, okay, well, this is used in these five different places in these different ways. And then you can then make a judgment of okay, can we actually safely refactor this as opposed to you might refactor something and then suddenly break something somewhere else because it's you know, usually apps don't have great code coverage. Um, you know, so there's there's things like that where obviously if you're if if you're just sort of saying I won't use AI because it could be wrong, I think you are like I said, you are you are missing out on the benefits. You could use AI safely if you had certain guardrails internally or processes in place, you know, like reviewing all the code. Don't just blindly like there's a there's a trend towards vibe coding at the moment, which is basically accept basically pressing tab and and enter and just constantly just you know accepting whatever the AI generates. And I tried it. I said, okay, well let's let's let's have a go. I vibe coded a little prototype for um for one of the clients. You know, it's like I'll spend a few hours on it. And it was all going really well. Uh it generated an interface that looked great, it you know, it was working quite well. And then at some point, it just got itself into a death loop. Uh the AI was just not, it got itself stuck, it couldn't unstuck itself. Um even with prompting and saying, Oh, have you tried this? No, it couldn't get out. And then another time I was building a little game uh in AI, another prototype, and it had introduced rather than using an off-the-shelf like WebSocket connection library, it thought, let's just write our own. And I was like, okay, great, you're gonna not use something that's open source, well tested, and you're gonna write your own WebSocket connection handling. Go for it. Um and it it just introduced a huge performance issue where actually the I was using lovable at the time. And lovable is a little bit different from things like cursor and other tools where you can actually modify it's more sort of local development. This is kind of closed within the cloud. It ended up doing a denial of service attack on lovable. And I couldn't load the editor to say, roll back, stop doing this. I don't know, it just kept opening WebSocket connections, maxing out my CPU, and I couldn't I couldn't do anything. Um I think I just had to junk it in the end and start again. Um I think that's I think for prototyping, you know, it's a very low risk. You know, you can you can easily just throw it away and start again. I think that's an area where yeah, you could safely sort of experiment with these tools in a in a prototype or something that's not gonna reach production. But yeah, I'd be very hesitant of pushing anything to production without proper review, proper testing.
SPEAKER_01:Yeah, there are certain like immutable laws of the universe, and no matter what technologies come along, those laws of the universe aren't gonna change. Um like, for example, you have to decide what you want and then implement it. Like you can't just come up with some super vague idea and then leave all the detail decisions up to I'll say someone else, whether that's someone else is another person or an AI or something like that. Um because it it's it's gonna be very unlikely that the details will match what you want. Um and you know, if we bring like test-driven development into the picture, um I code using test-driven development because it's it's a process of coming up with the specifications for what you want, um, encoding those specifications in automated tests, and then building the behavior to fulfill those specifications. Um and without having those um automated tests covering the behavior, you can't refactor your code without huge risk. And no matter how good AI technology is, um, that's not going to change the fact that you can't refactor uh without the risk of introducing regressions.
SPEAKER_02:Absolutely. I mean, it's not even a risk, it's a it's a reality. If you've if you use some of these AI agents, especially in in modes where they can just in a J in like in its agent mode where it just modifies the files, saves the file, runs the com you know, runs commands on your on your uh terminal for you. In those modes, you know, I've seen AI just delete perfectly good code. It just deletes huge swathes of code because it doesn't fit into its context window anymore. And it's like, yeah, I don't think this is needed. I'm like, well, that code was needed. Or or it'd be like, oh, I think I can improve this. I can improve it just has a I I don't know if it's something to do with Claude Sonnet 7 or if it's to do with um uh or if it's to do with uh just the way the sort of the prompts, the the system prompts that some of these tools are using. But sometimes it'd be like it'd be a function or a method and that works fine, and it'd be like, I think I can refactor it and it refactors it in a way that it doesn't work in the way. And then you could even ask it to say, oh, make sure this is covered by tests, and then you could write return true in the test, and the test passes, and you're like, oh, that's not that's not what I meant. Yeah, so this is a level of understanding that it doesn't have yet that requires a human to still be in that process.
SPEAKER_01:This this brings up another um important point, which I talk about sometimes, which is like there are the current incidental weaknesses of AI, um, and we can expect those things to potentially get better. Um and then there are principles that are true no matter how good or bad the AI technology is. Um and a lot of these things I can imagine and even expect to go away. Like problems where the AI refactors a perfectly good method and makes it stop working. I imagine that problem will go away as the technology gets better. Um but again, no matter how good the technology is, that doesn't mean the benefits of automated tests go away, for example. Um and the way I use AI is um I often use it to help me write tests because the hard part of testing, for me at least, um the hardest part, is coming up with requirements at the appropriate level of specificity and deciding what kind of tests it's gonna be, what level am I gonna test at, and stuff like that. And then actually getting the test to function is often just a lot of grunt work and boilerplate and stuff like that. So I use AI to help me with a lot of that grunt work and boilerplate, because I'm not gonna outsource the decisions to AI, but I am totally fine with outsourcing the grunt work to AI. And then once I have a failing test, then I'll have AI help me write the code to make that test pass, and then I'll repeat. Um, what could be really interesting is if even more of that grunt work is automated. Um, so if I can just tell the AI what kind of test to write, and I don't even have to spoon feed it as much as I do now. Um and then once the AI sees the failing test, you know, it can run the test itself, observe the failure, write the code to make it pass on its own, and I think at every step you'd want to like be prompted to accept it or not, or whatever. Um, but I think that could be a really compelling use case for for speeding up AI even more, rather than this like vibe coding idea where you just read it, let it run crazy.
SPEAKER_02:Yeah, spend$50 and get a half working app at the end of it. Yeah. Um yeah, I think I think for me, what it shows is also that some of the like ad like sort of in agile software development, typically the a lot of the thinking is delayed until when it's required, like say, you know, we're gonna plan out the next sprint, and then we're gonna break down the stories, and then we're gonna figure out exactly how these are gonna work now. I think given that AI will speed up that software development process, I think more of the thinking has to come. I don't I'm not saying we have to go back to waterfall, but I think more of the thinking has to come a little bit forward and you have to think about architecture a little bit more at the start of the process. That you typically would maybe say, well, we can delay this conversation until later on in the process, because we may not even need to build this thing. But if we're building things really quickly, then maybe we do need to think about this thing sooner rather than later. And you know, you don't want to sort of let your AI make those architecture decisions for you, which is what could happen if you let say maybe a developer who's got less experience, but then is using an AI tool to generate code, they may not be able to even know that it's picked the wrong kind of architecture or it's made a technical decision which is going to be very hard to unwind later down the road. Because they you know they they won't have that experience. They're just relying on the AI generation tool to do that. And I think I saw a study that there is a risk that overusing AI can reduce critical thinking over time. So even myself, like you know, senior software developers over time might stop thinking and just rely on the AI to be right as it gets better, and then that can then introduce really subtle issues that you know, subtle security issues, you could introduce performance issues, it could introduce, you know, all sorts of things which you won't really detect until later down in the process. And it may be very hard to then refactor or correct.
SPEAKER_01:Yeah, I think you really have to calibrate your laziness level. I've been doing that. Um, you know, a lot of the developers who I observe are not being lazy enough, they're still doing too much grunt work themselves, and I'm like, bro, we have like Claude and Chat GPT and stuff, like take advantage of these things. Um, but then you can get too lazy and just like uncritically accept the code that the AI gives you. And I've been guilty of that. And I've I've been introducing maybe like one bug per week on average by by accidentally not not scrutinizing the code as closely as I know that I should. Um and I think that's like an acceptable bug rate because I've been catching all these bugs.
SPEAKER_02:So that's also how much how much value are you creating in that time if you didn't use AI?
SPEAKER_01:Right, exactly. I think the benefit is worth the cost.
SPEAKER_02:Bugs are a natural part of software development. I think a lot of the times, you know, sometimes people come to us and say, Oh, you know, are you gonna be fix all the bugs for free? I'm like, um no, because a bug is either a misunderstanding in the requirements or a misunderstanding in the implementation. You know, it's not they're not really the they're at the end of the day, they're they're they're either human or in this case an AI error, you know, somewhere along the line. And it's if you if you consider that bugs are just a part of software development and you will you will have to fix and detect bugs as you go, well, yes, you're releasing maybe one bug a week, but are you delivering five times faster, ten times faster, a hundred times faster, you know, at an acceptable cost of the odd bug that you have to fix, as opposed to, you know, this would take this project would take you know orders of magnitude more without AI.
SPEAKER_01:Yeah, and I think I misunderstood the nature of bugs for a long time. Um like I used to think of them as like cases where the program is behaving differently from what I intended. Um, but once I really like thought about it hard, I realized that like most bugs are not cases where the program's behaving differently than I intended, but rather like some use case that nobody ever thought about. Or like you design the domain model or the system or whatever in a way that's like a little bit out of sync with reality. And so in those gaps between your domain model and reality, um, that's where the the bugs creep in. And it's not exactly like yeah, it's it's not I don't hmm, I don't know how to put it, but it's it's like kind of a gap more than a misbehavior most of the time.
SPEAKER_02:Exactly. Most of the time. I mean, some obviously sometimes it could be you know typos or you know, simple things like that, which you know, hopefully even now with AI, those will start to go away. Um but yeah, for me, I think it'll be interesting to see how how the community, software development community, adopts these tools and how you know how these tools evolve over time. Because fundamentally, you know, you could see yourself in a place where you're doing you know, the amount of work you're doing is you know, you're not writing code, you're not writing boilerplate code. That's boring anyway. Who wants to do that? Like we want to be doing more of the thinking, you want to be doing more of the architecture, you want to be doing more of the sort of deconstructing, like I said, problems down into domain on domain objects that you can then manipulate and and you know do certain things on. So I think from my point of view, um it's like we are on the precipice of a complete change of how software is going to be written. Most people, like I said, may not maybe some people say it's gonna be in a year, some people say it's gonna be two years, but whatever timeline it is, if it's three, four, five years time, I think at some point no one will be writing software the same way again.
SPEAKER_01:Yeah, I think we're kind of already there. You know, there's that quote of the future's already here, it's just not evenly distributed. Absolutely. I think a lot of people are like coding in the future right now, and you know, I'm sure there are still people out there who aren't even using version control, you know? And so there will be people who like never adopt AI.
SPEAKER_02:Yeah. Yeah. But I think that what you've just said there is a is an interesting thing because using these AI tools, you could be a non-technical person writing code. You know, I I've seen tweets, I've seen, you know, there's a viral tweet that went around where someone used cursor and then deleted four months worth of their work because they weren't using version control. And you think if you think about it, why would you need to use version control? I'm using an AI tool, surely it should just be saving snapdots as it goes and commit making its own commits, right? But unless you're, you know, the tools aren't tools aren't there yet. The tools aren't there yet, but to completely hand off all of software development to an AI tool. And the the sales pitch for that some of these tools come out saying, oh, you know, you can replace your entire development team with uh with our tool. I think that's very disingenuous. I think that is that is opening up people to huge security issues where things tweet where someone launched their app and it had, I don't know, some API credential on the front end and all of a sudden they've racked up like you know hundreds of thousands of dollars worth of bill, you know, because someone's just hammering their endpoints and stuff like that. So I think things like that where it's like, okay, yes, you know, these tools are extremely powerful and they're really useful. But I don't think we're at the point where you can just completely replace a software you know development team with one tool yet. Maybe in the future, but that future is is going to be a long way away, especially in some of the industries that we work in, which are like healthcare, government technology, sort of financial sort of sectors, which you know they have various sort of processes and guidelines in place to ensure that. You know, people aren't good software development practices, you know, writing automated tests, doing code review and things like that.
SPEAKER_01:Um that thing is so interesting about how uh cursor or whatever deleted somebody's like four months worth of work because they weren't using version control. To me, that goes back again to like these immutable laws of the universe. Like just because we have AI doesn't mean the benefits of using version control go away and the risks of using version control um go away. Um yeah, and the idea that AI can replace developers, I think is a misunderstanding. Um because it's you you can't like swap out an AI with a human, no matter how much you like anthropomorphize it. Because if I replace well, I've used this illustration before, like if I don't want to sweep my floors anymore, and I get like a floor sweeping robot to sweep my floors, I'll even make it different. Let's say I hired somebody to sweep my floors, then I fire that guy and I replace him with a floor sweeping robot. Um before it was this guy who was responsible for sweeping the floor, and now it's me. I'm responsible for sweeping the floor, I'm just using the robot. So it's it's not that I replaced the guy with a robot, I replaced the guy with me, and now I'm using the robot. So like if you fire your only programmer and you're using AI now, well now you're the programmer.
SPEAKER_02:Yeah. No, exactly. And I think I think that's not that's something that people don't appreciate, I think, with the current um the current messages that some of these tools are pumping out, which is like you you don't have to be a developer to to use our tools. Like, oh yes, you don't have to be a developer to use the tool, but if you were to ship this into production, I would strongly recommend that the developer was involved in the process.
SPEAKER_01:Yeah, you know, I think that's not true. Uh, you know, or at least it's not precisely true. You don't have to be a developer to use the tool. It's like by using the tool, you now are a developer. It's just that you don't have to be a very knowledgeable or skilled developer. You can get by with almost no skill. Um it's it's just that you're probably gonna paint yourself into a corner pretty fast.
SPEAKER_02:Yeah. You you'd have to go around after the robot poly, you know, sweeping the little corners that it's missed or you know, doing the stairs because it can't climb up the stairs yet.
SPEAKER_01:Yeah, exactly. And an important part of all this is like um like what is coding, what is programming? Um, because like coding is just like the act of typing code into the editor, and if we think of AI coding as like the AI, instead of you typing the code, the AI is typing the code. That's like one way of looking at it, and I think that's maybe not totally like clear and accurate. Um it's really like programming is designing a software system, and AI assisted programming is when the AI helps you design this software system. Um and if you're like letting AI do everything, then you're letting AI design the system and you don't understand the system. So now you have ownership over the system that you don't understand. And that can really only go on for so much time before you have a problem that you can't solve within a reasonable amount of time.
SPEAKER_02:Absolutely. But I guess interestingly, uh, you know, I think because we work also with software development teams where you know they have an existing team. And there is a challenge, there's a real challenge of ensuring that the work that we do gets integrated into the not just, I don't mean like just I don't mean merging the code, I mean how do we share the knowledge of what we've created with the existing team so they can support and maintain that going forward. And I think you know, the way we do it now is is very manual. You sort of you know, screen share, walk the person through the code, walk the person through how it's working, you know, answer any questions. Uh I think that the way I like I think in the future we could also use in some ways, you know, that that's how some people use AI as well. They just chat to their AI assistant and say, how does this work? You you know, show me the examples and you know, how's it where else else is it referenced in the code? I think using you can, I think it's a very powerful aspect of even if you've generated, even if you're using AI-generated code and you don't understand it, then asking the AI to explain it in a way that you then understand it, I think is quite important. Because if you don't understand what you've written or what you're deploying, how are you gonna maintain that? You know, you won't even know what to prompt the AI to fix your issue. I think that's the part of the problem. Sometimes AI gets into this loop. And if you don't know what the prompt is to then get it out of its loop or get it, you know, to fix the issue, you're gonna just be riddled with this buggy app and you won't even know how to fix. It's it'll be 80% there and 20% bugs that you won't even you know, you don't know the programming language, you don't know how it's implemented, you don't know why it decided to implement something from scratch rather than using an off-the-shelf open source library. You know, there'll be all these things that you just won't have any context. You won't be able to even say they can use this library because you don't even know that library exists. You know, I think there's a you know quite famous sort of uh open source vibe-coded flying game going around that coded it just in JavaScript or the TypeScript or whatever, and uh nothing else. Well, the entire thing is powered by 3.js. You know, like it's you know it's using open source libraries, and you name the the author didn't even realize it was using all these open source libraries to generate their game. And you know, I think that's that's the danger I think we fall into when you overrely on it and you're not really understanding what's being produced. If you're still if you're using it to produce something and you're understanding how it's produced, then I think you know, then I think that's a very safe safe way of using it.
SPEAKER_01:Oh yeah, definitely. Um and I had an idea as you were talking a bit earlier. Um, you were talking about like um kind of sharing understanding among a team and stuff like that is how I interpreted it. Um and it made me have a crazy idea, which I don't feel totally comfortable with, but maybe it's an inevitable future that we're heading toward. Um, what if starting at the inception of a project, um all of the working sessions of all the developers and everybody in the company, everything's recorded. Um and actually we're we're a good ways the way there, you know. A lot of the meetings that I've been in are recorded and then summarized by AI and stuff like that. Um, but take that a step further, you know, all the coding sessions are rec are recorded. And so the AI could understand, it could have the whole history and have some level of understanding of all the decisions and what's what and stuff like that. So that like two years into the project, a new developer joins and they're like, hey, what's this class all about? And the AI can explain it really well because it knows literally everything about the whole project.
SPEAKER_02:I mean, there's a I think there's a pattern called uh architecture decision records where you can document the decisions for certain pieces, you know, how why you structure certain things, why you use a certain open source library. And I think architecture decision records are going to become extremely important in the age of AI because when you're using the tool, yes, it has a context window for the chat session or the agent session that you're running. But you close your cursor down or someone else loads up cursor, they don't have that context. They don't know what you've prompted the AI and all the decisions and like the five five rounds of conversation you've had to had go with the AI for it to try and eventually get something that's passable for what you want. And so I think there's definitely a problem of how do we share the knowledge of some of the decisions we're making or some of the decisions that an AI is making for us amongst the wider team. You know, if you're just coding by yourself and it's just you and the AI, hopefully you'll remember. But even then, will you remember in two years' time? Will you remember in three years' time? Especially if you hadn't really written the code in the first place. The answer is probably no. So I think I think um there are maybe ways around that where you can document certain decisions um ahead of time. You can document um, you know, after you've made those decisions as well, if it's too late. Um, you know, usually when I'm looking at a code base for the first time, one of the first things I do is you know just go through the entire gem file, trying to figure out what dependencies it's got, go through the classes, trying to figure out what it's doing, you know, run have a run through of the app. I think all these things, you know, I'm thinking maybe I could write an agent to help speed this up. You know, I could write something that does this for me and generates assistant diagrams, generates the documentation. And that would speed up onboarding from weeks to you know hours.
SPEAKER_00:Mm-hmm.
SPEAKER_01:Yeah, I could definitely see that being the case. Um and an agent could look at the whole Git history, like for the gem file, for example. It's like, okay, line 26, there's this gem. What is what does that do? What's that for? And it finds the commit where that was added and looks at the other code that was added in that same commit, and it's like, oh, it seems like this was probably added because of such and such. And it would save you the time of going and manually looking all that stuff up.
SPEAKER_02:I mean, sometimes, like also, you know, I remember just looking at a gem file every time I look at a gem file, it's just the gem names. I'm I'm like, why couldn't someone just spend half a second just to document what this gem was for? Because sometimes gem is like, why do you have like three XML passes? You know, like why you know why we have so many gems that do the same thing but just in slightly different ways? Uh and you're like, okay, maybe there's a reason for this, but you know, without seriously jigging through the code, you won't know. Whereas uh yeah, sometimes I wish people would leave more comments in their gem files just to make make your future selves' lives a bit easier.
SPEAKER_01:Yeah, I talk about that a lot, like um keeping future archaeologists in mind when you're writing your code. Um, you know, don't make your commit messages just like WIP or did stuff or whatever. Um, because it's not a formality, like it actually makes a difference. Um, because there have been so many times where I like needed to um needed to see what a certain commit did or something like that. Um like a certain commit introduced a bug or something like that, and it's like, okay, what was this commit all about? And then you look at the commit and it's just like did stuff. It's like okay, that doesn't help me. But if you put something and then I've had other times where I land on a commit and there's a meaningful message, and it's like, great, this is super helpful because I couldn't have gotten this information any other way.
SPEAKER_02:Yeah, I think it's of it's often overlooked writing commit messages. And I think it's because it takes time and it takes effort. And I think to what's also not very useful is if you just have a huge commit that has loads in it. And I think that's the that is a challenge we will we will see with AI because AI can generate a lot of code very quickly. And it and it'll just take a commit of everything that's generated in that one run. It's not going to you know, it's not gonna do atomic commits making small commits for every little thing. It'll be like, okay, here's version of one of your prototype, boom. You know, try and debug that. You know, there's not gonna be much.
SPEAKER_01:Yeah, and this is why I think like AI is gonna make really good programmers way more productive, and it's gonna make poor programmers even less productive. Like they'll just be able to create a bigger mess faster. Maybe that way of programming will will make things faster for a period of time, but then you'll reach some point where you can't understand anything anymore, and everything falls apart.
SPEAKER_02:Well, I think the the author of that original viral tweet, which I which was you know, where he said someone hacked his app and then ranked up a huge bill, he's decided to shut down his his code and replatform it to something else, not using AI. And I think, you know, I think that just shows where like okay, some of these tools can be really great for prototyping, proof of concepts, putting together something really quickly. But I think the current state of plays, it's not something that I would vibe code straight into production.
SPEAKER_00:Mm-hmm.
SPEAKER_01:Yeah, and having an idea of what you can what's safe to just like toss out there and what's not. Like this thing that I that I had AI write where it renders a 3D shape for me, pretty low risk. Something that interacts with billable third-party services, like probably should be a little more careful with that.
SPEAKER_02:Absolutely.
SPEAKER_01:Yeah, it's like I've I've had the thought a number of times. Like, there are so many organizations that want to adopt artificial intelligence, but it's like, guys, first you need a little bit of like regular intelligence. Because if you adopt AI, like it can't artificial intelligence can't compensate compensate for stupidity. Like, think about this. It think about a smart person hiring another smart person and what they'll accomplish with that. And then thinking about think about a complete idiot hiring a smart person and what they will accomplish, like they'll accomplish so much less.
SPEAKER_02:Yeah, I think there is the there that's that's the thing. There's a level of critical thinking, there's a level of understanding that is required in order to even prompt the AI to do what you want it to do in a in a in a reasonable fashion. Um Will we get there? I think you will get there eventually. I think you will get I I I would disagree. I think at some point you will get to the point where you don't really need to know what you're doing, and the AI will still know the best way forward because someone else would have written some system prompts. Someone who does know what they're doing would have written system prompts. So, like, for example, now I think uh I think Vercel, uh that their team said they're like, oh, sorry, whoops, you know, yes, our AI can generate insecure code. We're we're gonna look at it. We're gonna say, well, maybe check all the code you generate against the you know ten you know security vulnerabilities. You know, maybe maybe yeah, maybe maybe write tests for your code, maybe make small commits before you proceed to the next bit of generation. I think I can see a future where people improve the tools to the point where maybe this dream can be realized where you can just talk to the AI and it'll generate something that's production quality shippable. Not now, not maybe in a few years, but at some point I think we could get there. And then I guess what does that mean for software development? You know. What if you're I think maintenance and support will never go away because someone still has to maintain it. I mean, yes, you could try and use AI to maintain your code, um, you know, but then who's watching the watcher?
SPEAKER_01:Well, there will always there will always be humans coding. Um because at some level you're specifying what you want, and that is coding. Um like, you know, you're just saying like make me a uh make me a program that renders a 3D object. Like that's coding is just at a much higher level of abstraction.
SPEAKER_02:Yeah. I think that's the thing. I think over time we'll be doing more of that, more of the higher level abstraction, more of the thinking, more of the discussing the edge cases, rather than literally typing the code. Which is probably a good thing for you know RSI and things like that. You know, typing boilerplate code, why are we you know you can, I mean, even if you didn't want to use Gen AI to its full potential, even if you just use it as a slightly smarter autocorrect or a slightly smarter like IDs or auto-completion, like you're already saving hundreds of keystrokes.
SPEAKER_01:Well, something else to keep in mind is reality is extremely complicated. And when you're designing a system that has to interface with reality, there's a million little decisions that you have to make. And in principle, AI can be smart enough to make those decisions for you and make sensible decisions, um, but they might not be exactly the decisions that you would want or even close. And so I I feel like we're never gonna get to a point where you can just say like at a super high level, like, make me a scheduling program with appointment reminders and stuff like that. Because it's like there's a question of like, okay, uh, how long before the appointment should a person get a reminder? And like, what if they cancel 24 hours before? What should happen, blah, blah, blah. Like, there's all these little decisions.
SPEAKER_02:I think if there exists open source or code that it's been trained on, that it can intuit some of those assumptions, it can figure out some of these problems. If it's in the training set, then yes. I think the biggest challenge is when you're doing something that no one has ever done before, it will struggle. It will, yes, it will I don't think AI is at this current, like without a fundamental shift in the technology underlying technologies of you know training on large data sets, if it can start to genuinely think for itself, then maybe at some point we will get there. But at the current the current state, if it's not if it's if you're not doing something that it's seen before, it's very likely it'll be making mistakes. It's very likely it will not produce what you want it to produce.
SPEAKER_01:Yeah, yeah, I think LLMs on their current path where it's statistically based, that'll always be true. If we have like AGI that can be truly intelligent, then I think it can make very sensible decisions about all those things. But it's like, you know, imagine like Bill Gates at the helm of an AI versus Steve Jobs, and each one of them says, make me an operating system. Um it's like without any further input, the operating system will just be like whatever the AI came up with. Um but there are things that come into the picture like taste and judgment, which will differ from person to person, and again, the AI won't necessarily coincide with what you want.
SPEAKER_02:Yeah, I think like I think taste is extremely, extremely interesting because that basically that is what is going to be coming, you know, that's what it's gonna come down to. What is your preference? Like what is your you know, what do you find good, uh like you know, and acceptable, because you know, AI can generate something that's a bit rubbish, and you're like, well, it's good enough. And but everyone has different levels of good enough. You know, some people say, Oh, that's not good enough, that it needs to be better. And I think you you run the risk of all code generated will be that's kind of good enough. Not perfect, not great. And maybe that's okay for most use cases, but I think for some cases where maybe performance is a real big issue or security is a big issue. Good at just like average code may not be good enough.
SPEAKER_01:Yeah.
SPEAKER_02:Basically that's it. You're taking the statistical average of what AI thinks is what it's trained on should be generating. You know, it's you know, it's probabilities, it's not it's not thinking. So that's the risk that you run.
unknown:Yeah.
SPEAKER_01:Definitely agree. Um before we start to wrap up, I want to ask you another question. Maybe we can spend one or two minutes talking about this. Whenever I see somebody with books in their background, which you do, I like to ask about your books. Um can you tell me a little bit about what you got back there?
SPEAKER_02:Oh, books in the background. I thought you meant books in like my when I try to write a book. Oh, I see what you mean. Um I've got a lot of books on agile development and uh some are some on running an agency and and sort of service design, software design. Um I also have a couple of books I'm quite interested in uh because I'm I'm half Chilean, so uh and from the Atacama Desert. Um so I'm still I'm still quite interested in the history of um the Spanish conquest of South America. Um most of the most of the sort of the stories are told from the perspective of the Spanish because they're the ones that wrote the history. So I'm I'm I'm quite interested in hearing some more sort of first hand accounts. That's that's quite good. Um and then uh yeah. Some some old books on Tai Chi that I I was quite into Tai Chi back in the day, but I hadn't actually practiced for a while. Okay.
SPEAKER_01:What was the name of that guy that like Spanish conquistador who is like I don't know, he he was like the kind of the um I don't know, Columbus equivalent kind of of of South America. Do you know who I'm talking about?
SPEAKER_02:So it's Pizarro is the one that went to South America, to Chile, just specifically.
SPEAKER_01:Um I was thinking of a different guy. I think his name starts with an E. Maybe he was he was involved with Brazil.
SPEAKER_02:Various ones. Someone some went to someone to sort of Spain and Mexico, and then up was uh someone down to Chile.
SPEAKER_01:Yeah, yeah, I find that whole time period really interesting. Um because like I guess the Incans at that time already had like metallurgy and like pretty sophisticated technology.
SPEAKER_02:Manipulating stone in a ways that are you know struggling to replicate today. You know, that's the you know, that's when you're like, okay, there's a lot of there's a lot of obviously knowledge lost um somewhere along the line. And that's I think the fall of civilizations and how that knowledge is transferred or lost is quite interesting. I find it quite fascinating. Um so yeah, I'm not I'm not really big into history, but yeah, I do try and yeah, keep up a bit. Although in terms of the book that I tried to write, which was on MERB, um if anyone remembers that framework, was it was effectively a it was a book on MERB, but I decided to approach it in a way of how how to how to create web applications. And MERB just happened to be the framework that we were using. Um so it was talking a bit about test-driven development, um, breaking down uh you know requirements into user stories, how to implement those um in tests, and use the MERB framework to then build the application. And I remember sending what was basically the final draft to the publisher, and then sitting down in Christmas and thinking, okay, the book is done. Don't need to worry about it anymore. And then the publisher goes, we have a slight issue. Um Rails had announced that it was merging with MERB for MO for Rails 3. And I was like, we can't have two Rails books, because effectively my book was now gonna be a Rails book. Like, ah, you know, I don't really fancy writing another book, so I'm just I'm just gonna park that for now. But I think it's something I you know, it's quite interesting. I I do I did enjoy the challenge of writing, so I wouldn't mind doing another book in the future, but uh yeah, I think that experience scarred me.
SPEAKER_01:Yeah, what luck.
SPEAKER_02:Yeah, but it's good.
SPEAKER_01:I think in some ways the best parts of MERB are basically now Rails, like you know, so yeah, that that merge predated um when I started using Rails, so I don't really know what's MURB and what's not MURB. Um, but I can imagine how that experience would kind of um not leave a person terribly eager to immediately write another book. Um before we before we go, is there anywhere you want to send people to learn uh to learn more about you and Bitsy and all that?
SPEAKER_02:Sure, yeah. I'm on uh I'm on Twitter, I'm on Blue Sky, uh Matthew C Ford. Um if you want to check out Bitsesty, it's at bitsy.com. You can learn more about the work we do uh and the services we offer.
SPEAKER_01:All right. Well, Matthew, thanks so much for coming on the show.
SPEAKER_02:Thanks, Jason.