Vibe Coding for Grown-Ups - AI in Professional Software Development

What happens when professional developers stop just writing code, and start leading teams of AI agents?

In this episode, Alex and Jason are joined by Sebastian Brandt and Paweł Stuczyński to explore how AI is changing software development beyond the vibe-coding hype.

From faster prototypes and AI-assisted reviews, to smaller pull requests, security scanning, QA, cognitive load, agent overload and token costs, this is a practical look at what happens when development gets faster, but the need for quality, governance and good engineering judgement becomes even more important... because AI can help you build more, but someone still needs to know whether what you’ve built is actually any good!

Transcript

Alex 0:09

Hello and welcome to Optimize to Innovate, a show where we help organizations stop wasting money on things that don't add value to their business and understand the technologies that actually will. We've got a couple of great guests with us today, so let's find out whether vibe coding is a shortcut, a superpower, or just technical debt with really good marketing. And let's commit to this week's episode, Jason. What do you think?

Jason 0:33

Absolutely. The world of software development and creating value through code has been changing dramatically over the last 12 to 18 months. And we did the first episode, which was looking at vibe coding and how it can help people who are not professional developers to really be able to do amazing things using some of these models which have been trained specifically to help bring coding excellence, like world-class coding capability to ordinary people if they know how to use it. Now, today on the episode, we're really digging into what it means for professional developers to have AI coming alongside them, working between them, amongst them as teams, thinking very much about the corporate, the enterprise world, and what the future looks like for this group of people. So I'm really pleased to be able to welcome two people who are very deep into this in our organization, Sebastian and Pavel. So Sebastian, would you like to introduce yourself first and then Pavel afterwards, please?

Sebastian 1:32

Yes, sure. Hi. Hi, I'm Sebastian Brandt. I've been a professional software developer for 20 years now. And today I'm working as a principal application architect here at Software One. yeah, my day-to-day job mainly is architecture decisions, code reviews, mentoring, and also a good bit on pre-sales for critical infrastructures and everything. So yeah, that's me. Thank you.

Paweł 1:57

Hi, I'm Pavel Stuczynski. Working in IT like 18 years, I think, for now, being ex dev game developer, DevOps developer, trainer, doing a lot of stuff, currently focusing on the platform engineering, consulting, implementation of GitHub platform, and of course a lot of copilot things around that one.

Alex 2:18

So it's safe to say we've got a couple of experienced developers on the call today, uh, I think. Um, so we're gonna start, I'll start you guys off easy, right? Because this is, or maybe maybe it's not so easy, I don't know. We'll see. what I was quite curious about was from your perspective, do you think that the definition of a developer has changed in this whole new world? Or do you think that really it's more about development as a role has almost just moved up a level? What do you reckon, Pavel?

Paweł 2:49

So I think it depends how who is looking at that one. Uh, in my opinion, as a developer, definition didn't change. It's the same person that needs to know the quality of the code is behind the scenes. But I speak of a lot of people that are not developers and they think now they are developers because they can vibe code something. Uh, but at ends, it's always finishing it. I look at the code was generated there, and there's a lot of like surprises. So, definitions, I think, the same, but we have now new experience that we need to have it. Like, we need to better review what is generated, what is in the our repository.

Alex 3:30

So shifting the balance of what you're doing day to day from the generation more to the review of that code.

Paweł 3:36

Yeah, I think that's more like about reviewing what was generated than writing the code. Uh, but I still I want to keep balance. I'm trying to don't use AI for every simple thing, like you know, at a new line, the code or something like that. And still doing like my best approach to like do as much as possible for the commits, so small commits, small changes, so it is easy to understand by other team members.

Alex 4:03

Yeah, interesting. And what do you think, Sebastian?

Sebastian 4:08

Yeah, um, I'm I'm definitely on on public side, so it it's the developer is still a developer, so he his goal is to write code. it's uh and yeah, some things have changed because we we've got introduced to a new tool, it's called AI or LLM, however you want to call it, and it helps us to write more code, more code, to write faster code. Um, it also makes some of the developers' life harder because they have to review more code, of course. Um, yeah, but at the end, we are we still need the developers, I think. Yeah.

Alex 4:44

Do you think just almost a bit of a follow-up there? Do you think that there's a danger that we just end up continuing, you know, the more of this code that gets generated, the more that that role shifts towards the review rather than the creation. Do you think we end up overly trusting it and it just ends up in a position where you know some of that code is gonna slip through that's not so good?

Sebastian 5:07

Uh of course, so this is uh this this chance is there. So that you it's it's totally easy to write some prompts and get a software out and click through it and think, wow, wow, this is definitely what I wanted. And it took me not one week, it took me two prompts or something like this. Um, but and when when you trust it, when you overly trust it and don't do this, what Pavis said, small commits, you still need to do small commits, read code, and understand what's actually happening there, because if you don't do this, you're getting into trouble sooner or later. Uh so at least that's my opinion, uh, my experience.

Jason 5:45

And thinking about the way that we use these specialized generative AI tools, you mentioned LLMs there, and I think it's really important to just point out to our listeners that you know when generative AI first came out, it was a very general purpose technology, right? It really was. But what we're looking at now is models which have been you know shaped through the data they're trained on, the way they're refined, um, to become expert programming assistants, right? Like world-class competition winning expert coders. So how do you how do you best use one of those in what you what you're doing, Sebastian? And the question I'm thinking here is like where where does it feel most useful to bring them into the process of developing code? Is it building prototypes? Is it actually helping create internal tools? Is it automations or is it actually just building production applications? Is there an area where you think they excel, or are they actually just so good all round?

Sebastian 6:54

Uh so the short answer is yes. I I would say even on most of it, on all of it, what you mentioned. So of course, prototypes, this is easy. So this we all know, yeah. So it's really fast to get prototypes out of this. and yeah, when you when you're in pre-sale situations talking to customers and and and they are saying that they want to see some kind of dashboard or something with these type of numbers, then you just put a prompt in, uh, go to cloud design or something like this, and put a prompt in, and you get a dashboard created and show it to the customer and say, Oh yeah, this is exactly what I need. Or no, no, these numbers are wrong. I need something, something else here. I need to see this more. This is you are faster uh in prototyping and talking to the customer there, getting feedback for the customer. You can do it live. Yeah, so normally you would talk to the tech. So in the older days, you went to the customer, you went to a workshop, you get all the requirements, or you tried to get all the requirements, went home again, prepared some dashboards, and so on, and now you can do it live. You can simply do it live, just with a prompt. Yeah, and this makes you much faster, and this therefore it's very, very useful. But also when you when you when you going to code, yeah, um, you have to implement your stories as a developer. Of course, you you you can you can create your development plan. You you can also use it for for checking if you if you if you took everything into account what you actually need to to fulfill the story is also there, AI helps you. And when you go down the bottom line, when you commit the code, when you push the code, the code needs to be reviewed, of course, by another teammate, but also now by AI. So we are also using AI for code reviews. Uh it also is very, very useful there because it can do much more than I can. just from context or brain context window. Um and yeah, so um, and of course, at the end, automation, deployment, testing, all of this, it's useful. Yeah, so it's not it's not the silver bullet, you still need to know what you know what you do, and you need your experience that from some time from time to time. Let's say it like this. Um, but it will help you everywhere. That's my experience.

Jason 9:09

And a and a question for you, maybe this is for Pavel. So if we're if we're we're saying it's great for prototypes and for automations, if we think about production applications. So say we we're using um, you know, Claude or a GitHub Copilot or something else like that, and it's building code for us for a production application. Um we we need people in front of the screen who understand what good looks like, right? Who can who have that reviewing skill, who have enough of an understanding of the what the structures and the way the modules are being built should look like to judge whether what's being created is good or not. So is that from your point of view, is that is that still the main bottleneck if we talk about AI helping make development more efficient, is reviewing still uh is it the bottleneck, I guess, is what I'm asking.

Paweł 10:06

So it my opinion it is bottleneck currently. Uh we are engaging a lot of clients currently with the advisory about onboarding to copilot things to like adoption of co-pilot. And the the like the question this is always happening from the clients is that how to how can how we can manage measure that is the quality of this the coders are doing is good with the AI. That's like for now is a big question mark how to measure it because like what two years ago we were working on this ROI thing, and the approach was very simple. Just we need to count how many licenses are there, how many licenses of this are used, and this is like RI for AI tools, like because it was only chat there at the beginning, and just maybe some more very simple agent that could edit your files. Now you have a lot of tools, agent swans, skills, and other on other things, and clients start task asking, okay, we have developed this kind of feature. The developers mark them as dawn, but it was productive or not. How big effort was behind this, how to measure the value, how to put price on it, because even now it's probably how to give values to the new feature, like someone is asking for the new future before we said hey it will be 80 hours, but now this is like a big question how much will time time take and how how quality will be behind the scenes. So going back to question, original question, and again this is bottleneck, but like but we have also roles in the company that are developed developers are doing only code review. Yes, is this code okay? Is this safe? Of course, we have HyperPress there, there's got scanning tools that can check quality. But that at the end, there are product managers, product owners, uh, someone who requested the feature for the application, that teams need to be involved now more, like to test it, QA quality tests. Uh, in my experience, there's a lot of clients that are looking at the cost of the project and they find that finding finding out that to lower the cost, we don't need tests, we don't need quality assurance team here. Why we need them here? The developers developers can test here. What is not true? Because I when I was as a game developer, the testing team was very important. That even fight between developers and testing teams, because there was for example, we were doing games for Nintendo, and the submission of the new game was so difficult from not even the functional way of the game, the you know the the story and the quest, but from the technical way, how the Game Boy will behave when you close the lead, when you open it. So that was that also need to be included in tests. How it will behave on the Firefox Chrome Internet Explorer, not only in my local environment that I have you know everything perfect.

Alex 13:15

Yeah, that's interesting because it's you're taking effectively, we're stripping out cost, if you will, for the cost of generating and writing that initial code, which is great. Um, we're accelerating the the left hand side of the process, if you will. But actually, what that is going to mean is we do need to have a more of a focus on the right hand side of the process, and probably that by that additional focus, then actually realistically costs are gonna probably go up a little bit around that QA and QA into production kind of a process.

Paweł 13:48

Yeah, I think yeah, we can now produce more. We can create more software, future prototypes, a lot go through the a lot, a lot of like ideas that you don't always have team, brainstorm, and we find out the best idea. Now we can select two ideas and go with two ideas and check them at the end. and but it means that at end someone needs to validate that one, and I think now like the companies that are you know they know what they're doing, they will invest more such teams like QA, Product Manager, uh, because it will be we produce more, so we need to test more it, yes?

Alex 14:32

Yeah. And the the you I'm just gonna pick up on something that both of you guys mentioned earlier on, actually in passing. Um you were both talking about the size of your commits and why the size of the commits is so important. So I'd like to move us into a few practicals as we go through this, so we can, you know, our audience can have some takeaways of what you're actually seeing day to day and what works and what doesn't work, etc. And so um, Sebastian, I think you you and Pavel both said it's incredibly important, regardless if it's a human or an AI that's doing the coding, that we should use small commits. Can you just dig into that?

Sebastian 15:06

Yeah, sure. Um, so it's not only small commits, but also small pull requests or or reasonable sized pull requests, let's say it like this. the important thing is that the very route needs to understand or to what's in there. Yeah. And of course, with a lot of small commits in a pull request, you have some kind of um roadbook how how the developer achieved the pull request. Yeah, and with the pull request also having it not too big, having it, I don't know. I mean, the number is there's not a fixed number. Yeah, you cannot say 10 is good and 20 is bad or something like this. But I'm pretty sure for me, 200 files are bad because I cannot read 200 code files and say it's working or it's not working or it's good or it's not bad, because I start reading, and after 10 minutes, I'm I'm my brain is dead. Yeah, so so this is so you need to understand what's there. When you review it, you need to understand it's it's like it's yeah, it's letting like reading a math book or something like this. You cannot read half the math book and send the say it's correct or wrong. It's not possible. So you can only read one page and and decide. Uh, and that's this is the same with pull requests.

Paweł 16:19

Yeah, I totally agree with that one. You you can always assign to review another AI tool and they can review for it for you. What will actually end up and end up with that massive amount of the comments and suggestions to change something? What would it be the worst situation? Like and it's like not ending loop. so yeah, in most of the cases, 200 files is a bad thing. It's good when doing some kind of formatting fixing on something like date, something like that. But uh, but yeah, I like to not mix features because no looking past when I developed new features, it will take me, for example, one day. At any day, I have some kind of few commits that I'm using locally to revert my job and one pre-request with this five or six commits, let's say. But now you can have features in five minutes, so it means that every five minutes you can do a pull request, yes? Not you know, and you can best be been bored with that. Oh, again, I need to create pull requests, oh again, I need to create pre-requests. Why I could commit direct to the main branch and get forget about that one.

Alex 17:32

So I guess to try and bundle together multiple different features and pull requests into one just for the sake of reducing your admin overhead, but actually then you're introducing way more risk.

Paweł 17:43

Yeah, it's easier now to have like one big commit at the end of the day with the five massive features. And what do you want to revert it?

Sebastian 17:56

One of it for for us is we we switched a bit the work uh or the work mode. so pull requests are not they they get reviewed by AI for us. So we have small pull requests and we have automations there, so which yeah, some using GitHub code uh how is it called. Uh for public quality code yeah, code quality checkers with some rules, some the the normal rules and some custom rules, and then we also used GitHub um agentic workflows to check these pull requests against the requirements. Yeah, so we connected it to Jira, or I think I think you have to call it Jira. we connected it to Jira and and and and it's checking back. It's actually checking back, okay, what are the exceptions criteria? Are they met or are they not met? Which or maybe are was the developer working against it, so it's doing something different and didn't read properly or something like this. So this is a fast check, and this fails if it's not correct. But if everything is fine and if these code comments, a code quality, the code quality scan gives you are resolved or at least closed with a comment, um, then the pull request gets merged automatically without having a look from me. I look at it later because it's not so we we cannot I cannot review every pull request in time now because if as Pavel said, if you have need five minutes, okay, maybe it's half an hour for a big for a feature, um, then you still get I don't know, 16 features a day by one developer, and I have a team of four developers, so you can think of the workload I'm now getting. Uh so we have to improve here a bit. But it's not that I'm not reading the code. I I just that I'm just using also AI to quanticize the risk for specific pull requests. And I have then at the end of the day or at the beginning of the next day, I have a list of pull requests in order, which I have to look at and with high risk and low risk and so on. And then I go through it. And if I have enough time, I go through all of it. If not, I skip something, and then we have meetings during a week, uh, one or two, one to two, depends a bit on the project, uh, where we go through the risky pull requests and talk about it together as a team afterwards. It's already merged, it's maybe even already in production, but we are talking about this. and also fixing things then because we are fast to fix things. So this changed completely, definitely.

Alex 20:29

Uh the the not just the volume and the throughput and all those things, but actually your day-to-day operations are fundamentally changing. Fascinating as well that you can also plug in, you're really thinking about it from a a requirements perspective as well. It's not just building out a whole load of code, you're actually double checking back on really what was being asked in human speak of the developer in the first place via the Jira ticket.

Paweł 20:54

Yeah, I love it. I love it that approach because I was doing a few speaks on the conferences last time about this again workflows, and everything's about if the developers or team know that the agent is not only for generated code, but you can also use it for the other tasks, like it quality, review something, even create this report for daily stand up or weekly stand up, whatever, but adding this other extra things that can be used to make it better, like to decrease this noise of the code generated, it will help. So the developers team seem to be aware that it's not only code generation things, it this is the thing that can also validate what's related if you will use proper tools. Yeah. So so so yeah, the documentation validation requires creation. Today I was working for the one client and we were working about effort for the immigration for one platform or another platform. It was like I think Azure DevOps or GitHub, but we use the outputs from the Our discovery tools that listing the number of repositories, number of pipelines, and etc. and stuff, versus our original original plan versus problems that client provided us. So we get the nodes from the meetings. I have one repository putting notes for every meeting and running opposite this notes using as a to modify the whole migration plan to find the spots what can be problematic, when when where you can save time, where we can't save the time to register all answers there. And and then at the end, I can even create some kind of automation scripts to help me with the bunch of the migration stuff and code generations.

Alex 22:41

Yeah, building out your stories as well, even in the first place, based on those on those meeting notes. Yeah. That makes sense. And I I guess it would also the this new way of working, if you're an organization or a team that is not familiar with say proper use of pipelines, uh, in your, you know, you're regardless of the fact you're using AI, you're maybe starting to maybe I'm wrong here, but it sounds like you might be setting yourselves up for failure without having some of those fundamentals in place. Because what you're suggesting here is because of this volume, because we're now using other AI agents to be actually doing code review to do static code analysis, security reviews, other bits and pieces, you need to have that pipeline in place, which has a mixture of traditional automation, uh, AI-based automation and review, and then human review within that pipeline.

Paweł 23:36

Yeah, yes, yeah. Simply triage of the issues every day. You can just throw out the the issues that don't have any comments or just two words there, like fix that. This is not working. The lot of things like you know, I have a lot of friends that are putting in the issues exception nothing that I could understand. Like we're talking about the problem now, we're writing in down like fix this problem, and naming problem with the two words, and after one month, because it was not important today, but after one month we stick down and we don't know what is what is to do. So a very simple task. Like, if you have a big project, there's a lot of communication happen. We can also use agent to trash issues or clean up the noise.

Sebastian 24:27

Yeah, and the thing is it's it's like this is why I also said it's not it's not a silver bullet because it's a tool, and you still need your your development background, or you need or at least you need to know how to use the tool. And if if you don't know, or if you for example trusted too much, it everything gets worse. Yeah, so it's I I think what what I see here is uh high potential teams are stay high potential, so they they're they they get faster, they get more output, and so on. But for not high potential teams, they're falling back. Uh, even when they're using AI, um, they are producing more bugs, having more problems. So you need this, uh you need to be a good team, otherwise it won't work.

Jason 25:12

Yeah, that that's a really interesting point that you know we Alex, you make the joke about um, you know, we just we just producing technical debt at a faster rate, right? Because if you if you're not doing it right, that absolutely could be the case. And we've already talked about bottlenecks. Um so it's clear that we're getting to the point where it won't be hard, based on what you've been saying, Sebastian Pavel, it won't be hard to produce more code than we actually have time to review. So it's not just a question of there being a backlog, but that backlog could grow exponentially because it's a limited number of people that can review or ways to review versus ways to create. Um, which makes me think about the overall process and makes me think about team. Um, because when we think about DevOps, you know, and Agile, these were effectively designed, they were they were people's ideas about how could we create the the ultimate or the optimal way for people doing software development, which involves communication, involves interacting with other people to work together, to be better together. And my question actually is around this concept of developers leading teams of agents. I'm interested in understanding what does it look like for developers to lead a team of agents, and how does that then change the dynamic between people within a team who are real human people, developers, interacting, communicating together?

Sebastian 26:53

It's an interesting question. Um, I I take it because I have an anecdote about this. Um, when when I I started to work as an as an architect, I got my team and we were working on and was tech lead on the project. And yeah, I of course I had to uh mentor the team. I have to look as as I said, code value is also very important, but of course, also talking to the team and or to the single developers and telling them, okay, we have to do it this way because it's okay to do it this way, but for our project here, we have we need to do another approach because of this or this or that. And the funny thing is now it it really for every developer, he has to do the same now, but not to the team, but to his agent team. Yeah, he also has to go to the agent team and tell him, okay, it's a good approach, good idea, but maybe it's not the perfect one for my solution because of XYZ. So it's exactly the same, or at least for me, it feels the same. Yeah, so it's it's yeah, this is definitely what changed here. Yeah.

Alex 27:50

I had a colleague who is working doing development, um, and one of the things he said was you almost have to limit yourself on how many agents you've got going at once because otherwise you can actually end up feeling overwhelmed and almost burnt out because you're trying to juggle mentally all of these different processes at the same time.

Sebastian 28:11

Yeah, yeah, definitely. So I also tried how how much parallel processes I can live with, or say, or which I can manage. And when you when you have a normal eight hours working day, yeah, something like this, and then you try to have, I don't know, five five sessions, five agent sessions in parallel with prompts running, and then always checking back to back, getting notifications. Here's a question, there's something, and you always have to switch, read something, write another prompt, and so on. Um, it took me easily three hours, I was done. So because this is your your head is exploding, and you just need a break, and you so it's not working this way. And um, yeah, so I maybe some someone else can do this. I mean, then they will be the god developers because yeah, so then they have much more output. Um, but let us later come also to to input. Yeah, I think what we also need to talk later is about requirements. We need requirements, and if we don't have requirements, we cannot put this code here.

Alex 29:11

So the you need to be a multi-threaded brain developer, that's what I'm hearing.

Jason 29:15

Yes, yes, yes, you know you are, you need to. But but it also raises the question, right, of not just do your agents know when going home time is, right? Do they know when to stop working? But how do we avoid the risk of developers getting burnt out, right? Because you could be running 12 agents, but if you know you're under pressure, then maybe you start running 15, and actually the like you're saying, the cognitive load you know during the day may just become so much that actually you can't even work five days a week. You have to stop and you have to do less. And it makes me think about platform engineering because platform engineering as a discipline really came about because we needed to reduce cognitive load for developers, right? To give them more opportunity to focus on creating value through their understanding of the business, their understanding of the need, and their understanding of how to write good code. And it was taking away some of those admin tasks, the things which weren't directly allowing them to bring that creative capability. I mean, apart from burning people out, is there a risk that we take away the things that they enjoy doing from the job?

Paweł 30:26

I think I could compare it with this with any kind of hobby you have. Like for developer, I you can see developers that are love it are doing it. You can have the product just doing it. Uh, I think here is I think nothing changed. Like, if you like to be a developer, you will dig into technologies and you will find a way to use them and will start loving some of them. but for me, what I see and what I'm struggling with now, to be honest, because I'm not active developer, I'm just developing from time to time, so I don't have form of against a lot of CLI terminals running and doing stuff for me. I don't keep being updated what is happening on the market, what is now a popular tool. So I'm more like I can think the developers are not scared but they're afraid that they go out from the business because they are not using the newest agents and tools that are you know now popular because I hit a lot of talking of my teammates that hey, have you heard of that one? No, this is new, this is crazy, this is amazing. And you can think that you need to be more updated, and you can like be thinking that you are behind behind all of this happening. You can think that you are not senior anymore. Uh, but like you say, the beginning, it's not about how much code you developed, it's more what is quality of drop. So I'm the person, I don't I don't like the household. Uh so I have friends to do doing everything who's new, a lot of hobbies, a lot of fancy stuff. I'm just like more oppositive. I don't like gadgets, you know. I just like to have things that are useful for me. So like like the agents, I have only a few of them, and my favorite one is thinking beast mode. So so yeah, so I think it's I but it is still the same. New tools, new technology. If you want to be you need to be keep keep peace with that that one, yeah. So if you stuck in the some legacy like classic ASP, when I was working that one, there are still, but there are still people, dinosaur, that that are very needed and paid very paid well to dig into classic ASP. So I think still depends on the person what you will take as enjoy from your work.

Alex 32:57

So just kind of thinking about that whole cognitive load thing. Um, this sounds very similar to the kind of challenges that we run into as knowledge workers in general with context switching. Because ultimately, you know, you're running five or ten agents, you're effectively it's the equivalent to trying to do five or ten meetings at the same time on five or ten related but slightly different topics and trying to solve five or ten different problems simultaneously. The cognitive load on that is is intense. Um, and it did make me think, well, actually, if we're able to, though, let's let's say we have this ultra god developer as you described before, Sebastian, and they're able to fire through and build all of this stuff, surely there's gonna reach a point where they're actually able to develop faster than the requirements.

Sebastian 33:44

Yeah, good point. Good question. Um, this this even this happened to us even before AI, and this is then the thing is code, producing code was was never be a problem, I would say. Yeah, so it's of course we can now do it better, we can now do it faster, but the requirements or talking to the customer, getting the requirements out of the customer is is a whole other topic topic. Of course, AI will also help you there, but you need time and you need to prepare your project very well when you say, Okay, now we want to go for 10 sprints with a development team of five developers equipped with AI, the throughput is is enormous. Yeah, so you need stories for that. And this you also have to keep in mind when now uh planning for projects, new projects. Yeah, so requirements engineering needs to be faster, maybe even more beforehand than during the project now. yeah, so this is what's what I observe here.

Alex 34:43

So you're you're almost shifting left some of the workload again. But you know, we keep talking about shifting left and right, but here, if left is your architecture phase, um, and in a kind of agile project, you're gonna have iterations of that architecture depending on various features you're working on and all that. But actually, the the again, much in the same way as for QA and test, is gonna start ramping up the need. Actually, you're gonna have an architect who's almost gonna have to be full-time requirements gathering and building stories for the rest of the team.

Sebastian 35:18

Yeah, we have uh for for us for our projects, we have a an business analyst in the project, always acting as the PO, as a product owner, who's who's doing this. so it's always doing the architect and the product owner together. We are we are creating the requirements or or sourcing the requirements, let's say it like this. But of course, when when we also need to do the QA part, so it's all time, and when you have it's all time you need to have, and when you have a fast development team, you need to plan your work properly. No.

Paweł 35:51

I have one question here. Like, what about we using this time to do more like experiments, A-B testing? Like I always problem with my teams that there's several ideas, and and we end up only with one, and then we find out that maybe the other one was better, but we never test that one. So, is there more time for like finding more place for like testing, experimenting with the project?

Sebastian 36:24

Yes, if you use your time wisely, this can happen because if if you don't have, I mean, you wouldn't stop the project when you don't have requirements. Of course, you have to sell the sprint somehow to the customer and to do a proper review after the sprint, of course. Uh, but when you try something out, you can, or if you have time, you can try something out. Definitely. Maybe one of the developers is doing this whereas whereas the others are creating new features, new value for the customer, definitely uh a possibility. Yeah.

Alex 36:49

That's really interesting because that opens up all different avenues of what it means to do a development project, doesn't it? Because obviously potentially a lot more complexity if we end up with you know lots and lots of different almost a tree structure of A-B testing of different features and stuff. But that actually sounds like a really interesting thing. If you're the product owner, you can say, hey, uh, this is my outcome. I know we could get there a couple of different ways, the development team say, well, actually, being able to build that fast, you could almost POV your features as you go, or POC your features, depending on how you want to put it. Um, and then your development cycle then becomes build multiple, choose the best as we go. Obviously, that also might lead to people going, ah, well, we don't have to make too many decisions now, we'll just build all the versions, and then you know that maybe becomes a wee bit wasteful. Um, but I could definitely see a lot of value to maybe also say not that I'm saying that it ever happens that developers would argue over the right solution, because that never happens. We're always you know completely on track. Yeah. But in those scenarios, it probably would actually help quite a lot too.

Sebastian 37:57

Yeah, and you can even use the or you will use the power of AI also to develop these A-B tests faster than you ever were able before. Exactly. So it's it's an easier decision to do it now.

Jason 38:08

It does sound interesting when we you know we were talking about prototyping the fact that you can, you know, you were saying, Sebastian, you can real-time prototype with people, um, which means you get the feedback quicker, which means you can then adjust and and you know, you can recreate and then show them again. It does make me wonder whether we, you know, will it compress the time frame of projects, right? Because we're actually getting there just much quicker, we're being much more efficient in how we collaborate with people, or or will we actually expand the work to fill the time, you know, because it's like instead of having one or two prototypes, we'll have 10, right? And there'll be this debate about whether it's eight or nine, you know, version eight or version nine, were they the best? Probably marginal differences between them. So there always has to be some constraint, doesn't there, you know, to actually get people to complete work, otherwise things just become open-ended. I guess come back to sprints. You mentioned sprints, the whole point of sprint, the whole focus of sprints was delivering something tangible that people can test, that people can validate, right? To make sure you're moving around that cycle. I'm interested in particular in security. And the reason I'm interested in security is we've talked about the velocity of code generation increasing. But one of the things that we're saying that AI can do when we think about models like Mythos and Fable, is AI can be very good at picking up vulnerabilities in code. So we know that we're moving into an era where releasing code that has vulnerabilities is going to be much more dangerous than it ever has been before. At the same time, we're also potentially turning on a new way or introducing a new way of working that's really geared towards producing code at speed. Okay. So there's been this battle of where does security sit in the development lifecycle? How can we make it fundamentally part of it? So DevTechOps, how can we shift left so make sure that the validation of security within you know applications is happening before they're released, not us running around afterwards trying to fix them and trying to find the vulnerabilities, then patch them and release them, which is a chasing your tail kind of cycle. And we know that with the AI models becoming better at spotting those holes in security, that's not the way you want to operate, anyway. Where where do you see us sitting with security? Are we gonna get more back from our AI coding assistants to help us actually meet this goal, or are they just gonna make it worse because the velocity will increase and we can't actually keep up with it?

Paweł 40:56

Yeah, so now the my experience with clients is that they're being more aware that they need to have scanning tools included in the process. Uh, even I couldn't say the name, but you have one big client, this is hundreds of developers. And think about that, they they're now looking for the security solution to scan software to do tests. So, what were they doing? What wait, what were they doing before? Like they didn't have security scan tools, or I think it was more like in my experience, each team was doing whatever they wanted. Like they would do one of them was using one tool, other team was using another tool. There was no no standard there. Uh and and now they are looking for the standard because they are aware that AI tools, coding has generating a mass of the problems, like even small dependency things like choosing the frameworks, packages, they are not secure. This is one thing. Second thing is generating the code that that it's not secure. Still, injection is top in top 10. Yes. I on one of the conferences I did demo demo of the SQL injection. Someone this someone said me, this is not happening anymore. I said, of course it's happening. If you put here someone that don't know what is that, it will happen. So so scanning tools can be need to be there. Uh companies are being more available at one. and AI tools can help here, but they can work, they need to work of the results. Some like real tools like static scanning tools, etc., can help, they can help to shift it left to make the provide the fix easier. For example, I shouldn't like promote anything here, but uh, as I'm working when I'm working, we're using the cap advanced security, and there's a small feature called autofix, so it can provide you suggestion how to fix this vulnerability find from the static scan. So it can bring can help you deliver fixes faster, not always precise, but can even give you a clue how to fix it. Dependency scanning. I think this is what we need for now. Like if you are doing this some kind of uh cloud solution, public website, fabric product, dependency scanning. This is like must have because there's you have a lot of examples that people find out that very, very, very popular package have small issue that you can break in, yeah.

Alex 43:32

Uh one one loan developer looking after that one teeny tiny feature that's buried away in the bottom of the operating system or in the bottom of the yeah, yeah. That's that's quite a common one, isn't it? So what what you're actually saying though to me is if I if I understand correctly, AI has shone a light on the fact that we should actually really have been doing these this level of security in the first place. Just that now with AI, the risks are even greater, but the risks were always there to an extent, and so AI has both shone that light on it, but at the same time given us additional tools that we can use by shifting left to capture those issues before they actually make it into production.

Paweł 44:18

AI is still learning from the code that we created. So if some package is very popular, AI will choose this package because it's popular, yeah. of course you can have some kind of custom instruction agents to that please check the OVAS, please, something like that, or check this the packages that aren't allowed. Uh it will skip to using this package. But it's like, you know, like you have some companies have some companies don't have control of the package register they're using, packages they can use. Uh, this needs to be still implemented in place, like at and on some kind of level. Uh so yeah.

Alex 44:58

I guess my question before we wrap up to you guys is if there were, you know, you've been working in this space now for for a while, and you've obviously you've bumped into and run into many of the walls, you've got the scar tissue, if you will. So if there were a couple of things that you would say, here's a really key lesson that I've learned working in this way professionally, day in, day out, what would they be? Sebastian, do you want to kick us off?

Sebastian 45:26

Yeah, I can. I can. first one that I already mentioned. So uh leading agents is like leading a team from a technical perspective. Uh but I have a second one. the agents or the LLM or whoever thinks differently than a developer. Uh I had one bug with where also it was just a POV. I developed something for a customer, was a translation application. And yeah, there you had the ability to just put in some words and translate it, like with you know translation tools there. And they also had the other uh or they also wanted to have another option for translating documents, PDF files, word documents, or something like this. And I also implemented this with the FFI. And this was, I think it was one of the yeah, it was the Pure, and I said it's a great solution to try out to max out on AI. So do as less as possible and see what's happening here. And and I did this. and yeah, then someday the customer came to me and said, Okay, we have a strange bug here. I said, Why? What's the problem? Yeah, when I when I look at the documents, the language from from what it was translated to the other one, it's changing. So it's it's wrong. So I look at it, it's wrong. And I don't know why. And I looked at it, and then yeah, the bug was easy because the AI needed these languages, the source language and target language for every document in the history. And but it was in the wasn't in the object, which I passed to the to the client. Uh so it took just the the dropdowns on the top for new documents. So everything that you change the drop-down from for the target language, the language in the list in the history also changed the target language. So a developer wouldn't never do this this way. I don't think the thing so I don't I hope, even, but yeah, but it's missing common sense ultimately.

Alex 47:10

It's it doesn't seem like a human, you would never do something as bonkers as that.

Sebastian 47:14

Yeah, yeah, yeah. So this was a nice learning. Yeah.

Paweł 47:18

To continue with Sebastian Sebastian's telling, don't be offended here, but it is like don't give a monkey monkey a hammer because it will break everything. And this is like with every tool. You can give tools, but you need to also give a proper training how to use the tool and put some guiders, what can be used, what cannot be used. so that's that's common rule. That's that's why what what I learned from using tools. And second thing will be like I will not be again original here, a security. Uh, because because now everyone can think that it's available developer because can wipe coding and create a solution. I have like two two weeks ago situation that I was checking application that created my friend, and he asked me to publish it. Okay, I can help you to publish it. And I didn't just review a code. To be honest, I just enabled some security scanning, like secret scanning and code scanning, and just to review and find out that he put in the code because it was front-end JavaScript application with but no back end, he put hard-coded in the JavaScript his path to GitHub account. A token to give up with the full scope of access. It was internal tool, but still it was like big fake because he couldn't read the code, he couldn't understand. The AI told him, hey, we need now a token to have it working. Oh, it's a token, put the clock at the application, and now Pavel, please publish it. Oh so yeah, so I think uh we can use wipe coding for the depends who is using the not developers can easily build some kind of a lot of prototypes uh to show the client to prove something. Uh but if you are a developer, you need to know what you're using at the end.

Alex 49:14

Yeah, absolutely makes sense. Security is job one, as they always say.

Jason 49:19

What have you guys learned in terms of the best way to be cost efficient using these tools? Because Pavel, you made this statement, if I remember it rightly, it was don't use a rocket ship to cross the road to get groceries when you could walk or something like that, right? So your point was there are many different ways of invoking these tools, and the impact of them from a cost perspective is going to be dramatic. Do you want to say just a bit about what you've learned from that point of view?

Paweł 49:52

I think I still learn because going too expensive, it is expensive. Is it using tools is expensive? But from other point of view, you need to convert this cost to the value that you're producing. So it can be expensive, but it can be bringing a lot of benefits. So it's like we are producing everything, even if doing you know, I don't know, we're doing some kind of building some car or airplanes, everything is part of the cost. Now we have new costs that can be easy, easy spent by running Swallow against, but at the end, we need to be sure that you bring in good value product because you can generate you have the demo for the client, and in the one hour we generate tokens costs around $300 by one developer. So even my son starts coding, and he gets I give you have access to a tool, and he was super happy because he was playing his Minecraft game and just he created some kind of launcher for that one, and he he sold it one license for a very small price, like a penny. But he consumed $500 of tokens in one week and sold it for like five dollars.

Alex 51:14

That's a that's a normal one. Love it.

Paweł 51:19

So but he's super happy, and I'm happy because he's happy. Uh comparing to me when I started coding, it was like Northern Commander. Remember the Northern Commander thing? It was option to create some kind of fancy menu there, it was like batch files, it was like it was my entry point there, just creating batch file with some kind of stuff. Now he chasing hundreds of lines and it's working and he sold it.

Alex 51:43

So amazing. What about you, Sebastian? What have you learned on this on this tokenomics front? Any particular tips?

Sebastian 51:52

Yeah, so at least I'm also still learning, definitely, because yeah, it's yeah, recently it got got just more much more expensive. So we we had to we are we are forced to to rethink all this, how how we deal with this, and yeah, of course, I mean always using Fable for simple development stuff isn't the right choice, I would say. so yes, you should choose your your your tool here. And yeah, also when you when you know, I don't know, I I mean fixing one line with some uh spelling errors, you shouldn't use AI for that. Yeah, it's easier to just fix it manually. Yeah, so I so I and I mean this was literally meant, yeah.

Paweł 52:33

So of course, as a developer especially if you can talk to agents, like you don't need to use use your hands to fix the stipo, yeah. You just can talk.

Sebastian 52:43

You need to think about this, of course. Um you also I mean, if with a simple model, you can also achieve a lot, maybe when yeah when you just need to write some biceps script or something like this for deployment, and you want to deploy something, and it's nothing special in there, nothing, so it just needs the context, simple context there. You want to deploy an app service, you don't need Fabit for that. You can just use a simpler model for that.

Alex 53:06

Yeah, basic task means basic model choice and saving potentially multiple times as many tokens. There was one really interesting thing somebody said to me literally just in the last few days, they were talking about. Um, if you look at the statistics on token usage and development, the percentage of tokens used for input are you know an order of magnitude larger than the tokens used for output. Because when you ask it to make a change, especially for brown fields, less if you're just writing a completely green field piece of text, right? But if you're modifying an existing code base, it effectively has to read and contextualize all of the code related to whatever you're asking. And so thinking very carefully about your inputs to then try and de-scope the number of files it's going to be working through and looking at can have a massive impact. So, for example, let's say I write I write a one-line change that I want to make. I say, go and change this the font on this table. I'll just keep it really simple. I'm gonna change the font on this table. I haven't I haven't explained, you know, which piece of code that relates to, where that table is, what document it sits in. So if I just keep it as simple as go and change this table and change the font, it's now got to search through all of my files, which actually all effectively get uploaded, contextualized. It goes through all that, you know, how many tokens have I burned just to find the change, and then it's got to actually make the change. And so the actual input tokens could be huge. Whereas if I had just said, hey, I've got this table, I know it exists in roughly this location in the code base. Instantaneously, I've already cut out a huge number of those tokens. So you're almost starting to think about how do you use, or sorry, how do you reduce, like you might do when you use a search engine, the amount of information it needs to troll through to be able to then get to what it is that you're going to change. And that can have a huge impact, like two or even three X on your token usage, just by being a little bit more specific in what you're asking it to do and having it spend less time acting as a very expensive search engine.

Paweł 55:12

True. True. Even don't say hello to the agent because it doesn't matter.

Jason 55:20

This is tough. You're being asked to be the manager of a team of agents and yet you can't talk to them, you can't be nice to them. It's just gonna burn tokens. In the words of Sebastian, AI is not a silver bullet. So, however you're gonna shape your team, however many agents you're gonna run, however you're gonna use them, weigh up that balance between what the human brings, what the AI assistant brings, and ideally what does perfect look like when they're both working together. So before we wrap up, I just want to thank our fantastic guests, Pavel and Sebastian. How can people reach you online? Sebastian, are you available on any social media platforms where people can follow you?

Sebastian 56:05

Yeah, you will you can find me on LinkedIn. Yeah, such search for my name, Sebastian Brandt, Software One Leipzig. You will find me there. Great. And Pavel yourself.

Paweł 56:14

I'm happy to answer any questions. Also linked in Pawel Stuchainski, Poland, Software One.

Jason 56:20

As for our next episode of the Optimize to Innovate podcast, we are going to be focusing on digital sovereignty. It's a topic which has become quite well um, let's say, publicized within the last six to nine months. But the question is, what's the value you get from implementing or embracing digital sovereignty? What are the pros? What are the cons? And we're going to be digging into that with some experts on the topic in our next episode. If you like this episode, hit subscribe on your podcast app and leave us a review. It really helps more people to find us. And if there's something specifically you want us to cover in the future, don't hesitate to leave a comment or let us know via social media. We're at Software One, just about everywhere. With that, thanks for listening, and we'll see you in the next one.