AA254 - QA Is Dead!?! Why a MASSIVE QA Boom Is Coming

Businesses killed QA with bad org design, but with AI, is there potential for a near-term QA boom?

Join Product Manager Brian Orlando and Enterprise Business Agility Consultant Om Patel as we discuss the systematic elimination of QA roles over the past decade and discuss why that decision is now backfiring.

That's right, with AI-generated code accelerating at breakneck speed and nobody to properly check or test it, Brian and Om argue that we might be heading toward a cliff of technical debt that will make skilled QA professionals more valuable than ever.

We discuss this potential future in five acts:
1. The Expensive Lie: Let's Dev Do the QA (until we lay them off as well)
2. The Coming QA Boom
3. When and Will Businesses Move Software Risk Upstream
4. Why Dev Didn't and AI Won't Replace QA
5. The Case for Human-In-The-Loop

Whether you're a QA professional worried about your career, a product manager who inherited testing responsibilities, or a leader considering QA cuts - this episode provides data-backed arguments for why the QA field may be on the verge of its biggest resurgence yet.

#QualityAssurance #AI #AgileLeadership

Stack Overflow Developer Survey 2023, Practitest State of Testing Report 2024, World Quality Report 2025 by Capgemini and Micro Focus, GitLab DevSecOps Report 2024, Google Code Review Quality Study 2023, McKinsey Technology Report 2025 (State of AI in 2025), Theo (t3.gg) video on the future of developer roles, Software Quality and Beer podcast by Bob Cruz and Matt Kubal (Checkpoint Technologies), Cooper Bench (AI coding benchmark study), W. Edwards Deming (quality management principles), Toyota Production System (quality ownership model), Eliyahu Goldratt (Theory of Constraints / systems feedback loops), Brook's Law, Melissa Perri, Playwright (test automation framework), Claude Code (Anthropic)

LINKS
YouTube: https://www.youtube.com/@arguingagile
Spotify: https://open.spotify.com/show/362QvYORmtZRKAeTAE57v3
Apple: https://podcasts.apple.com/us/podcast/agile-podcast/id1568557596

INTRO MUSIC
Toronto Is My Beat
By Whitewolf (Source: https://ccmixter.org/files/whitewolf225/60181)
CC BY 4.0 DEED (https://creativecommons.org/licenses/by/4.0/deed.en)

Transcript

0:00

Ohm, the software industry spent nearly a decade saying, we don't need QA. Well, we're gonna let the developers do that. sorry, do that too in addition to everything else. Of course. And then everyone is surprised now when quality and reliability are all clearly on the decline. Wow. So developers typically, uh, have hated testing. Business killed it with bad org design. Like, QA didn't die. Welcome back to arguing agile. I'm product manager and habitual troublemaker, Mr. Brian Orlando. And this is my co -host, the philosopher of process and enterprise business agility consultant, Mr. Rompatel. Today, we're going to talk about, one group that everyone thought was done, and that is QA. Software quality assurance QA. Some people say testers. I like to, I like to wrap the whole career field and say QA. so what the future might actually have in store that, could help these folks in their career journeys. Do we still have career journeys in QA? Oh, what? I think that's a topic for a different podcast. Oh, wait, that's what we're talking about today. That's, Specifically, the QA, a little bit, but the fact that, you know, career journeys exist or don't exist, I think that's worthy of a podcast on its own, maybe a mini one. So let's talk about our goals today, um, so by the end of the podcast, I hope everyone will, uh, have these three things I'm showing on the screen. We'll understand why QA's decline was a business design problem and not a skills problem. We'll recognize how AI has exposed the cracks in this developer -led quality model. sort of that and a few other things, not just AI, but AI is definitely, if every other, cut with the hatchet, widen the crack, AI was just a block that really just, you know, smashed the, the whole thing open. and then on the podcast, we'll see how modern QA can reclaim influence by owning systems level thinking, systems thinking. Cool, some of your good. good. I'm, I'm glad that, I'm glad you like those because I have some other ones that just because this is my particular career background, we're gonna have some fun. So these are for, these are for fun. They're, they're part of the goals that we're going to do on this podcast, but just, just because I want to talk about it. We're going to talk about the expensive lie that has been sold to everyone in the last, I don't know, 10, 15 years of developers own the quality. Developers will do the test. They'll own the quality. Although, listen, listen, I, I see where you're going with that one. Ooh, you know, you can sense it. It's not too early to say, like, the, the scrum folks here also bear some responsibility in this one with all the trainings because the team The team owns everything is also part of this. So I'll try, we've got to try to remember to bring that one up in the category where we talk about that, then the two other things we're going to talk about QA and testing equals the businesses. risk intelligence and if you have strong opinions pushing back on that one it'd be like no let the departments all individually do risk intelligence and then they'll be like the states where let we'll let the states deal with it and they can't then states never messed anything up for freedom I thought you're gonna say for free. No, this is true. This is very true. The expensive lie is real. We're living it now. So I think it's a very timely podcast. And then, you know, the last thing here is why, look, I went on, uh, Bob's podcast. Software quality and beer with Bob Cruz and Matt Kubal from Checkpoint Technologies. Part of what we talked about was, hey, there is a future in, QA, like for people that have the skill set, for people that come from the, from the, from the, for the career field because, it feels like all the QA roles are going away. I mean, even when I was working, they weren't necessarily going away. They were offshoring and they were kind of they were still there, but it was obvious as companies were trying to. get them for as few pennies. Cheapifying them. But now, but now it's like just straight going away, like there will be less QA folks. And, talked about with Bob that, that we went with was the topic that I something along lines of, hey, there's this like cliff of technical debt that we eventually will run ourselves off of if we keep doing everything like this. And part of it is that the downsizing and reducing elimination, whatever you want to call it of QA. And I said, what's going to happen in, in a short time period where we start producing, you know, seven times the amount of code that we have today. and nobody's testing it. Yeah, the obvious thing will be, we'll have more AI tools that'll test that too. So the AI, yeah, I know. Yeah, throw more iron at it. That used to be the case in the past. And now it's throw more AI at it. we're already talking about the first point. We might as well transition right into it. Sure. so for the last 10 years or, or more, right, companies have been either gunning QA completely, taking away all their budget, eliminating all the people. Right. Betting on automation, so skinning down the QA people, under the, under the banner of, We just need fewer people that are more well trained. Because you heard that in the media at some point. It's oh, companies are hiring people, but they're hiring the most senior people. Those are, you know, junior roles are going away. And the most senior people are still moving around, which isn't really true, like now, right? It was a while ago. I feel like saying, I'm only going to hire software development engineer in tests like companies type of roles. Yeah, in that world, you're saying, the developers are going to take ownership of all the rest of the testing and you're getting this automation person in their kind of supplement their workforce to make them better testers now AI is threatening to to replace the developers faster than the same industry did to replace qa right yeah absolutely so Now is when we're allowed to kind of like push the agenda off the floor and just talk about the high level of, did the industry shoot itself in the foot at this point, knowing now, you know, 10, 15, whatever years later, you're really going to need people every time a bill comes out to check that things are stable, you know, or at least construct systems that do that with your specific, you know, thing. Yeah, yeah, so I think a lot of companies are struggling with this idea of how to balance what they do with the QA function, with using AI under the belief that there's going to be better quality, so you need for your people. And then the people you do need, you can offshore, because again, it's all about cheapifying this whole role. so offshore, you know, it's, this is a known model, they'll do that. What they don't have are people with the domain knowledge and the experience who can work with developers, not to test things after it's coming off the line, but to build it in, right, to build the quality in to begin with and think of all those things that only experience can, uh, foretell. I have a couple studies in the section, practice, practice test, and, uh, survey. Stack Overflow .co. Co. Let's see that one coming, did you? No. Those are the Stack Overflow developer survey of 2023 and the practice, state of testing 2024. but the quotes from both of these, funny, whether you, Whether you doubt the sources or not, the findings are very similar. So on one, it says 60 % of orgs reduced headcount since 2015 from QA, specifically moving testing into the development teams from wherever it was in the business. I'm gonna assume operations or some, who knows, right? Maybe you stand alone in some case. Yeah, and then the other one was the stack overflow developer service said 62 % of developers dislike code review and testing. And I think that that is watering things down because I think if you separated those two and, and you threatened from, for life or limb, you would say, which one would you rather do, Mr. Mr. Mrs. developer, code reviews or software testing. They would do code reviews, I think. Yeah, yeah, I think that percentage is higher. But this is what the studies show us, so we'll go with those, right? so let's get into the four and against on these. the against here, they're straightforward home. Listen, developers, they should own the testing. Ooh, I like this. It's all, if you go back to this old, listen, the people that say this, I'm already gonna put the plank out on the pirate ship and walk them out into the sharks first. So at least if one becomes my time, the sharks are not hungry anymore and they won't eat me, okay? Dev should own testing. Because there's fewer handoffs, meaning the business just comes to one IT work center, dictates what they want, bounces out of the room, then you go off with your water cycle, oh, sorry, with your water cycle, with your waterfall teams, your water, so you go off on your water cycle with your water, shooting, shooting your squirt gun. Oh, man, I should leave that blooper in just to, and there should be fewer handoffs. in the old waterfall. I love that this assumes a specific way of working to be a defense point. Yeah, and I think this is also a fallout from this whole adage that quality is everybody's job. So it's the tester's job. It's, okay, why isn't it the developer's job, right? But it's funny how it stops there. It doesn't go beyond those two roles, right? Quality is not the BAs job. Quality is not the, you name it. right, fill in the blank here. Is it the business's job? Should they be specifying things more lucidly? Listen, quality is everybody's job is the most. This is, these are the reasons when we have, when we have normal, like half a year when Oman and I will discuss our working agreement. These are the ones where I lobby hard to be like, why, why do we not curse on the podcast? I'd remind me again, I need, I need written documentation of like, because the whole, that, that arguing point is the worst. Absolutely, yeah. Quality is job one. It's, it's almost, it's almost as bad as, disagreeing commit. It's almost, almost, but there's a special level of hell reserve for disagreeing commit. I mean, it's, it's close. I mean, it's on a different level. QA slows velocity, um, which is also sort of, like a sub point of, of, of, you know, dev should do their own testing, that, that kind of thing. and then, offshore QA became too expensive. Yeah, which, which is hilarious, by the way, now offshore has become super expensive. I think it's because they're saying, look, we're investing all this money in QA, in, AI. So why do we need this and that, right? That's what they're saying. Right, I mean, listen, is now the time or am I waiting? There's a Theo clip that started this. like the germination for the idea for this podcast and i want to play that theo clip you gzzi yeah the our friend of the show he now here we go here we go now i'm gonna mute us There's a pipeline of how features ship. Where at the top there is user problem, whatever the funny thing is the issue is that the user has. At the bottom is shipped solution. And there are always steps along the way here. problem identify a solution scope and assign the work write the code review the code test the code plan the release and then do the release generally speaking more So he's the old SDLC is basically what he's out there. Yeah, sure. Serialized, uh, steps. Yeah, I want to bring this up because I, One of the points that we're arguing here is going to be, it's actually the second point, but I think it's the most, you describe the most important point on this slide. It says, developers hate and are not good at. reviewing and testing and not in that order All right, back to, back to the OGGG. Generally speaking, more engineers helps here and here, maybe. There's like two dimensions to this column, though. There's how much is this happening and how long does it take for any one particular piece of work? sorry for anyone that's listening only. Theo just pointed out that the engineering works here and here. it's write code and review the code. Those are the two places that he's pointing out. Like 500 problems that users have, you'll be lucky to clearly describe 100 of them. You'll be even luckier to identify solutions to 50 of them. You'll eventually scope out and assign work for 15 of them. You'll actually get people to write code for five of them. And then from here down, things tend to stay pretty because by the time people are writing code, since the code is so expensive, the majority of the code people write at a company where they're paid to gets through everything else till it eventually comes out. this part was too expensive to be lossy. We put a lot of work from the top down to refine the things that made it to Eng because if it made it to Eng and wasn't actually worth it, you just wasted time. Also, he's pointing out from, from the right the code step, everything kind of stabilizes from the, from a right the code step. now you can literally take the user problem paste the screenshot into the agent and skip all the way down to here that's insane that changes for to hear he says to skip all the way down pass the right code pass a review code to the test code step and ignoring the identify the solution scope assign the work all that kind of stuff is me so he's going from describe the problem to problem which could be like a form or something like that or maybe a call where the transcription of what the user talks about goes into some kind of system that separates the problem that the user describes yeah If I can take a user problem, just take a screenshot of them tweeting at me, paste it into Claude code and get the bug fixed, this whole pipeline has been destroyed. And the stronghold we had here where everything from here down was too expensive to and also too high skill to easily staff up in. So if you wanted to increase the amount of work getting done, you had to hire more people to do the code writing. That's just over now. The person who scopes and assigns the work can do the writing now by kicking it off to an AI. And if you help them skill up a bit in how they spec out the work, the AI can get better at writing it. I would argue that the role of devs is shifting down here, where we'll do more and more of the review. It's that's even slipping due to all the AI review tools and now we're stuck doing the testing and manually verifying the stuff. Did you notice that he said, oh, my role now as a developer moves further down. And then the sections he pointed out were the sections that I typically think of as like typical QA in development. and by the way, that we weren't willing to pay QA and development to do. We just said, oh, developers could do that as in their spare time. Yeah. Yeah. It's interesting. Because, you know, using AI on the face of it, you might think that's not. too unviable. Yeah, you know, have your developers write code using AI and have your developers test the same code using AI. Yeah. What's wrong with that, right? We're gonna find out later. What's wrong with that? Yeah. I mean, you're gonna find out quick. What's wrong with that? Alright, let's finish his video because I think we're, I think we're, I'm gonna skip. I'm gonna edit his example up and then we're gonna get right to the end. Okay. Are ready for the harshest reality check here? Most engineers are really bad at this. Even like the best engineers I know are atrocious at QA. I cannot tell you how many times I got to work with like top 100 in the world type engineers. They shipped something that looked really good, seemed really good, made a lot of sense, was well aligned with the goal of the project. And I try and it just doesn't fucking work at all. Yeah, that's not even just like writing unit tests, writing integration tests or anything like that. It's using the thing. most engineers are terrible at that and historically that's been okay because there was such a desperation to fill the hole that was writing code that we would do anything for it and now that's over another way to think of this is as a funnel at the top we have like every single thing that's gone wrong in our product for every user and then all the way to the bottom here we have shipped stuff so that's to make it all the way to the bottom not fall out of the funnel along the way in each step along here required some level of refinement so describe problem right code you get the idea the funnel we've all seen a funnel like this before this is just kind of what reality looks like at least looked like as you go further and further down the number of things left would decrease and the really strange thing that is going on right now is that this looks like this now there is no longer a funnel at the line that we own as devs that's over That is a very different world we live in, where if you've gotten to the point where you can describe the work as tickets, you've gotten to the point where the work can be completed for the vast majority of development work, the vast majority of the time. yeah, so that was, uh, that was a video that I encountered from, from, you know, our friend. He's our friend, but also, like, I'm not getting invited to any chat jippity events, any Sam Alman events. Sam, if you're looking for a man in finance, I'm just saying, clip, partially to highlight that Theo also agrees that devs, they can't played that whole stand QA. And I would, I would pose that they're not very good at it, compare, comparatively to professional testers. it's a different mindset. at the end of the day, it's not just simply following a bunch of checkboxes. So I'm not surprised that they, you know, that they don't like it because if they fail at it because they don't have the mindset, they're not going to like it. I wouldn't if I was a deaf. the other thing here is, uh, like Schiff left was supposed to help with this. Schiff left did, It did no service to any professionals out there. I think, and I think I've hammered this enough in a podcast. Yeah, where I don't need to talk about it today. But yeah, shift left was we're gonna let developers do the QA's job, rather than embed the people from the right in and fold them into the process on the left. I assuming you even have different prices in the first place you need to do that, which is also part of the problem. It's an assumption. But yeah, so the, the QA elimination, I'm gonna pose that it has created higher rates of companies and volatility in burnout on development teams. Probably also burn out among, you know, good testers too, like good QA folks too. Yeah. I mean, a lot of people that I know personally have left the career field. People that I know that were, you know, excellent in tech at one point in time. And they just burn out eventually. Yeah. you, when you, you know, when you devalue their true earth, I'm not surprised. That's what's happening. AI, it magnifies the errors faster. it's, it's not true, but let's say it writes code 300 % faster. than your typical developer. We, we know because we did the last podcast, where we know it actually writes code 19 % slower than if you just did everything manually. so we have, there's actually actual studies and measures on our side here, but we're going to throw all that away and I'm going to ignore that. Go to this new world that I've been hearing about because it can't be left behind, right, FOMO, right? Can't be left behind. And we're going to say AI magnifies the errors It's not like 10%. Yeah. It's a lot more than that, I think. Yeah, absolutely. The last podcast really put a highlight on that, right? If you're cutting QA and you're just simply saying yeah, it'll work out because we're using AI and developers are doing more, right? were you doing QA right to begin with? I mean, if you're not doing QA right to begin with, the immediate gains from those proponents of cutting QA are easy to illustrate, right, on paper. Yeah, so later on, it turns out that, yeah, you know, the quality of the whole solution maybe suffers, but by that time, these people have already established themselves and all these cuts have become institutionalized and, you know, they will train show me. I mean, that would be the real issue under hood here that, it's your leaders that are there that make these bad changes that are gone two years later and never have to deal with any of the consequences. That's, that's a culture thing. It is. And, boy, if we want to have a podcast about changing that one, first of all, let me know what it is because I won't be there. Like, I only show up too productive conversations on. That's what I'm saying. So I have three things in the takeaway in this category. number one, it's compare and then you need to be truthful about what you're seeing and then take accountability. So there's three things. Compare is the, your defect leakage rates before and after the QA cuts. You might have to dig into the archives for this one if it's been a while, you if your organization kind of foisted it on you, it's been a year, right? you need to show. the actual cost and and the compare here leakage rates it'd be great if you can tie your leakage rates actual money that's even better yeah i mean if and it's again people say well it's hard to do it's not hard to do you just need to put your mind to it so figure out the impact of having these um escape defects or whatever yeah then you'd know right you could put some value on it but add to that the cost of rework and also the opportunity cost, while you should have been doing something else, it's a bit more productive. People say it's hard to do, but also, uh, can you just roll into your, uh, CFO's office if you have a relatively small company or maybe you know, director of finance, something like that, if you have a moderate size company, Obviously, if you have a giant company and like all finances done out of the Chicago officer or whatever, you're not doing this. Right. But if you're in a small, medium-sized business, you have a shot at doing this is, ask for help from your finance folks to say, hey, I want to tie my, my metrics and I'm saying, I want to tie those to cost. cost in terms of salaries the easiest one. Sure. to, they might not want to tell you exact salaries, but they can give you a number. Yeah, you can get a blended rate. Yeah. Uh, and the second one here is a truthful ROI, you know, subtract the rework and incident recovery costs from the quote, savings. So here's, here's the date we went offshore. Here's our mainline budget of, you know, salaries over time, all the downtime and everything you pay that, probably has cost to it, but the development team doesn't know that downtime has costs. Yeah, I'd say this is very difficult to do in practice. The reason being that in practice, these are different GL codes, right? yeah, the cost of quality, yeah, that's over there. Right. Right. Yeah. And that's the problem. you can't divorce it from the cost of, salaries. Right. Right. he shouldn't be able to divorce that. Yeah. to divorce it from the cost of salaries if you're going to you can't, you need look at, well, the system was down and customers couldn't sign up. The system was down. Customers couldn't submit transactions. Those kind of things. You know, those will never appear on on the payroll, so if you don't, if you're not tracking them, I mean, basically they're free. Right. Right. So basically what I'm saying is just turn your servers off at 5 p .m. when you leave the office and turn your service back on and then, you know, 8 a.m. or, look, we're all 9 ,96 here. Something like, what? 6 p .m. That's right. Which is funny because in 996, you're going to turn your service on at 9 a .m. because it's 996 and it's a little late for me. That's what I'm saying. Yeah, I got it. and then the final one, accountability. Identify who's accountable for released quality because it's not developers. Right. You're not holding a developer responsible. for release quality. In fact, this one, I wanted to bring up, like the, a lot of this podcast, because we started with Theo, a lot of this podcast goes back to developers. Yeah. really, uh, product management is the one who's taking ownership of quality in recent years. product managers oh product product boy i guess they're generating code too had no more developers what does this look like but i think we're gonna cover that in a future category so i'll i'll hold off for a second So if you're a developer and you're put into the situation of owning QA or a product manager, right? Let us know in the comments, is this something you really wanted to do. Is this something that you in function are in a world where we good at day to day. Right. or is this one more thing added to your old plate of corporate isms. I don't use them very much. Get a lot of my plate. Ohm. get off the plate. Shouldn't that be a good thing a lot on your plate? Anyway, anyway, uh, you just get told you're doing this and you have, you have no say in the matter, let, let us know. So we're going to take it one step further now. Theo is right and developers can't stand testing. They hate it. They're not good at it. He's known top 100 developers. Terrible at it. What happens when AI starts making all this worse? Awesome. Yeah. will do the testing for us. and before AI own, everyone thought that this magic automation technology would just automate everything and then nope, we didn't have to have any manual, manual testers because the automated folks everyone thinks AI would somehow magically know all the paths. Yeah. Yeah. But can AI really understand what matters to the customer and make a real impact? that's what we're gonna talk about here in a second. Oh, this is a good one. All right. So in this section, I have some research to bound our conversations here. It says, automation grows, business value confidence declines. 47 %, a 47% report more escaped defects. So business value confidence. So business value, reliability, that kind of thing, I guess is in this study of the world quality report 2025 by Cap Gemini, Cap Gemini, so Jetty. Micro focus. Yeah. For me, this, this number is staggering. 47 % reported more escape defects. that's a huge number. Yeah. Yeah. I understand what you're saying. Like, listen, you got 40, 47 % more escape defects. But listen, oh, all customers, when a customer just reports it, it gets fixed automatically. Listen, these tools are as worse as they ever will be right now. Oh, I thought I bring that back that one. I know we've had that we've had this one before a couple of podcasts yeah yeah yeah you know why is because I can't really this is this gets me out without having to argue and be like look they're only as bad as they ever will be there is they're most expensive as they ever will be today ohm that's another great one that we did on a previous podcast that was completely not true because then the opus model started coming out and they yeah yeah absolutely this is this stands on its head often So yeah, agreed. But yeah, the against here, some of them, I'll throw up on the screen right now. AI tools, they'll do the test validation of the future. They'll do the integration, the security tests, the pen testing. everything that we cry about now that you are i might hear on the internet is oh they leave so much so many security flaws and then uh obviously the trade off will be well they leave security flaws now but in two years they'll be nearly flawless or they'll be as you know impetrable as whatever pick your best you know operating system that's not windows oh boy yeah absolutely i so This is one of those things that we live in the moment and say it's it's only gonna get better from here on up, right? I think that's wishful thinking unless it is backed up with some deliberate measures to make it better. Yeah. Right. And that, I haven't seen a lot of. I mean, you're not going to either, so sit back and take a break. Like the LMs can generate full regression coverage home and faster than people can. And even if you need to dial it in, the speed of the LMs puts them over the humans, even if they make a mistake, whatever, 47 % of the time or whatever we pointed out, to speed to that 47. You know what I mean? Right. So they'll make mistakes, but they'll make mistakes a lot faster. Yeah, but they'll get fixed a heck of a lot faster because that, yeah, right. With any luck. Like in the future, in the future is what I'm saying. One day, maybe. I'm going to take out this national debt home now, knowing that in the future, my future generation will definitely make more money than I made right now. And then the final one is humans can't compete with the speed though. I already brought up a little bit about the speed. Yeah. The last one is humans can't compete with this. This is all the speed. This is all the speed, bro. you've captured the speed. like it's like you capturing you've, hopefully the the speed of light like hopefully you can capture the light and do something with it hopefully so otherwise it just goes away well we're only looking at speed rather than you know velocity which is a vector because it takes into account direction which direction is it going in let's so one of the fours is AI handles execution right not interpretation and it does That's right. Right? Yes. So interpretation, what value do we place on that? Who does, who whose job is, let me put it that way. Yeah. To ensure that this alignment along the lines of interpretation. It's, it's, it's not the devs. We've said that already. So who is it? It's, it's not doing it. I mean, it would. It's a, it's a gap. owned it for a period of time. That's, I'll, I'll a product management say that, that. Product management owned it and then, they were never really good at it. And it, and it still took a whole team to refine and break down work items in an interview. we had to have UX researchers, to do real long -term super deep work. the product folks were the people that had 10 % of everyone's skill set. But because one person had 10 % of everyone's skill set, they became the product manager. De facto. Right. in order to make a good call, you really needed familiar, familiar with a lot of different skills someone who was across the team to kind of pull it together and help break the work down and define prioritize and define and stuff like that. But yeah, now, now. Now everyone, I mean, even the, even the junior devs, they're aided by AI. So theoretically, they have access to all those skills. The problem is, they're, the advice, you can't tell bad advice from good advice if you're junior. Exactly, right. And guess what? We're hiring junior developers. Oh, that's right. So who's there to punch all the cogs and they work all the wheels and the, I'm pretty sure I reverse those, but it doesn't matter because there's nobody at the machine anyway. That's right. the tech of today is not quite there. Like what you'll hear is, you'll hear people say, well, you're just not prompting hard enough or the tech of today is not quite there in terms of the number of tokens. Like Maybe in the Cooper Bench experiment, I think, I think it was Cooper Bench. or maybe it was the AI episode we did where they asked the developers that were maintaining like 1.2 million. line of code that was cooper bench oh was it cooper bench i remember yeah yeah anyway those those developers like it is impossible for them to feed all of their code in all of the systems that they work on into the lm because the lm just doesn't have the that amount of memory yeah or amount of tokens in a working session to be able absorb properly all of the pieces of the system and they're not stopping and fine-tuning all the models and also, aside from all that, if they're a private company, they're not going to be sticking Claude codes, send all my code up to Claude Code access. They're going to be concerned with privacy and governance. They're not going to want to do something like that. Yeah. Yeah, absolutely. So overall, systems thinking. beats just pure automation coverage. Right. And so who has that? who has those skills to think along those lines typically? Is it, is it developers? Is it product managers? who is it? If it's not QA, I'm saying, who is it? it should be both. I mean, I was gonna say, yeah. It should be both because the, the, the, if you think about developers from the perspective of like someone who's a, system architect, I mean, the job of the system architect or, or the developer lead, right? Their job is to maintain how things work when things cross, across software systems. I mean, if you're, if the whole industry is saying, we're not going to have any more juniors, then the idea is that those skills that seniors have, or that leads have, every employee would have. the problem for me with this conversation is now the company wants to take no part in training anyone. So I understand what you're saying, but also, like, go, go, take your medication, grandma. Go back to bed. what is happening here? Wow, what an industry we're in right now. What a life. The next QA wave. should move testers to business validation not to have them continue clicking buttons because or automating the clicking of buttons this is an interesting topic is one of the clients i'm working with they have a traditional QA role. And the QA folks do everything, automation, testing, performance, load, all of that. We don't have separate people for that, right? A given QA person has those skills. However, we recently introduced a VNV function, as is colloquially known, validation and verification. And this is not... QA. this is not saying, you know, I click here and it should do this, does it do that? It's, why would I want to click here? What's the purpose? And does it serve the purpose, From a business lens. So I think that's what this is talking about, business validation, not just button clicking. What we found is initially there was some pushback. Well, why are we duplicating roles? We're not. And so after about, I want to say, a couple of months. it became pretty obvious that this was a gap, right? And now we're starting to fill that gap. So the reason I mention this is because initially people just don't understand what this is. We already have QA. Why do we need? These are just words that people think are synonymous. And they can be, but they're not. They're really not. Business people just simply say what they need. they typically don't even tell you why they need what they need so you have to elicit that information and it's the v and v folks that make sure there's alignment there with deliverables to the why not just the what takeaways here The category here is the coming QA boom, we have some like what I think makes testers really valuable here. I misread that. I thought it said tips for writing the broom, QA witches. Yeah, tips for writing brooms. I mean, you know, hey, whatever, whatever floats your. Broom. That was, that was a harsh one. That was a, that was a Harry Potter joke and Yes, along with the clip that we listened to earlier it's a family show. with from Theo about, developers hating, doing testing, I have some takeaways here for ratting the boom or the broom. Either one. I mean, the picture there is not the broom, but whatever, whatever works. they're very similar to what I expressed on Bob show. Number one, learn the business domain and learn how to show business impact in that domain. I would say, I would say that's a first thing. Okay, that's the first thing. because a lot of what you're doing is, is, predicting a lot. If we don't put XYZ dollars or time or effort or whatever into this. Yep. here's the prediction of, you know, our exposure or whatever, or here's the stats afterwards of our actual, you know, escapeage or whatever you want to, you know, escape defects. Number two, use AI for speed, not sense making. This is a great point. It does accelerate your journey from A to B. However, the angle there is efficiency, putting the human in the loop, you're now trying to become more effective and not just efficient. Yeah. And you can't replace that. So those people that say, you know, we just use AI for everything. without the human in the loop, oh my goodness, good luck. research that I left out of this podcast there's some because I didn't expect we would get here. Congratulations, we're here, where, you know, the AI, uh, it implies, it is not exactly say this. This is brain's interpretation, but, the AI makes you dumber. It doesn't necessarily make you, like, actually, like, doesn't lower your IQ, but, you know, regress, but yes, it makes you, it makes you, it makes you inclined to take what you're getting from the AI. and not dig deeper and not think about other structures that you could have had because you're focused on the result, aren't you? So you're taking things at face value to your point, right? Oh yeah, I've got all I need now. I'm going to move on. so getting back on track, the last item in our takeaways, advocate for quality metrics that are linked to the customer and the customer's risk. I guess over your risk too, right? Business risk, right? Sure, sure. that kind of goes with the first one is, hey, know how to talk impact in your business and, you know, to and in your business. And then the last one is the care about the quality that your business cares about. whatever that is. I feel if you can figure those two things out, you're well on your way, towards actual shifting left, meaning moving towards the people that are making these choices in the first place of, oh, we're gonna put out this risky release. I don't care what you think of them, Yeah, yeah, I think people should learn to measure that, that risk and also figure out, you know, what's the, what are some of the other risks? So we talk about customer risk, the business risk, there's, whole host of other risks, reputational risk, how you're going to be perceived as a result of one snafu, all of that. But yeah, I think the underlying messages don't just focus on passing tests through using AI. All right, can LMs ever really test customer outcomes, or are they just testing code coverage? So let us know your take in the comments below. we agree that QA is perfectly poised to become strategic, will companies, make that cultural shift home and will they All right. Even if reverse a decades long trend. So Deming tells us quality must be built in, not inspected in, but here we are, 40 years after Deming and, blink twice if they're holding them, Deming hostage, from beyond the grave, is what I'm saying. here we are 40 years later. and like we're still testing like automation tool enabled janitors it's yeah it's become a gate before we can throw things over the wall at the customer unfortunately right and that's not what deming was advocated by far so yeah you're right the shocking part in what you just said is 40 years later what's our learning velocity here as human beings don't answer that or if you can do it in the comments below You can be anonymous if you like. and I'll wait to find out how close to zero it is. The against here are straightforward, hey, you need to get in line with reality. Businesses don't fund early risk modeling. Like, maybe the most mature businesses, maybe they'll have that some kind of risk modeling. Yeah. But, before the most mature businesses that you'll ever find, not a function of the business, period. get off of my cloud. It can only be one of us up here. And then the risk lives with product and or architecture in the modern organization, which I hinted before. I said I'd, I'd put off into the later category because this is a category. Yeah. And, and I think even I on the podcast, has said in previous podcast when you're a small organization, or maybe when you're getting your product manager for the first time, they own the QA function because probably you have nobody else in business that doesn't. Right, right. They're really doing V &V at that point, which is probably the better thing to do in that scenario where you don't have testers, right? Because maybe you are meeting some of the needs, hopefully most of the needs of that the business, that the customers have. But perhaps the quality is a little subpar. Right. You can get away with that. But you can't get it with the other way around. Perfect quality, complete mess, right? the other one will be, oh, uh, more people doing QA upstream just means slower decisions. And obviously more people. Like that, that will be used as a two -sided weapon, both sides somehow weirdly being used against you. Yeah. Yes, sadly. Yeah. Oh, boy. Funding early risk modeling has no ROI. Right. Supposedly, which is why people won't do it, right? But, of course, it's a boomerang, it's gonna come back and, you know, hit you at some point. The four is here. I'm going to throw them on the screen on my side. Risk visibility upstream prevents compounding wastes downstream. That's a Deming rule of 10. This is, this is only visible in learning organizations. Organizations that really think through the overall value chain and say, if we do this, right, this will pay off. But again, in most organizations, this is, conspicuously abs. I can't, but I can't even think of how I would try to implement something like this. You would need to know how much things costs ahead and then downstream. I can see it like an AI enabled, defect system. sorry, I'm trying to design the system on my head because I'm thinking of my business and I'm thinking of how, yeah, yeah, yeah, how would you do it? I'm thinking of how I would actually implement something like this. the way I would implement something like this is whenever a customer reports a bug or a problem or whatever. of like a way to orchestrate if you're QA, right? A way to orchestrate the breaking up a bug so that you can put your enhancements that can fan out and you can put a risk value to it. Say, okay, when we usually touch this area of the code, it's got, you 1 .8, you know, times chance of generating a bug rather than this other side of the code, some statistics, some actual metrics. Yes, you could. Also, I'm saying yes, and, what you could do is you could also say through the journey of your product, I was going to walking through it, try to map out a, impact pain, measure. So. the overall risk would be a product of your impact in pain. So if it's, you know, low impact, high pain, high impact, low pain, whatever it might be. That would help you map out where you should be focusing, right? So if you imagine that as a, as a line with, I guess, troughs, right? Yeah. Figure out the biggest troughs, like where is the biggest impact pain happening and try and reduce that. And if you work on I'm already thinking those. over multiple time periods, you can try and reduce the amplitudes of all of those waves, right, down towards the, the, the zero line. And you're improving quality at the end of the day. what you just said is kind of, it's in our, uh, it's very similar to what is in the bullet points here. the bottom one says Toyota's model embeds quality ownership at every phase in every phase. So sort of what you're talking about, a little bit different sort of what you're talking about. And then another one that's in the, in the same vein is what you're talking about, Go rat proved that systems fail where feedback loops end. So if you have a piece of the system and there's no feedback loop from that piece of the system, there's, there's your failure points. There's your bottlenecks is your first places to inspect. So visibility is key, right? Right. And if you can't, you can't improve what you can't see. I've always said that, right? You can't improve it. It's not there. So you have to be able to see it and then decide, is this worth chasing now compared to all the other things that you see alongside? Yeah, yeah, go red. Smart cookie. don't go chasing waterfalls. That's what I'm taking. That's my takeaway here, but, I want to move takeaways, but also. sorry, what, what, what is, what is the title is that, will businesses ever, will businesses ever move this risk upstream? Will they make a seat at the table for, like, the answer is no. I mean, I, nobody got time for that. I enjoy, you're just creating ripples and waves. No. Sorry, it's just, but that's the reality these days, right? Well, not just these days. It's been the reality for a while. Nobody wants to give you a seat at the table if you're going to come up and say, but, but, but, but. Right. Quality. But we're shipping on time. Get out of you. Yeah. Sad. Listen, oh, you're saying sad, but in my brain, I hear easiest job ever is. You just get a job where you come in and you're like, I don't think it's quality enough and they'll say, you can't risk our deadline. I'll say, never mind, it's great. I'll see you next week. See you next week. Right. Keep doing that for as long as they keep me on staff. That's what I'm saying. There's an expiration at the end of that job and it's one bad argument. That's what. what the expiration is. Yeah. takeaways on the podcast? Solid ones. I think so. So if you want higher maturity, you have are there real to embed QA earlier in your processes or your processes. That is an interesting word you just used embed QA, not bolt on QA. Right. So yeah, absolutely. You know, this is another one of those things where in process design or, I guess, you know, you could do this at, uh, the value chain analysis, that sort of thing. You should always be mindful of the steps that you're trying to minimize and reduce the timings on and reduce waste, etc. Like, what's the impact of quality? Yeah. How is quality going to be impacted? Let me put it another way. How is the quality going to be impacted as a result of eliminating this step? or reducing this step. Is it going to be impacted? If the answer is yes, then take a closer look at what you're trying to eliminate or reduce, right? Because it's a direct correlation to quality. So what do you want to do? Do you want to make it optimal but low quality? Or do you want to make it reasonable quality, but to heck with some of the timings? That is a delicate dance, right? yeah, that's something that you do have to consider. So most people are not going to consider it. So thanks for coming to the podcast. No one's dancing that tango. But if you are, if you are going to dance the tango, sorry, Why is his accent like that suddenly? I Brian, Brian, I know you're listening. Let us know. Oh, boy. All right, so my suggestion here, that was a whole tangent on number one, unbelievable. Tango tangent. So my, my suggestion here would to embed QA at the level of wherever the business assumption testing happens. Now, let me take a big setback for a second. Business assumption testing means when your business folks or whoever, have a shower thought in the middle of the morning and say, I should buy a boat today. Sorry, I don't know why they're a cat meme now, but when they have a shower thought, they should come to the team and say, hey, I have this assumption I need tested. Because we have a whole team to do that. And you can say, well, vibe coding now affords everybody the ability to test all the assumptions that they want. I mean, right? Theoretically is cool, assuming they could test against your users or a dedicated pool of users, right? Right. So there's some assumptions, but that's what they'll say is now I could run all these business assumption tests with whoever had the test in the first place and only bring the product. the output, you know, that shows promising, except these tools don't exist right now. Yeah, that's wishful thinking. That's the holy grail if they could do that and then you could connect those with real customers and their problems. Right. But that's right. Still out there. so that's number one is, hey, we'll, we need to embed QA at the level of the business assumption test. So assuming that you have a product management function. and that product management function has regular meetings, regular stakeholders meetings. Yeah. Or maybe, I mean, everyone probably has quarterly, right? But I mean, more often than that, you would have some kind of reviews with all the product managers where you're coming there and you're talking about the assumption tests that we're running. A lot of times, the UX folks do this part where they talk about the assumption tests that have been knocked down and whether there's signal in places and whatever. But I would say, This is maybe 20 % of companies have this level of formality. I think you're being generous. I think I am too, but I'm I'm trying to throw out in the broadest terms, assumption testing out of all the different ways, because I haven't seen the same way in every place I've ever been. Sure. But different places have a sort of pseudo version of this, some more in depth than others. And I would say about 20%, you know. I would say 80 % of people don't even bother with this. That's a huge number. Yeah, absolutely. then point two quality findings should be treated as business decisions, not bug count. Yeah, maybe even business impairments, because if you continue like this, it's only going to trip you up. Yeah. uh, risk reviews alongside or with your sprint reviews to be talking about knocking down risk the same time you're doing your sprint reviews. I would, the only, the only, this is probably the weakest item I'm I'm going to have today because the only thing about this one is the people that you need in the room to discuss risk to make risk related decisions may not be the same people that come to your team sprint review. Yeah, I agree. So I think, you know, it's, If you have risks and you may not have risks to, to discuss at every sprint review, but when you have risks, you already know you're going to have a risk, right? Or you have a risk. Invite the people that matter into that sprint review. Yeah. Now, you might say, well, you know, some of these risks, they don't concern the stakeholders. Okay, fine. Then take it off line. It doesn't matter. But at least review them as soon as you can with the right people so you can knock this risk down. Right, right. Please mitigate it. meaningfully yes and then the last my last point here in the takeaway is just very quickly i'm trying to get through these is afford some sort of seat at the table with process design and make that business afford q a process wide for maximum impact. Because these are, I mean, these, these should be the process people. These have been the most process, like the, the, I mean, you do the disc assessment. These are the high C folks in the organization. I would argue they have a knack for this. So bring them into it. Yep. Even if nothing else, you have their opinion, which is more structured and thought out than anyone else's when they were busy scrolling Forbes or HBR or whatever. Yeah, I'm gonna put something out here. I know you have four here, 4A perhaps. Sure. and some people, maybe a lot of people just say, you're nuts, but that's fine. I've been call worse. How about bringing occasionally some of your customers into these discussions with the quality? that's the only way you're really going to know what they feel. Sure. Polls and surveys, they don't really, especially if they're anonymous, you know, they can kind of disguise some of that. It's just something to think about. Yeah, I think you're right. Yeah, you're crazy. I know. Insane. Like, we can't, we can't even bother bringing QA people in a QA meeting. Yeah. You're going to bring customers in a bit of a stretch, right? Outrageous. Boy, if, For all the rest of you living on my planet. Oh, apparently my planet is in the, uh, the 1920s and a sci -fi teleplay. Wow, moving on, um, today. Bethany News. All right, so that's it for this category. So will leadership ever give QA real authority? No, you know, hey, we didn't think so either. Like, that's cool. No problem. We can keep pretending that DevOps is enough, right? Absolutely. And when it's not, blame product tops. I don't know. Oh, yeah. Right. Yeah, you may not, uh, write angry tweets at uh, Melissa Perry, uh, is on the internet. No, don't do that. Anyway, let's be fair. Like some folks out there, plus AI or product plus AI because believe that dev remember we fired all of our devs. they, we can survive with them and, uh, without QA. can we close that gap. don't like and maybe we're about to discover uh that velocity without validation is just chaos scaled leading to a whole new world aladdin style of opportunity for for people who really have these deep deep qa skills um this this is probably the the main thoroughfare of the podcast that i wanted to do the whole time this is the crux the crux the lynch pin can they uh maybe i the tip of the spear. This is it. Mishwini Penny. Oh. we have some citations here. DevSecOps Report 2024 and, uh, Google's research, GitLab, uh, uh, their code review quality study from 2023. I have two stats here. the Git Lab DevSecOps report, devs report 40 % more QA tasks, but 56 % higher downstream incidents after the QA loss, the QA roll loss. Mm -hmm. So 40% more QA tasks and a 56 % higher downstream incident. That's significant, I think. Because remember, we're only talking about the chain thus far. And then there's beyond, right? So if it's 16 now, you know, it's only going to be higher later. It's 56 % higher downstream. Yeah, so it's 60. So they're saying 40 % more QA tasks. Yes. But 50, oh, yeah, sorry, I misread that. Yeah, 56 % higher. Their task, their, their QA tasks, meaning that the amount of tests they need to run that they have to do. Yeah. It's 40 % more than when they had their QA people. By the way, when they were offloading the role. So they're basically doing. almost double the work that the QA people were doing and they're doing it without the QA people and they're because of that they're generating 1.5 times more downstream incidents it's a lot worse than I was painting and but it's not surprising to me well you'll be surprising to me what would be surprising is they're generating 56 56 percent higher downstream incidents and then not fixing them that that would be that would be more surprising because you think the you got rid of your qa people they didn't want to management decided pay sixty dollars an hour they wanted to pay thirty eight dollars an hour or like sorry it was like forty four dollars an hour trying to think what the real numbers were back in the day yeah yeah yeah they were ridiculous um coordination costs that plus plus the cost of rework, which we already know is over like nearly 40 % the cost of rework. So you're generating 56% higher downstream incidents plus 40 % rework on top of it. Plus more if your AI enabled and you don't really know what change. You just have to go back to the AI and hope. Yeah. the slot machine hope see our last podcast yeah um it's it's i can't even i can't what i'm saying is i can't even count that high home i'm asking for mathematical help maybe a data sciences could help me someone or somebody somebody with an abacus if they're not too busy having their integers all stuck failing as decimals when they get converted it's that and the other study here that we have, the Google research code review quality study 2023, even expert developers miss over 50 % of functional defects during reviews, which lines up to our 56 % higher, item from the other study. Yeah, these numbers are huge folks. These numbers have a real life impact on your business. Right. Right. So think about this. Do you want to be short-sighted and just, got your loss because you're gonna be gone in a year or whatever or you're gonna basically ramp down this product come up with a new one and force everybody to upgrade in the sort of vain hope that these issues aren't there i mean there are so many things out there like that that i've seen instead dealing with the real problems right um so yeah shocking numbers really well it is shocking numbers but also um it wouldn't be the arguing agile podcast if there wasn't somebody crying wolf over here saying listen product can now own development and qa the developers were owning qa and it really wouldn't take that much of their time in the first place and also everything you every feature you implement you can spin up a playwright script at the same time and contribute it and say we can put that in the prompt don't don't do any coding without also doing a test at the same time and now you the the lm is handling you know it's handling a product can now own all development and all of QA and they just have a team of AI assistance or agents or whatever you want to call it. so, okay, first of all, if product is doing all of that without having the core skill set that the, QA folks had, or even developers for that matter. That's one thing, how good can they be? The other thing is, what are they not doing when they're busy doing this stuff? listen, oh, my product manager was that lady from LinkedIn that we watched the TikTok on and just got lunch all day and then had like a one hour meeting with her development lead out he's the one really is the product manager. Yeah, right. If I get angry, it's because the other against that you'll hear are as pedantic as the first one. QA was already made redundant. Oh, it was made redundant with DevOps. Good grief. Everything, everything gets checked through the CICD. And back and then, like, we have like our one. SDET for every two, two, two, three teams. Yeah. And he just automates what the dev team tells them to automate. So the dev team goes in the room, they come up with the stories or whatever, and they come out of the room and they just feed the SDET tasks. Yeah, again, you're only, you're only focused on passing tests here. Right. You're not really looking at how is your product or product increment? fit for purpose. You're not looking at that. You're not looking at the validation verification against real customer needs, right? You're not doing that. QA has those skill sets. They can look at those things, but you've already got rid of those people anyway. No, I mean, like the section category here is developers are doing QA with with a little AI help or product management. Basically somebody because we have AI, literally anybody, anybody could, like a person off the street. could be doing QA for us because we but what are they doing really i mean we're calling it QA but what are they really doing no you're right like it's there's you're completely justified in calling out the hypocrisy of what's happening in this like you're just looking for someone to check the box and sign their name literally anyone yeah doesn't like you you completely undervalued an entire professional side of software development and it's self-fulfilling if an AI agent can be that quote unquote person or entity because then you could just say who did that you know there's another there's another is the i that did that there's another side of this that uh is hot on the market uh it's it's kind of in the same camp of the people saying that product managers will just code everything in the future it's well one well maybe we'll have shops where you only have one or two professional software engineers and they're enabled with a whole bunch of AI tools and agents and stuff and then everybody in the organization writes code and those one or two professionals become like a not really a shared service but they become like the and they're not even necessarily the gatekeeper but they become like professional checkers kind of thing yeah yeah it's that that's an interesting one because maybe this one has some legs, right? The problem code coaches that is, one dev. for all of these other needs, like that, what are they doing? They're going to be overwhelmed. So they, they cut corners. I hear you. I think that one's a good against because the, the argument for it, the pushback for it is, you guys in corporate America, you're just going to screw this up like you screwed up every other ratio. You said, we're going to have one QA person split across two development teams because we don't really know what they do in the first half of the sprint. That, that kind of thing. Yeah, yeah, yeah. And then that leads to, we're just going to fire all the QA people from all the teams. We're going to have one QA person across eight teams, but that one QA person is going to be, they're going to have an automation engineering experience. So basically they're just going to write a test plan and automate everything the team does. And because it doesn't matter who writes the automation test could be the development team. They basically architect. So now we're going to create this cute. QA architect type of role where it makes you sound like you're like a dev architect yeah but of tests and then the developers implement the test that you asked them to do but the problem is since you're over eight teams you can never guarantee anyone's quality that you never go check stuff yourself because it's just even you wanted to it's too much yeah yeah yeah and at the end of the day all you're doing is just again focused on passing tests yeah So there's, there are some, there are some real arguments you'll see in this category. Those are real positions that I pointed out that I've seen or worked with in the past. but if we're gonna take a stand on this one, boy, the, the get lunch all day product manager that I pointed out that it's a real thing. These are the people they're gonna ask. Sadly, it's a real thing. Yeah, sadly it's a real thing. No, like my, I, I have no good rebuttal to this one home. I will just blankly tell you. I have no good rebuttal other Yeah, they're very expensive lunch getters. about the economic downturn that i was happy about is those people were gone for a while and now for some stranger's in their back and i can't explain yeah and also oh also where do i sign up short -term illusion. in long-term regression. Slippery slide. That's what this is. Yeah, absolutely. I mean, we talked about this already in this podcast, but, you know, let your devs, dev, dev led QA, the right? I mean, that's what they excel at. Don't add more to their already full plate because they're doing this and they're doing all these other stuff. In some organizations, I've heard arguments for getting rid of DevOps to say, you build it, you run it. Guess what? The devs are building it. They're the DevOps. They're also the QA. What else are they? Yeah, right? Crazy. Absolutely crazy. That is crazy because I, I lobby for my development team to do the DevOps work. I like to do that because, but also like, I wonder how, like how much shaking of my fist out clouds I can do, over the years. I know that that stuff's not super difficult. again, it is if there's anything cryptic. but, you can write that stuff to a template. You can say, hey, if I have to add a Q, I need these rules. If I have to deal with the S3 file, add these rules. Especially with you, with AI, you can do that a lot easier for sure. And AI is very good at that kind of stuff. It's all procedural. Yeah. And the last one here, AI can't validate the user's intent or the risk to the entire system. Again, maybe in the future, the people that are saying, oh, these tools are only as bad as, you know, will only ever be as bad as they are right now. Maybe those people will win because maybe something in the future will happen where the, context is much larger than today, you know, it can take in 1 .2 million lines of code as the background. Then you can ask you your question, but until you can integrate with a bunch of other systems or have agents that, I do this now because I have features in my site that integrate with social media platforms. So if I want to do one thing, hey, I want to post it a social media platform. Well, now I've got a dozen social media platforms to run that test against. They're all different paths of the code. does my code know how to get to all of them? Like, these kind of things. Yeah. The AI is not great at running the entire risk throughout the entire system. Today. Today. Yes. Yep. Yes. And, you know, that's, I mean, that's up in the air. I mean, that 200K token count. I mean, there's a few vendors that moved up to a million token count, but, uh, not all of them. Right. And, and, and definitely none of them through the front end UI. You know, all, all of them moved up, moved up through the back end, um, API. Yeah. So open your wallet. That's what I'm saying. What's in your wallet? Oh, nothing. Yeah, I took it all. in order to escape the black hole of QA without independence, right? That was one, one word, right? QA without independence. you want to trace the escaped defects in this no QA world. That was one of our earlier suggestions, so hopefully you're doing that. You want to reinstate independent validation for complex flows through the system. Hopefully you know what those flows are. You know what the complex pieces of your system are. if you don't go back to number one on the snakes and ladders map out your processes first that's right that's right if you do listen if you don't know how it works you have a very difficult time and i feel that that that goes back into the probably the the reason to have this podcast is i just vibe coded i don't you know nobody knows right how it works and then the final one we redefine what ownership means so we talked about the I don't remember if it was the Cooper bench where it said, the way to get the AI agents to talk to each other is to clearly define ownership structures. Okay, these are my sections of the code. I won't change those functions or whatever, those files or both, right? I'll, I'll pass over to you and ask you what I need changing and you do it. And that, that's the way that the system communicated. But this also works for people. Sure, it works very well with people. these are the decisions you can make home. These are decisions that I'll retain control over, and then, you know, never the two shall meet. Listen, as a formative habit, this would be a good place to start with people first before you start entrusting the AI to do this for you. Because you're the one feeding the AI to begin with. So yeah. Oh, man. Drink your own champagne or eat your own dog food, whatever it is. I'm kind of hungry. I was, yeah, not for dog food though. No. Maybe champagne. So, is. dev owns the quality or product owns equality is it a myth that we keep wanting to believe because it saves us a few pennies let us know in the comments what your opinion is on this cool all right so if neither a i nor dev solo can handle absorbing the qa function right by themselves what does our bright shiny new future look like maybe the winning model is hybrid human in the loop um is uh It's kind of complicated so falling out of favor in the modern kind of, AI world, human in the loop, especially with the rise of agents, because we have to promote that it. Human in the Loop early on with AI was a big. it was a big thing. Yeah, until, until agents came along. It was a big Yeah, until agents came along and then everyone said, oh, the agents can just do everything cheaper and faster, faster, faster. But let's think about this critically for a minute. Yeah. study. McKinsey Technology Report 2025. There it is. Human AI collaboration increases release confidence by 30 to 50 percent. Release confidence. That sounds like a good thing. It sounds like a good thing and it's probably from the there is a McKinsey perspective of the people releasing rather than people. that are being released too. I'd be interested in figuring this out a little more because yeah, sure, you have higher confidence if you're the, you know, the releasing party. But how is it landing, I don't know if they even measured that, but that would be an interesting comparo to say, yeah, we're very confident 30 to 50 % these are great numbers, but is it landing right? Or are we just shooting blanks into a vacuum? I don't know. The state of AI in 2025, agents, innovation and, uh, funniness. I don't know. It's key findings. Most organizations are still in the experimental pilot phase. have been some cuts here where we went to the McKinsey study to actually look at the graphic so we looked at this study and, um, that the data presented here. I'm not super happy with. Go, go figure right. Brian's not happy with the, how a data analyst presented stuff on the screen because they probably were just told by their boss. That's how they wanted to see it. And it says, human in the loop. have defined processes to determine how and when model outputs need human validation and ensure accuracy. All respondents out of this group, 23 % said they had a human in the loop as part of their strategy. Shocking, really. I would expect that to be much, much higher. Now, they divide the number of respondents. They slice that group into a smaller subgroup and they call it the... AI high performance. And they define AI high performance as respondents who reported more than 5% of their organizations, EBIT or EBIT. No, uh, I know. What happened? But five percent of their organizations, EBIT as quote, significant value attributed to the organization's use of AI. So even that is subjective, but the number, if you just consider, if it's, 5 % or more of your EBITA, um, If you just consider that as the divider, I'm not quite sure that is a divider, the number jumps from 23 % of companies using AI have human in the loop to 65 % of companies using AI have human in the loop. And again, I, I would only do these backflips to get these obscure numbers and everything for people to listen to if I didn't, work in the industry and I know that the industry is pushing back hard right now on human in the loop. because in my opinion, they see it as a slowdown. They see it as a bottleneck, sure, yeah, a bottleneck to be eliminated. in the, in the, um, yeah, it's right in the middle of the against here that I'm showing on the screen. It says, human becomes a new ball, human oversight becomes a bottleneck. You're the bottle of QA's a bottleneck home. They're just too slow. So we get rid of checking for defects. We'll never have any defects because we, fired all the people that report the defects you you see um if we fire all the people that take the reports of sexual harassment will have no sexual harassment in our organization that's what i'm pointing out right yeah there'd be no need to redact anything there's no look there's listen there's no need to write anything down if nobody ever does anything wrong that's right exactly my goodness uh so that yeah i mean these uh these numbers from uh from our source, they were quite obfuscated to begin with, but that's in, you know, it's in keeping with what you expect from that source. Yeah, yeah, well, you know, from that source. well done, well done, well done, well done, well done, well done, well, John Perw in the house. I name no names. Uh, humans in the loop is, and then the other, the other side of this will be humans in the loop home is just a, temporary phase because remember the tools are only as bad as they're not that one again oh my goodness it's so good it's so good i can use it in every i can use it in every category here the AI test in the AI can test automatically faster and cheaper oh okay and those tools are only ever going to get better so get off my back with this arguing point that that's yeah i think that's sort of analogous of saying well humans will be you know as stupid as they are now and no more going forward. I, I hope that there's one thing that says with people coming out of this podcast is that that bullet point going, getting used over and over now. You will hear it now sorry, there may like nails on a chalkboard the way that I hear of the tools are only ever going to get better and cheaper. that they, if we give all the money to the richest of companies, oh, it will trickle down. No, the trickle down economics. Hey, look, for all the AI in the world, there's an awful lot of, uh, NS, natural stupidity. And I do not see that degrading over time. I only see that increasing because people believe all the stuff that, eh, they don't understand, like, I don't know, a report we might have just recently covered. And, you know, a lot of hype in the, in the news cycles, right? AI is here to solve all your problems. It's only good for you. Yeah. Sorry, is that like NS, meant something else. I was thinking the whole time. Oh, my goodness. Yeah, never came up with anything. It's, so sad. our stance on this one, uh, is fairly straightforward and might I add sane, is, listen, the hybrid QA and AI teams, they're going to show better outcomes. They're going to show better outcomes because you kept your people that are focused on that thing. Yeah. Right. Even if you try to just scale them. right just keep your people with that skill don't throw away all that quote tribal Yeah, tempting as it might be. Yeah. You know, humans decide what matters, and then the AI scales how fast. So the, the AI says how fast you want to go, right? And that, that depends on how much money you want to, how many tokens you want to buy, right? And then the humans decide what matters out of that. Because again, even if you're producing 15 bagillion lines of code a day, if none of it matters, you're going to throw it all away. This is that race car or sports car analogy, right? I mean, how fast do you want to go, but it's a human directing the vehicle in the first place. and the last one is, uh, QA skills. equate to orchestration, right, not execution. AI agents do not possess that knowledge about what the pain points are that you're trying to solve for your customers, whereas QA skills can be grounded in that. So that's one of our, you know, parting takeaways from this. I mean today. Yeah, right. So I'm saying, until you pay me otherwise. That's awesome. Oh, boy. Oh, boy, yeah. Oh, boy is right. what is the takeaway for this section? Well, I'm glad you asked him because if you're becoming the quality orchestrator or whatever other nonsense title you have to make up to bridge the gap for right now. Number one, you want to stay in the loop. So the loop meaning the human in the loop, loop, you want to own the interpretation of the output. Mm -hmm. Okay, that's, that's a lot more difficult than it sounds. It sounds pretty straightforward. Just get yourself in there. No, because again, everyone can use a tool. So literally everyone in the company can question them, Brian, you don't know what you're talking about. I ran a test in the city fine. Right. Yeah. So I think experience plays a part here, right? So a savvy QA person could say, well, what test did you run? That's right. Let's look under the hood. Yeah. Right. the other thing obviously that goes about saying is you need to learn the latest and greatest AI tool configurations, metrics, evaluations, you know, how to use models that kind of like, you need to go deep in this technology. Sorry, y'all. I hate to tell you like that you need to learn the things that are taking your jobs away, but you really should. that's that's that again i'm not gonna spend a lot of time there because you need to know it okay it's being used against you you might as well learn it and then the last one is uh you need to tie your validation metrics to user and business outcomes and outcomes or impact or risk or financials this is a regurgitation of what we've earlier said tie your validation metrics to things that business cares about this one never gets old because it's always going to be true yeah so yeah well Speaking of things are, are we going to be true with QA's survival playbook. so let's wrap this, this discussion about QA, home. And, let's, let's wrap it by revisiting our goals. Sure. from the very beginning of the podcast. so we said, uh, QA's collapse, we talked about QA's collapse, being a design failure of modern organizations, not necessarily a skill failure, like the QA people didn't degrade in their skills, right? and then, AI, when we talked about AI and that it should magnify the value of QA skills and, and we talked about how if the next boom happens, testers, because they're already skilled in systems thinking. and, uh, how they may be uniquely positioned to come out on top of whatever ends up next, where we're all, we all have one software to Homer that we share with all the rest of our teams or whatever. I think the core skill sets that a QA person has, um, is going to prevail. Not, I'm saying it as singular, but it's skill sets, right? It's a set of skills. they're going to always matter, I think. I think what may change is, the titles, etc. a real QA person, I'm not talking about people that's simply care about passing or failing tests. Yeah. those people bring value and, uh, it's, you know, it behooves us to recognize that and incorporate that into our value chain. this was, uh, interesting podcast. I, I, I had a discussion and like I said, I went on Bob's podcast, software quality and beer by checkpoint technologies. And, it got my brain a spinning and all kinds of things that we could talk about in this category. And I feel, again, I could say on this for a long time. I think we're going to stop here just because that is the logical ending of our discussion. But if you want to hear another one like this, trust me, I got more. I got more things to say. If you have more things to say, I'm pretty sure our audience does too, so please let us know that in the comments below what you think of this one and what else you'd like us to delve into. We always value your opinion. Also, while you're there, like and subscribe.