S6 E5 - Farhan Sheikh – Reducing Variation: Reimagining Quality in the Age of AI

I have a conversation with Farhan Sheikh in this episode. We dive deep into W. Edwards Deming’s System of Profound Knowledge and explore how its principles can be applied to modern software development and AI-assisted coding.

Farhan shares his unique journey from developer to QA leader, where he applied Deming’s philosophy, particularly variation, theory of knowledge, and systems thinking, to transform how teams approach quality. Rather than relying on inspection at the end, he emphasizes building quality into the process through better test design, tighter feedback loops, and collaboration between testers and developers.

A central theme is Farhan’s “Darmok” approach: using structured examples and test cases to guide AI systems like Claude, instead of relying heavily on prompts. By feeding AI consistent, well-sequenced patterns, he demonstrates how teams can reduce variation in AI-generated code and achieve more reliable outcomes. This mirrors Deming’s principles, controlling variation and improving systems rather than reacting to defects.

The conversation also explores the evolving role of QA. Farhan argues that testers, with their strength in asking critical questions and creating clear examples, are uniquely positioned to become key players in AI-driven development. This shift blurs traditional roles, pointing toward a new kind of multidisciplinary “AI producer” who integrates development, testing, and system-level thinking.

Psychology and leadership emerge as critical factors. Farhan candidly discusses overcoming fear within teams, aligning motivations through data and experimentation, and the importance of leaders enabling systemic change rather than reinforcing silos.

The episode concludes with a powerful reflection: true digital transformation isn’t about adopting AI tools in isolation, but about rethinking the entire system of work through Deming’s lens, focusing on quality, reducing variation, and empowering people to learn and adapt.

Notes:

Linked In: https://www.linkedin.com/in/farhan-sheikh-fivetwofoureight

Github: https://github.com/farhan5248

Website: https://specificationbyprompt.com

Transcript

John Willis (00:00)
Hey, this is John Willis, another Profound Podcast. you know, some of them are, you know, every once in a while now these days, I've got my feet in a lot of different subjects, of course. So even though this originally is a Deming podcast, it sort of floats all over the place. I got some quantum ones coming up, a lot of AI ones. But this one gets us closer back to the original 10th, which is it starts out as sort of a Deming conversation that leads into AI. Farhan you want to introduce yourself?

Farhan (00:25)
Yeah, so I guess I'll go backwards. I'm a software developer. guess the last thing I did was I was a manager, a dev manager of a QA team. And there were like about 50 people in the team. Before that, I was a dev lead and a developer. I've done everything from deployment, automation, architecture, coding, testing, management. And I guess I just bounced around trying to solve problems. So I haven't spent too much time in any one thing.

John Willis (00:49)
That's cool. Yeah, that's awesome. Anyway, yeah, no, so good stuff. sort of the first sort of context here was when I did an original book club on my own on my Deming's Journal Profile Knowledge, and then I did one through IT Revolution. And I can't remember if you were on the original one or the IT Revolution? The original? No.

Farhan (01:14)
not

one the first one. I am part of the current one

John Willis (01:19)
yeah. and the thing I sort of like really enjoyed probably, there's a bunch of things I enjoy about what we're gonna get into today, but then also you had like the most in-depth questions, you like you'd hit me with a, which is cool, right? Like, and I know like, I love the kind of questions where I just don't know the answer right off the top of my head, right? Like, geez, that's a hard question.

And so it was, you were very engaged and I thought that was pretty awesome. And it was a great group of people. I know a lot of them still continue on, I've done three or four, maybe five. In fact, we did my AI book after that, but I know Farhan is constantly. So I guess the thing I wanted to go into is, you've got this sort of idea that you created this Darmok project.

And just what's the elevator pitch on Darmok

Farhan (02:13)
So I think it mainly has to do with Deming and QA. And my point is that if you were to build quality in and do this whole shift left thing properly, rather than inspecting in, a lot of the stuff that you would have to do that's not on test driven development, it just feeds into getting AIs to have less variation and better quality because the thing can work really fast. You're not going to be able to do mass inspection at the end.

Right. And I guess I'm a hammer looking for a nail. So, because I think what Dr. Deming is saying makes sense. The problem with all the white coding stuff is that it does this too much variation. So I'm like, if you want better quality code, you have to find some way to control the variation. And whatever I did with my QA team, I just feel it is continues to apply to AI, right? Instead of just trying to write a lot of code really faster than inspecting it in. And I guess the idea of Darmok came from, I mentioned, when you use the analogy of arrival,

Instead of trying to actually type out all the instructions and tell Claude, like, want you to do this or that, but here's how I want you to change my code, and that's what I want the methods to look like. I figured if I just give it patterns and communicate to it by saying, here's an example of something that worked, and here's test automation to make sure you don't go off the rails, it seems to do a pretty good job. And so I just had to keep giving it a sequence of similar examples. If the examples are unrelated, it doesn't do as good a job.

The more examples are related, the more similar they are, it does a better job. And it's just a matter of sequencing all of that rather than actually sitting there and prompting it. I just fire off a whole bunch of test cases and the test cases are written a certain way.

John Willis (03:50)
Yeah, no, think there's two things I love about this one, know, so the general, I feel like that, ⁓ you know, I'm sort of cementing is a strong word, but a place in the transition of deming knowledge to people, right, or specifically people in an area of technology that probably doesn't get exposed to deming, deep deming ideas, right? You know, so that, and so, and I think that, so I've been like,

You know, it's a small army that's been sort of an outcome of my book, The Book Club, right? A powerful small army, and you're one of them. But, you know, when I was writing the book, or telling people I was going to write the book, I got three responses from people, which one was, never heard of the a guy, right? The second was, you know, I really should learn more about this guy. And then the third was like, I know Denmig, love Denmig.

But the truth about it is even in that third category, very few and almost none that were in our field. Like if I just want to say Devops, DevSecOps, I certainly really understood. know, they were really good at the quotes and the quotes were amusing and Ken were definitely influential. But like what we broke down in that sort of book club was more than what just was in the book.

The people were literally, you know, we had people come in and actually give demos of SPC and statistical cross control. So, one, I love that you've taken the torch, particularly on SLPK. And then the second thing I love about it is you've been influenced by my second book, which is the history of AI. I tried to just take pure in there. I didn't really do a DevOps related AI except for the last chapter.

which I sort of focused on sort of the movie arrival, as you said, and that was this interesting thing about storytelling and how storytelling and you picked up on that and so the epilogue and if you haven't read the epilogue, I I encourage you to read both books, but look, but, you know, so, and you said that, you know, that was, you know, pretty cool on, but I thought what would be really fun then, so, you know, that backdrop is that

you were one of sort of a handful of people who not only attended the book clubs, but you also tried to put into practice what you learned. And let's be clear, Deming stuff is not trivial to It's easy to talk in the meta language and stuff at the meta level, know, joy and work and, you know, implement systems thinking. But like when you get down to

the nuts and bolts. It's, you know, it's just, particularly in a field where you don't really have any sort of, you know, in healthcare, there's tons of examples and manufacturing, of course, but in our world, there's just not a lot of examples.

Farhan (06:38)
So I guess there's two things I wanted to talk about. One is why did I like your book? And the second thing was why Deming, right? So the book was, so I studied software engineering and I learned all these things like graph theory, predicate expression, calculus, discrete mathematics and all this stuff, but we never thought about AI. So when AI came around, I'm like, I'm gonna try and learn it the old fashioned way. I went and found some coursera course or something, was boring as heck.

I gave up after two days. And so then when your book came around, I liked stories. I liked reading a lot of books. And it's the same thing, like, just as I tried to give Claude a whole sequence of tests, your book sort of, that's why I told you it's like, you could write a children's version of the book, because it was easy and digestible. And every day I could pick up a story, and every day one story led to the other, which helped me develop an understanding, without me having to go and understand.

the science and the math behind it, right? Or the how? Is this like, okay, when I understand the history of how we got here, then I developed the understanding as opposed to someone who's giving me the pure math or the science behind, this is what AI is. And just one way worked better than the other. I guess I'm just too old to go back to the way I was studying 20 years ago. So that's why I enjoyed it. And that's why I thought, okay, and so even the whole arrival thing or the Darmok thing, the team that is it storytelling, when Gene was on Dave Farley's podcast, he mentioned something about onboarding your agent, right?

And so when you give a whole bunch of stories, it's kind of like that, that I'm actually on boarding cloud to have a concept rather than typing out exactly what I want in terms of Java jargon. And then it is under somehow, I guess in its head, it forms some sort of a mental model.

John Willis (08:09)
Yeah, no, think it's really interesting about that. We just talked about how hard it is to implement. It's sort of the dichotomy of Deming, It's sort of easy to grock the concepts. They're like simple. They're simple. Get along. People first. These are like, yeah, yeah. But then the implementation is hard. And I think you pointed out something interesting, which is a science

in software development and QA and all like that is going to tend to approach these disciplines in a very sort of, you know, like you said, mathematical, but certainly, you know, deep engineering principles. And sometimes you have to sort of balance that simplicity of like what, how do I step all the way back? And I thought that was cool, but I thought it would be fun is to break down like what, you know, what, what you've done here and what you've learned.

Farhan (08:50)
Yeah

John Willis (09:05)
through the lens of system of profound knowledge. And just quickly, anybody listening right now, I'm going to do the quickest version. Go listen to another version. Go read my book if you want more. But there's really four, there's a lens that Deming said that you have to break down complexity and there's four elements to the lens. And I always like to put them in this order. I've explained in lots of places why, but theory of knowledge, which is epistemology, variation,

So theory of knowledge is how do we know what we think we know? Variation is how do we show what we know or the output of what we know? Psychology is all the human aspects of the biases, the sort of can't we just get along when we put all this together? And then finally, to me, the encapsulation of all those sort of like the, they're all equal parts, but this sort of lens that is this discipline that ties everything in is system thinking.

So I wanted to go ahead and start off with sort theory of knowledge.

Farhan (10:04)
Yeah, so I'll give a similar quick explanation. So when I joined the QA team, I guess once I'm what I'm doing with AI and Cloud, I have to explain why I wound up in the QA team. So I'm not a typical tester. I'm a developer. And when I had the QA team, I had zero QA experience. But just to go through the four things, the first one is the theory of knowledge. I didn't want to tell people what to do. I wanted them to do the whole PDCA.

John Willis (10:07)
Okay, yeah, good. Yeah, yeah.

Farhan (10:29)
and follow the scientific method to figure out like, this is how we're gonna determine what to do. Even though I had no QA experience, I'm like, folks on my team had 25 years and 30 years of knowledge. I wasn't gonna tell them how to do their job. it was a lot of bias and thinking they thought they knew, you know, like if you read the book, The Goal, and people say this is just common practice, but why? And then comes variation. So for me, the aha moment, so it's 2018, I have no one to talk to about this stuff, so.

I'm basically just making it up as I go along. And what I found out is that I focused on how long it takes if I treated the testers like a scientist, because I read Dr. Spears work and I'm like, okay, they're my community of scientists and they're conducting this experiment to figure out better way of working. The thing that I was trying to figure out my special cause variation was why does it take a tester so long to confirm their hypothesis in the test case?

And we tried to bring that down. It's still like, okay, there's a certain time, basically I'm saying, how long does it take you to run a test piece and make it pass? And so it's like, why does it take you so long? And then comes the psychology of change. When folks started understanding the problems, they were okay with this idea that I wanted the testers to go and work with the developers and run the tests. And I wanted them to do something for the benefit of others, but they had to develop that understanding. If I walked up to someone and simply said,

John, I want to go help someone else. It's going to make your life difficult, but it's going to make their life better. But trust me, this is good for all of us. That's a hard sell. But once they started seeing the data, they started conducting their own science experiments, they came to the conclusion that, yeah, this person on my team specifically, actually asked me, is there a way we can actually go do that? And like two years ago, they thought, that will just take more time. And then comes systems thinking. Once they started doing that,

Right, they could see that optimizing locally and making inspection at the end more efficient was not the way to go. They had to take a little longer to do a better job writing the test cases so they could let the developers take longer. Right. And that meant that the developers would have fewer defects and they would need less time in QA. And altogether, everyone takes less time. Right. So it kind of did work out that way. Yeah.

John Willis (12:32)
You

just nailed the whole, you know, we talk about, you know, um, debbing being hard and system knowledge being hard to sort of implement, but you just sort of like nailed it in summary because, know, because they all just intertwine, right? Like your variation is so entwined with the not the epistemology or the PDSA or whatever we want to call it. By the way, I, I played with chat GPT in preparation. found like now I can get so much better interviewing now.

people and by sort of, I'm not going to call it cheating, but one of the things that ChatGPT called your system is epistemological scaffolding. I thought that was brilliant. Well, I think what it was talking about in your theory of knowledge, I want to get into, I want to break down each one of these things because I want to meet here. But yeah, maybe it will make sense. But I think I just want to like sort of close the loop on what you just said, which is really awesome.

Farhan (13:11)
What does that mean?

John Willis (13:26)
because they all work together. And you gave at least two, maybe three examples of how they're like, in some ways, they're four different sort of elements, but they're really just one big old lens. And the one example is variation in knowledge. But the other example is you talked about how psychology and systems, like telling somebody that you need to make their life better.

but their life is going to be worse. But so psychology isn't as simple as just, you have a bias and therefore you need to fix it. It's sort of melding it with like, we need to see the vision of why this thing makes sense for everybody,

Farhan (13:53)
Yeah.

Yeah. And I mean, you tell someone it doesn't, so you know what psychosynch is. And when I was looking for information on how to deal with the psychology of change, I went through all these books, you know. So to explain why people need to help one another, have to like the book says start with why and I had to, and then they need to find their why. And so SPC helps them find their why. It helps identify the biggest bottleneck, the thing that was slowing them down when they

wrote their test cases. So it's basically a claim to dedication and it's an insane amount of math. And then, you know, like when you think of the Toyota story with the loom where they have to redo it. So now these folks have to redo all of this. When the bottlenecks of executing went away, they could see that it was in their best interest to go work with the developers so they don't have to waste their time rewriting, right? So Simon's right. You have to start with the why. They have to have their own motivation for the willing to go and work for the benefit of the developers.

John Willis (15:01)
No, no, this is really good stuff. mean, you know, I think one of the other things, the point is, think that, you know, I think like minds attract, but you're, I just did a podcast with Glenn Wilson, who wrote the DevSecOps book, just got his master's degree. And, know, the thing I think that I'm just so interested in the people like you and Glenn and a ton of other people, but we're just insatiable learners. You just mentioned like four, like bodies of work, you know, not even related to my, well, related to my stuff, but not directly.

from my stuff, know, Sinek stuff and obviously the Toyota legacy stuff. But all right, so let's dig in. Let's Theory of Knowledge you now, right? I think, so you've covered a fair amount of this, you know, so like trying to get people, your sort of hypothesis test observation refined your, like that's clearly like, ⁓ I think the cool thing from what I understood is this idea that you sort of were getting down to the, you know, sort of like removing

the code, not removing the structure, but removing the code, running the test and letting sort of Claude sort of control the cognition or the sort of, like it was like letting Claude be a PDSA engine, if I think I'm saying that right, right? And I thought that was really fascinating of a sort of Deming implementation. Again, it's easy to say, hey, do PDSA.

It's another thing to say, how do I do it in the real world? It's even more interesting of like, how can I use something like AI and specifically Claude?

Farhan (16:29)
Because so when you give it the test, and so I give it a previous example, right? And that it looks at it and it understands because when you have code coverage tools without getting too technical, people want to look at the production code. But I had it trace everything from the test code. And so now it looks at what the previous test was doing that is similar and it's slightly different. So it comes up with a plan. Then in one shot, if you're watching the log, it says, I know what I have to do. And it just does it.

checks by running the test and it's like, oh, I missed that. And it acts and it corrects it and then it's done. And tweaking the information that you give it so it can plan properly. Like if you give it two bigger tests, it goes all over the place and it's doing PDSA the whole time. And if you give it two smaller tests, it's too ambiguous. Now it's just guessing. If it's two bigger tests, it's like chasing butterflies all over the place. So it doesn't really know what thing to focus on.

I was trying to figure out what is the sweet spot that the tests are small enough that it just gets it right. And most times it takes three to four minutes.

John Willis (17:29)
Got it. And so that's an experimentation in general or experimentation that you've learned and now you've kind of honed it in on sort of any project?

Farhan (17:38)
Yeah, so that's what I'm hoping to get you. Like I need to collect all the log file timing and you know, even when you put things in Git tells you how much a file change. I'm trying to figure out what should I collect what data so that I can look for the variation because. Yeah.

John Willis (17:52)
This is the segue. This is awesome, right? The segue is the, well, then that's really cool, right? Like again, getting Claude to be the PSA. And one of the little quick questions for myself too, because I'm not, you I don't sit and code all day and like I have coded, I dive into this. I do way too many things to be great at any one thing, right? But.

But I like the idea as I dug deeper on what you were doing is like you seem to be focused, or at least right now you're focusing on the method level. Like, you don't leave the code, leave the class, go ahead and delete the code. And it seems like that had become somewhat of a sweet spot for where you're in right now. Is it an accurate description?

Farhan (18:33)
Yeah, because I think creating classes, so lot of the, all that code was actually created with me prompting Claude to do stuff. So it can actually create the classes if you tell it to do something. But the part that it was struggling with was implementing the logic or the bodies, right? So like, some more background. I was, I've been playing around with all this UML code generation for almost 20 years. And so you, there are tools and all to help you create all the classes and all the methods, but the hard part is the body because.

No one liked to draw a finite state machine to implement the method body. So for me, the main thing was if I want to have, if I want to reduce variation and code generation, I'll use some old fashioned tool. I don't have to use AI for everything, but the old tools couldn't solve the problem easily of figuring out what to put in the method bodies. And so that was to me the hardest part of the problem that I was most interested in solving. And now I'm just going through trying to figure out what, you know, how to get it to put the rest back and it can.

I realize, so I don't really care too much about the classes. I care about the log file entries because if you were having AI reactor code, you don't want to keep screwing with the log entries. Otherwise, how would you monitor anything in production? If it kept moving everything around, you wouldn't be able to time things if it just kept changing the log entries, right? So my only concern is logging to monitor, observe things in production. But I feel just like I don't know what the compiler does when it writes assembly code.

I'm caring less and less about what the classes look like. I can care about is this doing what I want it to, does it understand the logic? Because I don't want to give it a million test cases to specify exactly what I need it to do. I need to give it enough test cases so it makes something consistently.

John Willis (20:10)
Yeah, it's awesome. You know, it's funny too, like, you know, I felt like early on I had to defend putting my SLPK in order, you know, and like you're making the defense for me right now that I think theory of knowledge has to be first. You know, I mean, the argument that the deming purists will yell is like, they're all equal. How dare you do that? But there is an order and I, you know, and I'm gonna stick with that one until somebody punches me in the nose.

But you just described how, I talked about how did you figure out the sweet spot? So that opens up, explain what you've done so far with the variation element of the SLPK.

Farhan (20:51)
So I guess when I think of variation, I'm right now just looking at two things. One is why is it that if I give it a test case, it takes various amounts of time to write the code, right? That's one. Two, why does the code change a lot? Like if you use Haiku, for example, it'll make the test pass, but it's just copying and pasting a lot of code. It's all over the place, right? If you use Sonnet, it'll reuse stuff.

So the code is stable. it's always gonna make the same happen if you're things. Another interesting thing is if you use Haiku and you give it, let's say, six test cases, it'll do the bare minimum for the first, then the second test case is needed to correct mistakes and assumptions on the first one. The third test case is needed to correct the second one. And so you'll see it always has something to do. But if you give those same six test cases to you, if you use Sonnet,

It typically gets it right the first time. And then when it does a red green refactor for the remaining, there's no work to do. It says the test has passed because it got it right the first time. And so the only reason I have these extra test cases is as insurance in case it doesn't get the first one right. Like for example, yeah, without getting too technical. It's basically just, so that's another variation, which is how many attempts does it take

John Willis (22:03)
Good. Yeah, this is great.

Farhan (22:10)
if you fire up the same sequence, like, and I've just stuck with the sonnet now, I don't have involved with Haiku, and I'm like, it's good enough. And for example, Opus, this took longer, but the quality of the code was the same. didn't, it got it right in the first time, and then in them I had to use the others, right? But, so there's more to this. When I say, let's say you have, so know what, you have a thing called, hey, Junker in Lean, which is leveling, and if you have to make,

550 units of something and you can do 500 on day one. Typically you don't want to do 500 on day one and 50 on the other. You want to do 110 for five days or 275 for two days. And so to study the variation, don't want to give Claude, like even if I go through the graph model, I don't want to give it a four step test case, a 17 step, a 37, a 23, because then if something takes longer, I don't really know, is it because of the length of the test case? So instead I'm varying.

Each, so I'll go through this, but it's hard to explain. Let's say I have a four step and an eight step test case. Here's what I'm trying to feed it. A three step one, a four step one, then a three step one, a six step one, and an eight step one. Because what's happening is at most each test differs from the other by one, two or three steps. And I just, so I give it, let's say the first three steps and it's like, that's new to it and it'll make something.

then I give it three more steps for that same test case. Now it's slightly different, so it doesn't have to do that much thinking. And now I give it the last two steps. And so I'm basically trying to figure out how big a gap should the test cases have? Because if you give it a 20 step test case, it's too much. And it tries to go looking at the entire code base because those 20 steps reference so many methods and so many classes.

it just tries all sorts of things. And then the resulting code, because it's touched five or six files all in one shot, this is too much variation. So I'm trying to figure out what is the smallest sequence of new information that I can give it. I'm going from four to eight is four. If I give it the next thing is 27 test step, it's gonna have like 19 new things to worry about. Whereas if I just drip feed the 27 one in a sequence of three, six, nine, at any point in time, it's still a.

There's no more than three steps of difference. And I'm doing that manually. I need to use an actual graph model too, but I noticed that even if you do it manually, it just works really well. And the code is typically always the same.

John Willis (24:32)
That was going to lead into my next question. There is no way this is a criticism because I think this is fascinating. I think it's fascinating what you've done so far. But it sort of begs the question from the pure deminist in me, which is, have you been able to use pure SPC stuff to figure out special cause and common cause or just be able to check, it looks, it sounds like, and correct me if I'm wrong, that you're sort of doing an eyeball version of variation. Again, not a criticism.

Farhan (24:59)
No,

you're right. So I'm doing the eyeball version because what I wanted to get to is have a repeatable enough process and I could run it enough times and on more projects, I can collect more data because if I just like, gave it something really small to do and it is I plan on spending the next few weeks just turning all of my code bases into haunted code bases. Right. And then I'm going to actually automate creating the tests and then I'll start collecting the data. And I guess

So this is the difference between using, let's say a model-based testing tool to create a test where you can control things with a dial, like how big a step you want, how big a difference, versus just asking the AI to generate test cases randomly for you, right? And so you're right, right now I'm eyeballing it to see what do I really want to code and what do I really want to capture? And then I'm to focus on that and then build the graph model and have it automatically just, I'll keep, you know, change the parameter, differ from the last one by one.

by two, by three and start capturing and seeing what makes the difference. But I need to have a way to collect enough data.

John Willis (26:01)
Right.

Yeah. But do you envision, because I think one of the things is, you know, think statistical process control is gold, right? It's gold in other disciplines. We've been, I think, having a hard time in our industry figuring out how to create it in a way that has scale. You know what I mean? And do you envision at a point being able to, because I think the beauty of SPC, that sort of, or, you know, what Deming would have

called analytical statistics, is that you can start decoupling the subject matter expert from what the data tells you. In other words, data tells you patterns, and then systems, SMEs can then look at the pattern and say, yeah, I know what that is. And I think that's one of its most powerful, like what a special cause and a common cause is, doesn't really matter to the statistical analysis. It matters when the subject matter expert says,

Wow, okay. Do you envision going that direction or do you? Yes.

Farhan (26:59)
Yeah, okay. Yeah, so I, and again, maybe because I'm biased and do a hammer every problem's a nail, what I do

John Willis (27:07)
Hey,

by the way, Devon is not a bad, like, nap. Yeah.

Farhan (27:10)
But so it shows what I envision so it's like, know how you have like the Russian doll But it's one doll inside another one inside another one. Yeah. Yeah. Yeah. have a Claude You know process that's making the test, but I want to orchestrate that with another Claude process it's going to run the whole sequence collect the logs and look for the variation and then figure out where the variation so let's

There are all these test cases. So in my QA team, had something like 100,000 test cases for the regression for all these claims adjudication, right? I don't have 100,000 test case. Well, let's say you have enough test cases. You fire them off at this process. I'll collect all the data and I'll consistently see for a particular type of test case, it takes really long. I'm like, all these tests are three steps long. But why does this one take a lot longer, right? Only the domain expert can say, yeah, because that test case is talking about a concept that

you all the other tests don't because this rarely happens. But I wanted to be able to just correct that because, now this might sound crazy, but I think that tester becomes your new developer because they're the ones that are just going to come up with the examples. And so I want the SPC thing to point out to them, what's special about this test case? Why does this take so long? And my job becomes giving them that feedback.

John Willis (28:28)
And this is what Deming, I believe, wanted, which was, you know, he, there's a quote where it says, it's not the job of the statistician to identify the problem. says, And I'm sort of modifying my version of it, but it's their job to point out the pattern. And I think we, you know, the real hammer and nail here is the way we do things. We expect the practitioner, the expert, the demo and expert in your words and the proper words is to actually figure both out.

Yeah, and and and I think and then I one other meta point I think is really interesting here is you said something about like, know sort of testers become the new developer well truth of matter is Developers, know in a world where code being more generated by AI And in a world where tests is have to fundamentally shift to a different way of doing things there's really one unified new type of Sort of actor now, right? Yeah

Yeah, yeah, yeah.

Farhan (29:23)
Yeah, because I mean, for one thing, the person in software development whose job most closely resembles a prompter is a tester because they communicate an example. They the best qualified to come up with a good example. Like if you do BDD, they come up with a good example to communicate. If the example is bad and you give it to three or four different developers, they all might come to different conclusions. And then you might wonder why are there four different code bases or why there are four different implementations.

And so your variation goes up, right? Even for human processes. But if the tester wrote a really clear test case, then the variation goes down because all the developers have exactly what they need. They all come to the same conclusion, right? They don't spend varying amounts of time trying to figure it out.

John Willis (30:03)
That's awesome. All right, so this lends us into psychology. Hey, right, which again, like, all that's great. Like I always say, you you can basically do all the epistemology, you can do all the analytical statistics. But if you can't change people's mind. And yeah, there's a core here that like, I definitely want to hear your thoughts in general, but that

know, QA, you know, even with the DevOps and the DevOps has changed everything and I'm being a little ironic, satirical here, making fun of my own tribe. But the point is, which a lot of QA is in a lot of organizations is still territorial, it's fear based. And you talk about this in sort of your GitHub repo about the project.

Farhan (30:45)
Yeah, so I guess maybe I'll mention to talk about the AI thing first and then the whole fear of change because this is sound horrible on a Deming podcast. What I leaned into fear and I mentioned a little bit about that in my side, like I don't you ever heard of something called a benevolent dictator, right? What is it? Japanese like Shu Hari. So first you follow the rules, then you understand them and bend them and then you break them. So from the AI point of view, let's say your testers start using AI to create the code.

or they head in that direction. I mean, a lot of people are afraid and there's fear around, ⁓ we probably don't want AI taking over our jobs, so people might resist it, right? But I think if the manual tester sees that they can actually be more productive, and as a person who's worked in enterprise systems only, there's so many problems and there's so much legacy code, I think if people can see that they can start solving more of those problems, it'll take the fear away, right? And they can see that it's not as scary that,

They have value is my point that it doesn't replace you. It just adds more value to the tester. You want to, as a tester, do you really want to spend your time running tests or coming up with good test cases that clearly communicate how to make a quality product, right? And the AI helps you do that. And that's why I say that you have to treat the testers like a scientist. And instead of trying to make your developers more productive, they need to step out of the picture and make it easy for testers to come to this conclusion. I think how quickly can they conduct an experiment?

And then the next thing is like, would a developer let go? Because I personally, for me, I like solving problems, writing the same for loop or a statement. Once I learned the programming language again and again and again is boring. So if you like solving problems, don't you want to free up your time and do the more interesting things, right? And whether some company just wants to let people go or not, I guess is another story. And so then what I mentioned about the fear thing was, so trying to get people to

When I joined the QA team, so some more context, the QA director was let go and two QA managers were let go and two QA team leads were let go and contractors, contracts were being ended left, right and center. And so if I asked people to try something new, there was resistance because they're like, if we automate more stuff, then you'll be more efficient and more efficiency and you're gonna lose their job. And so I said something that might sound horrible, but I told folks,

Let's assume I'm going to package you all out and send all your jobs to India. Right? So there's two ways this can go. I can teach you these new skills before I package you out, or you can fight me teaching you these new skills. Now, which one do you prefer to be with when I package you out? Someone with all these new skills or someone without them? Right? So I did use the fear to say, well, you're going to lose your job. I didn't try to relay the fears and say, no, no, no, team, I'm not going to let you go. I told her I am, but I'm going to teach you the skills that you can put on your resume. So you pick.

how would you want to walk out, right? And guess which one? Everyone except two people actually wanted to learn, which is crazy how some people just don't want to learn something new. But yeah, so the fear is there, but Simon's book really helped, but this went back to why should you want to learn these new skills, right? And...

The main thing was people were concerned about their job security. So that's the thing. I'm like, this will just give you better job security. You can't control whether you get fired or not, but it will just put you in a better position in the future.

John Willis (34:02)
Did you find that like in that organization itself or like I definitely did like you're right, right? Like there's, you there's physics of like organizational dynamics, right? Like, you know, like, you know, it's sort of like the red bean game on steroids, right? Like if somebody above is saying, Hey, that's going to happen. Sorry, it's going to happen. So the choice to these people were, we don't have control of this. Would you rather have stuff? But I think, you know, the whole Jevons

paradox discussion that's going on here now is the sort of promise, and I'm sort of one foot in, one foot out, that even in that scenario, did you find that people, because they had the new skills, they could abstract and stick around and help be the sort of what they call human in the loop or human on the loop, right? And so they are now more on the loop of the implementation.

I was just trying to think that one of the promises that we're hearing a lot from a lot of people, I'm kind of optimistically hope it's true, I kind of intellectually think it's going to be true, is that most people in those scenarios you described are not going to lose their job because of these new things, because we're still going to need the people that understand the domain knowledge and

And so if you were the cyber tester that understood like your value versus your implementation time, then in general, the promise of all this stuff is that you're gonna basically not only probably stick around and not be outsourced, but your value to the company is gonna be more noticeable and higher valued.

Farhan (35:38)
So I think being an outsider to QA, I noticed that they see their value, at least I can only speak of, so, and let me give some numbers. The number of people on the team was 44 plus about 10 or 15 in Calgary. And they were part of an even bigger QA group. And so when my manager Jocelyn and I joined, believe it or not, I think officially there were almost 90 people reporting to Jocelyn as a manager, which was...

crazy. And so I got to speak to a lot of people. We spoke to testers in India, testers in the Philippines, testers in, you know, this company, Telus, has folks all over the world. Some folks came from the UK, some are from the US. And the consistent pattern is they see sometimes, or most times, they see themselves as the guardians of quality, right? They dare to police the developers. They see value in finding defects, not preventing it.

And that was the thing I had to explain to my team. How do you actually prevent a defect? Right? So like when Claude wrote that little one-pager and it's looked at all my experience and it made the connection between how, when the testers learn to help the COBOL developers prevent defects, that same approach when applied to Claude itself prevents Claude from doing the wrong thing because it gives it good tests. And so I had to explain to them, your value is not in finding defects, your value is in communicating clearly so that you prevent the defect.

from happening in the first place. And that is more valuable because if finding defects, well, a computer program could do that, right? Like AI seems to do that too. There are so many tools, asking like, you thought of this thing, a friend of mine, showed me this, you can ask your favorite AI agent, I want to go to the car wash and I want to do the more environmental friendly thing. And the AI agents always come back and say, walk there, totally missing the point that you're going to the car wash. Right?

John Willis (37:24)
Yeah.

Farhan (37:26)
Or even why do I not let Claude touch the test code? Because sometimes it's going to, it'll say, you know what, there's an L point exception. Let me just ignore that. I'm like, that's your problem. I'm like, no, you should question why do I even have that? It doesn't question. It doesn't have critical thinking skills. But every tester I met, they ask a lot of questions, right? They don't just blindly believe. They'll be like, but why is that that way? And...

they know how to ask the right questions. And I think that is more valuable than trying to run tests at the end. But the problem is that the input comes typically at the end. And when people try to include them in the beginning, there's a lot of talking. And when you communicate in a natural language, it's not the same as writing an automated test, right? And so,

without going into this too deep, this DSL editor factors all over the place. Because what I did was, so now think of Jidoka, people try to fully automate the test and they say, you must do TDD or BDD and have this. And I'm like, no, we'll do automation with a human touch. So we automated taking the manual test as tests and injecting the data directly into the developer's place of work, right? Where your Kaizen and Jidoka and all of that happened.

when we could take the testers data and continuously stream it to whatever the cobalt developers were, if you use Java with dependency injection. Now both the developer and the tester see the sort of mental model because the tester expresses themselves with data and the developer would see that data and be like, oh, is that the starting point instead of what they were thinking previously? So now that helps get them on the same page. And now they realize what to do. So like when I show Claude a failing test, I'm like,

here's your starting point, this is what good looks like, here's what I want this to be, here's examples of what it should be, and then it can figure it out. And so I think people focus too much on the inspection instead of the prevention, right? So let's say you have a car going down the assembly line. If at some point down that line, do you notice the car is getting all scratched and dinged, or you'll just notice it's scratched. I think people interpret this whole shift left thing as let's inspect earlier.

but they don't question why is the card getting scratched in the first place? It's like saying, let's scrape the toast faster. Let's automate the scraping of the toast. Let's use AI to scrape the toast. But not questioning why is the toast getting burnt in the first place and how can the QA team help you? And well, that's another whole conversation as to the mental model that most folks in QA have of developers. Like, but how can we run a test against a system that's incomplete? They're like, won't the test just fail? So, and I had to...

You could tell them, but it's only when they started working and they understand, yes, your QA test can run again, so this can be incomplete. And here's how, right? But so I won't go off on too much.

John Willis (40:06)
No,

no, no, this is good. it's like, again, it's also like the point you made earlier about variation, right? And variation or even in general about SPC or analytical statistics was what Deming would call it. The idea that like there's a variation and okay, we can say these three tests went really long, these tests didn't.

I mean, Claude really not gonna know what the domain expert knows. Thing here with, you talk about, like that scratch in a car, or the point that you, the brilliant point you made, which is, like, can we get testers to not be just the guardians of defects, but this sort of explainer, the understanders and this sort of sherpas I guess, if you will. Like, the scratch in a car is a great analogy, right?

Farhan (40:29)
Yeah.

John Willis (40:49)
let's go make sure that doesn't happen again. Well, then it happens again, right? Like what was, and actually that is a great segue to see what appreciation of a system or systems thinking, right?

Farhan (40:52)
Yeah.

Right. So I guess from an AI point of view, I haven't given it much thought. haven't gotten that far as to like how does this affect everything. But I think at least from the QA team's point of view, when they started creating these feedback loops, now I'm going to assume that I'm just thinking out loud here. I'm going to assume AI at least helps people learn a lot faster by having shorter feedback loops.

So you get to see the impact a change makes and then you get to see the system as a whole. Right now, when people take time to do some work, yeah, so okay, let me explain what happened when I was in my committee. We would make changes to the process. Maybe I would change, we had releases, three month release, six month release, three month release. And if I'm trying too many experiments at once, then I don't know what caused something good or bad, what changed the output.

So we would try one experiment per release. Kind of like the same reason I don't want the log statements changing too frequently or I can't study something in the long term. But if coding takes minutes to hours, then you can start to see the bottleneck a lot faster thanks to AI and figure out where is that bottleneck? Because you don't have to wait days and weeks like I had to to figure out, okay, is this going to work quickly or not? We had to try it out with the developers because even doing all these...

you know, small batches, it is a small overhead and you're trying to figure out this, this actually make it better or worse over time. What's the sustainable? I was still concerned that even if people did this, are they doing it out of fear? You know, and hiding the fact that they're taking 10 hours and reporting to me, yes, we took six, right? So I think with AI, I would be curious to know what, okay, this will help people because you'll see the bottleneck faster and then you can resolve it, right?

I guess the question becomes like, okay, as the whole system gets more efficient, does someone again tempted to let people go? Right. But, ⁓ yeah, I don't know about that.

John Willis (42:53)
No, I know. I think one of the things that's glared out from what you were trying to do here is the, you know, the one we've talked a lot about is like, there's a blending of sort of what QA means, what devit means, and what sort of

you know, like, you know, create how many bugs that I find, is how much did I ship, right? And managing is sort of like velocity to revenue, right? And so, you know, I think there's taking the SLPK approach where, you know, I sometimes like to refer to appreciation of systems or sort of systems thinking element of this lens is that ⁓ it sort of encapsulates it's, you know, I've gotten yelled at for calling it the fourth discipline, but

but sort of based on Sengei's fifth discipline. But it does sort of put the, like we're sort of, all the elements are sort of feeding to a way to sort of decouple stylization or decouple. I think that all of what you've talked about or so far discovered and where you're going is by definition leading to appreciation of a system.

Farhan (43:57)
Yeah, you're right. so like a couple of things my team realized, like one is the QA team can be used to prevent the defects so they don't have to inspect as much, right? And then people started realizing a couple of other things about the system. So this was basically a waterfall project and they would, you can imagine in typical fashion, all the developers have to deploy all the code into one big ball of mud into QA. And then they would have all these projects, status meetings, defect, reage meetings, status update meetings.

But later on when they realized that if the developers and testers could work in development and take a little bit of a little bit longer, it didn't really matter constantly, you know, like all the defect triage meeting. So the way you would manage a project would change is my point. You no longer needed to make decisions and decide, okay, you know, I'm going to remove some feature because we're running out of time because of this QA deadline. And so it started with changing the way the QA team worked, which impacted how the development team worked, which then...

the project managers are never really brought into this, but the developers and testers would realize, well, that's how we manage the project. You can just release stuff as it's ready, or with all the core bots, there are a lot of feature flags. At one point, a developer asked me, if we're running the QA teams regression in dev every single day, and all the developers are passing the entire 15 to 20 years of regression tests every day, that means they're ready to go to production every day. And I was like, yes, you are.

And so they started to see how the whole system of work would now change because they're like, well, then we don't need two environments, like a typical current production QA environment versus the next release QA environment. And they were like, well, that'll save time if you have one. So there is an appreciation of the system. guess it comes in the end once people start looking. I guess time is freed up.

John Willis (45:37)
Yeah, they're brilliant. And I was sort of looking for something. I had a conversation with Shannon Leitz yesterday, and I hadn't spoken to her in quite a while. And so last time I spoke to her, we were both getting our feet wet with AI, and now we sort of have some experience. Hers is she's a security expert, white hat hacker, extraordinaire. But probably other than Josh Corman and her, it's my two go-tos on everything I need to know about.

sort of risk and security. But she said something yesterday, which I thought was interesting, which really ties into your SLPK emergence of this new role. She said that what we need to get to or like what's happening is we're creating now AI producers. So they're not just like a dev or a QA or even a content or prompt engineer, which we know that is sort of like, like sort of two year, a year and a half ago, two years ago, that was like,

Make $200,000 a year, become a prompt engineer. Now it's context engineer, now it's specification. And then I do have some questions coming up about agentics. But it sounds to me like what we're really describing with the melding of knowledge and domain knowledge and variation and in psychology and in systems, we're leading the path to what Shannon would call the new role of an AI producer.

That make sense?

Farhan (47:00)
Well, I guess we could say AI producer, what are they producing?

John Willis (47:03)
Yeah, well, I mean, that's the point, right? I think that is the point in that there, there, maybe, maybe it's not a great name yet, but the concept I like the idea is that we lose there. They're not just coders who use Claude They're not just testers who use Claude, or they're not just traditional legacy coders or testers. They're not just prompt engineers. You know, they're not, you know, they're not like sort of people who sort of create

⁓ agents and sub agents and are using some of the fancy, you know, sort of tooling now around MCP and all that. There are people that are sitting above all this and they're sort of figuring out how to produce value based on all those things. I don't know. It's

Farhan (47:45)
I guess like is it that they will just be that they are now multi-disciplinary or like a poly skilled individual that they don't just have one role because they have more access to tools or skills.

John Willis (47:57)
I think that's it. then, you know, I can see how the word producer sort of implies maybe, or again, it becomes one of those subject mental models like what you're saying, think, but when I think about the, we talked a lot about like, what's the impact of risk with agents yesterday, right? And that's a whole deep discussion I've been thinking a lot about lately. And

But like the way she described it is that you know that the people sort of who are going to be the value adds are the ones that can sort of meld in all these things sit on your sit on top of all these tools and Be able to figure out how to put it all together which well, yeah risk You know like again like we can do all this stuff You know we can be what you're doing brilliantly with QA but like at the same level in an SLPK lens

you we have to make sure that we're like the system within the system, right? And this is, know, system that has systems in it. Anyway, yeah. So risk is going to have to be, so it can't be like, we'll do all that and then we'll figure out risk.

Farhan (48:54)
Yeah, no, I think that you're going to need people that I guess do look at the whole system of work and do look at all of those things in the future because for one, there would be that you're more capable now. Like for example, I don't spend that much time typing at all. It's mainly just talking to Claude and it's like everyone else just creates all the code. And if you imagine that a tester does that too, they could just work with, you know, the product person.

Right. And then they can start to do more and they can have ideas because they can play around with Claude and literally make a new product right then and there. So they no longer the tester. Right. And then that frees up their time to learn. So they can just keep organically evolving into new skills and thinking and coming up with their own ideas and what they have to do. So yeah, I guess, it's like folks want to stick. Maybe people like things to stay the same.

And folks are not thinking too much about what are these new skills. Like people say AI creates new jobs. But to me, for example, like one of the things I want people to get out of this is that, you know, lot of people see manual testers as they're not engineers, they're not your rock star, A player, whatever, right? But they have a lot of value because they can ask really good questions and give good examples. And in the day and age of using AI, you know, the developers really just slowing people down.

⁓ I think if you could just put your tester there and then give it a go to something else to do, like what I'm doing, right? I'm basically trying to figure out how do I make this process go faster? And if there was legacy code, I'd go rewrite that, right? And then, so what is the role of the tester? They're not really testing anything anymore because they're not running a test, even though I'll call them tester. So I don't even know what you would call them anymore. They're someone who can now leverage these tools and like you said, produce what they want and learn what they have to.

John Willis (50:36)
That's why I kind of like the idea of this AI producer. And I think we just exhibited how mental models can get in the way of that title. But if I think about the conversations I had within the last two weeks, right? And so you said something earlier, said that when you were basically, you told that group of testers that envision the possibility of this business packaging all up and sending this work to India. You got two choices. You can put your head in the sand.

or I can teach you what will make you more valuable. So maybe the organizations that are going to sort of survive and not fall prey to chaos are gonna have people like, somebody like Josh Long figure out how he can sort of create a knowledge of body in this AI producer. Somebody like you can infuse that same level of knowledge. And that becomes not just person, maybe there's a,

there's just sort of a corpus of information that evolves. And then somebody like Shannon Leitz or Josh Corman could possibly put risk into that model, right? so at the end of the day, a successful organization will have AI producers that have sort of a higher level understanding of what the coder knows.

Because they're not gonna be coding and they're not gonna be writing QA tests. They're going to be, like you said, talking to an AI system. so what Shannon knows, and that's idea that that's critical and can't go away. And so, development knowledge, like a Josh Long, Shannon Leitz or Josh Corman, you and QA and...

you know, and somebody like Patrick with infrastructure or, know, or, know, whoever we think is sort of, I don't want to say myself, but there are people that way better infrastructure than I, you know, and like that knowledge becomes a producer has got to have those characteristics. And if they do, then they're going to keep their jobs. B, they're going to show more value for their organization. And, you know, and I think that, and what, you know, what you sort of described is an interesting

way to look at the deming lens of SLPK to be able to sort of implement what their new role looks like.

Farhan (52:45)
Yeah, because like I think one of the things was I did I had no intention of letting everyone go. So I did want to bring everyone along. Right. And it's one thing to make it so all of this is possible if you have I guess it goes back to I think the two episodes ago you guys were talking about. It just comes down to leadership. Right. I wanted to make things efficient in a way that brought everyone along. Right. Without

making anyone feel like they were going to lose their jobs. Like, yes, I had sort of a fear to motivate them, but in the end, they understood things and I quit and I left when I realized they learned. So even like right now with AI, I still, I think you need a leader who has to see that now the roles are going to get all mixed and gray and we don't really know what people's roles are. Sadly, if I look at job descriptions for all the QE stuff, they still want you to do UI testing and inspection and use AI, right? So it's like,

It reminds me. So let's say I saw this post on LinkedIn. I typically stay away from social media, but I saw this post where this person use. AI to automatically create test automation, automatically run tests, automatically log defects, automatically now prioritize defect triage, automatically write test plans for all the new test cases. And because there were so many test plans and so many defects that they would prioritize those and set up your meeting so that you can now look at all of this new stuff.

And I thought to myself, with that much computing power, couldn't you just get the AI to fix the code? And that's it, right? And I feel that people still think that they're going to use the same roles and try to sit in their silo. And only the leader who looks at the whole picture is the only person who can do that. I don't expect an individual to know that person is doing what they're getting paid to do. And they worked hard and that's what their boss told them to do.

But if the leaders don't tell people it's okay to sort of blur the lines, then folks are not gonna do this on their own, right? And it is going to get scared if they don't know what's gonna happen next.

John Willis (54:36)
You know, this ties deeply into the webinar series I'm doing a workshop on sort of risk and AI, it's sort of the broadened version of this is, you know, something I came from inclusion and talking to Shannon again just yesterday is that, you know, I was talking about the normal risks, like you can't apply what we've been doing in ITSec and ITRisk to this new world. Like you have to sort of break it wide open.

But I think it's true for all the disciplines. And so like CIOs are leaders now. The ones that are sort of like, I don't know what's going on, just use AI. And the people are just using AI in their old, applying their old patterns to AI, like the one you just saw on LinkedIn. That's gonna be, I think, existential crisis for organizations. 9,000 developers running 1,000 each sub-agents.

100 QA people running, know, like, like it's going to be chaos. So there is this leadership ability to catch up and understand that there's got to be a systemic change to the way we do things. And, know, to tie it back to the AI producer as a very simplistic, you know, gross way to describe it, we have to start thinking about how do you create these new roles, not across QA, but development and risks.

and all these things in a way that is, has to systemically change. It's not just basically give everybody, know, Gene's book is amazing, Vibe Coding, it's a great book. I highly recommend it, but it's not just giving 3000 developers a copy of Vibe Coding you know.

Farhan (56:08)
yeah, because even if you like, I guess one thing I've seen, um, first it was a shift left buzzword and then it became QE. And there's still this idea that if you'll get someone else to automate the test to this day, you'll never, they don't really work together with the developer, right? So regardless of your programming language.

Imagine if I'm writing code and you are responsible for test automation. How was that supposed to work out? Will I take my hands off the keyboard and tell you now you work on my code base? You can't do that. You have to blur the lines. you know, it's like, would say paying someone else to automate a test case for you. It's like paying someone else to go to the gym for you. They're using the tools, but you need to know how to use it and they can be a personal trainer, right? But they're not doing the job of automating tests and you're probably doing their job of automating the test, but

I find it so fascinating that still today people just resist blurring the line.

John Willis (57:03)
It's ironic too, like what you just saying that made me think that we're maybe falling into the pre DevOps trap, right? The DevOps trap was you had a QA and you had you you had dev and you had operations and obviously, you know, know, tests was sort of another sort of wall. And we sort of figured out a way to at least,

descriptively describe a more optimum way of doing things, but in some ways, maybe AI is taking us back to like, I'll just give all, to your point, I'll just give it all to AI and I just created a new form of the pre DevOps wall of confusion. That's awesome. Yeah, that's pretty awesome.

Farhan (57:35)
Yup.

Also just imagine if you did replace your entire QA team with a whole bunch of AI agents. AI agents that come up with a test case randomly, AI agents that come up with this automation, they do all of this. They all move so fast that I can't imagine what will happen to your development team, right? Will your development team just be spending times and meetings with AI agents while they go over the defects instead of trying to say why wouldn't they use AI?

Right? Like one of the stories I had, that I documented was the story that one of the COBOL developers told me, where the junior developer not knowing anything about the system was guided by a senior tester, right? And he produced defect free code. And it's like that's, you know, like before QA there was another term, it was called quality assistance. I heard this last year on Spotify and they were calling it QA's quality assistance team because they saw their QA teams.

as people that assist the developers in ensuring quality. And I don't know if they bear that term came from, but I did the line with Dr. Deming's work of saying have quality built in rather than, there was this video where this lady, was a team leader, think at last year, and she was explaining how they evolved their process such that the testers worked alongside the developers to ensure quality in the process, not just inspect it, right? They were trying to figure out how to prevent it.

And in fact, all this model-based testing stuff does come from the Spotify QA manager. He put out a bunch of videos. And so I would listen to what he was saying. And he's like, the goal was to have quality built into the process. He never made any reference to Deming, but it just seemed to be.

John Willis (59:06)
I mean, that goes back to the Deming stuff is, know, it's in one hand, it's complicated to implement the other hard. It's not revolutionary. It's it's sort of human behavior in any silo, right? But let's, so where do you take all this stuff? What's the sort of game plan and then how do people, you know, I'm sure people, you know, I...

I just had a blast, you know, so I'm hoping people listening like had as much fun as I had doing it, you listening to you and trying to really dissect my

Farhan (59:36)
For me, well, so three years I quit my job to go chase all my hobbies. I like to do different things from gardening, to cooking, to baking, to woodworking, to I make these scale model airplanes. I have Lego technique trucks. I like to make things. And I guess for the past two years, I wanted to do this research to see like how does this change things. Because I realized if you go back to work, you'll typically fall into the same roles. And so

What I'm coming to the conclusion is what we discussed. I think in the future, your testers are basically your new developers. They're the ones that can actually help free up the developer's time. The testers will focus more on the things they like to, making good test cases, coming up with good questions. The developers will solve new and novel problems rather than doing the same drudgery all over again. My hope was just to show people that when people say that they try to do the shift left stuff and they need a budget,

My QA team had no budget. The testers had test automation created automatically, right? It's like things like model driven development might not have worked well for developers, but they work well for QA. So for me, it's finished the example just to show folks and then document it well enough so that if someone is trying to change their QA team, they can see there is a way out. But I think the main thing is, is like you mentioned earlier, there isn't a Deming success story. I'd like to see it some sort of DevOps conference where someone doesn't say,

CID saved the day, thank you Ken Beck. Or CID saved the day, thank you Jez and Dave. I'd like to see a conference where someone has a buzzword free approach and say, look, we applied systems thinking, we collected the data, we did all SOPK thing, we listened to Simon Sinek, right? We eliminated fear, no one lost their jobs, we have a lot of good people. Yes, some people were lost on the way, but I want to see a Deming success story. And so, you know, thanks to Shane for encouraging me to reach out to you because-

I'm looking for people that want to have a Deming success story and focusing on quality. I spoke to Bill Bellows a few months ago and the one thing I got out of that is I thought Deming, I mixed up Deming with Lean and then Dennis and Lonnie explained to me the relationship between the Toyota family, Taiichi Ono and Deming and what each person contributed to Toyota and Deming's focus is quality.

which is why I went to the QA team because I'm like, according to this Deming dude, you start with quality. have to see some tendency.

John Willis (1:01:54)
I loved Lonnie and obviously I love Bill Bellows and all that. I think one of the things that I think a lot of people miss about the DNA of Toyota, it's easy to fall into the trap thinking that Deming was just TQM. But like I interviewed Dr. Yoshino from Katie Anderson's book and got to meet him. And he said it wasn't just quality. He said that he started at ⁓ Toyota Production Systems.

in like 1966 or 67, right? He worked alongside Ono and Shingo and all those people, right? And I highly recommend reading Katie Anderson's book, Learning to Lead, Leading to Learn. But he told me specifically the biggest impact Deming had there, which was he taught people understand data. So that is, so when we talk about lean, like there are other things, there's Shingo's work, there's Med, there's, that's not Deming. We talk about Ono,

Farhan (1:02:38)
Mm.

John Willis (1:02:47)
They're just in time. That's not Deming. But there's a fundamental fabric that Deming, according to Dr. Yoshino, who was there for four decades and significant participate in the success of Toyota Production System, said to me, the biggest thing that Deming did for Toyota was to teach and understand data. Well, that is the fabric of all those other things that we today call lean.

Farhan (1:03:10)
Yeah, I'll agree with that because like when I was trying to automate things without finding the special cause variation that identifies, you how you have the Muda, Moody and Mura. I listened to this German professor and he says like in the West we read from left to right. So people go after the lean waste first, but in Japan they read from the right to the left. You're supposed to identify the fluctuation and the variation first that points you, right? It's only by looking for variation that I could find the biggest bottleneck that showed a huge improvement.

Without it, if I was just chasing after those lean weights, I wouldn't have been chasing after the real bottleneck or constraint. And people would be like, we keep trying and nothing is working, right? So the SPC stuff helps you figure out the actual big, and if you want to say this is working, it actually helps to find out the biggest bottleneck, which gives you the biggest win, which shows people, look, it's going to be different this time because something that you thought, you know, we couldn't have improvement work.

John Willis (1:04:05)
And that's not just quality, right? mean, like that's the whole analytical statistics is what then if you created that genre, it came from sure. You see stuff. But the point being is that isn't just fixing defects or shift from a defect that is truly understanding patterns in a statistical order that allows you to understand a solution. That's not exclusive to quality and what Deming impact Deming brought to Toyota was not.

Just quality was a way of-

Farhan (1:04:34)
I don't want to misrepresent.

John Willis (1:04:35)
I'm saying that sometimes we fall into the trap and know, and Lonnie and Bill are obviously way more, you know, they are pro-deming, but like some of the lean, you know, I'll be quite frankly, Dr. Sphere argued me for a year that Deming had nothing to do with Toyota. I mean, like, and I won that argument even before I met the Dr. Yashina. Anyway, like we could go and we should do another podcast, but the, do people find you?

Farhan (1:04:59)
I guess LinkedIn is the simplest thing. If you message me, I'll get an email. Then this is GitHub site, right? But I guess everything connects to my Gmail account. But LinkedIn is probably the best way to go.

John Willis (1:05:11)
And we'll put all that in the show notes. So that's good. This is a blast, my friend. I I enjoyed the book club. Like I said, you, you know, you, you're sort of in depth questions were like, awesome. Like, oh my good. Like this was a page and a half question and it was like well thought out. And I, you know, and I, I think I, you know, I have incredible respect for everybody that was on that book club, right? We put together, it created a great corpus of Shane, you know, all the people, Lonnie, but

But yeah, no, I think you have an incredibly talent of thinking very deep and great.

Farhan (1:05:45)
Thanks, John. I guess I'm also thankful to Rob for inviting me to the book club because there is a lot of knowledge from everyone, from Lonnie and Dennis and everyone else, right? And it's so hard, like for me to understand, I guess I do need stories and examples. And so this reading about a book, like I tried it, I could read the New Economics. It's one thing, but listening to their stories, but they explain these things communicates a lot better, right?

And then I find that's how I'm able to actually apply it now and think differently because I have a lot more examples of how this thing, you know, how other people's experience. So yeah, it is totally worth it.

John Willis (1:06:19)
I think

I wrote down something I want to say. don't know if I said it already, but like when you said I want to have a success case around them. yeah, Godspeed my friend, Godspeed.

Farhan (1:06:28)
All right. That was great. Thanks, John. Take care and have a good weekend.

John Willis (1:06:31)
my friend.