LeSS-Huge Adoption Case Study - James Carpenter + Mitya - Agile Lean Ireland

LeSS-Huge Adoption Case Study

Join James Carpenter and his mysterious co-presenter as they present highlights from a LeSS-Huge adoption effort at a large multi-national networking and server hardware company. After a quick walk-through of the case study excerpts, James and Mitya will focus on participant questions inspired by the extensive diagrams available in the pre-read content.

** PLEASE PREPARE BY READING PRE-READ CONTENT **
Please skim the case study by reading the following:

• Synopsis section
• All figures and associated captions
• Conclusion section

https://less.works/case-studies/large-server-hardware-company

Colourful information-rich context for the discussions is provided in the many graphics found in the pre-read content. Nuances regarding challenges from executive management changes, clever requirement area boundary definitions, complex engineering challenges, and the political dynamics of a multi-national engineering organization can all be seen in the figures and captions provided in the pre-read content

Speakers:
James Carpenter is a LeSS certified trainer, coach software engineer, and author who grew up on a Texas dairy farm. His coaching is grounded by the many years he spent as a hands-on software engineer and manager, along with a great deal of academic study, and in-the-trenches transformation work. He is focused on large LeSS and LeSS-Huge organizational transformations. His book "Forging Change: Agile Restructuring in Practice" is available in both e-book and print formats from most bookstores including Amazon (http://amzn.com/1732875111).

Mitya is an extremely talented senior engineering manager. He has extensive hands-on experience as both a hardware and firmware engineer. He was a pivotal director-level manager involved in the LeSS-Huge adoption being discussed. More concrete and glowing details have been omitted in an effort to protect the anonymity of the end client being discussed.

Find us here: www.agileleanireland.org

Transcript

So what we're going? To do here. We're going to go through my less my less case study, so if if, while you're watching this, you may want to bring up the. This link which is on the less website, the large server hardware company. You can also get to it on agilecarpentry.com and Scroll down and you'll see the timeline and you'll be able to. You'll be able to get. To it that way as well. So you know who the two presenters are? I'm just the external coach. I was fortunate to get to spend so much time with this client. And I was very grateful to get to know Mitrea better. We're going to go through this case study and I'm going to skim it, and then we're going to ask questions and my hope is that a lot of you have done some pre reading on the case study, and there's even a a skimming section on. The notes and so Mitty and I are just going to. Go through here real quick. Just a little bit of my background. I am a software engineer as as Anthony mentioned. So these are some books that I had that. I needed the dummy image from our website, so I put this up there. It's kind of the history of you know. I came from a. Physics background and. Did some Fortran programming in college and then oddly, I taught myself object oriented programming in Pearl, which is a strange place to learn. It and eventually I found myself doing a lot of Java development and over time I. Did a lot of. Trading systems and large e-commerce systems systems where you know. Continually highly available, always on hot rolls. You know, dozens of teams and you got to go to sleep at night and the phone didn't ring and other cases where things were a mess and the phone rang all the time. So. Been doing consulting for quite a while now. Again, a little bit about me. I am a certified less trainer now thanks to grateful to Victor Garrick who was my mentor and all the time that Craig Larman spent with me and bosses spent with me and very grateful for all of the less community. And you'll find most of the less trainers come from a pretty. Technical background Mitya, who we discussed is you know, very hard hitting hardware engineer manager and he is mithya in the case study and he's mitia here in a public forum. So he's he's our mystery man and he's Co, presenting as much as I am going to quickly overview, let's just focus on qualifying questions during the overview, but please do if you have, you know if it's a little confusing what I said, please ask, but what we really want to do is get to the end and do Q&A and spend the vast majority of our time in Q&A. And as we go, it's helpful to write write questions down so you can keep them. So when we get. To the good stuff. OK so here is. Here is an overview of the system that is involved. This is, you know. It's a server farm. This is a server form on the page. With all the necessary hardware firmware, the management system that you, the management software that you need to orchestrate this whole thing, all of this is a bundled product that, you know, forklift drops it in your server farm and ****, you now have, you know, a full blown sophisticated server infrastructure with a lot. Of blades. So here is it is a blade architecture and there is a there is a chassis here that has you know each blade is like not unlike your own a motherboard in your home PC except you know obviously you have a bunch of them. They're in this back plane in the chassis. There is storage of whatever flavour and it's modular, but some sort of persistent store is connected to this thing and up at the top there is a network device which is effectively middleware running an administrative interface, kind of a single pane of glass to allow you to administer the framework as a whole. The the types of customers vary. Some of the customers you know they might be Internet companies that are hosting various web infrastructure. It might be telecommunications companies would run in cell phone towers, it might be banks running trading systems that might be research laboratories. And each of those might configure the hardware a little bit differently. They might choose, you know. A a different number of CPUs on the board or or a different choice of more, more or less expensive storage. Or various things, but it is in large measure, you know, a single product that has some, you know, configurability as to how much money you want to spend in different components and. How do you want to configure it? The these are as you would expect, you know this is multi $1,000,000 systems although they make you know hundreds of thousands of them. But when you have a support problem, this is the, you know, first class support. The guys who work and support are paid just as well as the, you know, the more senior software engineers. And if you need them to get on a plane and come. To you, they. Do right so they may get on a private jet because you're losing money on your on. Your trading platform. This is the typical gold plated type support that you see in these sort of corporate environments. So that gives you a you know context. Doo Doo Doo Doo Doo middleware to low. Level. Ah, what's interesting here is. So in order to be able to manage the. Manage these blades. There's a variety of customizations that are done to the BIOS code, and there are, you know, many of these network network cards or custom network card. There is a node controller which is on all the time. So let's say you wanted to reboot that blade. Well, the node controller would always be on because it's running a, you know, a stripped down version of a of a Linux operating system. That is playing an administrative function. And that would be the equivalent to, you know being. Able to go hit the power switch. Yourself. Whenever one wants to make BIOS changes, those come down from the from the the GUI layer or the command line equivalent. They go through the blade chassis. They go they to the chassis controller, they go into the node controller and then the node controller goes and contacts the BIOS and has it do. Whatever setting changes it needs, or any other sorts of instrumentation. The what to take? Away here from a software engineering slash hardware engineering context is that. Down at this BIOS customization level, this is very very low level code. They're they're just lucky that they get the code in. C not C. Plus plus C instead of having to write an assembly and even their standard libraries are not there, there is no operating system in this early boot stage as you're trying to bring up the board. The you have to cross compile and deploy the target if you're monitoring any tests that are running, all you may have is a serial port that's coming back with those results. If you're lucky. So that's the nature very much. Low level firmware flavour of development. On the flip side, and it may be a problem with the hardware, because these are like perhaps custom chips from Intel that are that are pre production runs and there may be a problem in the chip itself. There may be a problem in the BIOS code that's being modified that's taken from a third party vendor that gives kind of a base set and then they build. Customizations on top or it may actually be in their own code. In, in contrast, if you go up here on the other side, if you go up to that management layer, this is stables. This is middleware, effectively middleware running on stable hardware. It is no different than you know from a from a software engineering design pattern, any kind of perspective. It is no different than any other big large corporate middleware system that's managing some complex system. In this case, it just happens to be managing a bunch of hardware.

James, this this management layer is, is this top right?

Corner right. This top right corner, yeah.

Just wanna point it up.

Yeah, so my background is mostly middleware where I to write code in here. It would look similar to what I spent much of. My career doing. When you get down here, this is a world that I don't know well. Mitra mithya does. But completely opposite ends of the spectrum, and that's what you that's the thing to take away here is that this particular product has thousands of engineers working on it concurrently and that even though it probably should have, you know, 1/3 of that but. Working on it concurrently and the breadth of the type of engineering varies greatly. Based on based on the. Based on which aspects of the? Of the product we're talking. About this is typically on premise deployments, so this is you have that additional complexity that absorption cost is relatively high. OK. Mitia, do you have any you want to give any other insights? That I may have missed here.

I think I think it was a good more than overview. You know good introduction, good detail. I would just add one thing that if you look at the left side of this blade motherboard detail while. For example, the the management layer as James James described this top right corner may stay the same, more or less. The blade chassis will stay the same, more or less the the left side. The Blade motherboard will. Change or we'll make it updated every 18 months, you know, depends on the the CPU vendor cycle. The Intel CPU maybe come AMD CPU, the the NICs will get also kind of upgraded with the industry to to the to the industry standard. With the memory also. So all of that gets. Updated via a new blade being brought in so so customer may have these these blades installed at a time of. Initial installation and then every 18 months or so as they purchase more they they will get new new kind of a. Up to date hardware and with it will come. Up brand new BIOS to the. Customer it doesn't. It's not really. That important, the bias is brand. New but actually to the. To this discussion and to what was implemented in terms of Agile, this 18 month cycle was very important and and the timing of it was also very important as. James will go into later.

Thank you. Thank you. Another thing I I failed to mention is that. This is from the customer perspective. This is the product. It's this whole thing. Everything you see in orange. Is the product from a less product definition perspective. This is the obvious product boundary. When you leave this boundary and you're outside of the division at this boundary, you can plug into. You can have, say, a router that's a competing companies router connect to it. This is the very natural product boundary for this, and it's how the customers would think about it. As to timeline, so there's this timeline that's in the case study a lot of things are coming and going a lot of things are happening that are discussed in the in the case study, they're not always discussed chronologically. It's mostly chronologically, but I might focus on what's happening at the diagnostics team. And then at what's happening at the BIOS team may be happening a little bit concurrently to that. The timeline is really useful in helping kind of maintain the overall context of what's going on. If you were going to print something out and take notes on it, this would be the thing to print out and take notes on. So you can put notes on it. When I first got there. There were a variety of teams that were claiming to be doing agile, but when I went and investigated what various teams were doing. It the only thing you would see is Agilent name only the realities were far from it. And so my key sponsor there, what wasn't Mitty, it was the person that was signing off on my pay cards on my on my invoices. We started looking for, well, where can we go and find something that we can demonstrate what a good feature team looks like? And the thing we found was the diagnostic capability this this system did not have the ability to do diagnostics well in the field. We would have hardware returns that you know, after multiple rounds of trying to solve a customer problem and you know these customers are paying losing millions of dollars. You know every every minute or so, these things are down sometimes. They would eventually give up ship them brand new hardware and then they would get that back on the bench, you know, a month later and they wouldn't be able to find the problem. And so not only were they losing, you know, millions of dollars or like probably 10s of millions, I forget what? How big the numbers were, but they were way more than the team cost per month to. Deal with returns in the field and what's worse is the the customer experience for these poor system administrators that were on the other end that the that the support teams were trying to help. You know they were frustrated this could be. With these sort of ghost in the machine type issues, even if you know it worked, most worked pretty well for most people all the time. There's always with so many systems out there, there's always, you know, some people that have problems and there are hardware failures that happen. And so this was an effort to build some additional diagnostics capability. Each of these various components, like the custom knit cards, the custom bio stuff, the throughout the system, there's a variety of custom hardware, custom custom ASIC chips. Custom firmware etc. And not always was. Did that even have a instrumentation to perform diagnostics? So we pulled A-Team together where? Every you know what was effectively A component team that had the skill set to work on, you know component. Everyone coughed up one person that joined this diagnostics team and then the diagnostics team built out this new diagnostics functionality. The vast majority of the diagnostics was kind of contained and was code that others didn't really need to touch much. But occasionally there was a need to go into a particular sub component. And add instrumentation to it and so this team had the skills and the access to go do that and it was the first team that had really done anything like a proper scrum environment. And you know, we put them all in a. Room, as you see here. We delivered in a way that they've never, never seen before. From a feature team adoption map perspective, it is mostly a vertical slice. Which is exactly what you want. It truly was a true feature team and within the narrow bounds of being a diagnostic component, it was, you know, end to end in that regard, there was even the ability to. Ship the custom ISO the the diagnostic ISO could be used in isolation on an individual node if you wish to test the system as a whole, you had to wait for. You know this waterfall release process to be able to catch a ride on to to make it into that high Level administration GUI. So you could like do it against a. Run test against it of a huge. Number of nodes all at once. But even incrementally, it had values. Sprint by Sprint increment by increment. So as I was doing that to to do healthy health product boundary oh. So that is. Simple. You'll see this diagram in the some of the less materials. This is effectively what we did. This feature team read is effectively what the diagnostics team was. We did not. I made the mistake of not dealing with enough of the organisational structure up front, which eventually came back to give us trouble and we did have cases where people were being pulled off the teams and refocused and we had to. Continually monitor that and and put some management pressure to avoid it. Very senior management pressure so that the middle management would wouldn't be problematic. And that had problems on its own, but nonetheless. It was about this time that. Mitty and I started talking because this team was up and going and starting to to yield some benefits and I had managed to. I'd managed to to slowly start talking to mitia and kind of bring him into the picture and Mitia was responsible for this. BIOS components at this point mithya can you can you speak to this for me? I mean, just in general like, tell the story of, you know, you and I basically started dating, if you will.

Yeah. Yeah, right. Yes, yes.

So, well, my wife, my wife would call my. Work wife started dating, right?

Yeah. So as James mentioned, he was working with the within the business unit with other teams. I mean not with other software teams within the same business unit and bringing them up to the agile framework. And you know, we kind of we knew each other when we had lunch together a couple of times, but I, in my mind, not only in my mind, but I was told by mentors and and and senior people that you know, former you don't you don't do agile and former BIOS you don't do agile and BIOS. It's so, so far down the the software stack and it's so close to hardware and the cycles are hardware driven, you know you're just the software who is enabling the hardware. You don't have your own kind of. Cycle your own. Destiny on schedule. You're all dependent on how things are going. In the lab. People are bringing up these fresh of the. Press boards and debugging and. Debugging new, bringing up new components. You know, new memory, new CPUs, new third party network adapters and so on. All of that is. It's not a, it's not that doesn't, you know, at at the at the onset doesn't seem like a natural thing to to put in a sprints and retrospectives and so on. Which wasn't really true at all, as as as we discovered with James working with James. But even putting that aside, you know BIOS. These days is. A whole lot of software, just pure pure software. Even though low level as Jim described James described, you know it's C without libraries, which is in. Innovation of the last 20 years. Before that, it was assembly, but you know, still just kind of pure C code. It's still software. It talks to just talking through this picture. The the BIOS communicates through the chest controller, through the management, through the GUI, and the administrator on the top right. BIOS communicates the status of the system to the to this high level software. BIOS takes configuration from the high level software and configures the system as desired. So all of that is simply software features. They just happen to be on this kind of a low level environment and so. So so I started talking more and more with James and James kind of was educating me. On agile and I was, I was simply one of those people who knew about agile without actually experiencing agile. And however you know, one meeting that James. So James set up one formal meeting to introduce aspects of agile in some somehow. So it was a. I remember the 30 minutes.

I was getting the director involved in the in the diagnostic stuff to help. Unless you it was a 30.

Minute meeting that left probably 2 and. 1/2 hours. And the reason it lasted 2 1/2 hours is because. It was a it was. Interesting to me to to hear the. The concepts that would be brought into the team with the framework you know, without the so much of A language of agile and when you start listening to the. The framework and the benefits and. Kind of what comes with it. You know, the engineers being in control of their own estimates and the, the, the, the, the concept of delivering code every every so often every few weeks, whatever the this print is and then retrospective. And kind of the team environment. And manager sort of being out of the way, mostly except for removing the obstacles for the team. And being the coach really for the. For the engineers. So. That that spoke to me as something that. Should be adapted by the team simply to make. That you know that what I thought and still think of manager's job much easier, which is remove obstacles and coach and provide environment where people people can. People can excel and and at at at that time we just we jumped in. I I don't think we it was a perfect timing also you know which I think I don't know this. Is a. Time to retrospect on this, I think. The timing of. The timing of this introduction of agile into our team was perfect because we were on one of the refresh cycles of the CPU and the generation of this blade and we were about to step into 18 months development. And test and and kind of QA, acceptance and release cycle and so that next 18 months. What it would have defined if we, if we ever do this agile method or not, and we decided to do it and and I'll let James continue with his.

Oh, sure, sure. Thank thank you, Mitia. It was a pleasure. And he wasn't volunteered. He wanted this. This was I was just looking for where there was, where there was interest, which gets into the org structure, which is interesting. So when I showed up, we had this original engineering SVP. The first time I said and I was actually hired by who's called Trent in the case study. You'll see him see him down here. That's who was actually signing my paychecks. But I was really only brought in because he got this gal on board. And and this person as well. And the first meeting I had with the guy, he said, James, the real question. There is not. What you can do for me, the question is how can I help you? And he made it really clear that anything I needed, all I had to do was go to him and he would make it happen. And I've never been as well supported at that level. As I as I was at this client, I've had another client where I had wonderful support from the person who was signing my paychecks, but not so much above his level anyway. So this was the org chart. The people that were working on that high level, that high level code, the stuff that had the the. That GUI, that is to say the. The people that we're working up here at the MCS GUI. They had a software VP that was actually sort of passive aggressive. To the whole. Idea. So that is to say the parts of the system that were. Most conducive to a standard software based engineering, better software engineering, technical excellence practises and where one might naturally look to do something like Scrum. In that area, we didn't have good buy in. In contrast, you know there's a like 4000 engineers involved here, so across several geographies in contracts, the quote hardware VP that Mitty reported into was very supportive and basically as long as Mitia was was on board, he would support whatever mitia wanted thought was. Made sense and so. It would have made more sense to have a vertical slice that cut all the way through what was really the real product boundary, but because of the level of. Because of the the level of buy in that we had and and didn't have, we started here. Over, sorry we started here, which is to say things that were under mithya's purview. We completely reorganised the teams there, even that was. People were working vastly more cross functionally than they ever had before, but we were still kind of constrained to what was really a component team. Even though the reality is that BIOS is so complex that there are components within components within components within components, think of it like a whole operating system. And you know there's a multitude of things down there. And we did overtime stretch ourselves so that we were had become an expanded, you know, an extended component team with an expanded boundary that was starting to stretch towards the surface. And we were beginning to get permission to go in and make changes all the way through the stack for things that related to the BIOS related aspects, but that never go out as far as we want. So from a feature team adoption map, the original scope was what you see here in this purple when we initially stood up the teams you effectively had this blue boundary and then we pushed ourselves, moved our tried to work ourselves towards what you see is the green boundary unfortunately. Three months into my engagement. I lost that key VP. And the new VP SVP was somewhat ambivalent. Yet we had enough momentum that things kept going for quite a while. In other words, actually for a year and we continued to make make a dent. But that is an interesting dynamic. In the end, when when Mitya lost his his hardware VP. As part of some major layoffs that were going to happen and people started jumping ship before it even happened, which is what? Happened in this case. Then all of a sudden Mitty, it was now reporting to that passive aggressive guy who didn't really want. In the first place. And that sort of. Thing started to fall apart and we could talk about you. Know how could I have better avoided? It's an interesting question for Q&A from the overview perspective to just be aware that there were some very significant management dynamics from a people distribution. There were people in Portland, people in the San Francisco area, Silicon Valley area, and there were teams in India. What we did as far as Mattias people is. Everyone that reported into mitia, we stood up. We did all the training, launched the teams, figured out definition, have done, figured out working agreements, figured out our cadence and for the teams that we're working on, the new version of the product. That's who we put in this new structure. And then. As some teams. Some US based teams. Finished up some work on fixing bugs and such for some older generations of the product. We rolled them into the structure and slowly grew it, but we didn't deal with India and we had kind of delayed that until we had proven that we had enough success and then we were going to use the use the conflict that would come up to solve that. And I think it would have worked. Have we not had a? Layoff that really kicked us in the ****. The other thing here. Here is some initial product backlog refinement that we did. We did brainstorming for two or three days, then over time, over the course of the first few. We flushed out and pulled things out of people's heads that had never been pulled. Out of their heads and threw it all on a wall and managed to slowly massage it into. This what you see here is really like a snake. A snake shaped product backlog where these this isn't a four strength order. You just have to read it right. And then we threw it all back into electronic tool. Here's our helper who would show up. This is Mithya's daughter. Here is an initial definition of done kind of drafted. Here is what it looked like a little bit. Yes, Sir. Go ahead please.

James, maybe, maybe going back to the definition of done and even pulling things out of people's heads, I I always you know, I think that this was, yeah, we can stay here. Something that James reminds me and and and even though this is written in with my handwriting, it's really me noting down what the team dictated. You know, this is. People. People really wanted. Those people really wanted to have. Accountability across the team. That's that's universal. People really wanted to have. Universal definition of done and you know with this number for example #8 point, I was sort of I was. Arguing that the. Severity 3. Defects. You know you can you can live with from Sprint to Sprint, but you know they convince me that. The no. Once it's done, it's done and something that is still very clean. It isn't an extremely severe defect, but it's still usually at least that that organisation, A defect that that that gets fixed by the release of the product or. Exceptionally deferred, but very exceptionally most of the survey that we were. Would have to be fixed by the release of the product, so the team argued that you know, no every Sprint we should fix this so that we don't go back and and and fix things so so it's. This is kind of a a good reminder, for me at least, that the the team wanted this and also you know the for pulling things out of. People's head. Was the timing of this engagement was critical? To being able to do that. You know something? Maybe not the place to go into details of why we have to do it, but just the nature of the CPU cycle and BIOS coexistence is that BIOS is. Largely new every 18 months. And depending on how carefully you, you you craft the custom code and segment off the common code, the process of CPU refresh is easier or harder and so in this beginning of the six month cycle, we really wanted to sit down and see what is our custom code. What are the? How? How does it function even? And how do we split it down into this as James guided us into user stories and? Kind of components by which somebody could assemble it in the future, you will. So those two. Yeah. Those two just wanted to highlight those. Two aspects of this process.

One of the things I found fascinating here was the degree of knowledge scatter. So among among say, 40 people, there was typically only one or two people that knew. Any particular facet of the system and they had basically done hardware bring. You know, half a dozen, a dozen times over the course of, you know, the last five or ten years. And they've basically been doing copy paste reuse over and over and over again. I didn't have to nudge much at all. They they were tired of the situation themselves. They had just never been in a in a place where they were empowered to solve it and there was an issue of they had to decide, you know, this working. Cross functional and cross sub component. It's really not cross component so much as cross subcomponent. The components within BIOS that was new to them because they were used to like hey only one or two people know how to touch that file. Not that subsystem but that file. I mean it was that severe. And what you see here wasn't really any outside person telling telling us what needed to happen. This was the teams. Brainstorming and that we were in a big room with where with Windows, a corner room with windows that we were putting sticky notes on the wall for about two days where we would brainstorm a bit, put sticky notes on the wall, merge these into a consolidated hole, do it again and. You know, I facilitated a little metius facilitated a little before long. We just had other, you know, random people in the team facilitating so they could, you know, they were learning. To do it themselves. And it took a long, long time to get that information out of people's heads and into the common. Into the collective knowledge of the teams as a whole that had never happened before, and then we drilled down a little deeper. The people who knew the most about a thing would spend some time flushing that aspect out more, and then once everyone who knew about you know what, we're basically really crude sticky notes. Had flushed him out in a little bit more detail than we got everyone together as again across the teams and we went through it again and you know, we did some affinity estimation. We had to figure out. What is a natural? What can we actually go to production with if we had to and that's that red line there and everything else is kind of nice to have, at least this has to happen for it to have functionality comparable to what came before for the previous generation of the blade. I put this in here for some reason. Oh. Mostly this is just slides from the. This is all the slides from the case study. This emphasises the point that you have to deal with structural elements and reward systems and processes and people and the whole system. If you really want to to change things. Here's some triage guidelines. Again, not not key to the conversation. The one thing before we switch over where it's really good stuff is I am. I have a plan to run a. A A less practitioner course in Atlanta, Dublin and Dubai, so Atlanta is the 21st and the 23rd. Dublin is 22nd through the March, February 22nd through March 2nd, Dubai is March 14th through the 16th. All of them are listed at my website. Just on that CLP. Portion of the website you can sign up for all of. And here is something stack leather wrote. I'll just let you read. So I'm new to this. I ran my first course last week. It's a real small course here in Milwaukee. I'm I'm not new to training, but I'm new to running this particular course and running it in a public. Context. It's new to me and. The good news is. So the the course is is based on a skeleton of a system modelling discussion where people work through how various design element, organisational design elements influence each other, how that influences adaptability. I go back and forth between the system modelling, which is entirely abstract of any particular framework, and then we spend time binding that to. How does less do it? Not so much, because it's important to understand how less does it, but. I find that if it's entirely abstract, people don't quite know what to do with. And so by spending some time saying, well, perhaps in your organisational design that wouldn't work. But if you had an organisation designed this way, you could see that it would work. Now that might not be the only way to build an organisation that would do that. That is aligned with the. In this case the optimise the system. Optimising goal is highest level adaptiveness in the service of learning and delivering. Class level value. But at least it's one concrete example of a design pattern of how you could build your org, and if you did. Build it this way it. Many of the the problems that people are teaching themselves, and they're teaching themselves as much as me, teaching them. That they're teaching themselves as they do the system modelling it instead of just being frustrated and depressed because how could I ever do anything like this? There's light at the end of. The tunnel they see. What would? Be possible and then that kind of helps with retention, which is my real intention is to it's not what I cover, it's what people will learn and retain. And the good news is this guy on the right. He had trouble sleeping the first night because he was so busy thinking through how he could do something with everything he was learning. So as a test of is my course working, you know, I'm new to running this course, but the fact that he's up all night trying to process. Such and and reconcile all the different things he's he's learned and taught himself in class. That means that I'm doing something right, and I owe a lot of that to the the value of the of the system model that Craig uses, that I've that I'm using as well. Most of what's on the right is at least half me, if not entirely me. Anyway, here it is. That's it. Mithya. Anything before, before we turn it over and go to QA, which is really where the really good stuff is.

I don't think so. I think let's let's do QA. OK, Karen.

So who's got cool questions for us? However, you want to moderate that, Anthony.

Yeah. So thank you very much for that, James and Mitra. Yeah. So if people want to put questions into chat, come off, Mike, whatever you feel comfortable with. So sorry, there's one question here in chat. I'll read it. Out, if that's OK, James. Sure. So if I understand there was no product owners, a single person or similar who defined what priority for teams, road map, etcetera, technical background or sorry technical back end product or feature collection, is it interesting to see how the team's need is met?

OK, so we did. Because our boundary. Mitya played that role. Because our boundary was so compromised. What what I would have preferred is it someone in product management? Sorry, it's a little slow to navigate this way. My preference would be that someone in product management was actually acting as a product owner, but in our initial product boundary way down here. The reality is it took someone very technical to understand. The this very detailed backlog and so we used unless there's a guide to use a start with a fake product owner, it's it's guide or experiment, I forget which. So we ended up with Mitya having. To play that role. And what you did well. But it's fair to say that that wasn't a real product. Owner. But it's also true that. You know when you're doing this kind of work? The teams know so much. I mean, so Mitia would make judgments as to. Where did this go? You know. Just how much of this can we actually go? Ahead and ship. But it was really a collective wisdom of everybody. Mithya you want to. Speak to that better than I can.

I I yeah, I can speak to, I don't know better, but I'll speak to that. So yes, I acted as a product owner effectively although you know. As as a BIOS you know BIOS as BIOS as a software component. In this system we had. Very clear definitions of what you know what needed to be delivered, you know which hardware components would support which memories you use, et cetera. Which CPU skills, which software features to interact with, integrate and interact with the higher level software management software. These blades, so all all of that. You know as an overall. Product that you know this this as a system. Was defined so the my job was to translate. Those definitions of the overall system. And the product specs a few little Doral system, the the features of the world system into what does bias need to do to support them, and how are we as a bias team by a software team organisers, those to support them? And when do we deliver? What? What is the minimum viable product? Et cetera. So so it's not, yeah, so I I. Was as mean as as James mentioned, a temporary product owner, although again I you know, I wasn't necessarily define I think that that's what made it possible for me to be the temporary. I wasn't defining defining. The overall product I was defining how, how and when BIOS delivers on on the software promises so that the overall system, who had its own. Product manager. Would function as as as designed. As promised, the customer.

And I would say in the in the next generation of the product had things continued, you know, as as they were, we would have been able to stretch that up and change that. So once we got to the point that you know for for years and years, they had been doing copy paste reuse and had not been using a pluggable layer and then bringing up a new. Bringing up a new board was basically repeating that work. Again, it's the reason it took 18 months instead of, you know, several months with really good build systems and automation and pluggable layers. Yes, there'd be problems, but it wouldn't be the problem that it had been in the past. And we would have been able to start defining product backlog items in terms of what does it do for the customer. The the true customer, that is to say the person who buys that whole thing on. A forklift? Not. What does it do in terms of? Implementing some API that that high level orchestration system needs to needs to see in order to be able to pull it into this federation of of hardware.

Yeah, I I would also this also add to this question. Something we didn't. Talk a lot about it is that there was. In this overall product delivery, there was a. Lot of interaction. Between the Agile team and the waterfall team teams. You know, there's lots of teams within this overall big product that we're, you know, in the in the right software send it to QA, get it back to software, get it back, the bugs, fix the bugs, send it back to QA, do regression cycle, fix all the bugs and ship that, you know, cycle and so. Maybe part of this question lane is that if we had if what James was trying to do would have continued and the other software teams within that organisation would have eventually become Agile. That, you know, having a single person for the whole product, not just bias, but the whole product would have been extremely beneficial and would make. Effectively my job easier and would let me focus more on the team, although I to to be very honest, I I I kind of question not question but. I wouldn't want to be completely removed from kind of technical discussion and just be the the coach and obstacle remover, as I know a manager should be. I still would want to somehow be involved in the technical aspect of it. But yes, having a product owner for the whole, for the whole, for this whole picture. Not just the bias component would have been beneficial. Maybe we shouldn't want to other.

Question. Well, well, to mithya's point. Yeah, I think what would have happened here is in the next generation of the product, if I had not lost my. Stop doing that. How do I get back to? Come on, PowerPoint. Due to do. Had I not lost this VP, this SVP, what would have happened in the next generation? Is Minty and I would have gone to this guy, you know, looping in the hardware VP and we probably would have managed to pull together a team that had the skills. That that took people that traditionally were working on these higher level components. And put them on. Make true feature teams and I have a feeling that this SVP here would have brought this software VP into line. Either he would have been ejected or he would have come on board. And that's what it would have. Taken to really do. That we were doing it sort of doing it. Sort of gorilla, you know, we're slowly, organically chewing into that layer, although we weren't officially, you know, responsible for that layer for these for these upper layers, we were getting into that code and the teams that were working on them were so overburdened that they were happy to have us do the work and then just. Have them review it. So, but eventually had that we've been more successful and continued that continued doing that, it would have been a threat to this software VP and somehow something bad mostly would have happened. Most likely, and we would have needed the positional authority of the SVP to to line things out, and this is what happens when you don't get enough buy in early on and you don't do something like certified less for executives. Type course, which I didn't. Have the the. The the wisdom and skill set to do. At the time. But that was probably a mistake. I would something I would do differently in the. Future and it would. Have avoided a lot of the problems you see in the case study, but let's go. Ahead next question. Whatever you think, Anthony.

Alright, thank you. So I'm just gonna take them in order. So what was compromise about the boundary? Did you mean the product self or a mixed buy in? From the organisation.

I mean what?

Sorry, what was? Compromised about the boundary? Did you mean the product itself or the mixed buy in from the organisation?

OK, I was referring to the product specifically I was referring to the product boundary and the fact that you know the real product boundary. That's the real product boundary, that orange thing, the whole down thing. Figure one, that's the that's what the customer thinks is the product. We started with this compromised boundary where it was just the BIOS teams. Even though we did pull in, as you'll see, we pulled in the QA. You see the sorry the, the the testers from the testing group that were formerly still in the testing group. We treated him as just another team member and some of the people that traditionally just did firmware development we're doing testing and some of the testers we're doing firmware development and you know that was the day-to-day operational reality, although the formal reporting reality was they were, the testers were still reporting through the the Quality Assurance VP. But because we had her buy in. The day-to-day operational behaviour was was healthy, perhaps not structurally stable, but healthy in the short term. So I was referring to the the that aspect is there another? Did I miss any part of the question or any whoever asked it, would you like to follow up clarify anything? OK. Thank you.

That's great. Thank you.

Hey, James. Question sorry. I just want to piggyback this is. Madeline. Hey, would you mind saying? More why a broader product definition right is more valuable because I feel like we're talking a little bit around the edges of that as you're trying to expand.

Well, because the customer doesn't give a blank about the. When some low level. BIOS functionality does all they care about is, you know. And nor does the CEO right they care. About stuff that helps the actual end user do cool stuff that they want to do that they need to do, which that whole thing does. That is to. Say that you know that orange boundary. That's how they look at the product. We have the advantage here of. Since this is a real product company that sells a, you know, packaged shrink wrapped product, yes, it's not completely shrink wrapped because it has, you know, consulting services and implementation services. And blah blah blah. So it's not quite like going to the Apple Store. And buying a iPhone. But in large measure, you know it's a thing. That people buy. And that's much different than, say, a banking system where the real product is, you know, equities trading or, you know, something like that. And the software is just an enabling component of that overall system, at least in this case. If you went to product management and you asked what the product boundary was, you know it would almost be a stupid question. They know what the boundary is. Take that with a grain of salt. Explaining understanding your product boundary is much slipperier slipper slipperier than it sounds. The truth is, the real product probably does include those consulting services and the broader aspects, and it does perhaps include maybe some additional network hardware that might be in front of the system, which may or may not be made by the same company. You could you could swap it out with a different competitor. Does that answer your question, Madeline?

Here I was just trying to think through, you know, and help maybe other people right who haven't had the benefit of taking even right a less practitioner class and things about the organisational implications of stuff right particularly like product boundary.

Yes, ma'am.

OK.

These people are lucky in the. Sense their boundary is obvious. This quasi obvious. So at least the idea the more ideal boundary is obvious, not the boundary that we ended up starting with, which was compromised.

Well, and though you. Didn't get to expand it as far as you wanted, right? And this transformation, did you see any increased product complexity and how did you work through any of that?

I don't understand the question, Madeline. I don't understand the increased product complexity portion of the quest.

Ohh, you know what? So sorry, let me be more verbose, right, so I'm reflecting a little bit of my own sort of systems modelling training. Right. And thinking about as we have a broader product definition, oftentimes that increases product complexity.

Yes, ma'am. Yes, that's certainly certainly true.

We have to rethink how they're. Like address that. So I was just curious. If you experienced that right, even at the stage you got to with this transformation. And how you?

Dealt with it. It certainly added technical complex. City, because now the team from the perspective not not at a systems level, it made things simpler because you know these feature teams would have had the ability to touch. Everything, certainly that was true within diagnostics. They did have the ability to touch everything. And it you. Know things got way better way fast. You know, there's an initial learning curve, but. Within a Sprint or two, they're doing stuff that's never been done before. But it's fair to say that you know the team struggles as they grow and learn these skills, and especially because we had that. Compromise support in the org chart as it pertains to the software VP. The sensible thing to have done would have been to had some of the people that were currently reporting into that software VP come over and join the teams as full team members. That would have been the sensible thing. To do, since that was not the case, you end up with firmware engineers that only know their firmware world that are now having to cross train themselves and and learn other aspects of the system that they. Haven't now the good news is we. Had the testers on the team. And whereas the firmware engineers that reported formally reported into mitrea, you know they've been living in that small silo for years. The the testers that came over that is to say. You know these triangle people. They may not have understood the low level details of any given component, but they understood how to test the system end to end. And they brought a tremendous that was a tremendous value that they brought to the table. Am I missing anything here?

Yeah. No, no, not not missing. I just, I would add to the technical complexity in certain areas certain the technical complexity increased and and I'm trying to answer this without going too much into the biospecific specifics. But it. Due to their not not not very wide usefulness. So BIOS. Has been and is less so now but. Historically, has been very. Kind of segmented. Knowledge, knowledge wise, segmented industry, so there was a, you know, person who does memory person who does processor, person who does, you know boot from hard disc and those people are kind of for the most part free to do. You know what? What? What they want as long as. The the product works and as long as the the external specifications of the are are the specifications of any kind of external consumers of what they produce are map and here we took we took all of that and we presented it to the whole team and. Now we have scrum teams. And by and large, every any scrum team can pick any component or any any user story. So so now a person who is who wrote the code that does boot from disc is watching another team take it apart. Art and another team is trying to take it apart and making something else with it, ultimately formalising a lot of interfaces, ultimately making the the segments more reusable in the future and for everyone's good. But oftentimes. You know, to answer your question directly, the complexity does increase with that because it's not, you know sometimes the the code that I write for something I know how to do. And I promise that it works. It may not be as complex as the code that has formal interfaces.

Thank you for expanding all, Matt. Really appreciate it, James Smithia.

You're very welcome, Sir.

It was another day. Another question, yeah.

Yeah, there are some more questions then. Can you tell us more about the huh wall? I'm sorry, the hood sticky category on the wall. Ohh.

Oh. Oh, OK. I'll get to it the other way, sticky. This stuff mitrea you you do it, you tell the story really well. The the couple days we spent in the room, mostly putting sticky notes on a on a window.

Was there a specific category that said, huh? Was this maybe there was a? There was a question about for some specific part of that sticky note or just in general?

If said the huh part, it's a hating gig.

Is it there? Maybe James. Maybe there was a.

Can the person clarify? Can the question the requester?

You scroll. Can you go?

John, Mike and explain the question a little better for us.

Yeah, Sherry. If you can explain.

Oh, the husk sticker.

We can come back to it if you want. OK, very available. I can go on to the next question for the introduction. Was colocation necessary recommended for new teams considering this or what is the extra you you got that that pertains to?

Oh yeah.

Non call it call non call.

Occasion. So these teams start when when we started, so that is before we did the the adoption at all. There wasn't a lot of attention paid to Co location Mitra may I may be a little off there, but but I know. That we had, you know, people that were in, in. Silicon Valley that were doing some of the work that was then, you know, perhaps being tested by a group in Portland by some people in Portland. And and there just wasn't a lot of attention paid to that. When we decided to do this, we set up our teams to all be Co located, so we when we did the training in in Silicon Valley, we made sure to go to that VP on the QA side and have her send a few.

And then. Your father's the only crime in life.

Investors to group. And we try. OK. And we trained them along with the Dimitri, sorry metrics. When it came to training the the Portland folks, we did the same thing. We were able to find some testers that were doing some other work on some other parts of the system and bring them into training and have them be part of the team. And then within, within the first Sprint or two. And we were to the point that the, the, the people in Portland had all the all the the the skill set that they were cross functional enough. That is to say they had, you know, enough testing capability, enough firmware, skill set, etcetera to. To work as an independent team and the same was true in San Jose, we stood up. Like two or three teams there and to start with whatever I said. Yeah. With step two teams there and then later added on more. And each of those were Co located from the day one from when we did this adoption. Is that? Any follow up question on that?

And no follow.

No. Perfect. Thank you.

Up questions.

I mean, you just can't beat. Co location for you know, if you're really after adaptiveness.

James, I'm gonna. I'm. Gonna. I'm gonna. Be cheeky. So if you had to do. It in a not like dispersed what's? The one thing that you need to have. In your back pocket or. You know, do extra if you have to do it in a dispersed manner.

OK, say say, say, say that again.

What is the one thing you need to do extra earth above that you wouldn't have to do because of the Co location factor that helps.

And by this first you're referring to people are working remotely and no one is in a physical physically Co located in an office or that you have a few clusters of Co located people, but they're not in the same geography.

They're not in the same geography.

OK. Well, except you're going to. Take a huge hit for that right. You may be. You may find yourself where you. Have no choice but to do it. Maybe there's a skill set that just tremendously missing. My first reaction is to. Build it Co located, you know, deal with that problem up front. Deal with the pain and then as you deal with the people will cross train themselves. The challenge is to the the problem is so severe that you really need to somehow get management on board. So instead of trying to live with the problem, if you can in some way. Change the structure. If you set up, you know people say, oh, here's a garden. Go please grow some wonderful, some wonderful plants that you know, produce lots of fruit in this. New garden. But by the way, we're not going to let you fix the soil before you do so. And you know, there's all these problems in the soil and it's mound area, you know. Unhealthy soil and. You're never going to be that successful. Now, if what you're dealing with is everyone works remotely for whatever reason, and there's some trade-offs there of. What is in the best interest of the product and being as adaptable as possible and delivering the most value as possible? Versus balancing work life balance and people being able to stay home and spend more time with their kids and you know that there's a trade off situation there and there's certainly a trade off situation if their compensation structures are not adjusted for when the product wins, they individually personally win, right. So. If you don't have the authority to go after the compensation structures. And and so then that's about like, we'll at least keep people in similar time zones, do everything you can to to lower the friction as little as much as possible. I'm using a teleprompter. I love this thing. It mostly lets me look. Directly at people. When I'm, when I'm speaking, especially in small meetings where there's only two or three people. The you know, Miro, et cetera, et cetera. And then of course. You know, coming together in person, occasionally whatever you can. But there's no way to get around the fact that you. Know having teams that are 12 hour time zone differences from each other that. You know, even if they're Co located in there each offices. You know the way to solve that is to. Make sure that each region is independent. Even though they're part is still part of the same overall product development group consumed from the same product backlog, et cetera, et cetera.

Yeah. OK. I got it.

You're welcome to the answer you want to. Hear but like I think. That's just the physics of the physics of the work.

Yeah. Yeah. No, there's a cost in in. Non collocations and I was just wondering if. There was anything that you come across that was? The silver bullet.

There's no silver bullet. There's lots of, you know, compensating ideas, right. And none of them are great, but they're. Maybe better than not doing them, you know. Next question, I guess, Anthony.

So do you use any method to manage cross team dependencies?

We don't have no stinking cross team dependencies. And that's a partial truth. It is true within those BIOS teams. That wasn't a problem, you know? And and they would. They could always go talk to their the their peers and another team to get help. And there's you'll see like, you know, component mentors and this sort of pattern. It is true that because of the compromise product boundary, they would have to work with other teams that were in this waterfall structure in order to get, you know, stuff that went up through the. The the administrative GUI working. And but there wasn't like some complicated, you know, dependency tracking mechanisms we used. We probably did something within. We were using Raleigh and we probably had, you know, some dependent tickets in Raleigh, but. This was much more organic than that mitrea. Any comments there I may have misrepresented. It's been so long.

Yeah, I think I think we. I'm also trying to remember if we had any at least significant cross team dependencies. We tried not to. Of course, you know, at least I can say that cross team dependency isn't something that stands out as either the problem that we solved or the problem. A problem?

Yeah, I just don't remember it being a a significant problem that we had. I mean, perhaps had we kept going and had other our other big problems go away, maybe it would have become so. But it it it just. We had a lot. Of big problems, and that wasn't. That wasn't at the top.

Of the list. Yeah, it it could also be unique to the, to the nature of the nature of the BIOS as as a software. And the, you know, literally if you imagine. It you know, if you open up a PC, yeah, like a desktop computer. You know, literally every every component of that visually that you see has. Some kind of a BIOS software equivalent that enables it, and so perhaps that's, you know, that's one reason that, you know, they're kind of. More or less independent, but you. Know we didn't drive. Certain teams to take on certain. Any any kind of specific user stories but. Whatever teams did take on. Yeah, I think the bottom line is I don't remember it. Being a problem.

Now, there were teams that had affinities, so yeah, there ended up being one team that had more of the middleware type skills on it that tended to do more of the work that that. That went up to that administrative console. And they were. Sort of. The leading team as it pertained to pushing into that boundary. So that's kind of a natural thing that happens sometimes. So, so teams would certainly have, you know, areas of stronger competency than the than their sister teams. Of 1 flavour or another.

Thank you. So do you believe collocation has helped implementing less effectively?

Yeah, definitely. I don't have a lot of. Everything you'll read about it is true about Co location. Forget about less, just it really does make a huge difference. I mean, I think Alistair Cockburn wrote about that years ago. Nothing has changed other than sure, the remote tools are better than they used to be, but they're still not the same as face to face.

Thank you. Yeah, I can see the pros and cons, but do you? I'm sorry, just another question. Do you have combined backlog refinement with all teams involved in a separate backlog? Refinement with each team.

So we usually did it. We pretty much did it as a group and it was pretty organic, but mitry. Can you remember this situation was a little different than the. You know, if you read the less books you'll see multi team refinement described which is which among other things, is focused on knowledge transfer between the the stakeholders who are and you know customers users that that are looking for functionality and the teams and there's a lot of effort on there's a lot of emphasis on. You know, direct interaction between the team members and these external SME's that that are using the product and have some need that that the teams are hoping to fix or to address. In this case, as Mitty explained earlier, to some extent. These blades had to come up and implement the same protocol that they'd always implemented in the previous generation of the blade, and in the previous generation before that, and in the previous generation before that. It is true that there was a tiny amount of additional functionality that you know, the administrative console might be able to do one or two new things. Maybe the new chipset enabled some new capability of some sort that needed to be integrated into the whole of the of the product as a whole, but mostly. They were implementing the same. API infrastructure that they had always implemented in previous generations. So it's 95% ninety 8% do the same thing that it used to do in the old generation of. The hardware. And so it. Is the the the multi team refinement? Activities were about the teams. Helping each other understand what needs done. And you know, we did. We did have refinement meetings and we would do. And I don't know and these didn't always align with Sprint boundary. Sorry. With team boundaries, it was, you know, whoever knew the most about these aspects would go and and do it. And then because of the weird nature of this particular thing, because we were, we were ploughing old ground that had been ploughed millions and. You know hundreds, not 10s of times before. So much of what needed to be done. Was flushed out in that in those first few weeks. And there was perhaps 3030 to 40% maybe of stuff that they weren't sure about that would come up as they discovered and did more work. Oh, we have the new thing to. Do that we didn't anticipate but the backbone. Was pretty clear and that's because of the. The nature of the product, the nature of where they were in, in trying to implement more technical excellence than historically it happened and so much effort was being put into improving the quality in this pass in a way that had never happened before. But they weren't at the they weren't in the place where the requirements were real or the needs were very dynamic in a very dynamically changing market. That would eventually. When the product boundary when you expanded it and things got better and you had a real product boundary, you would that would make sense. You know the market needs would change. You would need that dynamicism you would need people that are using the product in multi team refinement to do this kind of thing. But with this compromised product boundary. Or really component boundary. That wasn't the case to the extent it would otherwise be. Did that answer the question?

I might think so. Yeah. So, no other questions have came in. So I'm going to say thank you very much, James and Mitchell, that was very informative and yeah, so you can reach James on his website if you have any further questions or you can send them to agile in Ireland and we're very happy to forward them. And thank you very much, James. Thank you very much, Mitchell for your time.

Thank you so much for having me.

Thank you, Earl. Thank you. Goodnight.

One, thank you, James, for the second feeling to once again.