Bryce

There's an episode of the TV show Future Mama where um the protagonists discovered that cats are actually aliens, conquering aliens that have come from another planet, and conquered humanity into caring for them.

Conor

Welcome to ABSP the podcast episode 31, recorded on June 5th, 2021. My name is Connor, and today with my co-host Rides, we continue part three of the four-part conversation with Chamberrug and Patricia comments. On today's episode, we talk about strategic decision making, have this degree, and so much more. If you haven't listened to parts one and two, they will be linked in the show notes, and I highly recommend you listen to those ones first, as this is a continuation of that conversation.

Chandler Carruth

I'm just curious, should we try and get back to just keep going with the subject?

Patricia Aas

Mr. Cruz. Okay.

Chandler Carruth

Yes.

Patricia Aas

So one of the points here is not overcoming resistance to a directional change.

Bryce

Yeah.

Patricia Aas

What does that mean? Like can you give put that in example, explain how you would go about that?

Chandler Carruth

Yeah, I mean, I think this is so so first off, let's let's let's make sure we we understand what we're talking about. We're talking about times when there's you know there's there's a uh strategic decision or a directional decision um that needs to happen, and you have uh you're suggesting and you are proposing a particular direction, and that is not the direction taken, right? And you still need to, like you need to, like we're we're in the kind of disagree and commit phase, but it's not about like doing a single thing. It's about like a very large scale directional shift. And and I've seen this a couple of times, and I've I've I've had this happen to me a couple of times. Um so the particular like one of the particular sharp examples here is there's uh uh there's a technology that I I I do not personally kind of like. It doesn't seem elegant to me. Um and uh it's call it it's called uh uh thin LTO. Um this is uh a variation of link time optimization uh developed by uh Teresa Jackson and and David Lee and a bunch of other people at Google. Um and personally I don't like the way it works. Um I it it it seems like a messy workaround, and so I advocated really hard that we should actually invest in understanding why but but the goal of it is to make link time optimization scalable enough to use broadly on very, very large C applications, uh huge C applications. Um uh LTO normally, as it normally is, just doesn't scale very well. And so you can't use it for these applications, and thin LTO scales very well. I had argued that we should actually try and tackle the underlying scaling issues of LTO rather than taking the thin LTO approach. Um I thought it was much more clean. I think there's a lot of of really exciting untapped potential in that space. Uh basically every link time optimizing compiler uh that we had access to, which was basically one and a half LLVMs and some some uh early stage work in GCC at the time, was single-threaded, right, and built around linked lists, and so so like has the worst cache locality you can imagine. Um and and so it doesn't scale well, which is kind of unsurprising. And I was like, we need to fix those problems before we try and kind of change the fundamental model to make it scale. Um uh but Teresa had Teresa and David had a great uh idea for how to change the fundamental model, and it didn't require changing all of the internals of the compiler, it didn't require you know making things you know multi-threaded and efficient and all this stuff, and it still scales. Um and uh the decision was to basically go with their approach. Uh it was it was uh and at the time I really disagreed with this. I I thought this was just just the wrong thing to do. Um uh and it was it was kind of hard because right, like I had to I had to help them out. I was bringing up LLVM uh at Google, having brought up Clang as the front end, I was bringing up kind of the LLVM back end for actual compilation, and they were working on thin LTL and needed needed my support in LLVM. Um and and that was hard for me. Uh uh I struggled at first. Uh and and largely I I think finally we we got to a much better place where I found ways that I could kind of contribute to to enabling them um rather than kind of getting in their way. But it was difficult. Um and the funny thing is that like in retrospect, right, like if if if today me got past me and Teresa and David Lee's uh uh kind of different proposals, I would absolutely pick the NLTO. Um so as much as it's less elegant, it's also dramatically less risky. And at the time I felt like I I really didn't see this risk that was associated with the direction I was pushing for. Uh but in retrospect, it actually makes a lot more sense why we ended up going that direction. I think it was the right call, even though uh uh personally it felt really like, oh, this is this is a good technology. And it also worked. And for me, the biggest part of this kind of disagreeing committing at the strategic level was uh, you know, I I had a loud voice in the LVM community, and it was important to actually acknowledge like this is working. I had to do that very loudly, and so you know, making sure like Teresa could get up on a panel to talk about LTO. Um so I got invited to a panel, I was like, we should probably also bring Teresa up because like you know, I advocated for a different strategy of LTO, and hers is the one that actually got built. So maybe we should not Right, like and and doing that is you know, it's it can feel bad, but that's that's the important part of of kind of disagreeing committing at a strategic level, is that you actually have to own you actually have to own the strategy at the end of it, and because it's a strategic thing, right, it's not just like you're going to let the patch go in, right? You you have to contribute to the new strategy. Right? You have to help debug things, you have to help with the planning, you have to like figure out how the piece of you know LVM that I'm working on can actually call like work with the thin LTO strategy, how to integrate this stuff, and and kind of you have to pull yourself on board with that, uh, even if it feels like the wrong strategy.

Patricia Aas

But your your your negativity towards the concept probably spread the people around you as well. Because we are we are we you know we are uh tribal in in in how we work, we are in our groups in a and oh of course uh so did that cause any friction by when you were trying to to I'm sure, I'm sure it did.

Chandler Carruth

And and like I can remember specific places where it did, but because there were people who who really thought I was right and I should have been able to make this decision. Um and one of the interesting things, and again, this this is a great observation, right? Because when you when you when you want to actually overcome this kind of uh uh uh having a direction that you don't agree with being decided on, um you actually have to undo that negativity. You can't you can't prevent it. Like there's there's no hope, right? People are going to, as you said, this is this is a natural thing. But you have to figure out how do you undo that. Uh and and one of the ways that you undo that, for example, is is by going and doing the work, right? Do the do the do the groundwork, the dirty grungy work, to actually help out the direction you didn't want. And and let people see that, right? Because that helps people then understand that, like, oh, okay.

Patricia Aas

If Chandler, if Chandler's okay with it, then I guess it's okay.

Chandler Carruth

Right? Like And you have to be an active participant in kind of undoing it.

Patricia Aas

Yeah, because there are there will be people who will feel an intense loyalty to you, and and so therefore, yeah.

Chandler Carruth

I I have to Yeah. I also want to be clear, I don't think I handled this this particular case perfectly. Part of why I'm bringing this one up is that this one is probably the one of the rougher cases. Um and uh it you know I think I think there's still uh kind of uh ruffled feathers, and and it still is difficult for people who are on different sides of this to kind of work effectively, and we all lose as a consequence, right? Like that's not a good outcome.

Patricia Aas

Is there a way that you could have handled it differently that could have maybe given a better result?

Chandler Carruth

I should have been much more vocal earlier. Um so that there was a lot of lag time between kind of the decision getting made and when I was able to kind of you know do some of this legwork to support it. Um and I think that really uh amplified a lot of the problems. Um and what would have been better would have been to have kind of very immediate like decision made and immediately show signs of like helping implement the new the new decision.

Patricia Aas

Um but and for me, like when I I read these things, it seems it seems like an upward thing towards management that I have to get on board with this thing uh because the decision has been made and and you know, so that the management or whoever's above me will see that I'm a team player. But and and that doesn't resound much with me. But but what you're saying is more that I need to show these people who are loyal to me and who respect me that you know I'm okay with this and we should be okay with this together, and we should, you know, yeah, that makes much more sense to me.

Chandler Carruth

And it's it's wild the Offexus has like this this you know uh that delay, which I I I still really regret to this day. Um uh right it hurt team morale, right? It it caused you know parts of the team to drift apart, right? How many missed opportunities were there because we let this friction and I say we let because that is part of this, right? We let this friction develop. Um and and it's gotta be a lot, and that's that's you know, it's it's a regret.

Conor

There's a a part of my soul is sad that I'm bringing this up, but uh it is a good leadership principle. There's so there's at Amazon, there's these 14 LPs, which stands for leadership principles, and uh Amazon is famous for you know um really promoting these across culture and whatnot. And one of them is disagree and commit. And it's actually it's actually a pretty good one. Some of them I don't I don't agree with all 14 of them, but this one it's basically like there's gonna be times where there's different uh directions that you can go, either technically or strategically. And you know, uh they're really big on data-driven design and and stuff like that at Amazon. But at a certain point, a decision has to be made, and basically the LP talks about how once that decision is made, uh you you can say, you know, I don't agree with this decision. I think it's, you know, I prefer this one, but we're going with this one. And once that decision is made, um, like it's supposed to be like everyone's everyone's on board with it, commits to it, and they're gonna make that successful and work. And and there's an opportunity to sort of state that like I don't agree with this final decision. Uh but once that's that decision is made, it's it's time to you know go forward. And um yeah, it's I don't know how easy that is to do. And it's it's definitely not easy, but like having that as an LP at a company, like as as a sort of a tenant, like I definitely have worked at companies where those disagreements happen and then they just sort of like they just linger, and um it's not even clear like uh you know how do we come to the decision. And so they just sort of it goes on and on and on, and um it's it's I I think it's a great sort of tenet to have that like you know, there's inevitably at any company there's gonna be disagreements, but at a certain point you just gotta pull the trigger and um you everyone's gotta contribute to that that direction at that point. And uh, you know, maybe two a year down the road you'll figure out it wasn't the right decision, and you can reassess at that point. But you know, for that year when you're getting to that point, it doesn't help to sort of like have only 50% of the team that believes that that was the right decision to be really uh you know all cylinders ahead on it and and uh or or have 50% not all cylinders ahead. That you know, that's not helpful. Uh you you're gonna get to you're gonna realize the decision was the wrong one if it was the wrong one faster, if everyone is just helping out.

Chandler Carruth

Um I want to be very clear about something. I I having the tenant is important, but taking it back to Patricia, right, the the questions like like for interviewing someone in a leadership role, the the particular key thing here is that people in this role need to really show that they are operating by that tenant. And that's even more important than the tenant itself. We we have the tenant at Google, and it still happens because whenever there's a leader that isn't showing they're going to live by this tenant, it starts to cause like like it it starts that entrenchment and that that bad.

Patricia Aas

I I I think we uh I think it's a lot about the human condition. We are often very proud people, and we also have lots of loyalty, and and having these things will make it difficult to say, okay, I lost, and also I'm not only okay with losing, but I'm enthusiastically on board with you.

Chandler Carruth

Yeah, yeah, it's really hard to do. That's a great way of putting it, yeah. Um so some some techniques that I've come up that I've I've been trying to practice to help me overcome that human reluctance, uh, is is to change the the kind of frame of mind of what I what I what my goal is. Uh there's there's uh the like I'm not a huge usually a huge fan of corporate trainings, right? Sometimes they're they're real they're real they're real questionable. But there's one corporate training that I really love. Um even though most of it's garbage, it has this this one catchphrase that I I keep coming back to, which is uh uh you know, what do you really, really want? And then they clarify No no no what is the one thing that you really want? Like you must choose one thing and focus on that. And when you apply that here, the interesting thing you get to is that what you really want is a decision. Not your decision, but a decision. And so by focusing on the fact that you've made a decision, you're making progress, you basically channel your enthusiasm for progress as proxy for the enthusiasm of the decision you don't particularly like. Right? And this this kind of lets you get yourself enthusiastic and then apply it to something that you're not super enthusiastic about because of the.

Patricia Aas

Okay, so get a proxy. Yeah.

Chandler Carruth

Yeah. And it's all about picking, well, what do you really want? Do you want your decision or do you want to have a decision? Because you can't always have both. So you have to pick which one you care about most. Once you do that, it's much easier to be enthusiastic about not having your decision. Because that's not what you're enthusiastic about. You're enthusiastic about making it through the decision, getting the decision, making people start moving in some direction, right? Um and and this for me was really helpful in part because I noticed something uh and and a manager of mine, you know, years and years ago told me this, and I didn't I didn't get it. It didn't make sense at the time. But he pointed out that it is much cheaper in so many cases to let people make the decision they want to make than to convince them to make the decision you think is right. Um and then to your point, Connor, eventually, if it is wrong, they'll find they'll figure it out. It can be so much cheaper to let people make a bad decision and then learn how what the right decision is than to try and prevent them from making a mistake or trying to avoid the wrong decision. Just just make the bad decision. It'll be totally fine. We'll figure it out later. Um and that can be so much lower cost, um, especially at larger scales.

Patricia Aas

I I yeah. I have a baby example from when I was like a super junior. Um I I was at uh Opera and we uh we were working on the desktop browser, and we had uh uh this guy who was basically I don't know, he I I I guess in in in retrospect I would say he was like sort of like a creative genius, but he was also a coder, and and so he he had created all sorts of things, and so uh basically a lot of what was opera was basically his brainchild. Um and so um and then he comes and he was like he has this brilliant idea in his like his opinion, brilliant, and all of these people that you know worked with him for a long time, they you know he was like a superhero, they were like, This is brilliant, and and I thought it was dumb. I was like, no, nobody's this is silly. Like, are we gonna do this? This is a really silly thing, and I was so wrong. I was like the wrongest in the entire world. Because what he I don't know if anyone remembers, but way back when, in the early 2000s, when you opened a tab, it was white. There was nothing there, it was just a white rectangle in your face. And his idea was let's put like big icons and things there for your favorite websites. And he called it a speed dial, and it would have like the nine, like a speed dial, and he would even connect it to see if you tap the right like number, it would open that, but you could also click on them, and so you had like a three by three matrix where you could put in all of your favorite websites with like a picture and everything. And so when you opened a new tab, you would see the speed dial, and you could just click on where you wanted to go. I was like, this is silly, this is stupid. It's like if I was I was I was so wrong. I was I was absolutely completely terribly wrong about this entire thing. And and and for me personally, this was huge for me because and it's colored like my entire thinking for the rest of my career. Because I I learned that I can be fundamentally completely wrong about something and have no idea. Like and because I think we have this tendency to believe in ourselves. Like, I you know, I think this is dumb, and I will all data I will find will reaffirm my position, and I would kind of hold on to it no matter what happens. But actually sitting down and and like accepting the fact that this was brilliant and and it has been copied by all browsers ever since. You know, it's like 15 years later, this is what you do. Um but but so so it was brilliant. Uh but having that gave me the the the I don't I I don't I won't call it humility because it's not humility, it's it's a healthy um skepticism to to my own like uh opinion and a willingness just to like be wrong, yeah, and and like and see that like good things can happen even when you're wrong. Yeah, and just like okay, so this is what I think. I might be wrong, like there might be like a fundamental flaw here where I'm just totally wrong, and if I am, that's okay. And and that's like a superpower. If you if you can actually go walk into a room and be convinced by someone else, and on like really truly convinced that I think that what you said was right and what I said was wrong, that that can build like mountains. Like it's it's you it's massive. And it's such a small thing.

Chandler Carruth

I mean, I I do this, I I I freely admit I struggle with this. That's why I cheat. I cheat by focusing on on uh you know progress over.

Patricia Aas

Yeah, but I I think that's a good one because I think that's something that you know you can do because it I it hurts to be wrong, it sucks to be wrong.

Chandler Carruth

It really does.

Conor

It's interesting this is coming up because literally just on the last episode, Bryce remarked the exact same thing that his telltale sign of um really bright folks is the ability to like once they've realized that they're wrong, just to like unabashedly like change 180 degrees and start 90 degrees and start walking that direction as if like they never at any point previously thought otherwise. Like they'd oh, I'm wrong? Okay. That's fine. Yeah, yeah. And yeah, something something you were saying earlier, and this sort of goes to the your example, Patricia, is I I'm not I actually don't think this is an Amazon thing, but I've learned it in Amazon. They have this policy of like one-way and two-way door to decisions, and that like if it's a two-way door decision, meaning that you can go enter and then leave, um, those are the ones where it's a lot easier to just like you know disagree and commit. And we'll find out, and that's the thing is like designing that sort of nine three by three icon, if it's a terrible idea at the end of the day, you just you just trash it, sure there's a sunk cost there, but like there's there's very, very little cost to like reversing that. And there's certain decisions where it's one way. If you're gonna make this big change, and those are the ones that you really have to spend a ton of time up front, you know, figuring out is this The direction of the Chandler, you're saying maybe not even that one.

Chandler Carruth

No, no, no. Like I think I think that's that's that's actually a critical flaw. The goal shouldn't be to spend a ton of time up front figuring out the right decision when you have a one-way decision. The goal should be to engineer the door to open the other direction. Yeah.

Bryce

The goal should be like what one one-way decisions are never good. Um like what one of what the the classical example of this is standard C. Like the one of the big problems that Standard C has is that pretty much all of our decisions are one-way decisions. And we would have a lot less stress if uh if we could find ways to have that not be the case.

Chandler Carruth

Uh this is something that that I see in API design all the time. A lot of people feel like you know, an API is a one-way decision, because once you publish it, you have to support it forever. And and so so then a bunch of people invest incredible amounts of time trying to do good API design and to avoid mistakes. And and my perspective has really been warped here to a different thing where it's like, I mean, sure, invest some amount of time there, but you almost want to time box that very sharply, and instead invest all the rest of the time in getting out of the mistakes that you will inevitably make. Um because one of these things is is has a different scaling function on it than the other one. Right? If you if you invest a bunch of time making a good decision, that's one way, great. But the next one right? Like, it doesn't none of that investment pays off for the next decision you have to make. Right? Whereas if you invest a bunch of time making the door two ways, all of that time you invest pays off every time you get the decision where that door applies. Um and so it's it's a tremendously better scaling function. And so you time box the non-scaling investment, right? And put all of the rest of the energy into changing the problem space.

Conor

That's a great point. It's it's just it's basically like from finance or business, it's fixed versus variable costs. Like you do the two-way door and it's variable, and then that just distributes over every single decision you make in the future. Yeah. Actually, that's a it's a great meta point. Like Amazon, you should update your book. Update update your stuff.

Chandler Carruth

Uh but there's another kind of other subtle problem here, uh, which is the time spent before a decision is made is the most expensive time. So one of the interesting things about spending your engineering time on the doors instead of on the decisions is that you make the decision more rapidly. Right? So you time box your investment in the decision, you make the decision rapidly, and that that has its own value, right? And you're not paying for an unmade decision the entire time you're investing in doing the door. You're just making bad decisions.

Patricia Aas

So can I can I take us down from metaphor to concrete? Like what is a door?

Chandler Carruth

Yeah, absolutely.

Patricia Aas

Here.

Chandler Carruth

So so classic example. Yeah. Let's take APIs. I think C APIs are a great example here, right? We can we can try to do good API design, but the thing we actually want to invest most of our time in is the ability to change the API when we get it wrong. And so we build refactoring tools, and then we build large-scale refactoring tools, and then we build systems for running large-scale refactorings across a monolithic code base or across all of the shards of your sharded code base, and you and then you tune that system so that you can run them more rapidly and at lower cost. For example, at Google, one of our big costs is we're lucky we have a uh a monolithic repository that we can handle, but we we have to do code review. And so when we maybe we're making 10,000 refactoring changes, how do you code review those? So we've actually built tools and infrastructure to smooth the code review cost and to bring that cost down, build tools to smooth the testing cost of rolling out these changes, right? And what we're doing is we're because because you you almost never have true one-way doors. What you do is you have uh a differential cost function based on the direction, right? That it costs us so much money or time to go in, and it costs us, you know, so like this massive multiplier if we ever want to go back out. What you're actually trying to do is we're trying to drive that cost down over time. And we've we very specifically just attacked the economic problem of replacing an API when we discover a mistake. And every time we make that cost lower, right, this makes the kind of uh risk of each API decision lower. Because we're like, well, maybe we're wrong. And so we can still apply some time-boxed amount of like it's there's still a differential here. It's not it's not free, right? But now we can have a much more rational time box around like, well, we've we've we've consulted the experts, we've thought about the consequences of this API design, to the best of our ability, it's going to like we think this is a good one, let's try it out. And if it doesn't work, we'll replace it, right? And even for a particular API, we like you can you can turtle this all the way down, right? So for a particular API, past a certain point, rather than trying to theorize about how the programmers will will f will react to this API and how readable or not readable it will be, if we're particularly uncertain, we'll actually invest in being able to roll back that particular API uh API. So we'll have you know uh an allow list early on for who gets to use it so that we can avoid it you know being everyone on day one, right? And we can kind of have some beta testers and get some rounds of feedback. Um we may put special things into the API just to make it easier to change if we discover we're wrong. Um and again, it's and and the whole thing here is just about decreasing the differential cost so that so that we can make the decision and not worry about it, and then learn and then make a new decision when when we get new information.

Conor

Thanks for listening. We hope you enjoyed. Stay tuned for part four, our final part of this conversation, and have a great day.