Ep 146 AI: Premortems, Because A.I. Won't Wait for a Retrospective

The "ship it, fix it later" playbook was already broken. AI just made it catastrophic. Post-mortems assume you have time to observe, adjust, and course correct. AI has collapsed those timelines, and the damage isn't waiting for a retrospective.

In this episode, Stephen argues that pre-mortems aren't a luxury anymore. They're the only way responsible teams can keep pace with and scale AI-assisted work. He walks through six questions every team needs to answer before any project where AI touches the workflow — not as a checklist, but as the conversation you have before the pressure hits.

Stephen anchors the argument in a core frustration he's named on the show before: most teams are in the consequence business without ever acting like it. From the Pokémon Go failures to the output-obsessed cultures that celebrate shipping as success, the episode makes the case that individually good decisions can still add up to collectively disastrous outcomes.

This one is for any team using AI in their work who hasn't yet had a real conversation about what happens when it goes wrong.

In this episode:

• Why the post-mortem era is over — and what has to replace it
• The six pre-mortem questions every AI-assisted project needs answered upfront
• How to map the blast radius before you're in a war room during a crisis
• The headline test: writing the failure story before the project starts
• Why agreeing on non-negotiables is the easy part — and holding to them under pressure is the real work
• Why AI is a culture problem, not a technology problem — and why this episode is the proof

SHOW NOTES AND MORE:
https://thecrazy1.com/

WATCH MORE CONTENT ON YOUTUBE:
https://www.youtube.com/@StephenGates

WORK WITH CRZY:
http://crzydesign.com/

FOLLOW THE CRAZY ONE:
LinkedIn, Instagram, Facebook 

Transcript

SPEAKER_00 0:00

So we've been talking for a little while now about AI on the show. And today I want to sort of talk about what I see as this intersection between what I think are going to be two at best troubling, at worst, sort of catastrophic trends that I think are about to collide in a lot of different industries. Because I just think, look, it this isn't a new conversation. Because look, I think a lot of teams have been running the same playbook for a while. It's one they probably always run, right? That sort of like ship it, see what happens, fix it later. And the two trends that I'm seeing are are one, I think what qualifies as ship it has really slipped, right? It's really just gotten to a place where I think, like, look, they're all the poster childs that we know about. You look in the app space, you look at Spotify, right? Like they shipped an app that was so bad it got their CEO fired. I I think about something like the video game space, where there were titles like from big AAA studios like Anthem or Cyberpunk, where again they shipped and were functionally unplayable at launch and for a while afterwards, because there were so many bugs or mistakes or just bad mechanics. In the case of Anthem, like it cost them the whole the the entire title and something that you could tell they wanted to invest in as a platform. And I think the result of that is that consumer behavior has changed. I know it has for me. I I won't pre-order games anymore, right? I want to wait for them to come out to see what they're gonna be like, to give them a rev or two to be able to get to that space. I have the auto updates for my apps off because there's just too many times where I get ones that the new update, it's not functional. You can tell it wasn't coded right. It crushes my battery all of a sudden, right? Like there's just basic mistakes. I uh even with somebody like Apple, right? For a long time, I was in the Apple beta program where I'd get the betas of their, you know, iPhone OS and tablet OS and Mac OS, whenever they were in beta, not the developer previews, right? Like those are meant to be messy. But whenever they'd get to like the beta, and I opted out of that program because, again, there are just too many times where it would render my machine functionally unusable in really important ways. So that's been been going on for a while, right? We're just that setting the arbitrary deadline to launch a product, where again, the pressure to ship because we need revenue, because our margins are too tight, right? Like there, we now all know all the reasons why. But there's just sort of what qualifies as something that should ship, what qualifies as a good product has really fallen away. And I think a lot of that has been because of this kind of like ship it now, fix it later culture, right? Like that's a playbook that has been flawed for a while. The other trend that I see coming into this that I think is going to make it honestly kind of almost dangerous for a lot of businesses is AI.

unknown 2:57

Right?

SPEAKER_00 2:58

Because a postmortem assumes that you observe, you adjust, you course correct, right? AI is collapsing those timelines. It's it's collapsing the decisions and it's really speeding up the consequence. What used to take days to produce now takes minutes. And look, and I think that the what too many companies are finding out is that the damage because of this collision, it's not waiting for a retrospective. So that's what I want to talk about today. I want to talk about pre-mortems and why I think these aren't optional anymore. So, in this case, welcome to episode 146. As always, I'm your host, Stephen Gates. This is The Crazy One, the show where we talk about creativity, leadership, design, everything that matters to anybody that does anything creative. As always, if you want to go deeper, right? If you want to find out about that, you can follow me on social media. You can follow Crazy Design, which is my studio. You can, again, follow the show, subscribe, tell your friends all the good stuff that everybody says in every video. So I candidly don't even know why I say it, but I just feel compelled to, I guess. But, and so it's been interesting for me, right? Like pre-mortems are something I've talked about for a while. This was something that I first raised way back in an episode that nobody listened to on design ethics. 10 years of the show, least popular episode I ever did was on design ethics. And the whole concept then, before AI was even a glint in anybody's eye, was that, you know, as designers, as product people, as executives, as leaders, we are in the consequence business. And I'm not sure we always look at our role that way. That like a pre-mortem isn't just a checklist, right? And I think any process at its worst is a checklist. This is why, with almost every process, anything I do, I don't necessarily care about documentation nearly as much as I care about adoption. And I think this is the conversation you need to have before the pressure hits, before you ship, right? Before the deadline and the client pushes back and somebody says, look, we need to just ship it. Because we know that's going to be there, that it's not new, that doing things like, you know, actually writing a product brief or research or design are often labeled as slow and problematic, and why can't we just ship? So, what I want to talk about today, right, is what I've really come up with as what I think are the six things that I'm doing on every project that I think everybody else needs to be doing and the questions they need to be asking on everything they are doing without exception. Because I think these are the things, if you can think it through, if you are asking these questions before you do something, right? And and before the consequence comes. This is why for me I understand where retros came from, right? I deeply understand what agile means and a lot of that sort of stuff. I just think it it is also a methodology that's sort of just from a bit of a different time. Whatever it was, yeah, again, we're coming out of waterfall, we're doing those sort of things. Things were just different, and we need to evolve. And so these are the six things that I think everybody needs to start asking. I think that the first one, very specifically, is to actually go through like what does good look like? Right? Not the brief, not even what the brief necessarily tells you to make, right? This is about what is the standard you're going to hold the work to, whether that is quality, accuracy, integrity, what does it mean? Right? Like, what is something where we're gonna go like, oh, good enough, right? Like, this is why I hate the word just, right? Because it's always followed by a compromise or something where something's kind of belittled. And look, and I think this is something where for a lot of teams, what good looks like is just producing something, is shipping. That's what a lot of the episodes before have been about, is that for creatives and for a lot of disciplines, we've sort of been reduced to the executional aspect of what we do, and there's a huge problem there. Because here's the thing, right? At the end, if nobody can agree on whether it's good or not, right? Like, then it probably isn't something you should start doing. Too many creative teams, right? We just accept somebody else's dumpster fire, somebody else's mess, something that wasn't thought through. And then when we take it on, we are then on the hook for delivering something. And I've talked countless times until I'm blue in the face about how there aren't designs or UI patterns or typefaces or colors to solve some of the fundamental problems that are baked into the work that we are handed. And again, I think this is the question, right? If this ships, what would we be proud of? Would this be embarrassing, right? Like name both of them. Do it out loud, right? Before anyone opens a tool, before you do anything, define what those guardrails are, right? What are we going to be proud of? And then what is basically that line where if we cross this, if we go below this line, this is something we just should not do. And we need to develop a process where, again, we figure that out long before we are into production and coding and doing a lot of these sort of things. Because I think the other part of that, right, like is to the second question, going beyond just a word like embarrassing, right? Because I think that that can take on a lot of forms and it probably isn't specific enough. But I think for me, the second one is like to really map out, for lack of a better term, map out the blast radius, right? Meaning if this goes wrong, how bad does it get? How fast is it going to get there? Can we undo this? Because that's been the thing, is like, look, this isn't, I'm still such a huge proponent of doing A-B testing, of doing research, of doing design thinking, of like iterating quickly and putting things out there. That is not what I am saying, right? This is the we are skipping most of that just to throw something in market and see what happens. And I think this is where whenever you say this, right, when you start to map out the blast radius, I think the problem is this is where I think a lot of teams just have a really hard time being honest. Right. I I think they think it is about like the most likely outcome, not the worst realistic outcome. Right. Because it's it's who wants to think about it, right? Who wants to sit there and go, like, okay, like what's the worst outcome that there can be? And again, there is such a litany now of companies and lawsuits and missteps of companies that have put products out there, that have gotten them sued, that have cost hundreds of millions of dollars, that have cost CEOs their jobs, right? We're not wanting, for examples. What I think we're wanting for is that again, it is it's not, it's seeing it not as paranoia, but what I would view as like honestly, just risk mapping, right? What is that brass blast radius going to be? And then I think to go a step further to say, like, look, what are the triggers going to be? What are the sequence of events that have to happen that are going to get us there? And I think this is where AI sort of makes this critical, right? Because in a lot of cases, if we're turning things over to agents, to AIs, with in a lot of cases, I think we are unjustly trusting it with too much. Is this something we can back out of, right? Can we get it back under control? Because that's the thing, is like, look, in a lot of cases, like if print runs, that's physical, right? Like that's not something that, again, it's more controllable. The words are controllable. AI generation, in a lot of cases, if it is business decisions, if it is content, if it is experiences, if it is customer interactions at scale, that's something different, right? And again, I think Amazon did this for a long time, where they would write the press release of like, what is the press release when this is going to be successful, right? What does that look like? I think we need to do the headline test, but we need to do the other side of that too. If this goes wrong, what does the story say about us? Right. Name that headline out loud in the room before you start so that everybody is clear and remembers what the potential consequences are. Because that's the other part that I know so well that as you go through the process, of course, right, as you go from the spark of inspiration to the reality of execution, things change. Compromises are made, right? That is just a natural part of the process. But I think too often we get heads down in the minutiae and we lose the bigger picture of what could that potential outcome be for both good and bad. Right. But again, I think most everybody's cutting up the confetti and planning the ticker-tape parade and not thinking about the alternate. And I just think that that's my fear is that I've seen enough companies and enough cultures that if you are not strong on this, this is why I think plausible noise is running so rampant, right? Because if you don't have a standard, if you don't have a culture, if you aren't talking about this stuff, plausible wins. And then these outcomes happen. And I think, you know, the headline test is sort of an easy way to ease the team into this conversation. But I think you also need to be more specific, right? Like actually sitting down and naming what are the non-negotiables up front, right? Before the pressure hits, what are we absolutely not willing to do? What are the things we will not compromise? What are the things, right, that we will not sacrifice this sort of accuracy for speed? We are not willing to skip the legal review. We're not willing to let AI generate content that hasn't been reviewed before it goes out, right? Now, again, that can take a lot of different forms, but what are those non-negotiables that are agreed upon from all parties? And again, I think that in a lot of cases for me, as with doing any of this work, getting people to agree upon it is probably the easier part. I think the tougher part often is writing it down and holding people accountable to it. Because that's the reality, right? Like once you're in the middle of it, once you're under pressure, once there's a deadline, somebody's gonna suggest crossing one of those lines. Somebody's gonna suggest making a compromise or saying that it's okay or we'll figure it out later, right? Like this is where the eternal joke about, like, oh, we'll get it in, you know, v1.1 or v2 or something like that comes from. And I think if you haven't named them in advance, they're not lines, they're not boundaries, they're not standards, they're suggestions. And I think that's often the problem. That's why, again, like, you know, for me, this is where, and this is an example I cite all the time. You know, for such a long time, everybody loved to point Pokemon Go, right? Great app, fastest of 50 million users ever. I'd I'd wear people out with this example. This is a game that got kids killed, right? It sent them onto private land and got them shot. This is a game that put a poison gas-releasing Pokemon in the National Holocaust Museum, right? So, yes, while it was widely adopted, it had real consequence on things. And that was because there were teams that didn't talk to each other, they didn't map out and do a premortem of what could the consequence be of if this series of things happened, that there is private land, there is protected land, there are places where certain things are not going to be acceptable. They didn't map it out, they didn't name it, right? And so as a result, each team did their part. And that this is the problem with so much of this is that individually good decisions can make collectively bad experiences. And I think, you know, that's where too many of us just go heads down and go, well, hey, I did my job, I did my part. Who's watching out for the bigger thing? And I think now, especially as we get into AI, this is the fourth one, right? We need to define the guardrails specifically. And I think this is the part to me that I just don't really see many companies doing. And I think this is the thing that of all of these is probably going to cause the most damage. Because this is mapping out, like, what is AI allowed to do on the project? Where does human judgment have to come into this? Where does a sign-off or something like that need to take over? What is the review cycle we are going through, right? Like, what is getting what and this is as with all usual failures in culture, most corporate culture, right? What is the transparency and the accountability that's going to be here? Right? Who has the authority to hit the eject button and stop this if it is going sideways? And look, and I think for a lot of people right now, there's still this feeling this is sort of like a hypothetical question. And look, and I think that you know there's a difference between some of the smaller mistakes and then something that gets 10 times bigger, that it is trusted with too much, that where again, you know, it will destroy careers, it's going to destroy companies. And I think I don't know. I'm I'm part of me knows we're gonna have to have those examples, right? We're just gonna have to. We as humans just seem to need it that we can't stop ourselves before we slam into the wall. Because that's the thing, right? AI doesn't hesitate. It it doesn't have a gut that something is off. It is just there to produce, it is there to do it at speed, at scale, and it is going to keep going until you tell it to stop or you give it different guardrails, right? Like the guardrails, even then, they're not gonna slow it down. They're the only thing that stands between you and some sort of crisis you can't reverse. And so again, I think that's where we have to be able to sort of build in this process. This is why I keep coming back to you, and I'm gonna be screaming, I know for years to come that AI is not a technology problem, it is going to be a culture problem. And I think a lot of this is also when you go through this to get into the fifth piece, that you've got to take the time to really think and surface what are the hidden assumptions, right? What are the things that you are assuming are true and then you haven't actually verified? This alone has been a trend that has been rampant. I've seen it in so much of my career. Like when you work in a regulated space, I've worked in in banking, I've worked in healthcare, right? I've worked in a lot of those things. The number of times when somebody would say that this is something that we're like, we had to do or we couldn't question. Well, at the end of the day, whenever you actually verify it, when you took the time to talk to legal, to talk to a regulator, to do that thing, you found out no, it was just an assumption. And now again, I think assumptions can come in a lot of forms. They can come about the audience, they can come about the data, the message, the market, right? And I think it's to AI in particular, it is really starting to understand what are the assumptions that the model is making? What biases is it carrying into that work that maybe you haven't imagined or you haven't actually interrogated? What sources is it doing? What is the standard it is holding data and decision making to? And again, I think that this is often maybe even the harder version of that question is what's the voice that's missing in the room, right? Because people most likely, right, you don't like to think about the harm that's made by a bad decision, right? We we just don't. You just you want to assume the positive outcome, the winning the award, the doing those sort of things. And I think for a lot of us, right, like that the bad outcome is usually kind of the furthest from the table where the decision gets made. We're all thinking about our bonuses and thinking about things like that, right? The award, the executive recognition. But but I think to think about, as we always should, right? Think about customers first. Think about specific communities first, right? Like people who don't look like the people that are in that room or on your team, right? Are they being represented in that premortem? Are they like, is that part of that full blast radius? Or are you just seeing your own blind spots reflecting back at you? And this is why for me I've been such a long proponent of the of actual thought diversity in teams. That, you know, we tend to have the cognitive bias that we like to hire people who agree with us. We like to hire people that think the same way that we do. But the companies that are the most successful recognize that bias. They bring altering perspectives into what it is, they bring different skill sets, they bring those sort of things, and that's why they do it is in those moments to surface those hidden assumptions. And the last one for me is that, and we've touched on it a little bit as we've gone through what is the clear ownership and authority on this, right? Who owns what on the project? Clearly, not in theory, right? That who actually owns what part. And more importantly, who has the actual authority to say we need to stop? If something is going wrong, if it's going sideways, if we crossed a barrier. And again, this is a common problem. When you look at why 75% of most corporate transformation work fails, it's because people are not empowered to make decisions or to say this needs to stop or we need to review something, right? They are just quote unquote doing their job. And I think this is the thing that needs to be named and worked out before the work starts, not then improvised in a war room during a crisis. Because again, I think when something goes wrong, especially with AI, it is going to go wrong at speed. And you don't have the time to figure out who's in charge. The damage is going to be immediate. We don't have time to get everybody together and say, okay, now who's in charge of this and who's doing that, right? Like minutes will matter. And again, I think that because if you aren't doing that, the reality is going to be that by the time those conversations happen, the damage is done. The press is out, right? Like everything is going to be an ongoing crisis. And I think that this is the part that I think most leaders don't want to talk about. Because here's here's the hard part of this, right? Because I think naming authority usually also means creating accountability. And this is why in a lot of cultures, we don't, people don't like that, right? You just want to float around with, you know, and go with the prevailing wind of sort of go with the status quo or whatever this person said, that's what I'm going to do. It's easier, right? There's there isn't that accountability. There isn't that sort of clear, you know, kind of one throat to choke model of like this is the person that is responsible and accountable, and that there's transparency around that. We got to grow up, right? We just we've got to. And I think this this is the part where I think having this show for me gets frustrating, right? Like I'm tired of talking about the same shit for this long. I just am. But you still see it day in and day out and week in and week out. I still see the same thing. But let's let's kind of maybe let's let's try to figure out how to wrap this up in a little bit of a different way, right? Because here's what I know is gonna happen when probably some of you or a bunch of you don't listen, right? Or you don't carry the message, or your bosses don't want to listen to this, right? Is that I think you know, the the common response to this, and I know because I'm already hearing it with some of my clients and some of the people that I've been talking to, right? We don't have time for this. It's going to slow. Down the process. And no matter how many times I try to convince people that if you think doing it right is expensive and takes time, let's try doing it wrong. And I again I think that this is the thing, right? Doing it wrong is always going to be more expensive, right? The the companies that we're already seeing that aren't doing the pre-mortems, they're paying the price. And, you know, that's gonna be like, is it gonna need to be where every company's need gonna need to have one of these? And this is this output versus outcome problem we've talked about on the show too many times, right? Teams are focused on shipping, on the output, and they forget to ask whether or not what they are shipping is actually good. Because that's the other part everybody seems to forget is once you ship, yay, you hit it. That's not what anybody remembers, right? They remembered did it hit the goal or not, did it have an impact or not? Did it like, and that's the thing. If everybody's just celebrating like shipping goals as being an accomplishment, you're probably working in a wildly mediocre company.

unknown 23:01

Right?

SPEAKER_00 23:01

Because that's the thing. Whether it achieves the outcome or not, whether it does what it's supposed to without causing any damage, right? Like they didn't anticipate. AI is just gonna make all this worse. Because, again, like I've said, AI is like the greatest output machine we have ever built. And it's only getting smarter and being trusted to do more. And it's going to produce faster than any team can review it, right? Faster than any process is gonna be able to catch up with it. And the companies that are gonna win aren't the ones that use AI the fastest. They're the ones who figure out how to use it with intention. And that's why for me, this is where we need the shift. This is where the pre-mortems are how you do that. And understanding that we are just fundamentally building different products with different engines that are asking for different things that are thinking in different ways. That again, it is living in thinking code. And so, again, before your next project, before any project where AI is into that workplace or into that workflow, block out, just start with an hour, right? Not for a status meeting, not for a kickoff, not to just sit around and BS about how is your weekend or what are your plans, right? To do a pre-mortem and to ask those six questions. What does good look like specifically? What are the realistic outcomes that we again we need to get to? What are our non-negotiables? What are the guardrails we're going to put onto this AI? What assumptions are we making, or whose voice are we missing? And who actually has the authority to stop this? Write down the answers. Share it with your team. Empower people to be able to actually act on this, right? Because again, the documentation is not the trick and the answer. It is the adoption of it, and it's actually knowing that people are empowered to use it, not punished, not seen as being a pain in the ass, not seen as people that are slowing it down, but somebody who is actually acting in the best interest of everybody involved. Because look, I think that for me, that document is worth more than any post-mortem I've ever done. It's more than probably worth more than any post-mortem you've ever done. Because that's the thing, right? Like for me, this age of ship it and see is over. It's over, right? AI has ended it. I don't know that a lot of us want to see it that way or accept it. The pace is too fast, the scale is too big, and the damage is not gonna wait. And so for me, these pre-mortems, they're not a luxury anymore. This is how successful and responsible teams are going to work. This is going to be that non-negotiable and the shift that we're all either going to get ahead of and be able to avoid these problems, or we're gonna come the poster children for and learn the hard way. So, again, I think, as always, right, I'd love to hear your thoughts on this. Is this something you've tried to adopt? Did it work? Did it not? Are there other ways that you've been able to sort of implement what it is to get around that this is gonna take too long problem? Because again, I think I try to keep a broad perspective, but I also recognize my own blind spots in what this is. So, look, I think as always, I know the time is the only real luxury any of us have. I was wildly appreciative that you want to spend any of it listening to me. But start to change your thinking, right? Start to recognize that this is more systems thinking than output thinking. And hey, as always, stay crazy.