Episode 23 - Challenges for Humans in the Loop

Welcome to the second of two episodes with MIT Professor David Mindell.  In these conversations we are exploring the detailed realities of "human in the loop" systems.  In the first session we learned about the Apollo Program's lunar landing control system.  In this episode we are going to explore an important contemporary human plus computer control system failures involving a ship of the US Navy.


The first accident was a collision in the Singapore Strait in the pre-dawn darkness of August 21, 2017.  The National Transportation Safety Board, or NTSB, determined that the causes of the collision between the destroyer John S McCain and the tanker Alnic MC included a lack of effective operational oversight, insufficient training, and inadequate bridge operating procedures.  Also contributing to the accident were the John S McCain bridge team’s loss of situational awareness and failure to follow loss of steering emergency procedures.  This loss of awareness goes by the name Mode Confusion.

Transcript

SPEAKER_04 0:01

Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap, Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you. In these conversations, we are exploring the detailed realities of human-in-the-loop systems. In the first session, we learned about the Apollo program's lunar landing control system. In this episode, we are going to explore two contemporary human plus computer control system failures involving the ships of the U.S. and New Zealand navies. The first accident was a collision in the Singapore Strait in the pre-dawn darkness of August 21st, 2017. The National Transportation Safety Board, or NTSB, determined that the causes of the collision between the destroyer John S. McCain and the tanker Alnik MC included a lack of effective operational oversight, insufficient training, and inadequate bridge operating procedures. Also contributing to the accident were the John S. McCain's bridge team's loss of situational awareness and failure to follow loss of steering emergency procedures. The second accident involved a New Zealand naval vessel, the HMNZS Manawanui, which ran aground and sank off the island of Samoa on October 6th, 2024.

SPEAKER_03 2:09

Welcome back to Mind the Gap Dialogues on Artificial Intelligence, David. Thank you. Pleasure to be here.

SPEAKER_04 2:15

Let's look first at the August 2017 collision between a U.S. Navy destroyer, the USS John McCain, and a much heavier tanker, the Alnik MC, traveling westbound in the Singapore Strait. I'll just summarize some of the key points. The twin propeller destroyer was overtaking the Alnik, also transiting westbound in the strait. A single crew member was at the destroyer's helm, where he had control of both the steering and thrust of the McCain at the ship control console, or SCC. At 520, the McCain's commanding officer or CO who was on the bridge thought the helmsman might become overwhelmed responding to steering and thrust commands as the ship maneuvered in the busy shipping lane. The CO ordered a second watchstander, a Lee helmsman, sitting just to the right of the helmsman, to take over the control of the ship's thrust. Mistakes in carrying out that order and subsequent orders started a sequence of events that caused the McCain to veer in front of the Alnik. Moments later, the Alnik's bow crashed into the left side of the McCain, causing significant structural and flooding damage to the McCain and the deaths of ten of its sailors. Let's pause there. About a year before the accident, the U.S. Navy completed major changes to the controls for steering and thrust for the McCain and numerous other destroyers in her class. The Navy replaced the mechanical controls with digital touchscreen controls. In its report on the accident, the NTSB noted that mechanical manual throttles are often preferred in aviation and most Navy vessels. Unlike digital controls, mechanical ones provide both immediate and tactile feedback to the operator. Why do you think the Navy replaced manual controls with digital controls for the McCain's steering and thrust?

SPEAKER_02 4:22

Well, I think that kind of control replacement is part of a larger trend, obviously, of digitalizing many aspects of controls of these different kinds of systems. And there are certainly benefits to having a digital or software-based control of many of these things. But of course, when you do that, you're taking the control of a system from some kind of mechanically connected thing into some cyberspace software realm, you know, blinking lights on a graphical user interface. And what we've learned, particularly from aviation over the years, is there are dangers in doing that kind of replacement. Actually, in aviation, most modern airliners have automatic throttles these days, meaning the throttle is controlled by the computer, but they actually put a mechanical control in the throttle handle so that the traditional handle that the pilots use, which they still use, moves as the computer is changing the throttle adjustment to give the pilots this tactile appreciation. And there's cost and weight and complexity involved in that, but the consensus is, I think rightly, that immediate mechanical feedback gives the pilot some important cues, clues, and cues to understand what the computer is doing. In a larger sense, this accident falls into a category that in aviation has been called a mode awareness accident. The crew really wasn't fully aware of which of six, I think, different modes the steering and the control of the ship could be under. And in their confusion, that was one of the pieces that led to the accident.

SPEAKER_04 6:02

Got it. So as our vehicles have become more and more capable, they have exceeded the ability of the human operator to respond capably and quickly enough. We've augmented the operator with automation that can be can respond fast, but may not properly do what the human intends or need to do. What that did the designers do right on Apollo that they did not do right on McCain?

SPEAKER_02 6:29

Well, you know, that's an interesting question because you know the lunar landings, even though they were 50 years before the McCain accident, were all, you know, very highly selected, highly trained test pilots in an experimental mode. There was nothing routine about that at all. Whereas with a naval vessel, which I used to work on when I worked in oceanography, was a bit of a revelation to me. Everything in the military, for the most part, has to be run by 18-year-olds fresh out of high school. And the military is amazing at creating systems and training people to do that. But operating a destroyer or any kind of ship day in and day out with much lower train, lower people with less experience always opens you to different kinds of challenges than your Neil Armstrong or your Dave Scott landing on the moon. And so there's almost no comparison in some way between those two kinds of highly intense, very focused operation and a routine, you know, kind of normal day aboard a ship or a military vessel. That said, because the Apollo astronauts did have so much training, and also quite a number of them were actually involved in designing the systems, they had a pretty good mental model of the system. And that's one of the key pieces here, I think, is as we've digitalized these systems. You know, you think about an airplane or a traditional ship, you have a throttle control, you have a steering wheel. We grew up in the 20th century with a mental model of a system like that. When you put a digital computer in there at all these complicated different modes, we've learned that you really have to go out of your way as an interface designer, as a control system designer in training to make sure that the mental model of the operator corresponds to the model of what the system's actually doing. And that's why you'll see sometimes in if you ever peek in an airliner cockpit as you're boarding, you'll see that there's a little sort of skeleton image of the aircraft, and all the different fuel pumps and fuel tanks are there to give some kind of very clear analogy between what the computer thinks is going on and what the human thinks is going on, and try to keep those two things as tightly matched as possible. So as we introduce digital systems, they're able to do a great deal more. They're able to control in some very important and subtle ways that people have difficulty controlling, ultimately to augment the human controls, but you risk this mode awareness issue where the person doesn't have an accurate picture of what the machine is doing. And I think that's what happened in the McCain example.

SPEAKER_03 9:14

David, before you you go on to the next question with Mark, can we sort of double back to something you said earlier and in the previous episode? My understanding with the Apollo is that digital flight software was adopted because humans could not have performed all of the controls in the speed and accuracy needed to make that a reliable mission. What I still don't understand is why the Navy had to do this. They weren't adding 10 thrusters in different directions to this destroyer. Why do you think they shifted to it? Because I'm I I mean, unless it's a military secret, I'm missing something as to the necessity of it.

SPEAKER_02 9:54

Yeah, I I don't know the answer to that, although it's quite possible that they installed a better autopilot, that there are emergency modes here that would be more effective, that a digital system would be better at sensing failure and responding to it quickly. You know, there are any number of good reasons to add digital controls into an analog system. And airliners have done that over the recent decades, and it's a major contribution to the enormous safety improvements that we've seen in commercial aviation, better displays. There's no doubt you can do things that are that are difficult to do otherwise, but at the at the risk of inducing this kind of mode awareness problem.

SPEAKER_04 10:37

So the NTSB noted that the change to digital controls made control of steering and propellers of the McCain more complex. With tactile controls, a transfer of handling the twin propellers would have required one action. With the new system, a transfer of control of the propellers required two actions, as control of each propeller had to be transferred independently. Given that navigating and steering a destroyer is cognitively demanding task, can you comment on the risks of increasing the complexity of those tasks?

SPEAKER_02 11:14

Yeah, and you know, just looking at the pictures from the McCain report, right? Those look sort of like kind of 2005-era digital control type systems, where not only is there multiple steps, but you have to drop down a menu in order to find the thing to change the control. Today we would consider those poorly designed and too complex. In fact, you know, you would you probably would radically simplify the human factors of the display itself. And, you know, human factors folks are very good at pointing out all those kinds of weaknesses in those kind of display designs, which, especially in that era, were tended to be more done by computer science types than by human factors or people who specialize in human operations. I mean, I still notice all the time when an app gets updated and something that used to take me one button push even on my phone, now it takes me six or eight. And it's astounding to me that no one thinks about that along the way. And you know, there's no doubt that simplicity is a value for interfaces. And if you think of some of the great ones that are out there, including the automobile, like a simple interface is a is a great thing. And automobiles themselves have become much more, you know, poorly digital in many cases, where you have a lot of buttons that, you know, actually try adjusting the balance, the left-right on the stereo in your car. On my Subaru, it takes about six or eight button pushes, and it used to be one dial.

SPEAKER_04 12:45

That's interesting you you point that out. I mean, I I first encountered cruise control on cars 20, 30 years ago. And one of the interesting things about them is that they integrate with the controls pretty much comfortably. And if you take action, you override the cruise control. You know, if you put your foot on the pedal, on any of the pedals, uh brake or steer or uh accelerator, or if you, you know, if you do various other things, you override, you disconnect the cruise control, and that makes it sort of less obtrusive. It doesn't interfere, it doesn't fight with you for control of the car.

SPEAKER_03 13:26

Yeah, Mark, I want to build on what you just said because I think that's a really helpful analogy. As I recall in the NTSB report, the first big mistake was when the captain had ordered this transfer of thrust from the helmsman to the Lee Helmsman, and inadvertently during that transfer, the steering was transferred, and the helmsman immediately sensed that he had no control anymore of steering and reported to the entire bridge that he had lost steering. The steering hadn't been lost, it had been transferred, but the Lee Helmsman didn't know it, and nobody else on the bridge knew it. Which then leads us to ask this question. The NTSB report found that the McCain's crew was routinely operating the ship steering and thrust controls, not in one of its normal modes, but in what was called backup emergency manual mode. Why was the McCain routinely operating in emergency manual mode, which was a backup mode, and what risks did that practice introduce?

SPEAKER_02 14:28

Yeah, that's actually a very interesting question here. And let me zoom out a little bit also because I have developed a little rule of thumb, which is that when you read an accident report, rather than thinking, oh my God, how could they be so dumb and do that, you actually have to assume that they were doing that all the time, every single day, and that in this accident just happened to be the moment that it caught up with them. And this gets to what sometimes people call the reason Swiss cheese model of an accident, which is that it's only when a whole bunch of little things line up and the holes in the Swiss cheese line up that you get these accidents, and they're often chains of smaller problems. And so this was clearly one of those smaller problems that they were doing this every day. Who knows how many times they lost control of the McCain in the middle of the night or some other time, they just didn't happen to hit anything, and and nobody w was the wiser for it. I would assume that that had happened either on this particular ship or on other ships like that, right? When I used to do flight training, I still do flight training. In flight training, they always say, at the end of the day, there's really only two rules keep control of the airplane and don't hit anything. And in both of these accidents, they violated both of those rules. And if you follow both of those rules 100.0% of the time, you'll be fine. So, but it's also probably true that either the crew was not well enough trained or the system was too complex. And they found that backup emergency mode, you know, uh simpler and more amenable to their mental model for some reasons. And that it allowed them to uh circumvent some aspect of complexity that was a problem for them. And you know, uh anytime you you bypass warnings or emergency controls, you're asking for trouble. So these things, this was clearly happening here. And so there's also a pilot saying, which is one is none, two is one, right? Which means if you have one of anything, it's gonna fail one day, and you basically can't risk your life on it. If you have two of something, then it's like you have one, you can actually use it and rely on it. And and three is even better, right? So when you are operating with a backup, you're already reduced to an entire layer of your own complexity because a backup that you rely on on a daily basis is no longer a backup. And that's clearly part of the story here as well.

SPEAKER_04 17:07

Some years ago, I read a report by an expert in nuclear reactor operations. And he was reporting on visiting a number of different reactors, and one of his metrics of how well a reactor was designed and was operating was the number of post-it notes stuck to the console. And what people then do is they would write the correction on a post-it and they stick it on the console to remind themselves and their colleagues of the that. And so, you know, and he basically said if you go into most nuclear reactors, control consoles these days, you find them covered with post-its, which he views as a major red flag, and a sign that you know someone needs to go in and rethink the entire design thing. Of course, that's hard to do when you're actually keeping the thing from melting on a daily basis.

SPEAKER_02 18:15

But it's it's also a sign of a user of users innovating in order to make the system operate and safe. And engineers who don't either allow for that or take, you know, take awareness of it also invite problems because people got to get their jobs done. They're very good at figuring out what it needs, what needs to happen to get their jobs done. Sometimes it makes the system less safe, sometimes it makes them safer, because the systems as they're originally designed are rarely perfectly suited to getting the job done.

SPEAKER_04 18:48

This brings back a point you made in this previous episode, namely that part of the debate is between the operators of the of whatever it is and the people who designed it, who may not understand the operation properly. And there's kind of an ongoing struggle between the two for to make it work correctly.

unknown 19:11

Right.

SPEAKER_02 19:12

And that you know, that gets us into what is clearly a major issue in both of these accidents is this question of human error. And it's easy to look at these and say the crews screwed up in umpteen ways, and I think that's true. And at the same time, you also have to ask a system that is poorly designed will, by its nature, force people to make mistakes. And a well-designed system will recognize that people make mistakes and be robust to those mistakes. But then you also have to factor in were they properly trained on this kind of system? Were the work schedules and the other schedules leading them to operate with fatigue? And those are all kind of larger systems questions that we're much better at understanding now than we were 30 or 40 years ago. It used to be very easy to just blame human error, and that was the end of it. And in aviation, those were usually the pilots, and the pilots were dead and not there to speak for themselves. So you could sort of wrap it up in human error and wash your hands and walk away. These days, we're much better at understanding that many other factors can contribute. And human error is often just that tip of the iceberg or the pointy edge of the spear of many other things interacting to cause these accidents.

SPEAKER_03 20:34

David, I want to pick up on something you've mentioned, which our audience, which, if they've been with us on several episodes, we've talked about AI models. But you've been talking about the model that humans have in their heads for certain things. And it would seem to me, in addition to this tension between systems that are simple enough or too complex, there's a feature that seems to be at work, which is are the designers coming up with a system that respects or coaxes the human to adjust their model when security paradigms change. That's a major change in a model, and often we don't realize it until there's been a security, national security disaster. And then everybody realizes we've been operating with the wrong security paradigm model. But do you think part of what was happening here was that the system, the digital system put on the McCain, involved a mental model that the crew never mastered for whatever reason, but one of them had to be that it was that there was a real difference between the model and the analog system that they were used to working with and this digital model that they would have to replace it with.

SPEAKER_02 21:44

I mean, clearly the the digital system had a set of states that the crew was not familiar with. Okay. Whether that was because of inadequate training or because it was so far departed from the the model they had grown up with. Probably a a combination of both. And remember, these systems are what we call deterministic systems. They're still pretty much like clockwork in that they have you do X, Y, and Z and A, B, and C will come out. Start looking at an AI type system where that mental model is, well, the internal model of the system is changing by definition. It gets very scary very quickly, which is why we don't have any AI in any of these kind of systems these days. And we're probably a long way from having it, because how can I work with a system, even if that's sort of changing what it's going to do every day? That's just not something you can operate safely with. You know, maybe we'll need decades of experience before we really know how to do that. And we're barely at the beginning of it. We don't know how to specify those systems. We don't know how to certify those systems. We don't know how to verify or validate those kind of systems. And, you know, I I know where the FAA is on working on that, and it's it's very, very early days.

SPEAKER_03 23:02

I'm going to transition from the next to the following question, and we'll go back to the moments of the accident when the Hellsman reported that he had lost steering. Before that, he was making minor adjustments to keep the ship on course. Now the ship veered off course and headed towards the path of the Ulnik, which eventually led to the Ulnak colliding with it. So what I'd like to know from the NTSB report, it states had the John McCain crew pressed the emergency override to manual button at the ship control console when the perceived loss of steering occurred and was reported, the watch team would have re-established control of steering and would have likely avoided the accident. And if I'm not mistaken, they later concluded that even the ship's captain didn't understand the proper use of that emergency override to manual button, otherwise known as the big red button. Why do you think the crew failed to do that? And is there a general principle at work here?

SPEAKER_02 24:04

Well, yeah, I mean, there again, it's a mode confusion problem. They didn't understand the mode that the ship was actually in. They didn't understand the mental model or the model that the software system was operating on. And I think you probably see this in both these accidents too. There's a phrase in aviation called helmet fire, where when people get under stress, your your attention tends to narrow to a point in a survival mode that you can really miss a lot of things that to the outsider or in retrospect should be obvious, literally, including flashing alarms in people's faces and other sorts of systems. And you know, these crews got into a kind of mental hole or a rabbit hole where they couldn't get out and see that there was maybe a very simple alternative explanation for what they were doing. And you know, those situations are very scary, and we're all vulnerable to them to the degree that we're just ordinary cognitive beings. And I do think you can train around them to some degree, but that they clearly weren't able to respond to that here.

SPEAKER_03 25:14

Your statement reminds me of what a Norwegian official once told me, which was that in crises, human rationality tends to narrow. And I'm wondering if when you design these systems, you need to anticipate their failure or they're being hacked into by an adversary or they're being damaged during combat, so that you can't expect people to be able to have that broader situation awareness.

SPEAKER_02 25:39

Right. And you know, to go back to the Apollo 11 landing, one of the amazing things about that landing is there is this program alarm that's flashing in their faces. And Neil Armstrong has the wherewithal to say, you know what, I'm feeling this thing in my hands and it's still responding to my controls. I don't need to get distracted by that alarm. Until I'm not in control, it's not an issue for me. And he was able to keep that situation. And the overall system, including the people on the ground, were also able to provide the larger context, understand what was going on, and in that case, keep moving forward. And but again, that's an extraordinary crew on an extraordinary day who actually had built a lot of the system they were operating. That's very different from an ordinary crew on an ordinary day.

SPEAKER_04 26:28

You made this point earlier in your remarks that the the Navy operates with a bunch of 18-year-olds, and the Apollo program had a bunch of exceedingly experienced uh fighter pilots. What could the Navy do differently in its training or in its development procedures for ships or whatever that that would sort of build on what the Apollo program got right?

SPEAKER_02 27:00

Well, I think you know, every year that goes by we learn more about people and groups of people's interaction with these digital systems, and we do get smarter about both how to design them and how to operate and how to train for that operating. And there again, I mean, the the Navy is one of the most remarkable organizations in the world in terms of their ability to operate complex technology safely. And the nuclear navy, particularly the nuclear submarine navy, you know, is sort of the paradigm. They had literally invented the term and the principle of a high reliability organization, you know, to it's obviously not just 18-year-olds, but there's a lot of 18-year-olds involved with, you know, crew chiefs and young officers and so on operating these systems. And in general, they're remarkably safe. At the same time, you know, a little bit of under-resourcing on the training or higher operational tempos, and you begin to see these accidents come up. And if you notice, every time the country mobilizes for war, there are many more accidents, right? Even before the combat starts, you know, people are operating in higher levels of fatigue, they're in unfamiliar environments, the stress levels are higher. Accidents are just part of military operations at some level, even though they're, again, these are remarkably powerful, you know, sometimes quite unstable types of technological systems, often quite old. And it's also just amazing how much they, how well they operate, despite a lot of these challenges.

SPEAKER_04 28:44

So NDSB reports by statute are inadmissible as evidence in civil proceedings. What do you think of the wisdom of this rule?

SPEAKER_02 28:54

I think that's a very good rule. And again, I think it's a recognition that, you know, again, 40 years ago, you go find the people who screwed up and you fire them and you wash your hands of the episode. And now we realize that's actually not really a viable way to operate these systems. It may well be that the people were negligent and the people need to be accountable and even punished for their behaviors, but you also have to look at the larger system in which they're embedded. And, you know, searching for blame and finding the responsible parties who, as we all know from history, are usually convenient scapegoats more than the actual cause of the problem inhibits the deeper understanding and the improvement of the safety. And that's what the NTSB's larger role is, is understanding how do we prevent this from happening again so no one has to suffer the same fate. And giving them a free hand and a more, you know, uh, what's the word, inhibiting the use of their reports for legal liability is one way to open that investigation. And, you know, you read, I always say this true of your car. Think about my airplane. Every airplane by by law has a operating manual with it. It has to be in the airplane, in fact. But if you read those manuals, those manuals do not tell you how to apply the airplane. They are legal documents to protect the manufacturer in the case of an accident with a lot of I told you so stuff in it. But you can't learn how to operate the system by looking at the manual because that's not what the manual does. And those are the kinds of things you get into if you link the liability to the investigation.

SPEAKER_03 30:42

Well, if I'm not, you know, just to mention there were court cases related to this accident that tried to fix liability. And one of the things the court said is we don't know why the ship veered because they could not access the NTSB information and evidence. Through the course of these conversations with you, David, mode confusion has emerged as one of the recurrent themes for failures of human-in-the-loop automation. Are there other important themes that designers and users of automation should know about?

SPEAKER_02 31:14

Yeah, I think the you know there's a big idea over the last 50 years, which is a kind of evolution in how we think about accidents and an evolution in the kinds of accidents that happened. And so if you go back to the 1960s, it was much more common to have what we would today call component failures, component failure accidents. You know, I'm flying along and the computer on the airplane fails. Computers used to fail a lot more than they do now. And so I have a backup computer and or a backup engine, right? All commercial flights have at least two engines on the airplane because engines used to fail a lot, and they still occasionally do. And so the whole idea of a backup system and redundancy were solutions that were developed according to both in response to and to prevent these component failure kinds of accidents. And again, I think a lot of that blame culture around accident investigations was like there's a component failure, and the component was the operator, the human, or the pilot. We've evolved our picture a lot, again, partly in response to accidents, toward a much broader view that people today call systems accidents, where firstly, that's partly because the components have gotten a lot more reliable. The backup and redundancy is pretty effective, and we have a lot fewer component failure accidents than we used to. They still happen, but they're much more rare. And therefore, a lot of things are a lot safer. Again, we just had the first accident in American commercial aviation in 15 years, and that's with 50,000 flights a day. It's a pretty amazing safety record for commercial aviation.

SPEAKER_03 32:57

You're speaking of the accident at Reagan National Airport.

SPEAKER_02 33:01

Right, exactly. First one, there was one fatality that wasn't in a crash, but that was the first crash in 15 years in domestic U.S. commercial aviation. So those systems have gotten a whole lot safer for a bunch of different reasons. And the accidents that we do have are tending to be more subtle, and they're tending to be interactions between aspects of parts of the system that we didn't appreciate. And you know, a classic example of this that a lot of us teach with is the Challenger space shuttle explosion from 1986, where the initial characterization of the accident was oh, the the O-ring failed, and therefore you had this accident. Then people started to say, well, the O-ring failed because it was launched on too cold a day, and that was for politics, which was a beginning to get there, but a much too simple explanation. But then larger, in a larger sense, we began to realize the space shuttle system had been historically declared operational much sooner than it should have been. It was operating under much tighter resource constraints than it should have. And these larger systems were largely to blame. The congressional investigation of the Challenger accident really blamed the O-ring and a couple of engineers making the decision to go the night before. Scholars revisited that in the ensuing 20 years. NASA never bothered to look more deeply in it, and they had the Columbia accident in 2003, which was almost the same accident in a lot of ways as the Challenger accident. Totally preventable. In fact, there were literally NASA senior engineers and flight controllers who said, gee, I never read these newer interpretations of the Challenger accident until after Columbia. If I had read them, I might have operated differently. And the Columbia Accident Report, as compared to the Challenger Accident Report, so this is 2004 or 5, as opposed to 1987, is a very different document. And actually, there's a chapter that says titled An Accident Rooted in History and Culture, and brought a whole lot of this newer thinking to bear on how we imagine an accident and how we then go about preventing them, including these many other factors that went into right into the basic design of the shuttle, by the way. And you know, people used to brag about the space shuttle because it was the most complex machine ever built by man. To me, as an engineer, that's not something you should brag about, right? Complexity is a sort of failure. And but the both the Columbia and the Challenger accident happened, not because the shuttle had 58 million parts or whatever the number was, but it was because it had four parts. And both those accidents were kind of systems accidents where the four major parts, the two solid rocket boosters, the external tank and the orbital, interacted with each other in ways that were unacceptable and unpredictable. And those things are much harder to understand ahead of time, more subtle, and they include lots of aspects of both design, human decision-making, program, program funding, and other things, but are much more telling about why these accidents happen. And a lot of people spend a lot of time these days thinking about how to build those kinds of resilience into the system in different ways to help make sure they don't happen again.

SPEAKER_04 36:32

I'll interject with one quick comment. You pointed out that the number of parts is a is a bad signal. I was noting from an article I read recently that the number of parts in an electric vehicle are something on the order of a third to a half the number in a gasoline vehicle. So maybe there's a good thing coming. That's true.

SPEAKER_02 36:58

A reciprocating internal combustion engine is like this hopelessly complex thing. We all got used to the fact that you just need to get your oil changed every 3,000 miles and the head gasket's going to blow after 20,000 miles or whatever. And, you know, that's 20th century stuff. We don't need to do that anymore.

SPEAKER_03 37:17

Well, it's interesting the Navy's response to the McCain accident and another involving, I think, the sh one of its vessels called the Fitzgerald, the Navy decided a couple of years after the McCain accident to replace those digital controls with mechanical controls. So they went back to something simpler, which was part of why I was asking what may have been the motivating reason because apparently they felt they could go backwards rather than forwards with that design. Does the use or non-use of AI in an operational context change the answer to what we're talking about? And part of the reason we're asking this, and I was a little surprised to hear you say we're not using any AI. Maybe I've got this wrong, but I thought there was some AI involved in the 737 Max system that caused those two accidents. Am I mistaken about that?

SPEAKER_02 38:07

No, no. And those accidents, again, were really caused by these kind of system design mode awareness issues, very much informed by Boeing's desire not to change the training because there are certification legal reasons that that's costly to do. There's very sophisticated computers and algorithms, and even some of them are a little statistical, but there was no AI in that story. That said, the automation in the airplane was fighting with the pilots for control of the airplane, which is a really unacceptable situation to be in. And there again, because of Boeing's desire not to change the training, those internal state models of how the system worked were not revealed to the pilots. So the pilots were unaware and they could not have known about how that pitch control system worked, especially in that point of failure and contention. And so, you know, as in many of these accidents, that was not rocket science, that accident. That was just negligence.

SPEAKER_03 39:16

But it's it's interesting what you you seem to be describing as humans who were in loops, they didn't know existed or didn't understand the model that would have explained it to them.

SPEAKER_04 39:25

Exactly, for sure. So in the 737 Max, which is reminds you your point of earlier, one is none and two is one. The the particular planes that crashed that really triggered the whole thing had been what what do they call it, value engineered to remove the uh the redundant sensors that would have prevented that from happening given one of the sensors failed, that the one fence sensor failed.

unknown 39:54

Right.

SPEAKER_02 39:54

Boeing didn't even get the component failure part of that story right, much the less the systems accident part of it right.

SPEAKER_05 40:00

I was just going to ask Mark's question with a different twist, because I was really fascinated by the way you talked about the evolution of accidents, right? Being a very specific international trade regulatory lawyer, I never had an opportunity to sort of think about it in the context of component failure, systems failure or systems accidents, and then sort of what you call the third category was more subtle, subtle failures or subtle systems. And so it sounds like without getting into the specificities of that particular accident, was that a combination of sort of all three from your vantage point, or was that more an example of sort of a subtle thing that no one really saw going to happen, even though it was based on kind of training and the way the systems operated with humans in the loop?

SPEAKER_02 40:46

Well, I would say the the systems accidents and the subtle accidents are similar. I I think of those as the same category, really. I I would say that, well, it's actually not that uncommon, but the in some you might say the 737 was a systems accident that was precipitated by a component failure, right? And that's actually not that uncommon either, because the component failure, actually, the Air France 447, which was 2012, was a Airbus A300 over the Atlantic that crashed. Sort of similar story in that there was a component failure high over the Atlantic late at night, and actually cleared itself within a minute. But in the course of that, the crew lost control of the airplane and everyone it crashed and everyone perished. And so, again, it's not that the component failures don't happen, they do, but they can trigger systems effects, especially including the people, that put them into a different category. And again, that's a little bit of the Swiss cheese story of you know, little things go wrong all the time, right? And you know, I always say, like, you know, the keypad on your phone, what used to be a keypad on all phones, there's 10 numbers on there, and the human error rate at typing on a keypad is one in ten, right? So we make mistakes all the time. And the number of aviation, of commercial aviation flights that go exactly as planned is also about one in ten. And humans are extremely good at filling in those funny boundaries, often in invisible ways, that engineers building automation systems don't see very well. So, like the driverless car guys love saying, oh, so-and-so many, you know, people cause all these accidents in automobiles. Their statistics are wrong anyway. But yes, that's true, but people also prevent a lot of accidents. And if you think about it when you're drive, you're constantly preventing accidents because someone's in the crosswalk and they shouldn't be, or you know, you know, there are so many subtle things that we're the glue that fills in. And when we mess with that glue too much, we always invite problems.

SPEAKER_03 43:12

But David, you're suggesting that humans are exquisitely good at taking feedback from things that go wrong and making adjustments. And at the same time, in this episode, you've explained how poorly the operation of feedback from accident to accident causal analysis to re-examination of the actual cause to the people involved actually hearing or reading about it. Can you comment on whether there's a way to fix that? And at the same time, could you let's ask that question first, and then I'll be a question about AI.

SPEAKER_02 43:49

Yeah, no, that's a really interesting question because, well, first of all, I really only said that about NASA in that case, right? And you know, there's an amazing and I think Terribly scary quote in the Columbia accident report where the flight controller said the system was talking to us and we weren't listening. Meaning, in retrospect, there were all these clues that no one paid attention to. And so what I tell my students is that one of the really key behaviors to make systems safe, to avoid these systems accident, is actually curiosity, right? You have to be constantly curious about the system and when it's talking to you, and anything that doesn't fit, you have to pay attention to and really listen to. Okay. And the ultimate sin that NASA committed, particularly in the Columbia accident, where they had a week to mitigate that accident and they totally failed because the spacecraft was in orbit. They had all the time they needed, was in curiosity, right? They had stuff that wasn't right and they didn't pursue it further. Now, I think you can look at what happened last summer. It's actually still happening in a way, where the Boeing Starliner brought a crew to the space station. There were some troubling anomalies in the ascent and in the docking phase there. And NASA couldn't really bring itself to bring them back on that spacecraft in good conscience. And at extremely very high financial cost and very high cost of embarrassment, they left that crew up there. But at least they took the time and they made the decision. And maybe that's a case. I think it'll be studied for a while. We don't know exactly, but of actually learning and saying we don't have to make this hasty decision. There is an uncomfortable but option that preserves human life, which is the right thing to do in those cases.

SPEAKER_03 45:55

I take it that when you emphasize the value of curiosity in human beings, especially when they listen to something that seems anomalous and want to be curious about it. AI systems are not, to my knowledge, yet programmed with curiosity. What do you think AI users should be doing to better prepare for human in the loop systems so that the humans are capable of knowing when something's anomalous or awry in the output, maybe awry in their use of it, or a combination of the two?

SPEAKER_02 46:31

Well, to go back to a simpler case than that, in the McCain accident, right, if I were the captain, I would have those crews switching the modes on the steering all the time and just making sure that everyone on the bridge and everyone in the engine room, you know, tried every last mode of that system all the time. And what made it go from one state to another and move back and forth? And that's the kind of curiosity, the sort of exploration of the system under normal operations, so that when you see something under an abnormal operation, it's not the first time that you see it. I think you know there are people who do fraud and poke AI systems in, you know, in the at least in the sort of on-your-screen way, to understand what those limitations are. And those tend to be curious, technically oriented people who like to put a system through its paces. And there are many folks like that. But again, by the time a system gets you know institutionalized and put in a Navy ship and there's 10,000 pages of documents around it and so on and so forth, it's very easy to lose that. You know, when I when I fly my plane, I have many hours of kind of boredom and I don't read a book, I sit there and I poke at every last menu on every last touch screen I have to make sure that I've explored every last corner of what that menu structure is and know exactly what everything is gonna do. Because one, even if I never expect to use it, it may pop up one day for me. And so the more you can understand a system that you operate, the better off you are.

SPEAKER_01 48:13

Well, okay, I gotta jump in here too, because I use the phrase uh people are the ultimate critical infrastructure. And as long as people are in the loop, well, there's people in the loop. And there are those who are espousing the idea of take them out. And there are others who saying, oh no, never can do that. How are we gonna get past this fact that you know people sometimes goof? Again, people goof all the time, right?

SPEAKER_02 48:44

Like roughly one in ten we goof, right? And what's amazing about these systems is that we've built in whether it's two pilots or you know, other aspects of ways of correcting those very predictable mistakes that humans make. And you know, at some level, we are out of a lot of different loops, right? You're out of the loop of controlling the heat in your house. You have a little automatic system, you set that loop. You know, you we all sometimes we talk about it in control engineering, humans on the loop as opposed to humans in the loop, supervisory control, which is important. And really, there are many things we can't do, like the lunar landing, without being in those kind of on the loop. And at the same time, I like your phrase, people are the ultimate critical infrastructure. My version of that is people are the glue that hold integrated systems together because the world is pretty stochastic, the world is dynamic, the world is always changing. These systems like the McCain's control systems that were designed 10 years before are not very good at adapting to that, but people are very good at filling in those holes and you know, adjudicating or you know, just filling in the gaps. Sometimes people call it repair work in real time. You're not necessarily fixing a system, but you're you're doing that. And you know, I used to ask airline pilots, how have you ever asked of the of the cockpit automation, what's it doing now? And every single one of them would say, Yes, I've asked that question. Okay, but then someone said to me, Oh yeah, that's that's really only what the newbies ask. The experienced people say, Oh, it does that sometimes. So which is not very comforting, but at least then you're when you say, Oh, it does that sometimes, you at least have a basis of what you expect it to do to know that it's doing something unexpected and you can take action accordingly.

SPEAKER_04 50:44

So we know you have a new book, The New Lunar Society, an Enlightenment Guide to the Next Industrial Revolution in preparation. Would you tell us a little bit about it? Sure.

SPEAKER_02 50:56

It's actually published in two weeks. This is like the first copy. And it's uh it's less about these system critical systems in extreme environments as it is about um kind of large-scale industrial systems. And I go back to the origins of the industrial revolution, James Watt and Matthew Bolton and their friends who would get together for dinner over a 30-year period in a group they called the Lunar Society, included Joseph Priestley and Benjamin Franklin and many other important people who took the ideals of the Enlightenment and really very consciously created the Industrial Revolution out of those ideals. And we're at this moment now where because of threats to our global supply chains and global retrenching and changes in technology and climate change, we really need to rethink how we operate our industrial systems. Walking away from them is really not an option given the way that you know 99.999% of us live today. So we have to rethink them from the ground up. And going back to the Lunar Society offers a wonderful set of examples about how we might do that. What we're doing here today, remote work, is a one important component to it. And actually, the United States of America was very much founded by people inspired by the Lunar Society, the man who brought the Lunar Society together, who introduced Bolton and Watt, actually was Thomas Jefferson's teacher at the College of William and Mary. And through him, Jefferson actually channeled Isaac Newton and others into the Declaration of Independence. And the early years of this country were very much founded on not only the Enlightenment ideals in terms of politics, but industrial ideals in terms of national self-sufficiency and the value of making things and the importance of making things as a way to create the world that we want to create. We've gotten away from that value in the last 30 or 40 years, thanks to some economists who have missed the value of making things. And the book is sort of a bit of a polemic about returning to the not only the moral, but also the democratic value of creating the material substrates that we base our lives on.

SPEAKER_03 53:15

But it also sounds like you're describing in talking about the new lunar society the value of conversation.

SPEAKER_02 53:23

For sure. And the value of you know collaboration and getting away from sort of lone inventor myths of you know, one guy, always a guy, doing something in his basement that then changes the world. That's not how it works, right? There's collaboration and partnership and and people bringing things together out into the world and very different kind of heroism from you know, kind of classical Homeric heroism of conquering and swashbuckling, and much toward collaboration and building and creating.

SPEAKER_03 53:58

Which is then suggesting a different model to be used for these kinds of things. I would like to return to your emphasis on conversation because the four of us have immensely enjoyed the conversations we've had with you, both in preparing for these sessions and in conducting them. But I also want to urge our audience to read Digital Apollo. We have we found that book extraordinary. We wanted to have you on the podcast once we had read it. You've told us that your students have been reading it and enjoying it for over a decade. And it was a privilege to have you with us, and we want to thank you very much for taking the time to be with us.

SPEAKER_02 54:40

Thank you. My pleasure. Thanks for your interest in this important topic. As you see, it sort of keeps coming up.

SPEAKER_04 54:59

We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention the specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap. Dialogues on AI.

SPEAKER_00 55:59

Thank you for listening to the AVA Business Law Section Podcast Series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit ambar.org slash bizlaw. That's B-I-D-L-A-W.