Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 22 - Human and Machine in Spaceflight
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode of Exploring AI Matters, #22, and the following one, #23 we are going to do something unusual. We will explore the adoption of sophisticated control automation and look for lessons that may apply to the adoption of AI-augmented systems.
To help us understand these lessons, we are fortunate to have for our next two episodes Professor David Mindell of MIT. He is the acclaimed author of numerous books in this area, notably "Digital Apollo - Human and Machine in Spaceflight." In this book he examines both the design of the flight controls for the vehicles that conveyed astronauts to the moon in 1969 and beyond and how the people were trained to use them.
Dr. David Mindell is Professor of Aerospace Engineering and the History of Technology at the Massachusetts Institute of Technology. Professor Mindell has spent more than three decades researching the myriad relationships between people and machines and innovating to improve them. He is interested in human and machine collaboration; navigation and autonomy for transportation and mobility; ultra-wideband systems; history of aviation and spaceflight; and entrepreneurship.
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_05In this episode and the next one, we're going to do something unusual. We will explore the adoption of sophisticated flight control automation and look for lessons that may apply to the adoption of AI-augmented systems. To help us understand these lessons, we are fortunate to have for our next two episodes Professor David Mandel of MIT. He's the acclaimed author of numerous books in this area, notably Digital Apollo: Human and Machine in Spaceflight. In this book, he examines both the design of the flight controls for the vehicles that conveyed astronauts to the moon in 1969 and beyond, and how the people were trained to use them. Dr. David Mandel is professor of aerospace engineering and the history of technology at the Massachusetts Institute of Technology. Professor Mandel has spent more than three decades researching the myriad relationships between people and machines and innovating to improve them. He's interested in human and machine collaboration, navigation and autonomy for transportation and mobility, ultra-wideband systems, history of aviation and spaceflight, and entrepreneurship. Welcome to Mind the Gap, Dialogues on Artificial Intelligence. David, I'm Ama Adams, a national security lawyer.
SPEAKER_03And I'm Charles Palmer, a computer scientist. We are your host for this episode of Mind the Gap Dialogues on Artificial Intelligence. In addition, we have two more hosts.
SPEAKER_00Hello, I'm Roland Troop, a national security lawyer.
SPEAKER_02And I'm Mark Donner, a computer scientist.
SPEAKER_05Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves on.
SPEAKER_03Professor Mandel, your outstanding book, which is great on audio as well, Digital Apollo, Human and Machine in Spaceflight, describes a tension that arose in and continued throughout the design and development of flying machines through jet aircraft. On one hand, there's the need on to ensure flight stability, crucial for passenger aircraft. And on the other hand, there is need to maximize maneuverability, crucial for pilots to win in air combat. Can you explain how that tension affected the design and operation of successive generations of aircraft?
SPEAKER_04Sure. The tension between stability and control is really one that literally goes back to the beginning of aircraft. And one of the Wright brothers' great innovations was their recognition that the aircraft could be unstable in its own flight regime, but that the pilot would be the one who brought stability and control to the aircraft. The competitors who were trying to solve the flight problem before sort of thought of aircraft like ships, these big stately things that you kind of sit back and maybe turn the wheel. The Wright brothers realized that the airplane could be unstable and the human could make it stable. And when you think about it, that's perfectly logical because they were bicycle mechanics. And a bicycle is exactly that. It's a vehicle that is unstable without an operator. No bicycle will stand up on its own. And a skilled operator takes a little bit of skill. You sit on it and you make it stable and it's perfectly rideable. In fact, once you know how to ride a bicycle, it's hard to forget. They built that idea into their aircraft and in so doing really invented the pilot at the same time as they invented the aircraft. And for a while, it was clear that it was okay for aircraft to be unstable. And think about World War II. Sorry, World War I fighter airplanes were basically like the right flyer. They were unstable and they were notoriously difficult to fly. That conclusion kind of changed over the years as different types of aircraft evolved for different purposes. And as you mentioned, a passenger aircraft is really not something you want to be unstable. You want it to be inherently quite stable so that the passengers are comfortable and everyone feels safe. And it doesn't have to do loops and doesn't have to do barrel rolls, it just has to stay flat. And actually, the more stable the better, because you have all these disturbances out in the world, like wind gusts and other things. And so engineers gradually developed a trade-off between stability and control. And this is sort of inherent in a lot of different kinds of systems. You can make the system stable and always return to a stable state, but it's likely to have lower performance in some ways. And the higher performance, the things that operate at the extremes, like a fighter airplane, you probably want to make them much less stable and probably unstable. Over time, engineers found, and this is particularly true with the early jet aircraft, early supersonic aircraft in the 1950s, and the rise of electronics, that you could build an aircraft that was either unstable or very close to being unstable and improve the stability not by building bigger wings or larger control surfaces, but by putting electronics inside the airplane. And gradually over time, engineers learned to build airplanes that really had no inherent stability and at some level were unflyable by a human without a computer inside navigating and operating the controls to complement the human. And the the classic expression of that is the stealth bomber, very unusual-looking kind of V-shaped wing, has no tail surface, has no rudder sticking up, and the computer takes care of all that stability. In that case, it's for radar stealth as much as for performance, but you can actually use computation and software to make up for a lack of stability in the airframe itself. And those are all very innovative ideas that have impacted aircraft design for the last 50 years or so.
SPEAKER_03And so how did this stability and maneuverability tension evolve when all of a sudden humans began to conduct space missions?
SPEAKER_04Yeah. So the first extrapolations into space were, of course, from aircraft engineering and from pilots themselves, and many, nearly all of the early space engineers had kind of aviation backgrounds. And the question was: how should you design the spacecraft? Should the spacecraft be stable? And starting with the rocket. A rocket launching off the pad is extremely difficult to fly by hand. Many of the early astronauts felt that they could. Neil Armstrong, in fact, ran a series of tests and both flight tests and simulator tests, proving in his mind that he could fly a rocket off a launch pad by hand. And Werner von Braun came in and said, don't even think about it. You'll never fly a rocket off the launch pad by hand. And to this date, no American ever has, actually. It's just you're sitting up there at the top of this long, pointy thing, and the dynamics are very, very difficult. But then when you got into space, you had all this other set of questions around what are the astronauts going to do? What does it mean to fly? How do you fly in space? And there you have, for one thing, the the aerodynamics and the aviation precedents were less helpful because these things just didn't feel or look like airplanes. And you began to have the introduction of digital computers. Spacecraft were quite complex as well. So there was no, what is what does up even mean in space? How should you design the joystick? What do the controls look like? Up changes actually when you go to the moon, from being up from the earth to being up from the moon. All kinds of complexity came in. And those were all problems that were debated in the earlier space, human spaceflight programs, Mercury and Gemini, but particularly in the Apollo program.
SPEAKER_03And was it this continued increase in complexity, the driving force between uh making it impossible for a human to manage the flight without automation?
SPEAKER_04Yeah, I mean, many things. You know, flying in a lunar environment, very foreign environment. You don't have any atmosphere, so there's no drag on the airframe. So all of the intuitions a pilot would have developed would be not useful. The lunar lander, which I think we'll talk about, has 16 different thrusters on it. And so, how does a human control 16 different things at one time to command a particular action? It's really not something you can't imagine 16 toggle switches in front of you, or even a joystick with 16 degrees of freedom. You really need a computer to assemble those different actuators into something that looks like control, much the less when one of those 16 fails, and you have to know very quickly which one to compensate for.
SPEAKER_00And I just ask, to what extent does the change in the gravity, sometimes people think the area near the moon is zero gravity, but of course there's lunar gravity. And you describe in your book the problems of anticipating or predicting the sloshing of fuel during the descent.
SPEAKER_04Uh right. So the lunar gravity is one-sixth of gravity on the Earth. The that changes a whole bunch of the different dynamics. As I mentioned, there's no atmosphere, so you don't have just, you know, we naturally live in a kind of soup that aircraft pilots are very aware of. That's all absent. The lunar lander from the time it descended to the moon's surface to the time it took off again, the mass of the vehicle changed by a factor of 10. So the thing you're leaving in is 10 times less massive than the thing that you're landing in because of burning off the fuel and actually leaving part of it on the surface. And that's a huge adjustment for a human to make, is just that continuously changing mass. And these are all things that computers are very good at. And so it wasn't obvious really to anyone when the program began, but as it went along, it became clear that it would be very difficult to imagine a human controlling the lunar landing without the aid of computers.
SPEAKER_05So focusing on the descent, as you were sort of just discussing with Roland, in the book, you certainly emphasize that the last 50,000 feet or so of the descent to the surface was really the most challenging part of landing humans on the moon. Why is that? Can you help us kind of visualize and understand what made that final step in the journey so particularly challenging?
SPEAKER_04Sure. You know, at the most basic level, partly it's because you're flying through space, there's not that much to hit. And the instant you begin that descent, you know, you're gonna you're gonna bring this very fragile spacecraft with these very fragile human beings into contact with this giant rock. And that's a pretty risky thing to do. So the way that the Apollo missions were designed, they called them there was a series of what they called plateaus. And plateaus were were sort of positions of relative safety where you could sort of sit tight, take your time, figure things out, delay things if you needed to. So the first plateau is sitting on the launch pad at home. You don't have to go if things aren't just right. You're in a safe plateau. Then you light the rocket engines and you have a fiery few minutes, which is fairly dangerous, and then you're in orbit around the Earth. That's a safe plateau. You can sit there, get everything worked out. So the final plateau in the flight to the moon was a 10-mile, 50,000-foot orbit around the moon. And the command module is there, the lunar module is there, and they're basically pretty safe. They're orbiting around the moon, they can stay there for a long time if they have to and get everything set up, get everything figured out, don't change unless it's really right. And and actually, Apollo 10, which which was a previous test flight, which didn't land on the moon, went right up to that 50,000-foot orbit and came home. And so on Apollo 11 with Neil Armstrong and Buzz Aldrin, this 50,000 feet and below is the first time they're doing it. Everything else had been done at least once before. And when you light a rocket engine, which slows the velocity of the orbit down, and it that means the orbit is going to shrink in its size, in the next 10 minutes, three one of three things is going to happen. You're either going to abort, which is dangerous and unknown, you're going to land on the moon safely, or you're going to crash. Just one of those three things is absolutely has to happen in the next 10 minutes. And so actually, I think Armstrong said getting to the moon on a difficulty of one to ten was about a two, and landing on the moon on difficulty of one to ten was a 13. So everybody was very nervous about that one final phase of the landing and descent. And more generally, you know, for aviation, piloting landing is always a big challenge for the same reason. You're you're bringing a very fragile, soft thing up against something that's very, very hard and unyielding.
SPEAKER_05Yeah, no, I I think you you you you summed it up in a way that I think is is very easy for people to understand as to why that last part of the phase of the journey was most difficult. But curious from your view, you know, what did the flight software need to do that humans themselves could not do unassisted?
SPEAKER_04So at some level, what the flight software was doing was kind of creating a simulation of a spacecraft that could be flown in an intuitive way for a person. So Neil Armstrong had a switch in his left hand that controlled the rate of descent up or down. And even if you just think about that's what engineers called one degree of freedom, just how fast are you descending? Well, how much thrust you need to descend at a certain rate changes depending on how heavy the thing is. And as I mentioned, it it's getting 10 times less heavy over the course of this landing. And so the for a human, you'd say the controls would feel more and more sensitive as the thing you're controlling is getting lighter. That's quite hard for a human to keep track of, especially under that kind of time pressure. Where again, the computer sort of knows that and calculates it out. And Armstrong would just say, I want to go down 10 feet per second, and a computer does a great job of adjusting all the parameters in what an engineer would call a feedback loop to make that work. Similarly, in his right hand, he's got a joystick, and there's 16 thrusters, and well as an engine, a descent engine that pivots back and forth. That's a lot of stuff, really more than a human can keep keep in their mind at once. And the computer assembles that all into a set of controls that are much more intuitive for the person. So Armstrong could say rate of descent. He could say, I want to go here. He could actually look out the window and identify a point that he wanted to go and use the joystick to point to that point, and then the computer would kind of bring him there automatically recalculating the trajectory. And that had the effect of reducing the workload, really letting the pilot, in this case Neil Armstrong, control the flight at the high level that he wanted to control, which was really, I want to land there, I don't want to land there, this looks like a good place, bring it down fast, bring it down slow. And that's all under normal circumstances. God forbid one of those 16 thrusters should fail. And if it takes a while for a human to understand what failed, then they have to compensate. And by the time that happened, you could have flipped over and been dead. The computer, on the other hand, would have a very good sensitivity to which thing had failed, switch out the thing that complemented on the other side, and you could continue in a relatively smooth way.
SPEAKER_00It sounds like implicit in what you're describing, though, is that Armstrong had built up an intuition from years of flying. He also had built up a certain intuition based on his use of the simulators on Earth. And if the simulators didn't match what he was experiencing during that final 50,000, he would also have the problem of knowing what was happening.
SPEAKER_04That's correct. Yeah, the simulation in the Apollo program is was very, very important. And you know, starting with the fact that you're going to a place that you could never practice in. Pete Conrad, who who commanded Apollo 12, said, we're banking our whole program on a fellow not making a mistake on his first landing. And that was true, as well as national prestige. So they flew this landing in simulation many, many times, simulating all kinds of different failures and different types of scenarios over time, and introduced a great deal of familiarity to the crew, learning the procedures, learning the feel of the spacecraft. They even had some early kind of graphical simulations that, and so they had rehearsed it many, many times. Everyone knew that the actual landing wouldn't be exactly like the simulation. No simulation is perfect. At the same time, all they had to do and think about in a way when they were actually doing the real landings was what's different about the simulation. And that's a much lower cognitive load than dealing with the whole thing the first time. And so you actually find on the tapes of many of the six landings the crews commenting, oh, this is better than the simulation. Oh, this is easier than the simulation. They're constantly playing back and forth the simulation in their own mind. And in fact, one of the simulators on the ground was a flying simulator. It was a famous, became known as the flying bedstead, that actually did very loosely simulate the lunar module in the 1-6 gravity. And only the commander was allowed to train on that simulator because it was considered so dangerous. The lunar module pilot, the right seater, what you'd think of as the copilot, didn't train on it. And there are cases in those landings where the lunar module pilot thinks the spacecraft is tipped over too far, but the commander realizes that's what 1-6 gravity feels like. It feels like a helicopter that's tilted over six times more than you'd have it on Earth, and it's actually right and not a problem.
SPEAKER_05As you were talking about the flight software, it struck me that you know that the way you kind of described the innovation around the technology and then how that was doing things that humans couldn't do, you know, was very similar to what I was sort of reading about this moon landing, sort of the references you see when you read articles. It's sort of a was described it as a spectacular technological achievement, right? An array of innovative high technology, you know, all these buzzwords to talk about it. But it seems that that's only a part of the story, kind of as you were talking about it and as Roland was talking about Neil Armstrong's experience and expertise and relying on that as he was guiding towards a landing. You know, what functions really remained in through this landing for the human pilot and the co-walt pilot to perform, right? Particularly during the final stage of the lunar landing. You know, where where did the human come into play and was it was sort of a necessary component of the success?
SPEAKER_04Yeah, I think you know, one of the things that surprised me when I wrote the book, coming on 20 years ago now, was that the digital computer in this case, which was quite new, it didn't take over the landing from the people. It actually allowed them to design the human role in a way that was most effective. And actually the Soviet spacecraft at the time had much less sophisticated computers. They were old-fashioned analog computers, and they were much more automated than the American spacecraft were. And you had this the astronaut could choose the right level of computer control at any given time for what was happening. That was very important and ended up really contributing to the success. The computers of the day, for example, they couldn't look out, they didn't have cameras, they couldn't process where the landing site should be. And only the human at the last few seconds would be able to look out the window and say, I don't want to land there, I want to land somewhere over distant from that. And the system was very good at enabling the human to make that call at the last few minutes. Similarly, if there was a problem, the computer was not in charge of aborting the mission. It required a human intervention to push a button to abort the mission. And on Apollo 11, there was a moment where that decision was required. In fact, anytime a rocket engine of any kind was lit, it required a human to give a kind of manual approval so that the computer wouldn't run off and and start lighting rockets out of control. And so there were all these different boundaries that were put on the computer control. And it was ultimately the human decision whether to turn the automation on or off and at what level. Was the key one at what level to let the automation fly?
SPEAKER_03You talked about what NASA did to prepare the pilot and co-pilot to make sure they could fully perform their roles using simulators and both on the ground and in the air. Well, if you flip that, what what did NASA do to ensure the human pilot and co-pilot were actually in the loop? And thus capable of doing these, performing their roles and whatever else was needed.
SPEAKER_04So, you know, I think that the best word to describe what they did was negotiation. MIT and engineers here at MIT designed the computers and the software for the landing. And when they were first given the contract, they imagined, well, NASA said to them, what's the interface going to be? And the engineer said, Interface, who needs an interface? There's two buttons. There's one button that says, go to moon, and there's one button that says take me home. And NASA kind of said, Well, that's not really how we imagine it working. This this program is to showcase the great human pilots, great American pilots to the rest of the world, and the great skills and human decision making of American pilots. We need them in the loop. And the pilots themselves, the astronauts, said, you know what, we're the ones whose rear end is on the line. We need to have a pretty strong say in what's going on. They lost that battle in the launch phase, as I mentioned. They had next to no, other than their finger potentially on an abort button, they had next to no input into the process while the Saturn V was actually launching. But for the lunar landing, they had a fair degree of control and command authority over the vehicle and the mission. And not least because, on the one hand, their rear ends are on the line. On the other hand, they were the ones in the situation. They were always going to be closer to it and see more and feel more and hear more than anyone on the ground or any software programmer was going to see. So, you know, as I often talk to engineers who design missions today, if you think that the mission should be 100% automated and the human has no input, that's equivalent to saying we've foreseen every possible thing that could ever happen, and there's no chance that the person who's actually there in the environment at the moment could have something to add. And that's just, you know, real experience says that's not how the world works. And so there was any number of reasons that the crews themselves were the closest to the moon, the closest to the spacecraft, the closest to the situation. They should be the ones making a bunch of the calls, not least of them, where do you land and do you abort? And if so, when?
SPEAKER_00But implicit in your description just now about making decisions, where do you land? I think most of us tend to think that NASA had a very clear model of what the landing sites look like, but that's not really accurate. I mean, starting with Galileo looking through the telescope at the at the moons of Jupiter, we improved the model, but we never had a model that was accurate at the landing approach that a human would then be able to see. And I think that change in the model and that decision needs to be clarified, if you could.
SPEAKER_04Yeah, that's correct. So you have a model of the moon, you know, think about it as a photograph, and the photograph has some resolution, and you can't see things that are smaller than that resolution. In the case of Apollo, it was some number of meters. And so when the crews actually get there, they're actually able to see things that no one has seen before because they're just closer. That's a little bit less true today. We have very, very high precision maps of the moon that were not possible then. It's also true that you didn't know exactly where the crew was going to land ahead of time. In fact, one of the things that was striking about Apollo 11, very early in this 50,000-foot descent, they realized they were going to land long. And so they were actually coming into a place that was not where they had prepared to land. There were other errors in the system that pushed them about 3,000 feet downrange. And so they were landing in a place that had not really been contemplated or thought about before. And it was really Armstrong's eyes out the window that were required to designate that landing spot. You know, in engineering terminology, people would you would call it terminal guidance, right? You can you can concoct this whole flight on the way to the moon from Newtonian physics and all these kinds of measurements, but it's only when you're actually there that you're seeing the thing and navigating in relation to the thing you're actually going to touch down on. And that those are critical decisions that today were much better, actually. And and you could easily design a mapping system that does that. That's how we land rovers on Mars quite successfully and match a camera to other images or use a LIDAR to find a flat zone. But but those were things that we still needed people to do in the 1960s.
SPEAKER_03During that final 50,000-foot descent, seems like I remember hearing there was a bunch of alarms and things that went off that may have distracted the pilot and co-pilot temporarily, I guess, from monitoring the approach. What was going on?
SPEAKER_04So as they were landing, they got there was a light in the cockpit came on and it said program alarm. And you know, most of these aircraft, still true today, when everything's working well, the panel lights are all off. Everything's dark, so that you notice something coming on. And so Neil, sorry, Buzz Aldrin says program alarm, and then the computer spits out a little code. It's a 1202. And this was an unusual thing, and the crew actually didn't know what it meant. And so now you've got a couple of minutes to get to the moon. Are you safe? Are you not safe? Is the computer working? Is it not working? And two things happened. One is the controllers in Houston on the ground were became also aware of the alarm, partly because Aldrin had said it. And they went down and looked at their cheat sheets that they had prepared. And it turned out that that same program alarm had come up in a simulation a week or two before the flight. And in the simulation, the crew had called an abort. Maybe the ground controllers had even called the abort. And they aborted the mission, and it turned out it was not a critical alarm, and they would have aborted it incorrectly, which was as almost as big a no-no as not aborting. And so they learned from that the controller on the ground, his name was Steve Bales, made himself a little cheat sheet and said, if you get a 1201 or a 1202, it's okay. You can keep going. And he called that out. He said it's the same one we had. He meant a couple of weeks before they had experienced that alarm. They called that up, we're go on that alarm. Actually, they said, as long as it doesn't happen again, did happen again. But but also at the same time, Armstrong stayed cool, and he felt he actually felt that the spacecraft was still flying properly. It was still responding to his commands, and he wasn't going to abort just because of some alarm. He would only abort if he felt he wasn't in control in some way. And so he kept going. It turned out the alarm then recurred four more times. I think there were five total by the time they landed. And it was a problem, not least because it distracted the crew at a critical time. They had to deal with that, they had to think about it, they had to talk on the radio. What was happening was the computer itself was getting overloaded. And there was a radar sensor that had been incorrectly left on in the checklist, and that sensor was spitting all kinds of spurious data into the computer. The computer was overloading and was literally rebooting in real time. Now, when your desktop computer reboots, you've got to sit there and take a coffee break because it takes five minutes for it to boot back up. The Apollo computer was designed that it could basically reboot instantaneously and would keep functioning. And it was a great example of a robust design where the computer would then drop its lower priority tasks, things that didn't matter as much, but keep the spacecraft flying, which it did very well. And so they flew, in a sense, through these program alarms. And you know, later the newspapers would say, oh, the computer failed, and Neil Armstrong turned off the computer and landed by hand. That's not true, and that's not what happened. The computer did an amazingly good job at staying in good command and continuing to function. And amazingly, the ground controllers plus the crew across this 250,000 mile communications link diagnosed the problem, figured out what not to do, and kept going and landed successfully.
SPEAKER_00I want to make sure I understand this. You're telling us that the designers of the computer foresaw the need if the computer overloaded, in spite of the fact that they probably had a margin of error for overload, that if it did, there was a way to have it dump lower priority tasks without causing an abort, which to me is an extraordinary amount of foresight.
SPEAKER_04That's correct. So actually, and and that low priority task was driving the display. There was a certain amount of compute that was involved in displaying the numbers to the crew, and it decided that those were low priority, didn't matter as long as the spacecraft kept under control. And it dropped the display tasks, which is something that Bales on the ground recognized would be happening. But in his note, it said, long as the spacecraft is still responding, you can keep going. And actually then Aldrin stopped requesting the numbers be displayed, and that that stopped the alarms.
SPEAKER_02These days we call that load shedding. And in fact, it was designed into the system. But the alert came to say, oh, I'm shedding load. Right.
SPEAKER_04Yeah, actually, if there was an error in the design, the error was that the alarm signal was sort of more urgent sounding than it really was. And it really wasn't a very big deal, but the way it lit the alarm light to the crew made it sound like a bigger deal than it needed to be.
SPEAKER_05Well, well, you had said that there were sort of three options to this whole journey. It was abort, land, or crash. So it does sound as if they did put in place safeguards to try to prevent, you know, option one abort or option three crash from happening. But I was intrigued by something you said that you know it wasn't true, or there was a lot of media or news reports about what Armstrong did or did not do, right? Um, some of the stories said he turned off the computers and landed it himself. If you could just tell us a little bit more detail, sort of what did he spot, how did he respond? I'm assuming they're in the small craft, there's all these alarms going on, sort of, you know, what was his role in that moment?
SPEAKER_04Yeah, so a little bit later, Armstrong does turn off the automatic targeting system, which was something that was basically expected. So rather than him pointing the spacecraft to a spot and the space the computer flying there, he changed the mode to one that was slightly less automated that enabled him to fly over this crater that he saw in a in a still pretty automated mode, but one where he had a little more control and set it down under that kind of control. So he he lowered to the engineering term, he's he used a lower level of automation than it had been programmed in. But he didn't turn off the computer and he didn't, and it really wasn't because of the program alarms, it was something that had been understood he was probably going to do anyway. And what's interesting actually, if you look at the climactic moment of the first Star Wars movie, which is only five or six years after Apollo 11, the climactic moment is Luke sitting there trying to destroy the Death Star, and he can't get his computer to lock on. And the voice of Obi-Wan Kenobi says, Trust the force, Luke, and he turns off the sighting device and he fires the missiles by hand and saves the day, right? And nearly every space movie since then has some moment like that where turn off the computer and land it by hand is the kind of legend of how it went, based on an incorrect reading of the Apollo 11 landing.
SPEAKER_03So what did Armstrong use the force for there? What was he trying to do? Was it just don't let anybody cancel my ride, or was it something else that he had to do to make sure everything was safe?
SPEAKER_04He felt that the landing spot that had been designated had a bunch of large boulders in it, and he didn't want to land among those boulders. Now, it's interesting because that didn't require him to turn off that automation. He could have redesignated the spot. There was also a large crater in the way, and he flew the spacecraft across the crater and landed on the other side of it. He said it's an old pilot's trick. When in doubt, land long. And he also said he had some great quotes about what he was doing. He he said, like a frog finding the right lily pad, I was looking for the right place to put it down, and I was absolutely adamant about my God-given right to be wishy-washy about where to land it. He also knew very well how much fuel he had, which was still 20 or 30 seconds or so, and he was going to use that time to just be extra sure. There was no reason for him to put it down before the final moments when he had to. Sometimes you hear people say, oh, they only had 30 seconds of fuel left, they were about to crash. None of that is really true. There was about 30 seconds before they had a mandatory abort if they ran below a certain amount of fuel, but they weren't going to run out of fuel, they were just going to have to abort. Not something anybody wanted to do. But he he intentionally pushed it pretty close to that limit because why not use that time?
SPEAKER_00It also sounds like you're suggesting that when reporters and the readers of the reports interpret what happened, we're putting ourselves in those positions without the experience and training that he had. Tell me that I have only 30 seconds of fuel left, and as a layman, that sounds terrifying. To him, it obviously sounded different. That's correct.
SPEAKER_04And you know, also I think people have a very different sense of what digital and computer systems will do these days. They're very different this today than they were then. And so people had a a much simpler view of automation and what it did, and the automation was much simpler. And so we have so many more digital interactions in our lives these days. I think it's easier for people to understand different kinds of automation than it would have been then.
SPEAKER_00I'd like to take you out of the Apollo mission that you've been describing and go to Apollo 14, whereas they neared the time to initiate the descent, they got an erroneous message that was to have the computer order an abort when it shouldn't have. Could you describe what happened and what they found was actually the problem that it caused and how they resolved that problem so they could go forward with the landing?
SPEAKER_04Sure. So on Apollo 14, again, they were in this safe plateau of 50,000 feet above the above the moon. And as they were preparing to descend from the 50,000 foot, they were preparing the spacecraft, they got a signal that someone in the cockpit had pushed the abort button, which they hadn't done. And they did a little bit of research with it, and they found that the button wasn't pushed, but there was something in the button that was sort of short circuiting. They found out later. And so because they were in this safe plateau, they actually spent, I forget whether there's one or two more orbits of the moon, they could take time to troubleshoot the problem. They weren't in a hurry. And actually, maybe it was more than two additional orbits. And the crews on the ground, some of the young programmers here at MIT, particularly Don Isles, who's still around, a friend of mine, they didn't quite change the software, but they reprogrammed some of the registers in the computer to ignore that erroneous abort input and took a little while to make sure they had a good procedure for it. But then they were able to proceed because they could basically tell the computer, don't pay attention to that abort button if it feels like it's happening. And they they ended up landing quite normally. It would have been a bit of a challenge if they really did have to abort. Instead of pushing a button, they would have had to, you know, issue a bunch of keystrokes and there were rooms for errors, but that didn't actually happen. So it was a case where there was a hardware failure. It turned out to be a little loose piece of solder that was floating around inside the switch and shorting it out. There was a hardware failure that the software was able to compensate for and in a sense fix in the heat of the moment.
SPEAKER_00Well, your description of it in the book reminded me of the way we used to adjust early black and white televisions. If they started to be too noisy or the signal didn't seem right, we'd go over and tap the side of it or hit the top of it. And if I recall correctly, one of the members of the crew tapped the panel at ground control's request, and that temporarily resolved it, which is may have made them think there was something floating in that gravity-reduced environment.
SPEAKER_04That's correct. And uh, you know, as an electrical engineer, I see most problems with electronics are actually mechanical. Although much less so with computers today. We don't you can still tap things, but it very rarely will fix the problem.
SPEAKER_03So speaking of computers today, what kind of lessons would you draw from the Apollo program for introduction into what we're facing now, which is AI, artificial intelligence, augmentation of safety critical systems on future vehicles?
SPEAKER_04So, and this is partly why I wrote this book, because even 20 years ago it was clear that there were a lot of phenomena that were coming up at the time in airline cockpits, as we got computer-controlled and fly-by-wire aircraft, and you began to have accidents when the computer and the person were really fighting with each other over control of the aircraft. And Apollo really was the first time that a digital computer was in a life-critical real-time situation for human beings. Previously, all computing had been done kind of offline in a different way. And now we have computers everywhere in our life, including in our cars and our blenders and our micro ovens. And people in the 1960s, including the astronauts themselves, their sense of the computer's intelligence was every bit as intelligent as we think of ChatGPT or any other AI system today. It was it was mysterious, it was a little spiritual, it was maybe not to be trusted. Those things were very, very similar in how people perceived that intelligence. At the same time, to really get the job done, the kind of collective group of people working on it realized that you still wanted people involved, you wanted the computer to augment their skills, but when it came time to the life or death thumbs up, thumbs down type decisions, you really wanted people making those decisions. And I'm not sure we've really I think that's probably a pretty good guideline for systems today. All the AI you're reading about is still really non-life critical AI. You know, airliners, fly-by-wire systems and airliners do not have generative AI systems running in them, nor do CAT scan machines, nor do really anything that has human lives based on it. It took 40 years just to learn how to use ordinary software in those types of systems, and and they've been made very safe. And at the same time, the places that they're unsafe, like in the 737 Max crashes, have as much to do with improper engineering of the relationship between the machine and the and the person as they do with the machine itself failing or making a mistake.
SPEAKER_00I want to go back to a phrase that I thought was really elegant in your answer just now, where you said that the problems often originated with humans and machines fighting for control. And what we're going to try to do in our next episode is discuss with you where humans were fighting with machines and not understanding what the machines were doing because they hadn't been trained as exquisitely and meticulously as NASA trained the astronauts. And it resulted in two major ship accidents, one involving a collision with a destroyer and an oil tanker in the Singapore Straits, and the other, more recently, just last October, resulting in the sinking of a New Zealand vessel off Samoa. So I'd like to keep that in mind, that that theme that you've highlighted for us of how to keep humans both in the loop, but not fighting with the machines that they're trying to operate or the digital systems that are intended to assist them.
SPEAKER_04Right. And and when I in all those cases, when the humans are fighting with the machines, they're really fighting with other humans who are the humans. Who designed and programmed the machines. And the human-machine relationships are human-human relationships at their core. And even though the people who did that designing and programming, they may not be around. They probably don't work for the company anymore. They may not even be alive anymore, but they're still their sense of what the user might do is built into the machine. And if the user doesn't quite do that, that's where those conflicts arise. And so, you know, I think over the years we've developed some best practices around these things, including always trying to have the machine be as absolutely clear as possible about who is in control at any given moment. And not all designs live up to that. Not all people are trained to see that, and so on. And there's a term that is used in airline cockpits that I think we'll see when we talk about those two ship accidents called mode confusion. And it's a it's a very dangerous, huge source of accidents when the when the humans operating are really not clear what state of control the vehicle is in.
SPEAKER_00I think it's important for our listeners to know that you don't take a detached view of these things in aircraft because you're also a pilot.
SPEAKER_04That's correct. I I actually learned to fly while I was writing this book on Apollo, partly because I felt like I needed to understand piloting in order to understand the issues that were at stake there. But now I fly all the time, and my life depends on these issues and behaving properly with regard to them, and that uh, you know, of course, take them very seriously.
SPEAKER_02I'd like to add one more point that that you that came out to to me very clearly from some of the things you said during the earlier parts of this discussion, which was that there was a fairly intense negotiation between the astronauts and the software designers and the system designers during the overall design of the spacecraft and the software and everything else. I think that that is a the sort of a signal or a flag to me that says, gee, I wonder if in the things we're going to look at in the next episode of this podcast, I wonder if that level of discussion, really peer-to-peer discussion, not here we're building this, you know, what do you do you like the color of the of the joystick, right? It would be a real, you know, your your butt is going to be on the line, to use your phrase. You know, now let's talk about what could go wrong. I mean, it was obvious that that NASA really got the astronauts deeply involved in that discussion. And as a result, there was an awful lot of corner case exploration that was done before anybody actually stepped on top of a rocket. But I'm wondering if that was the case for these things we're going to look at next time.
SPEAKER_04Yeah, that that's a great point, actually, because there's a couple of things that are special about the Apollo program that are not replicated, right? One of them is that, especially for Apollo 11, everyone who was anywhere near designing or operating the thing was watching at the time it happened, right? So usually a system gets designed and the designers move on to other things or retire or whatever. In the case of Apollo, everyone involved was there at that moment. And that doesn't happen very often. The other thing was that the because the astronauts were already kind of national hero media figures, they spent a lot of time at MIT actually working with the designers. Several of them had master's degrees from MIT and actually participated in the design. But they had fairly high level of prestige, power, voice in the design conversations, more so probably than most of these design things that happen. Another aspect of that that is too often done is people say, oh, these are human factors issues. Bring in the human factors consultants at the end and they'll tell you how big to make the display letters. I reject that, right? I don't I don't there's nothing wrong with it, it's just not going to get the job done. And in this case, you saw it was all these conversations were conducted at a kind of basic engineering level as the system was architected. That's the right way to do it. And you can't just add a pretty display as a band-aid at the end. A bad display will doom your system for sure, but a good display isn't going to save a badly architected system. And I don't know about the McCain and the design case there, how those things were conducted. They're usually not documented very well. I don't think it's in the accident report, but there are many cases where designers and engineers are not talking to the users in quite the right way at the right level as they think through these design issues.
SPEAKER_00Mark, let me just pick up on that. Having advised foreign militaries during acquisition of complex digitally based weapons systems, that negotiation that you astutely highlighted can occur at the contract and specification stage, can reoccur at the preliminary design review, can reoccur again at the critical design review because they are making decisions on trade-offs. If we can't meet this performance spec, what are we going to give you instead rather than simply fail to meet it? And then most critically, it's important to have end users at the system acceptance tests because they will understand what a failure will signify in onboard operations that the land-based engineers will simply not understand. And there's always an argument as to whether that's interfering with the acquisition process or making sure that it stays on course.
SPEAKER_04Those are complicated conversations because as in Apollo, the users are also the people whose jobs might be automated away. And so there's a lot at stake.
SPEAKER_03Yeah, that's one big difference with the Mr. Armstrong and today. Well, this is fascinating. And it the fact that we're running full-time here, I think, underscores the fact that we want to talk some more. And in our next next episode, Professor Mendel will help us look at the causes of the 2017 collision of the USS John McCain with the tanker in the Singapore Strait. Thank you very much, Professor, and I can't wait to continue this conversation.
SPEAKER_04My pleasure. Thank you.
SPEAKER_05Thank you so much. We thank the Business Law Fection of the American Bar Association for their generous sponsorship of the production of this podcast.com. We read all comments and questions and we try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention the specific episode number. Please do also include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.
SPEAKER_01Thank you for listening to the AVA Business Law Section's podcast series to the extent that the section offers a robust collection of content.org slash bizlaw. That's P I Z L A W.