Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 14 - Adding a Dog to the Cockpit
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Send us comments and questions.
In this episode of Exploring AI Matters we talk with an accomplished leader in national security. Air Marshal John Blackburn retired from the Royal Australian Air Force in 2008 as the Deputy Chief of the Air Force following a career as a fighter pilot, test pilot, and strategic planner.
Defense procurement processes struggle with rapidly evolving technologies. These next-generation systems are so expensive and so critical that nations demand strong governance in acquiring them and keeping them current.
Artificial Intelligence promises great benefits for national security, but a significant challenge is how to adapt this new technology to sophisticated weapon systems. [2023-10-11]
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that in addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_02National security is a complex topic for Australia, situated at the edge of Southeast Asia, an increasingly contested trade and security space. The Royal Australian Air Force has recently demonstrated leadership in the development of AI-augmented, uncrewed aircraft. Technological change is both a blessing and a curse in this environment. Successful adoption of new technology can confer substantial advantages, but cost and complexity make success anything but certain. Defense procurement processes struggle with rapidly evolving technologies. These next generation systems are so expensive and so critical that nations demand strong governance in acquiring them and keeping them current. Artificial intelligence promises great benefits for national security. But the challenge is how to adapt this technology to sophisticated weapon systems. Welcome to Mind the Gap Dialogues on Artificial Intelligence. I am Roland Trope, a national security lawyer.
SPEAKER_05And I'm Charles Palmer, a computer scientist. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence.
SPEAKER_02In addition to our hosts.
SPEAKER_00Hello, I'm Alma Adams, a national security lawyer.
SPEAKER_02And I'm Mark Donner, a computer scientist. Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. In this episode of Mind the Gap Dialogues on Artificial Intelligence, we will be talking with an accomplished leader in the national security arena. Air Vice Marshal John Blackburn retired from the Royal Australian Air Force in 2008 as the deputy chief of the Air Force, following a career as an FA-18 fighter pilot, test pilot, and strategic planner. He served a further 11 years in the reserve, where he worked on the RAAF's fifth-generation Air Force Strategy and subsequently the Joint Force Plan Aurora Integration Strategy. John is today the chair of the Institute for Integrated Economic Research, Australia, where he has been leading the Institute's national resilience project since 2019. He also advises government agencies and companies on defense and national security, national resilience, and the implications of emerging technologies such as AI. Welcome to the program, John.
SPEAKER_05Thank you very much, Roland. It's a pleasure to be here. Thank you, John. Yes, for joining us. As a senior leader in critical Western national security establishment, how and and when did you first start thinking about artificial intelligence? Uh well, Charles, it's about 25 years ago.
SPEAKER_01Um we didn't call it artificial intelligence. So in the late uh 1990s, I one of my jobs was Secretary Defense Commander for basically looking at the defense of Northern Australia. And when we were doing very large-scale exercises and uh war games, we had a real problem. How do we make sense of a very large dynamic data set with information gaps, potential system attacks upon us? And one of the things we wanted to say, well, what is normal? We had so much information coming in. Can we have some way of having normalcy data? Tell us when something out of the normal is happening, or it looks like we might be attacked. And we started to talk about what is now called a digital twin. So we wanted to create a digital model of our system, one of the adversary, and then look at experimenting on attacking them or them attacking us. What did the system look like? Because we were having a lot of problems, just basic technical. We brought so much data into the system, for example, from civil air traffic control systems. How did we know that data wasn't manipulated? But we had quite a few system errors. It took 10 years after that to get a basic normalcy data system in place. The problem wasn't just technology, is that when that was developed by our scientists and industry, you couldn't inject it into our command and control system because that had been built by a company as a project. And we didn't have open architectures or any of these things. So it was a real challenge not only to recognize a problem, but find a way of how to insert that into a procurement or management system for capabilities with stovepipes. If I step back to more recent times in the last few years as a reservist, thinking about AI and what defense was doing, there's a lot of machine-based learning tools coming into place, very basic concept demonstrators. But what they're doing is like injecting little pieces to see what it's like. Ignoring that one of our major problems in modern defense systems is the integration at the system level. How's the whole thing going to come together? Uh, and I think the other one is a concern that a lot of people don't really understand much about this yet. It's an emerging technology. Uh, some of the AI startups I've worked with are brilliant on the AI technology, but they don't understand the context or environment we're trying to apply the AI to actually have some effect. And what that means is the data sets they use might be technically correct, but they miss out on things like well, what's the culture or the training system that we're trying to actually apply AI to? But also I think we have to really recognize that the limitations of where we are. I think about uh military platforms like fifth generation, we use that talk a lot. The AI systems that we're trying to introduce are Wright Brothers Letter, the first generation. And we've got to really understand that what that difference means and don't assume because it's a fifth generation platform that the AI on board is the same generational capability.
SPEAKER_05That does sound like quite a challenge. Not only are you adapting your existing system to use this new widget for which it was not designed to play, uh play along with, you also have to figure out how to buy it. How do you do this through procurement? In addition to the operational questions and doctrine and uh rules of engagement and all that stuff, you've got to be able to buy it. That must have been a huge challenge.
SPEAKER_01Well, just let's think back to the start of the last century. I mean, probably knew how to buy balloons, but someone has this flying machine and you go, hang on, this just doesn't fit into any of the model. Uh and that's the other issue is that you know, that we're all getting really excited with chat GPT at the moment of running around with our hair on fire because something almost sounds coherent. Uh, but we've got to recognize some of the basics. So I had a little bit of fun yesterday. I tapped into the chat GPT. Um, yeah, what why are the Chinese sending surveillance balloons over the USA? And it's a wonderfully coherent response. Said, I'm not aware of any reports of Chinese surveillance balloons being deployed over the USA. It is possible you may have heard misinformation or misinterpreted information, which I think is a bit unfair because I know their databases only go to 2021. But we've got to understand the limitations of what they've been programmed to do. So apparently uh the balloons are misinformation or misinterpreted.
SPEAKER_05At our share of uh creative information over the years. Yeah. But yeah, and and I see your absolutely see your point because the the training of some of these models might take uh days or weeks uh and a whole lot of power, and that might not be might be one of the many challenges you're gonna have integrating this not necessarily real-time technology into a very real-time environment.
SPEAKER_01Well, in some ways, if you if you're trying to do massive data as we have to right now with the computing power, you almost have to think in an Air Force sense of having airborne nodes with massive, with incredible connectivity. And that might be doing the AI functioning and purely providing the results like a thin client to the air, other aircraft or platforms. But what you've got to do there is you've got to rethink your architecture of how you actually construct your force, deploy it, and use it. Um, and when you don't design a force as an integrated architecture, you just buy the pieces.
SPEAKER_05Uh that's gonna mean an even bigger challenge with AI. And and once again, the pieces that were not necessarily designed for uh to play nice with the other components as well as in that particular environment. So I'd ask about procurement and doctrine. Yeah, have we made have you made progress uh in these areas? Are we are we figuring it out?
SPEAKER_01Uh I don't think so. Look, I've had a look at the US uh DOD directive on DTNE and OTE for autonomy and weapon systems. And when you read it, it's a good attempt to taking existing policy and saying, well, what is the difference now we're trying to do? It has statements that you'll go through rigorous hardware and software B and V and realistic development. Yeah, that sounds great. Uh my background, as well as Fiverr's, was as a test pilot, so I've done a lot of DTNE and OTE work. And then it says technologies and data sources that are transparent to, auditable by and explained by relevant personnel. Now it's a wonderful aspiration, but let's just do a reality check here. We're going to end up with systems that can't explain themselves. The bit I liked also, where it said that DOD personnel will exercise appropriate levels of judgment and care while remaining responsible for the development, deployment, and use of AI. We can't do that, frankly, with current tech or yeah, historical technologies. So, once again, I said, okay, well, how will DOD personnel do it? So I put that into chat GPT as well. So, how would an AI program tell us how defense personnel will exercise judgment? It came back and it was interesting. It sounded pretty good. They'll exercise appropriate levels of judgment and establish ethical principles, will ensure they're transparent and explainable, it'll conduct regular assessments to make sure it's functioning as intended, and it's aligned with DOD values. Now, when I read this, I said the AI system seems to be scooping up rings of corporate speaking banalities, and it repackages them in some form of coherent sentence, which excites us, that I think allows shallow individuals to regurgitate rather than actually think for themselves. And that's the early trap. We'll get oh so excited, I don't have to think I'll use it. So the guidelines at the moment, understandably, are simplistic. I don't think they're really feasible. You know, I thought to myself, perhaps an AI system actually wrote the policy itself, but I'm not going to be that cruel. Um, but the real problem then comes back is that you've got procurement and military systems. And in Australia, in 2015, there was a major review of our defence organization. It said, look, you can't build a current or future generation defense force just by buying lots of projects and hoping they integrate through data links or something later on. You've got to build a system. Our defense department has not been able to implement the recommendation from that 2015 report. So what we got, and I would be surprised if it was very different in the US, the culture is project focused, not program or system. Few inside the bureaucracy, apart from the deep experts, really understand this. But is that something unusual? No. In 2017, I did a year's work on integrated air and missile defense, both as a reservist, but also as a think tank. And I started to realize very few people who said IEMD understood exactly the complexity of the system. And therefore, all they would ever be able to do is buy little pieces. When you look at these areas, and AI is just the latest example, we're not going to be able to do much about implementing testing without significant changes to synthetic environments to test at scale. Because what we were trying to do is identify existing and emergent behaviors in systems, and you can't ever test that in the real world. Not unless you go and sit in the middle of the Ukraine right now and watch what's going on, but you've got to take a whole different approach to DT and ATE. And of course, there's huge risk in doing that as well.
SPEAKER_02John, from my experience in advising militaries during testing of weapon systems, the big challenge is what happens if things fail. And listening to you describing um the sort of stovepipe development, the lack of system conception, and the failure to be doing the kinds of things you recommended several years ago. If a you know a new AI system doesn't seem to work or can't be integrated, is the is the likely result that the military will default to the previous generation or to the or to push on and somehow try to make it work.
SPEAKER_01This comes back to another cultural issue, which is risk aversion. You're a senior officer in a position, you don't want anything to fail whilst you're there. So you can get this risk aversion and they defer the problems to the next guy. Look, I think we're gonna be risk. There's a risk of that. Or what they'll do, and I've seen this happen even in recent times, when you talk about a complex system problem, the answer back is look, we're gonna fix this piece here because I can do that whilst I'm in the job in the next two years. I'm gonna do that. That's where I'm gonna focus my team. But nobody's looking at the bigger picture. The question is, is how do you how do you know if you succeed or fail? Let's be honest about it. The failure rates with humans in the decision-making process are quite significant. We accept that in some ways, you know, we we lament it, but we accept it because we're humans. In my view, we've got to look at this in a different way of developing it, and we can talk about that a little later. But when the failure rates of AI start to come down to the failure rates of humans operating in a complex combat environment, to me, that's fairly good success. Uh we've got to make sure we don't think this thing is gonna be perfect or whatever. It's designed by humans, it's developed by humans, and so you're gonna have to do this. But as I said, the only way we're going to explore this is basically in complex digital twins and very complex synthetic environments. And we're gonna have to accept that we need red teams established, not just for an exercise, but continuously to work out how they detect, we'd attack ourselves, how we attack an adversary, because in this case, it's so much more subtle where you can start to interfere with data sets and how the neural engines are actually being trained and things. If you could fiddle at that level, very hard to detect. So I think we have to be realistic. The idea that we're gonna have a perfect system, uh, that's gonna happen, uh, because that's not real the real world.
SPEAKER_04So, so the you you you basically have sort of established sort of a spectrum between the ideal, uh, which is, you know, as as you sort of say, not achievable, and and we're used to being imperfect because we are.
SPEAKER_00Yeah.
SPEAKER_04Okay, and and in some sense where we are today, which is you know not great. One of the things that that that immediately makes me wonder about is are the adversaries that we are thinking about today or the ones that we don't know about that might come along next year, um, are those adversaries in in any better shape in terms of adapting or applying or acquiring this technology than we are?
SPEAKER_01I don't think so. Look, over the decades, let's put it this way, we in militaries have had the tendency to inflate the capacity of the adversary, you know, based on looking at how many platforms they have or how many people they have. The fact that they could be a complete cluster and not able to operate because they're so you know, like if you look at the Russians, so hierarchically dependent, they kill innovation and they become incredibly uh inflexible. Or some other cultures that we'd be worrying about now, and you look at how do they think, how do they make decisions. Uh, we've tended to inflate this in the past for the reasons of saying why we have to have the latest fifth generation, this or that. Uh, Vera's complete confused. And I think the advantage we have in the West, if we allow it to work, is the innovation, the contest of ideas, the dynamics in some of the younger generations, as long as we don't squash it. And I think in some of the adversary areas, whether it's you talk about Russia or China, I think they have a cultural disadvantage in one way, but a national discipline advantage in another to execute their goals. So, no, I don't think anyone's better off than us because introducing this type of thing requires a lot of debate, investigation, innovation. And that's an area in the West that we could exploit if we're prepared to.
SPEAKER_02Well, let me pick up where you left off. Um, Western militaries have reportedly gotten much more comfortable uh with bottom-up strategies in recent years. I could have made the same statement in 1944, based on the way soldiers innovated out in the field. And we've been reading a lot in the press of how innovative Ukrainian forces have been, uh, which I take it the you know, US and Australia are monitoring closely to see because that instead of digital red teams, this is the real thing. Do you feel that bottom-up collaboration among troops on the ground has developed leadership acceptance for this mode of work? I know in Australia, for example, if there's a new bit of kit, uh, it often goes first to the special forces. And if they find that it works, then the other branches try to get in on it. But um, if you could give us a sense, at least on your side, whether that's because it was a bottom-up collaboration or a contractor sold it to somebody on the top.
SPEAKER_01I I think it's a mixture. The majority of big system develops obviously come from contractors, they've been through huge amounts of testing. How you use that new capability in an innovative way gets really interesting. So I think this bottom-up collaboration has to be there. It varies depending on what's happening in the organization. When uh I was doing the mentoring of the plan Air Force's plan Jericho, which is a fifth generation Air Force plan, we were looking at how do we do rapid innovation and how do we go somewhere. So the top-down design model that happened happened, you say, Well, here's the requirement, the concept, here's the concept of operations with limited scenarios, do a gap analysis. Now we work out a program of work that says, okay, we've come top down and we've looked at this. Now go off for the next two years and implement it. But what you end up doing with that is you end up being reactive to system failures or problems without anticipating or exploring it. So with Jericho, what we did is, okay, let's not um get the generals to lead the fifth generation plan. So what I proposed to the chief was we would get two O si's kernel equivalents. We put them together with a small team, and we'd get them to go and look at you know, opportunities, technology, and in perhaps small vignettes, look at how you could use these in different ways in a future operational environment. And the idea was to give them freedom to experiment, to go outside the boundaries and say, well, using what we've already got plus this other stuff, we could do this quite different to what we currently do. And what we wanted to do is start to define the characteristics of this future force so it allows us to be far more effective and inject that into the top-down system so that when we're doing a gap analysis, we weren't only gap analysing on what the problem was today, we were seeing what the problem could be. This worked reasonably well for a couple of years in that particular mix of leadership that we had at the time. But as we started to define all these things that needed to happen, then we had some new leadership come in who said, Well, I've got this list of things to do. Now let's all just focus on the next three years to implement this list of things that we want to do, rather than keeping that innovation process running in parallel. It's difficult to do, um, but I think if you don't do it, you become reactive, narrow, and um you actually limit the potential of your force. So it's absolutely essential.
SPEAKER_02John, we've heard from experts in other fields on the program that there are two challenges to the adoption of AI or an AI-enhanced product. One is the trust deficit. For example, will the operators trust an AI system if it's introduced into their cockpit or uh an aircraft that's working with them that's AI operated rather than uh having a human pilot? Another is the loss of skills normally gained through that kind of experience. Uh, surgeons uh using AI, uh, especially if they're trained with it, may not do the kinds of things an experienced surgeon would have done because they're relying on the AI to do it and they may be missing things that the AI is missing. What is your thinking on this, especially as applied? To AI in the military field.
SPEAKER_01Yeah. I've been listening to a podcast where the ghostwriter of Nelson Mandela's autobiography got the tapes that he recorded when he was speaking to Nelson Mandela. And at one stage in there, there was a comment that he made, and I'll sort of paraphrase it. He said, You can't really find out if a person is trustworthy until you trust them. Because if you don't trust them at the start, you'll never find out if they are trustworthy. And I really thought about that. So I thought, well, how do you apply this to AI? I think we've got to think of a spectrum of simple processes to complex and risk levels. And so if it's a simple process or decision, then go through your verification piece. And then you have to say, well, I'm going to trust this. It's a simple process. It's a lot of noise, it's a lot of activity. If I can take it off the human decision maker, it will help us focus on the more complex problems. So I think there's an area where we should be trusting it, where it can be quite defined. If you get it into a complex dynamic environment and systems, uh, then I think we've got to take a slightly different area. I'd always have a degree of distrust. But mind you, when we first got moved from the Mirage to the F-18 fighters, we treated the F-18 systems in those days in the 80s with a degree of distrust. And I'll come back to explain why in a moment. So what we have to do is treat it differently. Take it as an advice, but have the human layer in there looking through their experience and context to say, does this make sense? Is there a disconnect here? And you have to think about it being an advisory rather than a decision authority. That means that your command and control systems get it interesting because you know you often have a yearly or even two-yearly software update. Think about your iPhone. You know, I accept the update because I trust it. These things don't work at times. What we found in some of our complex systems was a multi-layer software system. And so with one of our huge sensor programs we had, we were running a series of software layers where you could inject it, introduce a new capability out of, say, a layer above, work it with the operations. If something was concerned, you'd very simply, with almost a flick of the switch, revert back to the previous layer. So you didn't have to reload all the software. So what you have was a complex software management system where, for say 90% of it, it's running as it was. We're going to produce new capability and watch it very carefully. If there's any at all, we immediately revert back to the known running system. And it was a very flexible system. I think we need something like that with this. Uh, you don't want to be stuck in like with an airplane with a you know the annual software update happens and it's been through the whole process. It won't work. We're gonna have to do it differently. On the loss of skill suit experience, in the Mirage days, uh, this is the Mirage 3 when I started flight fly. Um it really was basic. There was no uh inertial navigation system, there's no GPS. Uh yeah, we had sort of navigation instruments that were uh very imaginary, they didn't do much for us. So everything you did you kept in your head. You had to have a mental picture of where you were, and we know you're running around at you know 480 knots, 540 knots at times, very fast, but you also need to be able to calculate how to get to a particular place, say plus or minus 15 seconds, so you don't blow other people up when you're doing combined strike. What we found is that when we introduced the F-18 in the mid-80s, the folks who'd fly in the Mirage kept that same mental model. And so when we had some system problems in the early days of the F-18, you went, well, that doesn't make sense. But when we brought new pilots into the system, uh, they saw these wonderful displays. Well, they must be right. You know, like we hear of people driving Google Maps into a into a pond or something. And we did see a difference there without keeping that mental back. So I think what we saw then was the start of a loss of analytical or deductive skills that then presents a concept or framework of how the world works, which then gives you what you feel is intuition. It's not, it's just pattern adding. And this is, I think, the risk with this. The education that we're gonna have to give people and the experience and training that we should be doing, I think, in virtual environments to understand how to use the tools is going to be the fundamental piece. It's gonna happen. You're not gonna push back on AI, but the use of it intelligently is gonna be our biggest problem because I don't think our education systems uh are doing really well in some areas already, and this is gonna make it harder.
SPEAKER_02John, you remind me uh that I once learned what certain navies and their submarines have the captain observing the executive officer developing a firing solution in reliance on the computer, but the captain is calculating it in his head to check the computer's accuracy.
SPEAKER_01It is, and that's what you need to train. I mean, there is a logic that when you fly fighters or strike aircraft, you either fly at 420, 480, or 540 knots, or perhaps 600 knots. They're all multiples of 60. So what you're doing is it miles per minute. So when you need to adjust something, you go, well, I need to go from, you know, I need to, you know, I'm two miles behind, so I need to do this for two minutes. So it's it's very simple mass in your head, but you've got it in place. You know, the 60 is a magic thing because you know, one nautical, one degree at 60 miles is one nautical mile. So if you need to change this, is that you've got this maths going in your head the whole time. Sounds a bit simple, but you fly in these ways that you can use mental maths. We're gonna have to think about that of how we look at the world. What is that mental processing? Because to me, the most powerful thing is you know, humans working together with their own concepts. And this is why airliners, this idea from some business managers that you could put one pilot in the cockpit because all the automated systems will take care of you is mad. The value in an airliner when things go wrong is you've got two brains in there with competing ideas. And a good friend of mine was the captain of the A380, that the engine blew up after takeoff out of Chengi, and they had five very experienced pilots in the cockpit, and systems just started to fail everywhere. How they got it back up and saved all of us' lives is amazing. But when those systems started to fail, there is no way that a single person in that environment could have dealt with it, no matter how great the existing technology at the time. So I think this is the thing is how do we keep those skills in place and how do we train people to use tools intelligently, not slavishly?
SPEAKER_00I'd be interested for your thoughts on this. As you were talking about the trust deficits and sort of the spectrum from simple to complex, and then the loss of skills. You had said earlier in the discussion that you know there's concern about a potential default presumption or bias, assuming that because it involves computers, it is it is accurate or it is right. And as you were walking through those iterations of the question or your responses to Roland's question, you know, it came to mind to me, do you think that the national security establishment has that deep appreciation of the trust deficit and the loss of skills and how you cannot just assume because a computer or machine learning is involved that it's going to be right or it's going to be accurate? Is there enough understanding to push back appropriately when those situations come up?
SPEAKER_01In pockets. So I think if you look at, let's say in a society of the natural distribution, you've probably got five or 10% of people who are really attuned to all this sort of stuff. The advantage in America is that you have what, 270 million people? In Australia, we have 26. So that 10%, you folks have probably got 30 million really brilliant, absolutely brilliant people. If you look at the same statistics, we've got three. So, yes, I think uh when I look at the discussion happening in the US in the think tank areas, all these other areas, yeah, you've got enough people doing it. As you go to smaller organizations, it gets problematical because you know, I've been in strategic policy areas and government, all this sort of stuff. The majority of people are, you know, good, intelligent people, but they're working a process. And the process they work is take the last iteration of the strategy, make a couple of tweaks, and then put it through a filter that's politically acceptable for the politician to say something. And so you become a slave of the past. It doesn't mean they're stupid, it means they're trapped with an organizational process and culture. And again, what I found in these areas are bureaucracies then also start to behave the way they're treated by the present government. So for the last nine years, our government told our bureaucracy, you're there to provide information, not policy. And if you keep beating somebody over the head with that, it's like if you keep telling a child they're stupid, they behave that way. And uh yeah, no, there's not enough people thinking about this because the machine's running uh and doing what it did before, but there are enough people in our society if we listen to them. Uh, and to do what you folks are trying to do here, get a different discussion going, bring other points out. So it's not a lost cause, but the bureaucracies, I think, in some ways have become self-limiting.
SPEAKER_04So your your your comments about developing trust resonated very uh loudly uh to me in a in a related area. There's a famous problem in game theory called the prisoner's dilemma that I'm sure everybody knows. Um and what's interesting is that there's a there was uh uh a uh famous mathematician named uh John Nash, who came up with the most interesting uh way to address that by basically saying, oh, well, this isn't a one-time game. It's actually a set of repeated games. And if you play it in a repeated game with a very simple strategy, namely whatever the your uh your opponent did last time, you do the next time. Uh you create a stability, you actually get to a stable solution. And the interesting thing is that that game theory insight uh it underpins things like the um the the thing called the I forget the exact name of it, it's for uh uh stability and uh security in Europe, OSCE, I think is the uh the name for it, which is basically developing trust over time by small tests. And that's precisely what you said uh in your comments on that. And I think that's the clear, the clear insight. As you say, you can't trust it totally instantly, but what you can do is you can build trust incrementally over time, and you can design that building of trust into your, you know, your acquisition and onboarding processes for systems.
SPEAKER_01Yeah, absolutely agree with that.
SPEAKER_02John, let's take an example where it didn't sound, at least to me when I read about it, incremental, but a real technological leap from a country with, as you just got done saying, smaller numbers of really bright people, but clearly very capable. In the last year, I saw media reports of an unpiloted combat jet called Loyal Wingman, developed jointly by Boeing Australia, the Royal Australian Air Force, and numerous other companies. It's reported to be aimed at accompanying a piloted aircraft to perform at the, at the very least, reconnaissance and defensive tasks. Can you comment on Loyal Wingman's use of AI in this and the significance that the Department of Defense in the US has now arranged for Loyal Wingman to be brought here for additional testing?
SPEAKER_01Look, I think with the unmanned platform, you know, unmanned platforms, AI is going to have to be an essential part of it. There's no doubt whatsoever. And what we've seen here is that Boeing Australia, really taking you know the knowledge and everything out of the US, has done this development program in Australia at a small scale. Uh, I think they're doing it there primarily to get to avoid some of the ITAR problems. Uh, but of course, the US has its own development programs that are going back to.
SPEAKER_02How to explain ITAR. The odd not everybody in the audience knows what that is.
SPEAKER_01It's the technology um restrictions that go on the export of critical technologies for the United States. So when you look at these projects, there's a whole bunch of regulations and limitations that the US government has about what can and can't be exported, depending on which country it is and how it's used. So if you want to develop something uh within the US, then it will obviously have all those regulations and limits. If you want to develop something, say in Australia, it doesn't have the US ITAR regulations if it's not using US technology. And therefore, in a market sense, you could sell it to a lot of other regional countries. So the but let's not kid ourselves, this is a small-scale development here in Australia by a US company. Uh, and in parallel, there's massive stuff happening. So AI is absolutely essential, I think, in this. But as the comment I'm at, Loyal Wing is a great example of a looking at a fifth generation level integrated force design producing a platform, hopefully with some low observability things about it, to work as an integrated one. But the technology they're going to put on that platform is Wright Brothers level, it's first or second generation AI. So the trouble is we're going to be looking at a you know shiny thing that's operating with a JSF, therefore, this must be the latest. And this it's like the equivalent of having a dog on board that barks if it sees something. Um, yeah, great start, we have to go there, but let's not get too excited. It's like getting excited about chat GPT, and then when you lift the lid and go, it's just a dog, yeah, it's barking.
SPEAKER_05That reminds me, John, of of the the um astonishment people had when they saw that the computers running the Apollo systems, the the systems in the Apollo series of uh flights had core memory, for example. And they were they were far behind uh modern systems, quote quote. And of course, the explanation is uh well, um bleeding edge is nice, but it's really nicer if everybody else works out the bugs and uh we get to take it home. Along those same lines, um, I've often heard that a modern fighter pilot, uh fighter aircraft really can't be flown by a human, and that it has to make you in the design, you have to make a boatload of uh accommodations. Uh, you know, don't freeze the pilot, don't asphyxiate the pilot, uh, don't crush the pilot when you need to turn. Oh crap, we've got that. So let's turn real fast. Oops. It seems like it just might be better if the squishy part wasn't in there.
SPEAKER_02Oh what do you think?
SPEAKER_05Is it gonna be remotely flying remotely?
SPEAKER_01Or uh are we just not gonna go there? That we have to head towards getting the humans out of the cockpit. So it's not only the limitations of the human, but I mean the extra weight assistance you put on board this thing. I mean, you're never gonna get a fire department airplane unless it's got an injection seat, I'll tell you. Uh, and that that's a fairly big bit of weight. So look, it the human in the cockpit today has been the greatest strength because the ability to make rapid, you know, very, very quick decision making, to work as a team with the other aircraft. But it's also the fundamental vulnerability in that, you know, particularly in the early stages, a lot of working that team was on the radio. Now, the radio can be jammed, but also you've got to say something or listen to each of you guys talk back to you. Where we're going with, you know, particularly the JSF and the later systems, where you've effectively got sensor fusion, communications that are very secure, and you're sitting in one airplane, but the sensors in all the other airplanes, it's as if you have them in your airplane as well. And that's where the huge leap has been. We have to get to where we do take the pilot out. Remotely piloted, yeah, it's fine for something like a you know Reaper or a Global Hawk or Triton. Um, but the speed of decision making and the environment we're going to be in, it's going to have to have autonomy. Uh, you can't make those decisions and act adequately without it. But what will happen is the tactics using remotely piloted vehicles or in those autonomous are going to be very, very different. You know, when you haven't got the concern about having to get a get the pilot back, or if they eject, try and go and rescue them. Yeah, so you don't need the CSAR stuff going. It'll change performance capacities, weights. I mean, even with the best G-suits, you don't want to be running around above 8G, you'll have a whole bunch of other sort of problems with your pilots. But to be able to maneuver adequately in the complex environments, you need to do that. So, no, we have to go that way. Uh, but we have to understand how those systems might be attacked by an adversary so that we can ensure that they're not used back against us.
SPEAKER_02John, just a quick follow-up on that. Uh, one of the two things that humans do better than AI is they tend to recognize when something's amiss, something's um, you know, not right, at least from their perspective. And they tend to fail gracefully rather than just simply defaulting. In making this, getting the pilot out, are we what's the risk that we're going to be giving up that capability to say, I think I'm lost, I think this is not working. Uh, I can improvise.
SPEAKER_01I think you uh probably not valid, but I think you almost need a separate AI system on board the airplane. We used to joke that you know, if you wanted a fully automatic air uh airliner, you'd have the pilot and you'd have a dog. And the dog's job was to stop the pilot touching the controls and just leave it to the autopilot. And you know, in this case here, you almost need that second system on the side here going, is this AI pilot lost the plot? Is there something that doesn't make sense? And so in in my uh, I guess traditional mind, the slight part of my mind that might be traditional, I'd be thinking about you want to have the checks and balances. So it's not a single AI system on board the airplane. It's like uh, you know, you've got another system watching this going, this is going well off the rails. Uh, but by having a parallel system, it's going to be possibly hard to attack them both exactly at the same time. And that's what I like about the the checks and balances, uh, that we need to have a think in there. But yeah, keep the AI dog in the other seat, making sure that the other AI doesn't do something stupid.
SPEAKER_05Well, you raise an interesting point too when you said if we did do uh remote piloting or even just remote uh data feed, because maybe the an autonomous jet wouldn't have sensor capabilities that that uh downrange or somebody on the ground or a satellite might have. But what about the risk, like you said, of of uh someone interfering with that data feed? It's one thing to confuse the pilot and cut off his or her connection. They've got the creativity. It it all comes back down to the failure and who's in charge, uh mistargeting and you know spraying into the wrong airspace and violating this or that. Sounds like it's it's it's not gonna happen soon.
SPEAKER_01No, I think look, let's be real, AI is decades, a real AI. I'm pretty sure it's still decades away. Uh as some of your previous discussions you've had on your podcast that I've been listening to. But it's not just an AI in a cockpit in this case, it's AI battle managers. So you're gonna need other AI routines that do the job that a battle manager does today, which is looking at all that sensor coming in, the management of the various assets in the area. And that functional needs to go in that area as well. So you still have humans in the command centers, but the battle management and the dynamics of this working so rapidly with these other elements. But then you've got to come back and say, okay, I've seen AI systems that are very simple, might be identifying this as a particular aircraft. So you've got disposable remote sensors you throw around the place. It does it from basic AI and it sends a very simple message back. I'm seeing this particular thing here as a sensor. So, what you've got to then start thinking about is what information has to flow to and from those platforms in a complex environment and what's the architecture of getting that information around. Because our way of thinking about data links today is not going to survive a high-threat environment. If you start to think about the potential of what could happen to satellite communications in a contested environment, you've got to have multiple layers, high bandwidth communication systems within the combat area. So you've actually got to take a whole architectural approach to say, I can come up with this wonderful autonomous platform driven by AI, but what's the architecture it fits into to do its job? And that's where I see a lot of information lagging. Because I think the thinking about those architectures and battlespace stuff is I don't know, I'd say probably a decade behind the thinking about platforms. Because you can sell platforms as a package. Talking about a battle space architecture, you've now got this, well no, yeah, we've got multiple companies trying to do this, and it can't happen at a single service level. It's a joint issue. And uh the authority of joint uh military groups, even in the US and Australia, is is variable, let's just say it, because of the culture of organizations. So the complex design problem. hasn't it, I don't think it has been properly looked at. I still work in that sort of space as a consultant, but it's very early stages on battle space architectures. And of course, but let's learn the lesson, as probably the Ukrainians learning now, when uh with the Starlink, they're now being told they can't be using Starlink for drone operations. So they've actually built up a model of what they're doing, and Elon sounds the reports of the Elon Musk comments for something about not allowing them to be used for offensive operations. So let's really understand all those complexities of the enabling architecture around the outside.
SPEAKER_02John, I want to ask you a couple of questions about protecting AI systems. But before I do, um, I'd like to ask you a more personal question. Uh, this is not my first conversation with you. And one of the things that I'm impressed by is how differently you think when you move to the next stage in your career. You were a fighter pilot, then you were a you know, deputy chief of the Air Force. Undoubtedly, when you stepped out of the cockpit, you started seeing things differently than you did when you were a fighter pilot, but you never gave up the experience you developed there. Now you've been out of the command and control structure of the Australian, the Royal Australian Air Force. You've been working in industry, and you seem to be much more reflective than I think somebody who's in that chain of command right now. Can you tell us how your own thoughts on AI have changed as you've moved through these different structures? In other words, do you now think you're in a better place to think about it? Or would you do you wish you were back in where you were able to make the decisions?
SPEAKER_01Well, uh, let's be honest about it. When you're back in the squadron, your decisions you're making are very narrow because they're about the media environment. The thinking process happened is that when I was a commanding officer of a fighter squadron, I did a master's on uh defense of strategic studies, and started to think about not just the military part, but how a national security framework works, and realizing that we didn't have one. So that's what triggered it. The masters I did when I was a commanding officer of the squadron. The second masters I did and the next rank level developed on that. Uh but what happens once you get past colonel level, certainly in our military, you get so overloaded with process, politics, the system, that your ability to sit back and think and reflect is quite limited. When I got out from full-time service, and even though I was doing a huge amount of contracts as a consultant, I did a lot of work in think tanks and I had the opportunity to educate myself. And so I started doing studies into things like logistics, base policies, a whole bunch of other weird stuff, and realized that despite the depth of expertise as a military officer, my understanding of how the world worked was very limited. So in the next 10 years, I spent a lot of time doing different studies as well as my consulting work. I started to learn about business, which you do as a consultant, everything from the large companies to the small. And then I came to the conclusion that most other people are so busy in their jobs they don't have time to think either. So I used to have some academics who have time to think, but are not in the middle of the reality of well, how the hell do you implement something? You have politicians who, unfortunately, most of whom there's some brilliant ones, but most of them are looking at the next three years. But you've got to then step back, take the time to educate yourself and rethink and just take your thinking up a level. And that's the journey I'm still on. When I had the opportunity to start working with small AI companies, I was asked by a defense organization to give them some mentoring advice. And then I had to look at what they were doing. I thought, brilliant in their area, but no understanding of context or the actual thing they're trying to work on. Not not they're not stupid, they're very smart people. But that's where the education gap is. We educate in pieces. We have an industrial age policy, politics, and education model. Train them all on the bits. When they get out there, they'll sort it. So that was the opportunity of the journey. And the best thing I ever did is when I retired from full-time service, is I never worked for a company again. I only worked as a consultant. Because once I've seen with my peers, you work for a company, you're now back on those railway lines again for the company's reasons. Uh so yeah, it's been an interesting journey. Uh, but it really comes back to is how do we educate and inform uh people to allow them to keep learning. If you stop learning uh any time in your life, I think you're missing out on something pretty important.
SPEAKER_02Well, I'm glad you gave that answer because I want you to speak from that perspective now on how confident you are in our capabilities to protect AI systems from adversaries. And to put that in context, the DoD directive says that before fielding an autonomous system, the DoD will verify, among other things, that system safety, anti-tampering mechanisms, cyber survivability, operational resilience, and cybersecurity capabilities have been implemented to minimize the probability and consequences of failure, and a monitoring regime will be in place to identify and address changes in the operational environment, which is probably going to be pretty dynamic, data inputs and use that could contribute to such failures. Is that all aspirational, or do you see that as something that is incrementally possible given the perspective you now see these things from?
SPEAKER_01I think we will stumble forward in small steps in reaction to failures because we're in junior school, in my opinion, here, because the stovepiped approach to acquisition or systems therefore produces a stovepiped approach to threat that can mask potential system level problems. And so our vulnerabilities and the gaps between our capabilities, um, not necessarily within the capabilities themselves. So when you look about the risk, I think data modifications is a huge risk. If an adversary can determine the data sets that are being used to build a particular system, then manipulate them. The consequences are really ugly because despite what you might do in some DTE and OT work in a narrow area, you get large-scale loss of confidence or trust in an AI augmented system that's in use. Now you've got a problem. Because once people lose confidence in that system, what do they do in the middle of a complex operational situation? What are the reversionary modes? So we have to design with a realistic look that there are going to be failures and mistakes. How do we actually think about managing those as we go along? And one of our biggest problems there is honesty because when things start to go wrong, it's often a human nature to cover it up. Oh look, don't worry, it's under control. And governments don't want to spook their populations. So again, using security boundaries, which sometimes are overplayed, they don't inform the public. And so it actually compounds the problem. So I think we're going to have a hell of a mess uh as we go into this because we have a naive belief in technology on the one side, a shallowness in thinking in general, which is being driven even more that way with social media and TikTok and all this other stuff. And I think AI will actually amplify that in our society. It's like turning the volume up to 11 to use an analogy of a very old cynical moody. Uh, this is going to actually amplify the shallowing of thinking and the consequences in our society, where our responsibilities now, those who want to think about this, is how do we deal with that journey we're going to be on? Because it is a reality that we're going to go here. We do need AI. It's going to be a very difficult transition journey in a society that's largely distracted to shallow issues. So, on the one hand, I think we've got the brains to do it. Um, but man, it's going to be a difficult transition.
SPEAKER_02I I can't help but ask you this question. Given the competition between adversaries right now, or peer competitors, to put it differently, um, do you think we're better investing in systems that defeat and uh compromise the adversary's AI system rather than trying to build one that they can't do that to us on?
SPEAKER_01I don't think we'll ever build one that can't do us to us. So therefore you've got to counter it. My biggest concern would be is not in a sense of military operations, is the use of AI to reduce confidence and uh, I guess, self-belief in a society. So you look at you know the reported use of uh Russians and Chinese in terms of elections, whether it's in England or America, you look at the manipulation of media, the manipulation of interest groups, that AI will only allow this, and particularly some of the fake AI generated capability, this is going to amplify and drive the divisions in our society. And I think that is a much bigger risk than the AI implications in a military sense. Because without cohesion in our society, without a degree of belief or trust in what's happening, then Western societies can fragment very, very rapidly. To me, that's the big threat, not the military part of it. Yes, there's a lot of complexity in the military, but the military, I think, is you know, is the hammer. But if you've got no idea how to use that hammer or no willingness to use that hammer because your society is imploding, that is, I think, the national security risk that we're facing in the next few decades.
SPEAKER_02That sounds like what the grid operators face when there start to be cascading outages and they lose situation awareness and trust in what awareness they still have.
SPEAKER_01So when I think about potential roles of AI in our society, what came out of the National Resilience Project we've been doing for the last few years is we postulated three aspects in society that are important. I just ripped these off in the military, and then we started looking at civil society. We said there's three things that make our society resilient. Improve situational awareness. That means you've got to assist in clearing the fog of war, not only in a military sense, but in society. You need to have near real-time assessments of the situation, merging risks and opportunities, again, not just for the military, but for society. You have to be honest with your population, say, look, this is the reality of the world, yet things are messy, but hey, we've got a problem. That's number one, situational awareness. Number two, you need to use AI to help team your ability and performance. None of our societal problems can be handled by an entity or an individual or a political party. We're only going to deal with the challenges that we face in the next decades, whether it's economic, environmental, conflict, uh, energy. So we have to work as a team, but we're not good at supporting and enabling teaming because there's this competition stuff half the time. If you have shared awareness and you have their teaming ability, the third one, again, from the military, is preparedness and mobilization. My experience in preparedness and military assessments, they were always made after the fact where we had failed in something. And you'd sit in these chiefs committee meetings, finding someone to blame for something that had gone wrong in our preparedness last year. We've got to shift that and look at preparing, again, not just for military, but for the inevitable problems we're going to face with energy transition or climate. We need to think about how AI can think us, help us understand the changing situation, the emerging risk rather than the threat, and say, well, how do we better prepare for it and track our preparedness? And the last one is mobilization. We mobilized before World War II pretty effectively. I can't think where we've mobilized ever since. And when you think of the scale of the challenges we face, society we have to mobilize our society to act, which means we can't all keep living necessarily to the same standard and freedom that we have, being very cautious about those words. But if you mobilize to deal with an existential threat, which is some of the issues we're facing, then frankly, the complexity of that, we're going to have to think about how would you use AI to help you understand that. So situation awareness, teaming, preparedness, mobilization. And that's the line that we're pushing. And what I'm most pleased to see in Australia is the discussions that are now happening. We're having our first national risk assessment will be coming out in the middle of 2023. And that will be an ongoing annual process. We don't have a national security strategy, but as we've pointed out in our work for the last few years, you can't have a national security strategy if you haven't got a national risk assessment. Because what's the strategy for? But the government's taking this on. There's groups of us who are involved behind the scenes with that. So I have a degree of optimism, understand your risk, be honest about it, and then let's get on and do something about it. AI could play a great role there, but the people developing AI are technically brilliant in a very narrow area. You've got to understand the context, the society, and the problems we face at a society level to work out how do we apply it there. If you don't do that, I don't care how good your military is, it's not going to be a value to us if your country's falling apart.
SPEAKER_02Well, let me just say, John, listening to your comments, I have learned a lot, but I'm also, as I've always been in conversations with you, extraordinarily impressed with how articulate and clear you are about what you're thinking.
SPEAKER_01Well, thanks. And uh yeah, look, I'm still optimistic. You know, the sort of conversations we're having are happening in a lot of places. So we've got a lot of smart people who's just trying to nudge society in the right direction. But thanks, Ronald. Thanks, folks, for the opportunity to talk to you today. It's great and I'll I'll keep listening to your podcast. There's some really interesting stuff coming out on this.
SPEAKER_02Well, thank you very much. We we were honored to have you as a guest.
SPEAKER_05Yes, indeed. Thank you so much.
SPEAKER_00Thank you.
SPEAKER_02We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. We welcome questions and comments from listeners. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention a specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.
SPEAKER_03Thank you for listening to the AVA Business Law Sections Podcast Series to the extent that the section offers a robust collection of content. To explore more about this topic, or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.