From Action Plan to Mission Impact: Conclusion

In this final episode of INSA's From Action Plan to Mission Impact: Key Takeaways from America's AI Action Plan podcast, hosts Chitra Sivanandam and Dr. Yevgeniy Sirotin reflect on the key themes and lessons from the series. They discuss the challenges of moving AI from strategy to operational capability, including workforce readiness, data quality, trust, governance, and responsible adoption. The conversation highlights the importance of clear standards, model provenance, public-private collaboration, and international competition, while recognizing the progress already being made across the national security community. They conclude with a look at what it will take to sustain momentum and translate America's AI ambitions into lasting mission impact. 

Transcript

Chitra 0:00

Welcome back to uh the final episode of Insays All Source Podcast from Action Plan to Mission Impact. I'm Chitra Savanadham.

Yevgeniy 0:08

And I'm Yevgeny Sarotin.

Chitra 0:10

So we've spent some time this series looking at what it actually takes to move AI from strategy on paper to capability in the field.

Yevgeniy 0:20

And we've been fortunate to have some great conversations along the way, um, hearing from leaders across industry and academia who are navigating these challenges every day.

Chitra 0:31

Yeah. And um we went through some deep dives into data and architecture, uh, discussions on trust and workspace. And the goal was always to see how the action plan was holding up against reality.

Yevgeniy 0:44

Yeah, we've seen uh where momentum is building and where the community is still running up against a few roadblocks.

Chitra 0:51

Yeah. So as we wrap things up today, let's look at the big picture. You've been looking back at the series, what's the one thing that really stays with you?

Yevgeniy 0:59

I think the one thing that, you know, really stood out to me is the caution that we've heard from some of our uh uh guests. You know, one of them mentioned, you know, before the technology destroys the world, those were those kind of a powerful, uh, powerful statement. And then, you know, how do we even work with it? How do we make sure that we harness this technology in a way that we as people can can work with? And that that those two things kind of stood out to me. Um, because, you know, as an early adopter of of this tech, I'm like, yeah, let's use it. Let's I see the potential, but you sort of get that, okay, maybe we need a little bit of caution too as we we move forward.

Chitra 1:45

Yeah, no, I agree. And I I do think that probably, yes, um Mark's words on let's not destroy the world is probably the one we uh leave with. But to me, I kind of look at that as like less doomsday and more like, hey, yeah, we all think it's like quite capable. Like it is, we are definitely on the verge of something. And it's no longer like this discussion on can it do enough of the job? It's it can do enough of the job. And let's try to make sure we are doing the right smart thing so that it doesn't take us down a path that we don't want to go down. But I think it's great, right? And I'm in the same boat. Early adopter, and let's make sure we don't like set the wrong tone and uh do something that we're gonna regret later.

Yevgeniy 2:33

Yeah, and uh going forward and doing it uh, you know, responsibly. So looking a little bit uh uh on on where things are, you know, where are things breaking down, right? So looking at, you know, we've got these ambitions and you know, we need to execute. The technology is capable. Where are the barriers? What are the what is what struck with you about um uh things that we've heard?

Chitra 2:58

I think um across the board, it felt like maybe a lot of the barriers felt more cultural than anything else. Um also kind of murky, maybe related to guidance, right? So I think where people are like, I'm really not sure. Do we do it, do we not do it? Um I I think this particular group of folks are all inclined to to do the things and figure out how do we get there smartly. But I could imagine with those same reservations, how if you were less familiar than maybe all of us were and less inclined, it would, it would make you feel a little, a little nervous, right? I mean, there's there's a lot of gray areas to navigate.

Yevgeniy 3:36

Yeah. And and and I think that part of that gray area could be even what kind of AI um we should we should be using. So we mentioned, I think at the beginning of this series, we talked about that the big change is about these large models, large language models, multimodal models, foundation models. But there's still that good old-fashioned AI. And I think there's maybe some confusion about, you know, when should I be using what? Um that could be a little bit of a of a barrier too. Like, do we all need to use a large language model for everything?

Chitra 4:08

Aaron Powell Yeah. And I think we did hear some really good examples of how people were pairing more classic algorithms with AI and um achieving some good mission outcome. I think I was also very pleasantly surprised at um how many people have like legitimately deployed things. Um so that gives me some confidence that um our ability to move out on the action plan and you know do more while we're trying to navigate the right ways to do it is is probably a good approach, right?

Yevgeniy 4:40

Yeah, yeah. No, that's great.

Chitra 4:43

Um I think the other thing that I felt like was on the risk side, it seemed like a lot of folks, you know, whether there was a lack of clarity or not, it it didn't feel like there was an emphasis on the lack of clarity. It it seemed like we were actually all talking collectively about how do we like solve the problems and address the gaps as we go forward. Did you get the same feeling from talking to folks?

Yevgeniy 5:07

Well, I think to to me there was uh one thing that struck me is you know, we need really good terms when we talk about these systems, the vocabulary. I think a couple of folks mentioned that maybe the vocabulary is still a little bit of a um a challenge. Like I think I mentioned it like, okay, what do we mean by AI? What kind of AI? What are we using? And then the other part that I wanted to kind of bring up and that we talked about was data. And do we have the right data to you know, do we have the right labeled data? Do we have clean data to be able to really feed into these systems? Because ultimately, like sometimes they're only as good as the the information that they're getting. So I think those are still things that we need to work on is, you know, let's get a clear vocabulary about these systems, and then let's talk about how we feed in good quality data into these systems and gather that up.

Chitra 6:02

And I like that you're referencing the systems because I think that became a big piece of the language conversation too, right? So what constitutes the systems and how do we think about testing against them? And um basically what are those bounding boxes? I think um we might all have a different notion of it. Um, but you're right, the the vocabulary, I think, was something that we all dis discussed and decided we needed we needed to come together on a little more clearly. So what do you think um you took away from when the speakers were talking about like the future and what happens next and how how we kind of think about implementation and the good and bad, I guess?

Yevgeniy 6:44

Well, I think I I think I I come back to the human element. I think we need to do a little bit of learning. Um, we need to get a little bit more savvy about how these systems work so that we understand, you know, what to expect from them and you know, to reduce that sort of automation bias and over-reliance. I think that's stuck with me from our conversations with Missy about how, you know, when she was talking about how you know AI might be over relied upon, and then you're you're not able to adequately detect when it's making a mistake or where where when something like that happens. So I think we just need to make sure as a workforce we're ready for it and that we understand the tool that we're now being um asked to use.

Chitra 7:28

Yeah, no, I like that. Um I think similarly, we have to think about the scenarios and the experiences and learning that everybody brings to bear. I I think in particular, I remember several of the anecdotes that Ian had that were a little more tangible. I feel like I feel like sometimes when we think national security, people go down this killbot um, you know, thought process. And there were so many examples, good and bad, on how we think about those things we have to be cautionary about and those steps we can take as we implement in the future. And I think they were um awesome examples that allow me to kind of paint a picture in the back of my head, right?

Yevgeniy 8:06

Yeah, yeah. No, I yeah.

Chitra 8:08

What do you think the the kind of government and policy decisions and things like that? What what what should we think about as we like look at like government leadership and what we would recommend them to think about based on the industry perspective or former government folks perspectives?

Yevgeniy 8:23

Well, I think I think it is going to be really critical for, you know, as we uh as as an industry, we need you know good methods for understanding what our requirements are. So we're putting together these systems. It's all very new, it's not like a typical software system. Um we need really good requirements for what are these things that we need to be responsible for. Uh what are the things, what are the guardrails that we need to set, and how do we know that our system is operating at the level that that that can be relied upon. So I think where the government and the public sector can really help is in understanding how to evaluate the quality of our systems, how to help us understand when or what those requirements might be. Um I think that could help everybody develop a uh a better product.

Chitra 9:17

Yeah, and I think that kind of goes along the lines of what I think Sean mentioned about there are certain things that probably the government's better equipped to do than industry, and how do we figure out where that split is um so we make sure we're working this together? Because there's certain there's certain things that we're just not going to be able to do with the same effect. So I think that's definitely spot on. What did you think about um Chip's kind of comments on kind of the um the PRC position and the adversarial threats?

Yevgeniy 9:48

Aaron Powell Well, I yeah, no, I mean we're not gonna be the only ones using AI. Yeah. Um and where I find it to be a little bit scary is when we talk about open source software, um, that's been a really arguably a good development from the perspective of security, because things, security flaws that are detected could be identified in the code and then patched. And then we have these robust CVE programs for uh you know, understanding where the vulnerabilities are, and then they can be patched. With respect to AI, I don't know that we have that yet. And there is a lot of AI available in the open source, uh, but it's very difficult to know whether or not that AI is trustworthy because we don't know the provenance of the data that it was developed on sometimes. And it's hard to uh look at the model weights. So in terms of like how our adversaries might um taint the systems that we rely on, we really need to focus on that model provenance and know exactly where things are coming from. So, yeah, no, absolutely, those are big concerns. And the other part of the equation is standards. Like if we as the US don't shape AI standards, then our adversaries will. And systems might be engineered to fit those standards instead of the ones that we set as a country. So I think it's really important for us to have these leadership in that standards community.

Chitra 11:16

Yeah, I agree. And I think um the more and more we see what what's happening in industry, the more we realize each of us have the responsibility to understand what's happening under the hoods.

SPEAKER_02 11:26

Yeah.

Chitra 11:27

Um, and do a little bit of extra um uh assessment of what model and and maybe what model variation and what what was it derived from, right? So I think going through that sourcing and pedigree uh becomes an important part of the software supply chain, especially as it relates to how quickly our AI tools um can can become like culturally dependent um kind of assistance in our lives. Like even for me, I feel like I use all these things all the time to a point where it's easy to weigh the risks and say, well, but I'm getting so much lift off of it, so maybe I don't have the time to investigate. But it we we do have to take the time and the responsibility because so much is happening and the threats are real.

Yevgeniy 12:16

Yeah. And I was wondering, Chitra, what you thought about, you know, when we had Sean Batir on and talking about the experience from Project Maven and and and you know, d that was a really important development. Um so how do we like accelerate the adoption and deployment of these tools in operations and what should be, you know, what's the winning recipe that we've learned?

Chitra 12:39

Yeah, and I think um the way he described it, like they they had um really strong partners on the government side that allowed them to do those experimentations with Maven, right? So I think we have to find those opportunities, not expect a one-size-fits-all across the board, but find those right entities where we can figure out like how does this work and what does it mean to operation deploy and and how do we think about the testing? I think nowadays there's a lot more emphasis on guardrails and guardrail development. Um and so I think we have to we have to continuously evaluate these things and say, what's the next big piece that becomes the sticky part that we have to experiment with to figure out like, is there um is there some issue that could prevent us from operationalizing the capability? And I don't I don't know that there's gonna be an end to that because as these AIs advance, um I'm sure that the uh the threat space rapidly advances as well. And I think that's what we're seeing with the agent models as well, right? Um I am, I will say, generally again, pleasantly surprised to see how much is happening across the community. I think in a previous time, I wouldn't have expected the government to be um as closely caught up, I think, to at least enterprise customers in industry. So maybe we're not bleeding edge in government um in comparison to maybe the Silicon Valley tech companies that are building the foundation models. But I think we're not as far behind when it comes to like large enterprise.

SPEAKER_02 14:19

Yeah.

Chitra 14:19

What's your thought? Like do you feel like we're way behind, or do you think that the action plan is creating the right call to action?

Yevgeniy 14:27

I I think the action plan is creating the right call to action. I I think um as a as a deployer of systems, the government often has very esoteric use cases, but they also have use cases that require really solid quality assurance because you know the government has a level operates at a level of transparency that that that sometimes is not there in other domains. So when a mistake happens, it can also affect people in a very different way. If I can't log into my Google account one day, it it inconveniences me. But if I get detained at an airport and miss my flight, you know, that's a different level of impact. So sometimes the government has a higher standard that they need to meet. And because of that, I think our applications of these AI systems are generally done very um, very, very thoughtfully. Um what I've experienced in the in the national security domains that I've uh uh uh worked in is that the government does a tremendous uh job of both assessing these um you know workflows that they're integrating these systems into and ensuring that they're providing this sort of like you know quantifiable quality. Um in that sense, I don't know that the government is behind. Um I think I think that's quite good. But as a community with these new technologies, you know, I would say that we need to, like as not on the government side necessarily, but as providers of technologies, we kind of need to do a good job of like sharing uh the knowledge that we might have about these use cases, because they're also new. And so I I think right now, like a lot of folks think that, you know, they're hesitant to share. But I think that as a community, especially as we do uh more of these committee meetings in INSA in the AI subcommittee, I'm hoping that folks can start sharing their use cases and their experiences operationalizing these technologies for national security missions.

Chitra 16:23

Yeah, I I couldn't agree more. And I think it's gonna be interesting. Like we um we've had a lot of conversation about key things like privacy and trust and responsibility. Um and all these things become interesting related to like we're we're used to working in stovepipes, and the way that we protect and manage data in these stovepipes is you don't get the data out, right? And I think the um agent frameworks that we're all moving towards, we still protect a lot of these basic things related to authorities. But it's possible that data is still getting commingled and utilized to inform analyses and still protect the sharing, but then by virtue of that, is the the risk related to exposure? Like did you didn't technically share the PII or you didn't share but but you might have made an informed decision as a result of, and did that trip the intent of the sharing issues we have currently with PI? So I think there's a lot to unpack and think about as we look at those privacy considerations uh as we're implementing for the government. But I think those are going to be interesting and good problems for us to resolve.

Yevgeniy 17:43

Aaron Powell Yeah. And and I I'm sort of I was approaching sharing from a sort of a lessons learned perspective. I think there's there's a lot of opportunities right now uh in that regard.

Chitra 17:53

Aaron Ross Powell No, I 100% agree. And I think every time somebody is doing something experimentally, um I'm fascinated at an alternative use case or perspective on that same piece of learning um that we're all able to learn from like rapidly, right? Our our own, I think, LMs in our head are uh advancing at a rapid pace. So it's it is quite good. And uh again, like I could probably, with any one of the speakers we had, go through like a whole day of anecdotes because they seem like they had like a treasure trove of uh, you know, hands-on experience that would be a wealth of knowledge. And hopefully we can have more of these conversations, even with those folks or with others to share with the broader insta community.

Yevgeniy 18:38

Well, Chitra, this has been a great series and a lot of really strong perspectives and honestly a lot to think about.

Chitra 18:44

No, I agree. And what's clear to me is that this path from strategy to execution isn't simple. Uh, but the work is happening and it's happening in a lot of places, and these conversations are a big part of what's moving forward.

Yevgeniy 18:57

Yeah, the A Action Plan sets the direction for us, but it's going to take a kind of sustained effort across government, industry, and academia to deliver at mission speed.

Chitra 19:08

Um, to everyone who joined us throughout the series and to the subcommittee members who were part of this and gave us their thoughts along the way. Um, and to our speakers, of course, thank you very much for being part of this conversation.

Yevgeniy 19:21

And a big thank you to all our guests for sharing your insights uh and helping us break down uh what it really takes to move forward from action plan to mission impact.

Chitra 19:31

Perfect. And if you're interested in continuing the conversation, we encourage you to get involved with INSA's AI subcommittee.

Yevgeniy 19:39

Yes. Uh this has been an the all source podcast from action plan to mission impact.

Chitra 19:46

Perfect. Thank you all for listening.