From Action Plan to Mission Impact: The Human Side of Mission Data

In this episode of INSA’s All Source Podcast series, From Action Plan to Mission Impact, hosts Chitra Sivanandam and Yevgeniy Sirotin sit down with Dr. Ian McCulloh to examine what it takes to transform raw mission data into operational advantage. Drawing on decades of experience across military operations, AI, and data science, Dr. McCulloh explores why data quality, integrity, and pragmatic mission application are essential to successful AI adoption. The conversation breaks down how leaders can better align technologists, domain expertise, and real-world mission needs to operationalize America’s AI Action Plan and turn data into a true strategic asset. 

Transcript

Chitra Sivanandam 0:05

All right. Welcome to another episode of INSA's All Source Podcast. My name is Chitra Savanandam.

Yevgeniy Sirotin 0:11

And I'm Yevgeny Sorotin. And today we're going to continue our discussion of data or bust. America's AI Action Plan makes a clear argument. The race for AI leadership is also a race for data leadership. And the plan calls for high-quality data, a national strategic asset, and emphasizes building large-scale, AI-ready data sets, improving standards for data quality, and expanding secure access to federal data. Trevor Burrus, Jr.

Chitra Sivanandam 0:38

The plan also links data directly to operational AI adoption through evaluation frameworks, AI test beds, and secure compute environments capable of supporting national security workloads.

Yevgeniy Sirotin 0:50

Today we're going to be exploring what it takes to move from raw mission data to repeatable operational advantages. And our guest today is Dr. Ian McCullough, who's an associate professor at Johns Hopkins University and a former leader of Accenture's Federal AI practice. He also served more than 20 years in the U.S. Army, retiring as a lieutenant colonel. Ian, welcome to the podcast. Hey, thanks. It's good to be here. So, Ian, I was wondering if we can get started. You have a really diverse background that spans public service and the U.S. Army, as well as AI leadership roles in industry and in academia. So, how does your background inform your perspectives around data and AI?

Dr. Ian McCulloh 1:31

Well, it's kind of interesting. You know, I I've been very fortunate to be kind of throughout my career involved at this intersection of data and operations. My first day as a second lieutenant in my first unit was the days to Ecom dropped off the first tactical computers for battalion and below. And so I was working on the aviation mission planning system. And, you know, back then it was uh, you know, there was like no UI on the computer interface. You know, all the people are like, this is stupid. We would never have computers in a tactical environment because how do we have batteries to support them, right? There were all these kinds of concerns. This was in the in the mid-90s. And uh, you know, I mean, some of the some of the early innovations were things like the sit stat, which was the precursor to Blue Force tracker or creating threat templates. And then, you know, you you fast forward into the early days in Iraq, you know, there were issues of integrating command posts of the future, which relied on very well-structured data that was very well validated. And uh some people might remember the Tiger system, which was kind of like, you know, a little internet cafe on the file where people just kind of free texted whatever they thought about their mission. And of course, we found that people, when they had that flexibility, would use it more and put more great information in there. And people didn't like formatting data in a structured way in CPOC. And so then people tried to put the structure on Tiger and people stopped using it. So, you know, some of the work I was doing there was using AI and machine learning to see if can we do natural language trans uh processing on the free text of what people are saying and populate uh command posts of the future for enterprise use. I was also doing work with like universal translators to you know translate from Arabic into English, you know, for people. Um, and and then really uh kind of I I guess my my big thing in my career was uh network science based targeting and trying to advance more, you know, AI graph analytics for targeting. And my biggest frustration during that entire time was trying to get leaders to understand and trust data-driven intelligence.

Chitra Sivanandam 3:35

I think going to- nothing really has changed, I think, Ian, is what you're saying.

Dr. Ian McCulloh 3:40

Well, you know, I I feel like um I was focused so much on the mission, but you know, obviously I didn't create peace in the Middle East during my time in uniform. But um, you know, I felt at that time I was like an army of one trying to trying to use AI and data. Uh now I see hundreds of leaders that are far smarter than me, uh, far more engaged than me, and um, and are making a difference. But, you know, I'm sure they, like I was, are are still very frustrated. There's this, you know, leader to technologist communication. And, you know, I think leaders try to struggle to figure out, you know, who these guys are and and how to like, you know, use this, right? It's it's data is a combat multiplier, and just like any, you know, we we do combined arms and joint joint operations, but but we need to also add this uh element of AI and beta into it too, to kind of advance insights and and that's still a problem.

Chitra Sivanandam 4:40

I love that. Um it it's it's kind of making me go back and and think about this stuff in the context of like I feel like it has been a many, many decade-long journey on creating structure out of unstructured stuff. Um and so that kind of brings me to one of the the questions we were we had around there's this construct of like data as a national strategic asset. Um what does that mean and what makes data AI ready and what what does it mean to be a strategic asset?

Dr. Ian McCulloh 5:10

So I I think uh data is a strategic asset. I mean, it probably depends on who you ask, but I think the important thing to think about data is the importance of uh of data consistency and of how we consolidate data. So I I would tell you, in my experience, I have I have I have never really gotten double-digit performance boosts on playing with the algorithm or getting a smarter, cleverer algorithm. I almost always, if I think back on it, it's always data consistency, data labels, right? And how do we get that, how do we get that consistently labeled? And and that's where the the gains are. And um, when I was at Central Command, uh I ran information operations strategy, and there was a large program to combat the digital caliphate at that time, right? This is in the, I don't know, 2080, 11, 2012 time frame, somewhere in there. And so, you know, part of that was assessing the internet for you know forums and websites. Are they pro-Al Qaeda, pro-American, basically, right? And you know, the internet's big, right? So when I'm looking at how much they're covering on this, right, it's about four million dollars in proportional labor. I'm not saying they were completely doing that, but like of the people that were engaging, a certain percentage of their time was assessing the internet. And uh, you know, that that was a proportional cost of about four million a year. And I'm like, I can make an AI classifier to do that. So I made my classifier and it was uh 64% accurate. And everybody's like, and that sucks. That's like flipping a coin. I'm like, it's double digit, better than flipping a coin, thank you. But how often do the people agree? You know, they didn't know. They wouldn't, they wouldn't stop and pause to have two people looking at the same data point to see agreement. It wasn't until I brought it up, um, you know, much to the the irritation of my boss in front of some Cream Gradual Stanfords that they forced CentCum to pause the program, do an interannotated agreement study, and you know what they found? They found that the people agreed 68% of the time.

Chitra Sivanandam 7:02

It's as good as people.

Dr. Ian McCulloh 7:04

So you're you're spending $4 million a year to get a 4% marginal improvement over what an AI classifier could do while completely not addressing or be not even being aware of the fact that 32% of the time people can't agree on whether a website form is pro-Al Qaeda or pro-America. And the decision is not as important as the fact that a decision is made. And leaders don't know their role in engaging in that process, and it's not up to the technologists to decide. Now, if we fast forward to something that's maybe a little more close and dear to people's heart, is the basically popularity of deep learning began in about 2012. By 2015, computer vision AI is better at doing image classification and object detection than human beings. So General Shanahan and Odie and I at the time decided, hey, why don't we start a project of adopting AI for you know image classification, object detection? We we know this as Project Naven, I'm sure many of your listeners are familiar with it. So, you know, I was involved in the early stages of that, and and when we were kind of bringing people together, I used that example of the digital caliphate problem to say it's important that we have multiple analysts looking at the same images. So have we learned the importance of data labeling, data consistency, and its impact on our AI programs?

Yevgeniy Sirotin 8:23

Yeah, so that's I think that's really important, and that's a good point. I think you raised two really important points, if I can highlight them, is one is the need to have really solid quality data set. You know, not all data is gonna be a strategic asset. If you have messy, poorly uh ground truth data, that's gonna be an issue. And then the other point that you brought up is when we run these AI models with these AI systems, even though they're becoming even more capable, sometimes, you know, what is the right benchmark to run them against? And and I think you raised the issue of the right benchmark being sort of human performance and human consistency um on these data sets. Would you say that that's right?

Dr. Ian McCulloh 9:04

Uh it is. It is. And and that's not just a technology program problem. Do you know what I mean? Like um, I I would say, you know, I mean, I think you you talked to me about this before the podcast, right? And I think this is kind of a well-known statistic that, you know, 80% of AI projects fail. But they fail for three very common and specific reasons, right? It's the communication between the domain leader and the technologist, it's the the pragmatic use case, and it's you know, wrong data. But you know, if I may talk a little bit about the the leader engagement and communication, right? Um, your data scientists out there in the forest, right, in the field and in IC or whatever, can become not just frustrated, but you know, if they're anything like me, they can become a little bit arrogant and narcissistic in their in their knowledge of everything. But, you know, I I think back to when I was at at SOC Cent, Special Operations Command Central. So I had had success in in uh in Iraq. Uh General Cleveland brought me to SOC Cent to establish a strategy and assessments branch and kind of drive some of this data-driven uh intel. And I was frustrated that nobody on in the J2 was really listening to me, right? And so as I'm sitting there, you know, General Tobo, uh, he was the he was the two-star commander of SOCSEN at the time, and he he likes walked by and he says, Hey, I want to report on what you're working on. So I wrote this report for him, I handed it to him. I have to go in and brief the senior leaders. And to give you an idea of how absurd this report I'd given him was, right? Like when I went in, he says, Ian, why don't you start by explaining what multi-kill learning erity is and why it's important to counter terrorists, right? And I just say that not because it is, right? Like you can imagine how absurd this report I gave a two-star general was, right? Now, at this point in time, I was uh a lieutenant colonel, I had a PhD from Carnegie Mellon and AI. I had been a special forces team leader, I had had a second command, I had deployed to Iraq and Afghanistan, right? I had gone and done sorts operations, right? Like I was fully a domain expert in special operations and a PhD. I should have been the bridge. And I completely failed in this mission. But General Tobo, to his credit, relied on his engineering degree from West Point some 30 years earlier to read my paper and figure out what I was trying to say. And then after he asked me that stupid question, right, then he began to explain my paper to the other senior leaders at the table. And once he did, then they started coming up with strategy and operations and planning for how we were going to conduct special operations in the Middle East that have humbled me to this day. I mean, this was like 15 something, 15 or so years ago. I still use it as an example. Do you want to be the arrogant and narcissistic Ian McCullough that's frustrated? Or do you want to be the like engaged leader that Ken Tovo was to actually lead his organization to success? And it's not always on the technologist as much as it's on the leader.

Chitra Sivanandam 12:05

I love this story. And like I'm curious, um, how much did he um kind of extrapolate in what you wrote versus what you thought you were doing with the data? Like, did he did he overfit it and actually like do something different with what you had actually like engineered beyond what you envisioned?

Dr. Ian McCulloh 12:24

Yeah, uh he did. And you know, you know, it was interesting. Like I remember this one, I knew I he had this one quote, right, that I think is okay to say on this forum. But he said, you know, he goes, Man, I knew Lebanon was important. Now I have a big dot to prove it, right? Like I think that like, you know, there was a certain amount of data that was confirming what he and other senior leaders already know. And that confirmation is important, right? Because it, you know, the people read a lot and they get an intuition, but actually being able to have something to back it up and confirm it is important. Uh, but then there are other insights, right, as far as where are central nodes for, you know, various types of trafficking are important, you know, or I mean, if if we even go back to more of a tactical example, because I think this is, you know, I I gotta be careful of you know what what's it's probably 15 years old now, but I that doesn't mean that it's declassified. Um, but you know, when I when I think about you know threat networks, right? We found that um, you know, in Iraq, financial brokerage was a huge piece in threat networks in Iraq, not as much in Afghanistan. In Afghanistan, it was education and knowledge brokerage, right? Like, right? And so the differences are important because people will easily take their last deployment and whatever worked there and just assume it works everywhere, and they're not going to actually use the data to kind of see, well, what are the insights that work in this time, this mission right now? And and I think that can be important.

Chitra Sivanandam 13:54

If we can extrapolate on that a little, like I feel like on the mission use case, the um the strategic net assessment type of thing that you went and did, that type of data is very different than the tactical mission side. So, how do you think about the the data and the the mission fit related to the data as we kind of go down this, and especially as we put more AI into the picture?

Dr. Ian McCulloh 14:16

Well, I I think of the data as separate from the algorithm, right? A little bit. So, you know, for me uh in my path, right? Initially when I was at West Point, you know, we were all feeling guilty in in you know 2003, four, right, when when there's a war on and we're at West Point. And so, like uh, you know, I had a cadet that was taking open source videos of roadside bombs, and we were able to identify forensic clues, and we would use network science, right, or graph analytics is the academic term for it, to to identify forensic clues that would lead to actionable, targetable information. And back then, we were literally like calling our friends that were forward deployed on like cell phones or using Gmail. I know that's like, you know, scary now, but like that's what we were doing right back then. Um and we were giving them actionable information of like find this license plate tied to this truck and this video, right? And that had a huge impact in uh ID and sniper activity at the time. Um, so that's open source video in that, you know, fast forward, right? I I got sent to uh show some folks in the IC, uh various IC agencies, how to do this kind of network-based targeting. They had access to large volumes of data. Uh fast forward a little bit further. I'm in Iraq and we're taking forensic data that we have people actually going, like EOD teams, taking weapons technical intelligence off a roadside bond to kind of populate these models. Right. So the data can be very different across those three examples, right? And then at the strategic level, we're actually taking, you know, we're mining and doing natural language processing on Intel reports, you know, IRs, and we called it the PIT at SOCOM, but it was basically a repository, the Library of National Intelligence of, you know, and that was back in a time when they didn't have good controls over need to know. So you could basically mine everything in Intel and all of that stuff, and you could figure out where the threat patterns were. And uh, and so that's at like a strategic level. So many of the algorithms are the same and the you know maybe different in data volume, but um, but those data sets and the mission uses uh can be very different.

Yevgeniy Sirotin 16:21

So, Ian, I I'd like to pull on this a couple of top things that you've raised and and were really interesting to me. So, so going back to your anecdote of your, you know, interpreting your report. Um, so you've you were doing, and I think the at the time, you know, we were using sort of AI systems that needed to be, you know, carefully composed by analysts and then, you know, or data scientists, AI scientists put together and then communicated as a report to leadership, and all of those concepts needed to be explained. And now we're sort of moving into, I feel like with modern AI systems, especially those that can produce natural language, they can almost generate some of that content themselves. And so senior leaders can now use these AI systems to ask them questions directly. How do you feel like that'll change the paradigm and sort of the uh buy-in that senior leaders now will have in terms of being sort of data more data driven or less data driven when they are able to interact with these systems directly themselves? Do you think that'll change things?

Dr. Ian McCulloh 17:28

Well, I so I think the biggest thing that that I've kind of I've I've really been happy to see is um, you know, at Johns Hopkins, we um we do executive ed and professional ed. And so we've had the privilege, you know, um I haven't taught in these courses myself, but like, you know, my team at Hopkins has been able to kind of do a lot of executive ed across the Department of War for senior leaders, you know, general officers, flag officers. And uh, and I think that's made them better consumers and and better equipped to understand how they can integrate that, right? It's it's helping equip the kind of modern-day General Tovo, if you will, with that education. And I think that investment is huge. I think possibly, in my opinion, one of the best things the CDAO in the Department of War has done has been to educate senior leaders, but I but I think those educational programs are essential because it's not going to be the Silicon Valley guy that puts on a uniform and comes into the force that's going to change things, right? You already have just as brilliant technical leaders across the force. Um I think, you know, I think even when we look at um cyber leadership, right? Um, you know, a number of years ago, you know, like when I went to uh when I went to get my PhD, right, I I have to, I was having success applying uh AI methods to um to to targeting, right? And so this is from West Point, even, right? I had a, I called it a pseudo-skip. We could work at the secret level. And uh, you know, I would work targeting sets and we would pass them forward to Iraq and Afghanistan. I had a number of cadets cleared doing the same thing. And we were finding that these were making a measurable impact on the war on terror. So, you know, myself and a few others decided that we wanted to continue doing this work. There was no slot for data scientists or AI guy in the army at that time. There was no skill identifiers at that time. So when I uh when I went to PhD, my uh get my PhD at Carnegie Mellon, um, the projects I was working on was actually funded and sponsored by SOCOM. But I had to sign a letter saying I would never make lieutenant colonel if I went or get promoted again if I went to get a PhD. And myself and several other people decided, you know what, it's more important to have mission impact than our personal careers advance, right? Um, you know, another guy that copied my pack and did the same thing was uh now Lieutenant General Paul Stanton. Now, when when Paul was working for Army Cyber for General Cardone, right? Uh, you know, following his uh PhD, General Cardone engaged with Paul and saw the value of what he was doing and gave Guidas the promotion board that he needed for the cyber branch to promote PhDs over battalion commanders. And that was a very controversial decision. But what it did is it enabled a whole generation of amazing officers. Now, in doing that and then allowing them to get back into the cycle, you had this whole generation of really talented and educated uh leaders. And so I think now we we look at the education, people that go and get these educations and AI or these technical fields as non-warriors. So, like when you when you look at that, there's this viewpoint that the education is not a combat multiplier. And I think leaders can forget that the only army uh educational experience that has a higher attrition rate than Ranger School is fully planted PhD program. And I think that people that go to STEM PhD programs that are technical, all of us that I know of, uh, I can't tell you how relaxing by comparison my deployments to Iraq and Afghanistan were, right? Like that was that was just so restful in comparison. But like people don't recognize that. And then I think they don't, they don't see the value of how to integrate those in. And I feel like we run the risk today of leaders that that haven't done that missing it. Like people that haven't gone to ranger school will still respect the ranger. People that don't go to the Q course will still respect the long tab. But when it comes to higher education, that's important in the AI world. Um, people don't necessarily respect the work and the effort that went into that. And then I think that hurts their ability to integrate them into the fight.

Yevgeniy Sirotin 21:50

Well, no, definitely, definitely good, good points. So I want to get us into talking a little bit about how we leverage data for. for for for these for these operational challenges. And I wanted to keep pulling on this notion of the current generation of AI systems, you know, so-called foundation models, really shifting this paradigm, right, about how we're approaching AI. Used to be, right, back in the day, you would have a data set, you would have a smaller AI system that could be trained, you know, on that data set, you'd do some machine learning on a particular data set. Now we're trying, you know, retraining a full model for like a foundation model is could be really out of the question because this thing is trained on the full corpus of human language, scrape scraped off the internet, you're not going to be able to fit all those parameters in any reasonable amount of time. So how do you see sort of the impact to national security organizations and being able to leverage their data, right? And supposing we have the right experts in place, um being able to re leverage their data in this changing paradigm, you're no longer just going to be training it on your data set. Or do you still see a a you know a role for that? What are the approaches you think are are are are going to be needed?

Dr. Ian McCulloh 23:11

I think the two things that people need to understand if they're not familiar with it is probably mixture of experts and agentic AI orchestration, right? Are the two kind of things that we're in and people probably heard about agentic AI. But you know mixture of experts is uh it's not always the case that a big model is the best thing, right? There's this concept of distillation where you can take a large model and you take the output probabilities, so not just the output, but the output probabilities and you use that to distill it into a much smaller lightweight model. And those can be just as effective at certain tasks. You know, there's um uh there's a course we teach uh it's a public course right uh in applied generative AI and uh you know I I had there's a physician that was funded by the National Institute of Aging for Alzheimer's detection Alzheimer's research. And in Alzheimer's uh you know a physician typical um test you know several hours of tests with physicians probably about 70 ish percent accurate at predicting you know detecting Alzheimer's uh you know there's uh blood tests that's like 52% accurate at detecting Alzheimer's right so he's funded to use AI methods to do this and and his AI approach is about 60% accurate with his colleagues so he decided to as a physician to take our applied generative AI class he learns how to code in Python in that class right he learns uh you know generative AI and and how all of that's working ends up with a little bit of uh introduction to agentic AI at the end and so he goes and he makes a model and it's like 30% accurate and then he recommenders the conversation like we just had about the integration of the domain expert like the Ken Tovo and Ian McCullough you know he's like I want to be like Ken Tovo like people should and so he uh uh he started thinking about how he goes through the process of diagnosing a patient and what the challenges are and as he engages in that degree right even though he didn't know how to code in Python months earlier right he ends up coming up with a smaller model or series of models and so one of them as an example is a test to say you know think of all the words you have you you can that begin with the letter T. And a person will go like I don't know tiger truck tool uh tool tool tool tool uh what else what else what else oh you know Timbuktu right and what you have to parse out of that is you notice how I said tool, tool tool because I was thinking a little bit I said some other words how do you actually parse out how many of the words are T that's something that a foundation model can actually do to parse out the data and then he can design a much smaller model to look at the frequency of words, the how many times they were repeated, you know, like these sorts of things and he thinks through how he runs it and he in a much smaller model. And you know what he ends up with an AI model that has about 86% accuracy at detecting mild cognitive impairment right and so like this is an example of how we actually build AI systems with a domain expert and a leader that works, right? So you're using the LLM for coding the data, but you're using much smaller lightweight model for actually classifying. And so this is kind of these ensemble models or these mixtures of experts is really thinking through the process. And now with the agentic AI framework, right, it's not just about having an AI go off and do it, it's about thinking about what are the steps and can I make these small agents do different tasks. And so now we're doing this in the IC workflow where if you think about a typical Intel analyst in one of the larger IC agencies, like what does their workflow look like? They have access to probably a couple hundred software tools that through those tools give them access to thousands of data sets and they go through with a query and they pull some data they append it to an Excel spreadsheet they go to another tool they append it to an Excel spreadsheet oh the columns aren't exactly right and then they spend hours manipulating the data and then they try and tabulate some simple algorithm because that's all they can do at the end of the day. And then they they they try and make a decision and then guess what? That's thrown away and then they go on to the next problem set and like how can we automate some of those steps right can we automate data query absolutely we can automate data query can we do automated consolidation of data uh that's probably in my mind is the biggest billion dollar multi-billion dollar challenge facing the the entire data enterprise for the IC right now we waste so much money on manual or semi-manual data consolidation instead of doing integrity by code. And it's it's it's it's not I'm not talking about SQL queries and having machine learning that's probabilistic and error pro. I'm talking about good old fashioned formal methods but instead of doing formal methods for cybersecurity we can use formal methods for data integrity. And so what that means is you can be assured mathematically prove that when we pull these two data sets together that the integration is correct, here is the inconsistencies, here are the gaps, and we can focus our time on real problems and you know in doing that we're we're we're automating this huge inefficient labor challenge so we're faster, we have lower costs, less labor, so we don't have the problem of clear labor, and we have higher emission assurance and that as a process I I I only know of one IC agency that is testing it at all right now.

Chitra Sivanandam 28:49

I mean that that's not just I mean maybe there's more I hope there's more right I mean I think I would think what you're bringing up though is super interesting because I think that's what I'm seeing as the lift that we're we're tangibly experiencing with everybody building all these MCPs, right? The the burden that existed before on trying to conform all these APIs, get everything to standardize, I spent you spend 80% of your budget on the data engineering before you can actually do anything with it, right? All of that now it seems like it's shifting because you've got these MCPs that take that pain away and maybe inherently in the way you're describing it Ian there's a possibility of that quality and and trust and reliability fit kind of baked into the MCP.

Dr. Ian McCulloh 29:34

Well I I I I would like to think that it could be but I I'm I'm a little bit more uh uh I'm a little less optimistic on on that right like I uh you go to some of your friends that are in the government and they don't have the ability to access the tech outside of a contract vehicle right because the way contracting works and so because of this spot we have failed to to really adopt commercial best practices that that will work and and moreover you know the the the other challenge is um you know in the in this kind of I don't know quagmire is we we we talk about data mesh but you know really data mesh what it does is is exposes incentive misalignment you know so when I talk about incentive misalignment we have this idea that like okay I think we understand that hey let's say let's say there's some data deal in this is going to be a silly example but let's say there's some data about what's going on in Venezuela, right? It's probably in Spanish there's probably a certain set of vendors that are providing the data there's probably certain kind of telecom industries and and whatever that is going to be set up there, right? So they have their fields, right? Their schema. And then we want to go and say hey what's going on in Iran and so different language different data sources different vendors different schemas and we're like yeah but but doesn't some of the money flow from Iran through Venezuela and like isn't that all connected? Yeah okay so we want to have this enterprise picture but the data sets are off right they're not they're not aligned right so then we're gonna say well let's have enterprise data standards and we're gonna govern this so we're gonna tell the the guys in Venezuela they have to add a certain amount of work and fields and we're gonna tell the guys in in Iran they got to add a certain amount of work and field, right? Even if you gave them the money and you create a policy to do that, does the guy working in Southcom on a Venezuela mission, does he care? It's not his mission. He doesn't really want to spend the time doing it. It's not about the money and the policy direction it's about the mission incentive right he worries about his mission, his people, his immediate task in front of him that he cares about. And so data mesh is making these things exposed. What I'd say is formal methods, you know, category theory, right? What it can do is it can reason over that and it automates that process process. So you do integrity by code. And and it then you can make a better business decision of do I really need this data? Is the guy in Iran going to be informed by this extra data that I think the guys in Veneguilla may or may not be able to provide and if so you can facilitate that conversation. And so it's just it's just um uh we think that there is a technical solution to an organizational problem right or maybe maybe I think that we think that there's an organizational solution to a technical problem is maybe the way I should say it. But we're not adopting integrity by code that would significantly automate these challenges and actually make the data a strategic asset. Instead we're trying to mandate at a labor cost that we can't afford right now and we don't have we don't have enough cleared labor to do it we don't have enough time and resources to do it but we're still pushing it up a hill instead of adopting a uh a you know mature AI that could actually come in and solve these problems.

Yevgeniy Sirotin 33:09

Ian do you think that some of these generative models offer an answer here because you know they can perform some of these data transformation tasks that you're talking about and maybe they wouldn't have the incentive misalignment that you described with human operators. And you know if you let them loose in those different environments maybe they could mine out the the you know some of the data field.

Dr. Ian McCulloh 33:33

It's a clever mistake um and and the reason why I say that is a generative model is inherently a probabilistic model, right? Which means that there is room for error. So um you know I'm not saying that generative AI can't automate like create like you know it can make creativity to some degree or it can't it can automate you know trying out a bunch of different candidate patterns. But what you have to do is you have to have the mathematical proof of correctness right and you can't rely on humans to do that at the scale of data. You see in the 60s when people were coming up with database theory there were two general approaches uh horrible names but there's relational algebra relational calculus right relational algebra is procedural right where you're gonna say hey I'm gonna join these two databases together I'm gonna merge this I'm gonna add this I'm gonna do this and it's you just kind of make it up whatever makes sense and you kind of think about it. It's very procedural relational calculus is you make logical statements about what must be true about data. You know what I mean? Everybody has a social security number they have only one social security number right everybody has an address they might have multiple addresses right like you know like these sorts of things and then with relational calculus then what the algorithms and all of that does is it is it infers data structure and it infers those schemas right from from that and it mathematically proves the correctness of that. Well that's as you can imagine a little more difficult to wrap your head around. So in the 60s when data sets were small you're automating your little store you're automating you know your your little manual process right people didn't adopt uh uh uh the relational calculus approach so SQL grew as the predominant means of moving data around and nobody has ever gone back to say at the scale we are today that is replete with error and inefficient and so nobody's gone back to kind of more more um they're kind of good old-fashioned methods that actually solve some of our modern day problems and I think what you're getting at Yevgeny is the importance of hybrid methods. So let us use these generative models where they're good. They can interpret natural language and and convert them into queries for schemas pretty well right for search. They can um they can reason over a lot of data sets and kind of like bring in editive patterns together but we want to be able to have a formal method to actually prove the correctness or the integrity of the data so that we address that early problem we were talking about about the inconsistency, the messiness, because that's what's going to get our algorithmic performance up. And with the agentic AI orchestration there's no reason why we can't have a large language model agent to take the user request we can use a large language model, foundation model type thing to actually format the query and request the data. But then we should have a formal agent that actually does the data consolidation, tests for consistency and correctness, highlights those issues and probably at that point surfaces it to a user before we finish the process of running it through a probabilistic machine learning model to detect patterns of potential attack or threat.

Chitra Sivanandam 36:44

Yeah I I love the way you're positioning this because then it makes me think that you know let's say um that there's a natural bell curve distribution on how people implement integrity by code maybe it's a small subset that do it really well where you don't even have to think you can trust it. Everyone's kind of somewhere there's a large chunk somewhere in the middle and some people who will just never do it and it's always going to but there might be something weird and unique in a certain place that you need to like use it right. So I think where I'm seeing and I don't know if you agree with me here some of maybe the opportunity has to be moved into more agentic frameworks is that we don't need all the data. The concept before of like more data is always better is gone and it's really how do I discriminate and find that right data and bring it together for my mission knowing that there is this they're not all ever going to be following it uniformly and implementing properly right.

Dr. Ian McCulloh 37:38

That's absolutely true. And and I think what you're you're also getting at is another concept here which is the difference between data accounting and data science right people have this uh mistaken belief that data has to be a hundred percent perfect right and and it's not always right so I mean if you're doing things like invoicing you know subcontractors or dealing with money like that needs to be right. We can't say hey you know we invoiced to about the right amount of money you owe us right we can't we can't say hey we we've we've gotten most of our soldiers home from the combat theater right like like some of these things right you you need to have data perfection on right but when it comes to um you know uh the the the the whereabouts of of you know every bad guy or every you know every weapon in a terrorist organization or whatever like do we really need perfection is that even realistic or hey do we just kind of need to say hey the big cash is probably right here you know the like you know there's there's certain times where what's more important is to sample, conduct a a decision support analytic fast to get inside the adversary's decision cycle and recognize that the time it will take you to have data perfection, the adversary will outmaneuver you and they will be gone, right? So so when is speed and agility more important than perfection in your data and does like when you're dealing with with a lot of the AI and machine learning type things, they're all most of them are probabilistic that we use. And so there is some they're never perfect. There is some room for error. And the other concept that that brings in when people are concerned about that is um you know perfection is not our goal better than human baseline is our goal. Right? What I find is very rarely does anybody measure the human baseline. So the the issue is is if the AI is doing better than people, what do you do? So human bias or human decision making is not just a matter of convenience it actually costs lives in many situations. And I would argue as much as we're talking about the ethics of putting a human in the loop when when is it ethical to take a human out of the loop right? When are we morally obligated to consider AI?

Yevgeniy Sirotin 40:01

Yeah yeah I it it's absolutely right um and you know underpinning all this I I really appreciated the discussion today um you know show talking about the need still for high quality uh data sets the need to use AI perhaps to to to leverage those data sets effectively and and and and that's been really um a a thought provoking discussion so that we can have AI that that that works, that's trustworthy, that's accurate. I think that's been just a terrific discussion. So I I wanted to leave a like a closing question here and and we had a really thoughtful discussion so far but I for the for those AI leaders listening to us uh or people just interested in this what is the number one thing that AI leaders should consider when they're operating when they're operationalizing this America's AI action plan and the data agenda? What is the main thing you think that that they should think about?

Dr. Ian McCulloh 41:04

Well I think that uh I I think the main thing they should think about is the sobering statistic that 80% of AI projects fail. But the good news is they fail for three very specific reasons. We've talked about the communication between the technologist and the domain or industry leader and that that coordination and and don't be the arrogant narcissistic Ian McCullough be the be the engaged leader Ken Tovo right uh in that situation. We've talked about some of the issues with wrong data and it's not about getting all of the data labeling all the data and driving up your cloud storage and and costs or a data mesh solution right there's some solutions there. What we haven't really talked about is the pragmatic use case. And selecting the pragmatic use case really comes down to understanding your current process what are the bottlenecks pain points and and and mission critical areas to to improve uh it is um understanding your human baseline for how how the current manual process is actually measured and most leaders can't do that. And I think that if we look at AI as a way to take out the trash a little bit more and how to be pragmatic, then we're going to be more effective in our AI adoption. And the last thing I would say even the simple thing this the simple task of mapping your process and measuring it is going to give you the roadmap of where to have real achievable ROI increases. It's also the first step in actually developing an agentic AI orchestration layer to automate some of these tasks and you will find that many of the tasks are are really great for people to do not AI. And other tasks are really horrible for people to do and AI is better. And knowing the difference and knowing where to deploy it and being able to measure that makes a huge difference in AI success.

Chitra Sivanandam 42:59

Thank you Ian I think this was awesome I think like appreciate all the time and for everybody out there listening this was uh INSA's all source podcast from Action Plan to Mission Impact.

Yevgeniy Sirotin 43:10

Uh thank you for listening our next guest will be Mark Mansell the former chief AI officer from NGA tune in next time