Why Are We Trusting AI with Intelligence?

In this episode of Unspoken Security, AJ Nash and Faith MacGregor tackle the private sector's sloppy relationship with the word "intelligence" — the gap between raw data, contextual information, and verified, actionable intelligence — and how vendors, circular reporting, and fragmented disciplines like "cyber threat intelligence" have eroded that discipline. From there they turn to AI, weighing real utility against real danger: hallucinations that persist even with vetted sources, analysts pulled toward premature conclusions under time pressure, and the EY finding that roughly 40% of AI-generated conclusions had to be recalled. Nash's sharpest line - if a human analyst behaved this way, you'd fire them - sets up a back half that's cautiously optimistic, pointing to Recorded Future's human-curated, AI-assisted model as a template, while widening the lens to AI in government targeting decisions and the quiet brain drain hollowing out the intelligence profession. 

Transcript

[00:00:00] Faith Macgregor: AI needs to be a tool to help enhance our collection capabilities and our vetting capabilities.

[00:00:53] AJ Nash: Hello, and welcome to another episode of "Unspoken Security." I'm your host, AJ Nash. I spent 19 years in the intelligence community, mostly at NSA, and I've been building and maturing intelligence programs in the private sector for about 10 years now. I'm passionate about intelligence, security, public speaking, mentoring, and teaching, and I hold a master's degree in organizational leadership from Gonzaga University.

[00:01:12] AJ Nash: So I continue to be deeply committed to servant leadership. Now, this podcast brings all these elements together with some incredible guests to have authentic, unfiltered conversations on a wide range of challenging topics. It's not your typical polished podcast. My dogs make the occasional appearance.

[00:01:26] AJ Nash: People argue here. We even swear — at least I certainly do — and that's all okay. I want you to think of this podcast as a conversation you'd overhear at a bar after a long day at one of the larger cybersecurity conferences — the kind of conversations people have when nobody's listening.

[00:01:41] AJ Nash: Now today I have an exciting guest and a good friend of mine, Faith Macgregor. She's an intelligence professional and author currently serving as senior manager for threat intelligence at The Cigna Group. Faith built her intelligence career over 14-plus years in the US Army National Guard, followed by roles in intelligence, governance, and strategy, at US Bank, Atomic Data, Wells Fargo, and Cargill.

[00:01:59] AJ Nash: Now, beyond that, and perhaps more interesting, Faith is an award-winning author of historical fiction and historical fiction fantasy. Anything you want to add to that bio, Faith?

[00:02:09] Faith Macgregor: No, I think you covered it good.

[00:02:12] AJ Nash: Well, we're off to a good start then. I want to jump right in. We have a really interesting discussion today. Both of us have intelligence backgrounds, military backgrounds, and we both live in this modern world where AI has become part of everything.

[00:02:25] AJ Nash: When I left the government for the private sector, my joke was that everything had apparently become intelligence. The label got slapped on everything. Data and information — it was all intelligence — and suddenly everyone was a network intelligence analyst — and security intelligence professionals and business intelligence analysts. I think my dry cleaner was a specialist.

[00:02:43] AJ Nash: And now it seems like AI is the thing. It's just getting shoved into everything — the new hotness. So today I want to talk about the future of intelligence in the age of AI. Before we go too far, I think most of the audience probably has a pretty good understanding of what intelligence is, so we won't belabor the point.

[00:03:00] AJ Nash: But for those who might not, let's at least quickly cover the foundations. Starting with the intelligence cycle — planning and collection, processing, analysis, production and distribution. Can you talk a little bit about that?

[00:03:20] Faith Macgregor: The community will argue, but at its core, in traditional intelligence, information is just data. It's not corroborated, it's not fact-checked — not what we'd call backstopped.

[00:03:43] Faith Macgregor: "How do I know?" Well, someone told me it was true. I want independent sources to verify — usually not the same source. To turn data into intelligence, it has to be vetted, cross-checked, and verified through multiple independent sources.

[00:04:03] Faith Macgregor: So if a human source tells me one thing, I want to see a satellite image that backs it up, and then maybe a signals intercept that triple-corroborates it. Then we can start to say, "Okay, this is now verified information.

[00:04:23] Faith Macgregor: Now we can make a decision off of it." And that's really what intelligence is — information you can act on.

[00:04:34] AJ Nash: The purpose of intelligence is informed decision-making. Data is just raw — a point of context, perhaps. I used to say in technical terms: a list of IP addresses is data.

[00:04:47] AJ Nash: A list of IP addresses with some context attached is information — which is usually what gets sold in feeds as intelligence, but it's really still just information. To get to actual intelligence you have to validate it and say, "This is the list of IP addresses — these are the threat actors associated with it.

[00:05:02] AJ Nash: associated with this actor. These are the actions taken on this infrastructure. These are the malware families and signatures. Here is some context, and maybe even this is what we might expect to see next." That much deeper picture — built through multiple sources.

[00:05:14] AJ Nash: through multiple sources — not just one. And that's been a real struggle in the private sector, partly because it took time to get enough people with deep intelligence training and tradecraft experience to cross over from government.

[00:05:30] AJ Nash: And then there's the marketing. Vendors turned a lot of things into intelligence that aren't, because intelligence sells. It's more expensive if you call it intelligence rather than data. I think that's caused real issues.

[00:05:42] AJ Nash: And then there's the question of standards and tradecraft. In the intelligence community we have rigid structures everyone's trained on — ICD 203, 206, 208, things like that. The private sector doesn't really have that. I know people who've come out of the government — myself included — have tried to push that forward.

[00:06:00] AJ Nash: But what are your thoughts on where we are in terms of tradecraft maturity and consistency of messaging in private sector intelligence?

[00:06:08] Faith Macgregor: There are enough people who come from government or agency backgrounds with solid classical intelligence training. My current team has a good mix of people with that experience and others who came up through the civilian side — civilian agencies, some DIA folks and the like.

[00:06:26] Faith Macgregor: I think the biggest challenge in the private sector is speed. What no one likes to admit is that intelligence is a process, and you can't skip any of it. Sometimes intelligence is slow. In the last part of my government career I worked in human intelligence and counterintelligence.

[00:06:45] Faith Macgregor: HUMINT is slow. No commander wants to hear, "I've been nurturing a source." Or if you're doing a private investigation, "I've been working this and haven't found anything definitive yet."

[00:07:04] Faith Macgregor: And I think that pressure waters down the intelligence in the civilian sector. When people go looking for it, they pull articles — a CrowdStrike post, a tech blog — and if you read them carefully, they're all citing the same source.

[00:07:25] Faith Macgregor: And they cite the IBM Cost of a Data Breach report. Then you read the second source and dig to the fine print and realize — that one is also citing the same IBM report. So they're not independent sources.

[00:07:47] Faith Macgregor: Since everyone's doing their own thing, it's hard to find a second source looking at the same question the same way. Especially in cyber — everyone gives you slightly different statistics. How are they actually measuring it?

[00:08:05] Faith Macgregor: And without seeing the methodology behind it, source validation becomes very difficult — which is exactly what intelligence requires.

[00:08:17] AJ Nash: You made a good point. Circular reporting is something we watch for in the intelligence community — something that looks like five or six sources, but when you dig in, it's all the same source. Bob told Susie, Susie told Mary, Mary told Sally — and it all traces back to one original source.

[00:08:29] AJ Nash: It was just the one rumor. Having other people repeat the same story doesn't validate it. We lose track of that a lot in the private sector because you see blogs, journals, and vendors quoting each others, et cetera.

[00:08:42] AJ Nash: And it becomes: IBM said this, CrowdStrike said this, FireEye said this, Flashpoint said this. But they all said the thing IBM originally said. Unless they were adding their own findings on top of it,

[00:08:54] AJ Nash: There's nothing wrong with quoting sources, of course. It just needs to be understood that you still have one source. You can't say it has more legitimacy because other vendors quoted it — unless they were also involved in the original research.

[00:09:06] AJ Nash: But you do see it happen. A lot of what gets sold as intel feeds, when you dig in, is really data or information — or sometimes just news with some perspective added.

[00:09:20] AJ Nash: People don't always know how to accurately calculate the real value of that. A news story about a threat from a foreign nation is just news. If you have a senior intel professional on your team whose entire career has been dedicated to studying that nation — and they add context based on deep cultural and historical knowledge — you've created something of genuine value. That analysis gets you closer to actual intelligence.

[00:09:51] AJ Nash: But otherwise, if your team is just packaging headlines into one product and sending it to you, that's not intelligence. That's a nicely curated news feed, which is useful — but not intelligence.

[00:10:02] AJ Nash: The difference between data, information, intelligence, and news is something I've spent a lot of time on over the years, and it still ends up being frustrating. I think you and I share that frustration — most of the world doesn't appreciate the distinction, even just in the terminology.

[00:10:15] AJ Nash: Cyber threat intelligence is a term anyone who knows me knows I dislike. I'll use it occasionally because that's where the market has gone, but I don't particularly care for it. It's intelligence. Cyber is a subset of it. Threat is a subset.

[00:10:28] AJ Nash: Cyber threat intelligence has become a normal term, I get that, but to me it's all intelligence — you break it down into its subcomponents. And I know you feel similarly about this. How do you see intelligence as a discipline?

[00:10:44] Faith Macgregor: Yes, we agree completely on this one. Traditionally, if you're not an intelligence professional, intelligence is just a big umbrella term. Within that you have things like OSINT — open-source intelligence — which most people have heard of, HUMINT, human intelligence, counterintelligence.

[00:11:04] Faith Macgregor: But there are lesser-known disciplines like MASINT — measurement and signature intelligence — and ELINT, electronic intelligence, which kind of morphed into modern cyber on the public side. I remember when I was doing my first master's degree in intelligence, I was writing about building an intelligence program,

[00:11:31] Faith Macgregor: they were using the term CYBINT — this was maybe 10 or 12 years ago.

[00:11:38] Faith Macgregor: cyber intelligence, not CTI. It's just another discipline. And really what we need is a higher-level intelligence function within private sector organizations that works across multiple disciplines.

[00:11:55] Faith Macgregor: And we do it. I work in cyber threat intelligence, but do I report on geopolitics sometimes? Of course. Especially when it's relevant to what a threat actor might do. We have people in different parts of the world and I can get information from them — someone saying, "Hey, this happened outside my window this morning."

[00:12:16] Faith Macgregor: That's human intelligence — I'm vetting that source. Or if HR suspects a fraudulent worker, they can bring it to the intelligence team. We can do the technical work under the hood, but we can also run what we'd call an intelligence debrief with that HR person.

[00:12:39] Faith Macgregor: And we could pull details from that person that add context and support the hard data — giving us another source. We just don't do that often because cyber moves so fast, and we're constantly responding to incidents. We don't have 17 intelligence agencies to call on.

[00:13:02] Faith Macgregor: Most cybersecurity programs — not just intel — are lean. Most intel programs are two or three people.

[00:13:12] AJ Nash: Severely underfunded and under-supported. Normally a handful of people doing a little of everything. An intel team that works with an insider threat team — you can call that counterintelligence. It's the same concept.

[00:13:23] AJ Nash: We do signals intelligence. We do OSINT. Most of us are really all-source intel professionals at some level, pivoting from one source to the next to build a holistic picture. Which is another reason why I've always advocated for just saying you have an intelligence team, you know, and the rest are pieces of that puzzle.

[00:13:42] AJ Nash: I don't have to get on that soapbox. Anyone who knows me knows my thinking on having a chief intelligence officer at the C level. We won't relitigate all of that here.

[00:13:51] Faith Macgregor: I do love that idea. I want to go on the record saying the chief intelligence officer concept is probably where we need to be if we want this to mature.

[00:14:01] AJ Nash: I agree, and I know some people are working on that. So that gives us a baseline for anyone just catching up. And for intel professionals, you probably skipped all of that and are now thinking, "Okay, good. Let's move on."

[00:14:11] AJ Nash: If you jumped ahead, good for you. If you needed the baseline, great. Now we can get into where we're really going — the discussion about AI and how it fits into this. A lot has changed,

[00:14:26] AJ Nash: it changes every day, but certainly over the past year or so. ChatGPT launched two or three years ago now. AI has been out for a while, people have used it for varying purposes, and it continues to expand.

[00:14:39] AJ Nash: We have agentic AI, and now we're seeing mass layoffs as people are being replaced — or theoretically replaced — by AI. We'll see how that works out. Let's talk specifically about the intelligence space. Keeping in mind the purpose of intelligence —

[00:14:55] AJ Nash: is informed decision-making — these are critical products we're providing. Leadership makes strategic decisions off of this. Sometimes it's restructuring the entire organization. Sometimes it's how you handle security in the near term or over time. These are high-stakes decisions.

[00:15:11] AJ Nash: What do you see right now as the risks and rewards of this exponential increase in the use of AI?

[00:15:18] Faith Macgregor: Honestly, the final answer is still TBD. Changes are coming in fast. MITRE has rightfully raised some healthy fear about what threat actors can do with AI — and a little healthy fear isn't a bad thing.

[00:15:38] Faith Macgregor: And when it comes to intelligence specifically, as we discussed — sources quoting the same source, like the IBM example — how do we know that AI is using vetted information? How do we know it's reaching conclusions based on actual analysis and not just opinion?

[00:16:02] Faith Macgregor: What sources is the AI plugged into? If you don't have access to classified sources, how do you know that information has been vetted? You ask AI for a brief or a summary of some event — how do you actually know?

[00:16:16] Faith Macgregor: The most important thing we can do in intelligence is validate our sources. And I have used top-of-the-line AI programs from vendors where the summary simply does not match what the underlying sources actually support.

[00:16:39] Faith Macgregor: All the pieces are there, but the AI made a leap — and that pushes the analyst to make a leap too. You're on a tight timeline, so you go with it: "I think this is where it's going," but you don't really know.

[00:16:56] Faith Macgregor: You don't get to say, "Give me six weeks and I'll get back to you." That's never an acceptable answer. You're forced to make snap judgment calls under pressure.

[00:17:17] Faith Macgregor: We've seen examples of this. My favorite to cite is the EY report they had to recall because several of the AI-generated conclusions were incorrect.

[00:17:29] Faith Macgregor: Something like 40% of them. Not the entire report —

[00:17:34] Faith Macgregor: but this is a company with serious funding that has produced solid work before.

[00:17:37] AJ Nash: And they charge a lot for them.

[00:17:39] Faith Macgregor: Exactly. So if something like that can't be trusted, it comes down to trust. We talk about zero trust all the time. Why are we extending unconditional trust to AI?

[00:17:50] Faith Macgregor: AI is not going away. So how do we put controls in place to ensure the information it produces is valid and can actually be trusted? And I don't think the term human-in-the-loop is sufficient anymore.

[00:18:11] Faith Macgregor: Maybe human-in-the-loop works for low-level SOC tasks or logging, but for complex critical thinking — decisions that affect budgets, businesses, and jobs — that's not enough.

[00:18:27] AJ Nash: I think you're exactly right. The tenets of intelligence include timely, accurate, relevant, and complete — there are actually seven. Those are probably the four most critical.

[00:18:35] AJ Nash: As we bring more AI into the process, timeliness seems to be the piece everyone cares most about. It's very fast. But how accurate is it? How relevant is the research? Is it complete? Those are much harder questions to answer.

[00:18:46] AJ Nash: Even with good AI platforms, even with solid prompt engineering, it's always a challenge. It's a challenge even for humans to say, "Do I have everything relevant?"

[00:18:57] AJ Nash: Normally in intelligence we work to an information cutoff — I can't wait forever, so I report what I have. But with AI you don't really know where it stopped. The AI does its thing and comes back — but did it get everything relevant?

[00:19:10] AJ Nash: It almost never tells you. It cuts off at some point and decides that's good enough, and it may not be. There might be more out there. Then you get into the accuracy problem, which you illustrated well.

[00:19:19] AJ Nash: Even if you curate all the sources — say you pull eight validated, vetted products and ask for a summary — there's still a reasonable chance the platform will come up with conclusions that aren't in any of those sources.

[00:19:37] AJ Nash: conclusions that aren't in any of those eight sources. It hallucinates. The platform creators have said they don't completely understand their own technology — they're honest about that. And yet we're pressing forward anyway because we're in a hurry.

[00:19:49] AJ Nash: Everybody's in a hurry. Timeliness seems to matter more than anything else, which is incredibly dangerous when massive decisions get made off these outputs. And that puts enormous pressure on people to cut corners.

[00:20:05] AJ Nash: Humans are designed to be efficient, so after 28 validated products you start thinking on the 29th, "I checked a couple. It's probably fine."

[00:20:18] AJ Nash: And then you validate less, and less, until you're just shipping it out. That's how people operate. And now you're making major corporate decisions based on things you haven't verified.

[00:20:33] AJ Nash: The EY example is a good one. Frankly it's embarrassing for them given what they charge and the reputation they carry. It wasn't one or two errors — as you said, roughly 40% of the product was flawed.

[00:20:46] AJ Nash: That means they offloaded a significant amount to AI. A human in the loop can't catch errors fast enough when it's that widespread, and it went out the door before anyone caught it.

[00:20:59] AJ Nash: People are just pushing forward: AI can do it, we don't need people. And keep in mind — there is no intelligence in AI. It's a brand, it's marketing. It's not intelligent, it's not sentient, it can't reason — at least not yet.

[00:21:16] AJ Nash: If you had an intelligence analyst who occasionally just added fabricated lines to a report — most of it good, but a few lines completely made up — you'd fire that person.

[00:21:29] AJ Nash: They'd have to explain how it happened. AI won't be able to — it'll just say, "I don't know why I did that." You'd fire that person. You can't trust them with products you're using to make major decisions.

[00:21:48] AJ Nash: I don't want this to be all negative. Some genuinely good things are happening with AI, and there are valid ways to use this technology. What do you see as some of the positives?

[00:22:03] AJ Nash: How are people using it to do better work? What are you seeing as positives when it comes to intelligence specifically?

[00:22:11] Faith Macgregor: There is some good news. Back in the day we had a reach-back capability — if you were deployed you could reach back to a unit in the US for information. But to get it you had to draft a formal request,

[00:22:30] Faith Macgregor: "Do you have this information?" Humans would go find it. Then we got better — computers started doing that, and you could write a simple SQL query to pull what you needed.eed. So one way I see this working great is with Recorded Future. They.. When they first did this, they were one of the first intel vendors to adopt like a GPT model. And so what they do is they have their INSIC group of human analysts that curate reports and do all the source validation and stuff like that, classic intel reports, and then they put it into their system, and then you can use their Recorded Future AI to find those reports.

[00:23:09] Faith Macgregor: Now you can do an advanced query that pulls not just data but validated source reports — finished intelligence. You can pull those into your assessments, and I can make that based off of like good conclusions, and then they have their sources too that have been independently verified, hopefully. Now Recorded Future still does like kind of the classic, you can build a report with AI and stuff like everybody's doing that. But, I do think that's like a great use of intelligence, artificial intelligence in the intel sector is, kind of curating reports. We have.. 'Cause we have all this data and sometimes we've talked about analysis paralysis.

[00:23:50] Faith Macgregor: We have too much data — there's no way to go through all of it manually. AI can help sort that. It can help you find the one IP that was masking its origin, verify where a login came from, identify the host country. AI can surface patterns in ways that can actually help us get through some of those ways that threat actors mask where they are.

[00:24:12] AJ Nash: You make a good point. We're swimming in data and information and have to consume so much of it. If you're working in a mature organization, it's even more so. More funding and resources is great, but it also comes with more volume.

[00:24:25] AJ Nash: All these vendors, all this open source, all these collection systems — an overwhelming amount of content coming in. We rarely have enough people to manage it. When AI is tuned properly, and that's the key piece, I've accidentally become something of a prompt engineer.

[00:24:39] AJ Nash: If you work with a technology long enough, you learn how to use it or you struggle. Over time you learn how to ask better questions, and the platforms improve too. One of the things you mentioned was that paralysis — a challenge I've seen often in the intel space. You keep going down rabbit holes, which is why information cutoff exists: to force people to stop researching and start writing.

[00:25:03] AJ Nash: Sometimes just getting to the writing is hard. How do I want to structure this? How do I get started? And I find AI is great for drafts. If nothing else, it gets you started.

[00:25:16] AJ Nash: It's easier to edit than to author from scratch. Assume everything in that draft is flawed — just as a baseline safety assumption — but at least you have a structure to work from.

[00:25:27] AJ Nash: Then you check sources, validate, move things around. AI doesn't do confidence language well, as far as I'm concerned. It'll throw in high confidence, low confidence, but it doesn't understand what that means.

[00:25:38] AJ Nash: It's not applying any tradecraft standard. You'll have to go through and apply those yourself. But summarization — as you mentioned — can be genuinely useful. Drafting can be helpful. If you configure it properly, it can do a lot of valuable groundwork.

[00:25:50] AJ Nash: As you said, it helps to work from a corpus of already-validated content so it's not inventing things. Even with open source, if you put in the time to build the right models and limit what it draws from, you still have to check everything it returns.

[00:26:06] AJ Nash: I've found it genuinely helpful as a research assistant. You can set time parameters, create an information cutoff, give it a defined range. At some point you may just have to accept that you've done what you can.

[00:26:20] AJ Nash: There have been improvements, but for anyone who thinks they can just hand a tasking to AI and get back a finished intel report — I haven't seen that work yet.

[00:26:33] AJ Nash: Maybe you've seen things I haven't. Has anyone found that easy button — where you hand it to AI and make strategic decisions off what comes back?

[00:26:43] Faith Macgregor: Not at all.

[00:26:44] Faith Macgregor: But I will say I've seen people make decisions off a headline, and that's just as problematic. News correlation is a big part of where things break down. My bigger concern is cost. If AI costs ten times more than a human analyst but still requires a human to review everything it produces, would a second analyst be cheaper? Like any new technology, cost has to come down.

[00:27:44] Faith Macgregor: AI needs to get a little more controllable. It's user-friendly, but it needs more parameters around it. And for the AI developers out tThere's a mix here — give us some controls.

[00:28:00] Faith Macgregor: Could I tune my AI for intelligence specifically? Tell it: only use sources validated by two or more independent references. And then enforce that. That kind of intelligence-specific tuning would be genuinely useful.

[00:28:28] AJ Nash: Yeah, I'm not sure

[00:28:29] AJ Nash: There's enough demand yet. Not enough people understand the difference between data, information, and intelligence to drive those requirements. For what it's worth, I've been experimenting with creating collection tiers —

[00:28:46] AJ Nash: trying to teach AI confidence language through source tiering — tier one, tier two, tier three — so that if it has at least three tier-one sources it can assert moderate confidence. Hasn't been fully successful, but at least it's a start.

[00:29:00] AJ Nash: it's a direction worth pushing. If I can say, "These 20 sources are the only ones you can use," then at least I know what's coming in. I still have to verify it didn't hallucinate URLs, but at least I know it's drawing from sources I've already vetted.

[00:29:15] AJ Nash: It narrows the problem. But no, I haven't seen the easy button yet. They'll sell it that way, and people will buy it — AI intelligence, no need for an intel team. For anyone listening: don't do that.

[00:29:26] AJ Nash: Don't make decisions based on that. The over-reliance on AI stems from a belief that it's cheaper than people. It's not, as you said. We need the technology to get cheaper and better — like all technologies do eventually.

[00:29:40] AJ Nash: Before that calculus makes sense. And what we're hearing now is that this technology is about to get significantly more expensive. It's been subsidized so far. Now that it's embedded in corporate environments and people have been offloading work onto it and laying off people to justify the switch, these platforms have created deep dependency — and now the pricing is heading up. Companies are starting to notice.

[00:30:02] AJ Nash: The pricing is going up. People's tokens don't expire. A salaried employee costs the same whether they work 40 hours or 55. AI platforms bill by usage, and that adds up fast.

[00:30:17] AJ Nash: People have a tendency to trust machines that behave like people. And what we're saying is there's a mix There's a mix here —

[00:30:32] AJ Nash: real possibilities, genuine opportunities, but this technology is not what it's being sold as — at least when it comes to intelligence. And if intelligence drives decisions, those decisions need to be based on the best available, most rigorously validated information.

[00:30:51] AJ Nash: Some people are still making decisions off news. A lot of my job working in intel organizations was helping people — specifically the CISO — understand what not to worry about — before even getting to what they should worry about. Most mornings started with explaining why the latest headline wasn't actually relevant to their threat environment.

[00:31:10] AJ Nash: Not everything requires a reaction. Have you experienced that as well?

[00:31:16] Faith Macgregor: A little bit. And we keep coming back to trust — we keep using that word. Trust is foundational. Without it, intelligence fails. But intel people are human, and they're going to make mistakes. They.. I was in Iraq once, and it wasn't me, it was a different unit, and someone made an assessment that someone was gonna attack the base in Kuwait, and highly unlikely to anybody who understood situation But, they saw the information, and they're like, "This could be the thing." Now after that, you make a mistake like that, and the human is gonna be like, "Okay, so the next time I go into this again, I'm gonna quadruple verify my sources, and then I'm gonna ask three friends to validate this, and then I'm gonna call another buddy in Singapore and be like, 'Hey, are you seeing what I'm seeing?'"

[00:32:08] Faith Macgregor: make mistakes. And every time you do, you're going to learn and get better. Your next product improves. AI doesn't work that way. Its programming doesn't change because it got something wrong.ot something wrong. not yet. We're we're not at live learning, whereas a human analyst learns. And also I think someone show this to me and prove that it works, but, can.. aI is sorting related information that makes sense. What about unrelated information? The little reserve.. You remember that story about the reservist who found Saddam Hussein?

[00:32:45] Faith Macgregor: There's a story about an analyst going through reports that trained intelligence professionals had completely ignored — uncorroborated, seemingly unrelated material. But he saw patterns, made connections,

[00:32:58] Faith Macgregor: and acted on them.

[00:32:59] Faith Macgregor: 

[00:33:00] AJ Nash: Right.

[00:33:04] Faith Macgregor: And 9/11 is the clearest example — the information was there, it just wasn't connected and shared. So how does AI help or hurt that kind of pattern recognition?

[00:33:15] Faith Macgregor: Yeah.

[00:33:16] Faith Macgregor: I don't know.

[00:33:20] AJ Nash: That's a really good point. Everyone who has done intel for any period of time has gotten something wrong. That's the nature of the work — we're not perfect, it's not 100%.

[00:33:30] AJ Nash: That's why we have caveat language. Very few people will say high confidence. Nobody I've known says, "This is certain." High confidence still requires a lot of sources and strong corroboration.

[00:33:44] AJ Nash: Confidence language is something we live by. And we've all gotten something wrong. You take it personally if you're an intel professional — a lot of introspective discussion with yourself about, "All right, what do I do differently?

[00:33:59] AJ Nash: "How do I fix this?" Depending on how bad it was, you might get external feedback too. And that shapes you as a human — either as a dedicated professional trying to get better, or sometimes just out of fear — because there are consequences for getting things wrong.

[00:34:13] AJ Nash: fear of getting it wrong again. Even though everyone understands we're not perfect, there are consequences. AI doesn't have any of that. There are no consequences for getting things wrong. Most platforms don't remember one project to the next.

[00:34:27] AJ Nash: So it can make the same mistake again. You can teach it to some extent — memory can be created — but it doesn't work the way human learning does. And so if you're dealing with someone who has 15 or 20 years in intelligence, they carry all those battle scars.

[00:34:45] AJ Nash: an experienced intelligence professional, they carry years of iterative learning — ingrained in ways that can't be captured in a resume. AI doesn't have that. It doesn't understand the consequences of its actions — or the consequences of someone acting on its judgments.

[00:35:01] AJ Nash: Not yet. We may get there, but we're certainly not there now, and it's dangerous to race down that path — because if AI makes a serious mistake that you relied on,

[00:35:17] AJ Nash: given the same inputs, it may well make the exact same error again. What they call hallucinations, I'd call fabrications — where the AI fills gaps

[00:35:31] AJ Nash: with invented content. Humans can do that too, but intelligence professionals specifically don't. We flag gaps: "We don't have this information." Ask any AI platform a question and I have yet to have one say, "I don't know."

[00:35:45] AJ Nash: Humans will say that. An intelligence professional will say, "Best I've got, I honestly can't give you a confident answer either way." AI will always give you a confident answer, and that is disturbing when the stakes are high.

[00:36:03] AJ Nash: I'm worried about where that leads. Where do you think we're going from here? We've talked about where things stand today. What do you see in the next five years or so?

[00:36:23] AJ Nash: And let's also talk about the government space too — we're starting to hear about AI being used in classified work and operational planning. What scenarios do you see playing out?

[00:36:34] Faith Macgregor: Well, I guess we should first address

[00:36:36] AJ Nash: Sure.

[00:36:37] Faith Macgregor: that I haven't independently verified this — it's a news report, not an intelligence report, caveat noted. But there are reports that the government is using AI-based models to assist with target acquisition for kinetic strikes. I remember early in my career — 1-194 Armor, Minnesota — intel folks were always trying to give commanders a confidence percentage. And the commanders would say, "I want to know what you think, S2." You're standing in front of 100 people in a briefing and you say, "Here's what's likely. Here's where I think this could go. I'm not certain, but I have a theory, and I want to put collection assets on it." That's the right answer. Teams need to get comfortable saying, "I need another day," or, "The data just isn't there yet."

[00:38:38] Faith Macgregor: And some of that has to involve knowledge gaps being filled by systems we don't fully understand yet.

[00:38:42] Faith Macgregor: That actually leads into the good part. What could work well is not blanket replacement, not removing the human from the loop. AI needs to be a tool that enhances what we're doing — improving our collection and vetting capabilities.

[00:39:05] Faith Macgregor: collection capabilities, our vetting capabilities. It's like any other technology. The Army had the BAT and HIIDE systems for biometric collection — retinal and fingerprint scans in the field, instant database checks. It was fast. Instead of sending a report back to the States and waiting three to six weeks, you had an answer immediately. Technology as a force multiplier — that's the right frame. Air power is great, smart bombs are great, but at the end you still need the infantry to walk in and take the town. There's no substitute for going in and verifying.

[00:40:12] Faith Macgregor: But it wasn't meant to replace judgment. A scanner doesn't tell you what decision to make — that's still on the operator.

[00:40:24] Faith Macgregor: We need to use AI that way. There will be good applications. We just have to be disciplined about it.

[00:40:28] AJ Nash: You make a good point and it connects to my biggest fear — the speed component. People are in a hurry. You give an intel assessment with moderate confidence based on conflicting reporting, this may or may not be relevant — moderate confidence at best.

[00:40:43] AJ Nash: what it appears to be — and the combatant commander says, "Just tell me what you think." We don't do that. We go back to the caveat language. What I think is caveated intentionally.

[00:41:01] AJ Nash: because the caveats matter. But I fear that as AI gets more embedded, combatant commanders are going to say, "The hell with the J2." For anyone who doesn't know, the two is intel, the three is operations. "Too many caveats. Just tell me what's going to happen."

[00:41:19] AJ Nash: AI, what do you think is going to happen?" And the AI is going to give you a confident answer. Which is the opposite of the caveated answer that is actually the correct one.

[00:41:35] AJ Nash: That bias toward speed and confidence over prudence is a problem in cybersecurity — but in kinetic operations, it's a much bigger problem. If someone is making decisions about real targets based on AI outputs.

[00:42:00] AJ Nash: Real targets based on AI outputs, the consequences are catastrophic. I've seen in my career significant errors that resulted in significant harm. Go back to the Balkans — public knowledge — the US accidentally bombed the Chinese embassy. There's more to that story that I won't get into,

[00:42:15] AJ Nash: but the official account is that maps and targeting data were out of date. And then more recently there was the strike on a school full of children in Gaza. One might say, "Well, humans make mistakes too.

[00:42:33] AJ Nash: The Balkans was human." The Gaza strike reportedly may have involved AI in the targeting. We've only had this technology for a short period, and that may be one of the most catastrophic targeting errors in recent memory. Not fully validated — I don't know all the details —

[00:42:47] AJ Nash: but that's what the reporting suggests. And bombs on targets have consequences beyond the immediate harm. If you hit a school or a hospital, you create longstanding damage that goes far beyond the immediate harm.

[00:43:08] AJ Nash: People don't forget, and they don't care that it was a mistake. And I fear that's where we're headed — combatant commanders saying, "I don't need all the caveats.

[00:43:20] AJ Nash: briefing. Tell me what to do." And the technology is perfectly happy to do exactly that — because it has no awareness of consequences. I worry about the rush to deploy this in spaces where the consequences are severe: government intelligence, combat operations, law enforcement.

[00:43:34] AJ Nash: I'm hopeful we don't go too fast there. It seems like we're moving fast. It may fall to the platform providers to pump the brakes, but that's not in their business interest. So this is something we're going to have to watch carefully.

[00:43:47] AJ Nash: I agree — this technology has real possibilities and can make intel professionals smarter and faster as a tool, but not as a replacement for what we currently do. That said, we do need to wrap up. We've run a little long.

[00:44:04] AJ Nash: and it's been a good one. We probably could have kept going. The takeaway, if there is one, is that AI does some genuinely useful things and is also genuinely dangerous — don't trust it blindly.

[00:44:16] AJ Nash: We always close with the same question. The name of the show is Unspoken Security. No pass just because you're a friend. What's something you've never told anybody before — something unspoken?

[00:44:31] Faith Macgregor: Hmm.

[00:44:32] Faith Macgregor: On the spot.

[00:44:37] AJ Nash: Yes.

[00:44:39] Faith Macgregor: Well, intelligence professionals never tell people things. There's always something we're never going to share. But,

[00:44:45] AJ Nash: True. Go ahead.

[00:44:47] Faith Macgregor: Right. If I think of something fun and personal —

[00:44:51] AJ Nash: Yeah.

[00:44:51] Faith Macgregor: doing this kind of work

[00:44:53] Faith Macgregor: — you never truly go to sleep.

[00:44:57] Faith Macgregor: You're always on. It's Sunday morning, you're sleeping in, you open your phone, see a news story, and immediately think, "I'm going to have to do something with that later." You don't just let it pass.

[00:45:16] Faith Macgregor: That's just the life of intel. I always thought I was a city person — I live in St. Paul, surrounded by technology. But the longer I do this, the more I think about buying a farm small farm, like far away from people and just, I don't know, raise sheep and plant fruit trees and know, maybe just kinda while.

[00:45:48] AJ Nash: That's validating to hear someone else say what I say all the time. I go down rabbit holes constantly. You pick up the phone for something light and four hours laterrs later, like you said.

[00:46:02] AJ Nash: you don't even know why you're still reading. You don't work for anyone who cares about this right now. But it becomes second nature — as intel professionals we're wired to dig in, to want to know more. you read some wild story and you're like, "No, that can't be real," and then you wanna validate it or invalidate it, and you can start digging through all the details.

[00:46:17] AJ Nash: I've burned more hours than I care to count. But what you're describing — I've heard it from a lot of people across intel and tech: I'm ready to step back, to check out, get a farm."

[00:46:32] AJ Nash: to farm, raise livestock, just get away. Which maybe tells you something about how hard this work is. I'm not sure people on the outside appreciate how hard it is to do this for decades.

[00:46:46] AJ Nash: People define hard work as physical labor — factory work, mining, farming. All of that is genuinely hard. But I think people underestimate the weight of this kind of workuse, you sit at a desk and, you may drink fancy coffee and whatever, and there's good perks in jobs.

[00:47:02] AJ Nash: because it never turns off. And people burn out. It's very

[00:47:10] AJ Nash: difficult to sustain for long periods of time.

[00:47:12] Faith Macgregor: Just in my circle, off the top of my head — I know one intelligence professional who became a welder. Another who became a coach.

[00:47:19] AJ Nash: And I know people who moved to farms. That's just within my small circle. I fear that AI is going to push people out — and intel is not something you learn in a year. I've been doing this a long time and I still have a lot to learn. W- And once that, once, once you get away from it, like it, it takes so long to get back into it. And, if we lay off all the people for AI and then we try to bring them back 'cause it wasn't working, that resource isn't gonna be there anymore. And then risk. And so,

[00:47:57] Faith Macgregor: We've talked about burnout in cybersecurity for years, but specifically in intelligence — you're essentially on call 24/7. People burn out. They're going to be done, or they're going to say, "I want to retire early," orr, "I'm gonna switch careers 'cause I can't do this anymore," and, or, "I don't want to do this anymore, and anymore."." And you know it's like And you know, it's like you can do a thousand things right, but one little thing is off, and that's all anyone remembers

[00:48:27] AJ Nash: People don't remember the victories in intelligence. There's a

[00:48:29] AJ Nash: reason we're called silent professionals. You

[00:48:31] AJ Nash: do a lot of things every day that work, and people don't remember those. People focus on the failures, and to some extent that's appropriate —

[00:48:37] AJ Nash: some things need scrutiny. But you're right, it feels thankless. Over years it accumulates, and you start wondering why you're doing this. You've done a lot of good, but you don't think about it because most intel people arere pretty humble. and they thing.

[00:48:51] AJ Nash: wired to move to the next problem. And nobody calculates the lives that weren't lost because of something you prevented. Prevention is very hard to measure. When something goes wrong, the harm is obvious. The prevented disasters are invisible.

[00:49:01] AJ Nash: So it happens regularly — someone who is very good at this, who has done remarkable work, and leadership says, "I'm not sure why we're paying these people. Something bad happened and they didn't prevent it." It's a very difficult position to be in.ring after a while. So yeah, I think you're right.

[00:49:15] AJ Nash: There is a brain drain happening, and despite what some might think, you don't develop this expertise in a year or two. It takes time, energy, and experience — good and bad. I think we are losing that institutional knowledge, and we may find ourselves in a position where we have a very hard time in the future if we keep pushing too many people out.

[00:49:33] AJ Nash: that out the hard way. If you push out the junior people, nobody grows up to be mid-level and senior. Seniors retire or move on. That pipeline matters. And that, I think, connects back to the unspoken thing — the personal cost of this work that most people never see.

[00:49:46] AJ Nash: I hope you stick around for a while. I really enjoy these conversations, and hopefully we can keep contributing to this community.

[00:49:52] Faith Macgregor: Not going anywhere yet.

[00:49:58] AJ Nash: Outstanding. I appreciate you being here today, and I'm sure everyone listening did too. Thanks for taking the time to talk intelligence, AI, where we are, where we're going — the good and the bad.

[00:50:10] AJ Nash: Any last thoughts before we close?

[00:50:14] Faith Macgregor: Just a shout out to my fellow intelligence community: find something completely unrelated to the work that makes you happy and turns your brain off. That's why I write historical fiction fantasy. You need something that is just for you.

[00:50:27] Faith Macgregor: Keep yourself sane, keep yourself going — so you can keep doing the work that matters.

[00:50:36] AJ Nash: It's good advice. Yeah., turn your brain off or at least turn it on something else for a while. All right. Well, let's that with that, we're gonna close it out. Again, Thank you. ver- very much for being here, Faith. Thanks for taking the time and having this really interesting conversation. I look forward to getting feedback from everybody on how others, saw this and felt about it.

[00:50:49] AJ Nash: And for everybody out there, again, thank you for taking the time for listening or watching, however you chose to, to tune in today. If you like the show, please, provide feedback. If you don't like the show, okay, provide feedback too 'cause I still wanna, see what we can do to make it better.

[00:50:59] AJ Nash: But please and follow and share this with other people, so we can continue doing this show and bring, amazing guests with interesting ideas like Faith and others. So thank you all for tuning in. I appreciate it. With that, I'm gonna close this out.