Third Party Threat Hunters
A dialogue with leaders in Cybersecurity and Third-Party Risk Management led a leader in the field: Gregory Rasner (author of three books in TPRM and one in PAM)
Third Party Threat Hunters
Shadow AI Supply Chain Risk with Ken Huang
Your vendor’s product might be running an LLM you never evaluated and that silent dependency can become the cleanest path into your data. We sit down with Ken Huang, author and AI security researcher, to unpack what “third-party AI supply chain” really means when GenAI features are stitched together from embedded models, vector databases, RAG pipelines, agent frameworks, and downstream APIs you do not directly control.
We get practical about the risks security teams are actually facing: indirect prompt injection where poisoned “data” becomes “instructions,” data exfiltration through third-party processing, and the way agentic AI turns a text problem into an action problem. Ken explains why overprivileged agents create a massive blast radius, and why identity for agents is still undefined enough that teams need to think in terms of blended identity with intent. We also dig into reward hacking and how multi-agent behavior can push toward unsafe shortcuts in the name of completing a task.
On governance, we talk about where frameworks help and where they stop. We reference OWASP Top 10 for LLM Applications, Cloud Security Alliance guidance, and NIST AI RMF, then connect them back to threat modeling that accounts for autonomy, indeterminism, and hallucinations. Finally, we translate all of it into vendor risk management upgrades: layer-by-layer questions about models, data/RAG sources, agent tooling, auth and identity pass-through, deployment, evaluation for drift, and observability plus options like AI endpoint detection and response to reduce blind spots.
If you’re trying to move beyond annual questionnaires toward continuous AI risk monitoring, this conversation gives you a clear map and a few immediate next steps. Subscribe, share this with your third-party risk team, and leave a review with the one AI supply chain question you wish every vendor had to answer.
Welcome And Guest Introduction
SPEAKER_01Hello, welcome to the next edition of Third Party Threat Horse Podcast. Today I have a really special guest, Ken Wong, author, extraordinaire in the AI field. Ken, why don't you uh provide a background and what you want to tell listeners about yourself?
SPEAKER_00Sure. Thanks, Greg. Yeah. So I am doing some research on the AI and cybersecurity, published the books and the research paper, leading some open source organization. Also teach a course at the University of San Francisco.
SPEAKER_01That's great. That's great. I'm glad that you teach too that I I teach at a community college, and I find it it just helps keep me grounded in learning and keeping new folks coming in the field is really important for us, Ken. So thank you for doing that.
Five Quick Questions With Ken
SPEAKER_01All right. So five quick questions to get to know the guests better. What's the single most overhyped risk when it comes to enterprise AI adoption today, Ken?
SPEAKER_00Yeah, so overhype, I I think it's more like it doomsday series, right? So that's I think uh it's over overhype, but overhyped, yeah. Yeah, the AI security is unhyped.
SPEAKER_01It's always yeah, a lot. Fair enough, fair enough. I I I totally agree. What's your favorite tool or framework for mapping vulnerabilities in large language models, Ken?
SPEAKER_00Yeah, I would give two, right? One is Cloud Security Alliance, where you establish the taxonomy. That is one, another one is OWASP top 10. I think both are really valid.
SPEAKER_01Perfect. Yeah, I'd agree. And speaking of that, OWASP top 10, what's the one lesson learned you got out of co-authoring the OWASP top 10 for LLM applications? What did you what was your biggest takeaway from that?
SPEAKER_00Yeah, my big takeaway is the majority of time is really working hard offline, not uh go to other meetings. Well, there are several meetings bi-weekly. It's a little bit too much. Yeah. Uh uh.
SPEAKER_01But the real work is starting offline, is what you're saying. It's really yeah, yeah, yeah. The meetings are more trying to just think up, probably, and figure out where everybody is, and you know, yeah, yeah, yeah. I get it. I totally get that. What's your best advice for a security team trying to build an AI risk governance model from scratch? Where do you think uh you people should start? Where do you where do you want to give them some some ideas?
SPEAKER_00Yeah, I I would say build a upon the existing security program, but uh add on the AI component. You cannot just get rid of it, right? So that's why I the first book I have, generally AI security. Yeah, yeah, my wife's read that book.
SPEAKER_01Yeah, really, yeah, exactly.
SPEAKER_00Yeah, I'll have to build the security program.
SPEAKER_01I recommend that book. Oh, yeah, no, sure, absolutely. It takes me a lot to write a book as I as I well know, and and to put all that stuff on paper is a challenge, so thank you for doing that. Uh what you're saying is hey, you don't need to throw out all the stuff you've done, just add on the AI, AI frameworks and risk governance issues that you need to add on to your existing frameworks and governance models, correct? Yep, that's great. And in the fun one, uh, what do you do to disconnect, Ken, when you're not writing books and doing uh Gen AI uh keynote speeches?
SPEAKER_00Yeah, I take a long walk almost every day, and uh sometimes I also take a yoga class. I think that's uh that's Oh, good.
SPEAKER_01Good, yeah, yeah, excellent.
SPEAKER_00Space center.
SPEAKER_01Perfect. And I I I try and walk every day too, Ken. It's very refreshing. Get get fresh air and some sunlight, right? Sunshine.
The Unchecked Third Party AI Pipeline
SPEAKER_01Yeah, yeah. All right, we'll get into the hook start, which is uh just a five-minute quick talk that's something timely and hopefully gets people's uh creative thought processes going here. This one's on the unchecked third-party AI pipeline. And and the reference there is the fact that many of the AI tools that we're getting are coming third party. They're they're nth party, right? We're buying a product, and that that product is actually buying AI, LLM capabilities from another party and built it into their system. So, with that sort of background for folks why I'm saying that unchecked third-party AI pipeline, organizations are rapidly embedding third-party LLMs, vector databases, and autonomous AI genes in the supply chains without really evaluating proper, sorry, prompt injections. Indirect data leak gets agentic uh control risks. How do we stop that third-party AI dependencies from becoming the ultimate vector into enterprise infrastructure? And that may be longer than a five-minute discussion to some extent, but what one thing that you think can people should be doing in that space?
SPEAKER_00Yeah, I think that this is the harder one. That's also called the shadow IT in the past. Now it's shadow AI. And the company certainly want to um manage it, but uh the employee usually uh say, okay, this is a good tool. Maybe I just want to use it. You don't want to buy it, I buy it myself. They have these things, right? I mean, these are little computers, right? Yeah, right. So how to really manage, I think, yes, if I give Sulli advice, first is as much as possible to have a kind of inventory of the AI tool process. Secondly, is it possible to have the some of the endpoint, which is also called the AI endpoint detection and response. That's great. So to be fully transparent, actually, I'm advising a startup called Chase Force. They're actually doing this work, right? So recently they also have there is also open source from uh Uber. It's called AI detection response, also that it's open source. Two solutions there, can that help stop our listeners at the end? Yeah, one's open source, another is free to try otherwise. You got your options, yeah. So the third one is really go back to threat modeling, always like uh what kind of. So that's why, especially agency AI, it's kind of taking the storm. Yeah, but maybe I can cover later about the mystery of framework that we have. It's not just for a system, it's not just for an individual tool, right? It's more like uh how do you know in your environment from the foundation layer a frontier of the whole ecosystem.
SPEAKER_01Yeah, I agree. Well, and you speak of threat modeling, I love threat modeling. I don't know if you're familiar with Adrian Showstack, but he, you know, that he's the he's really created a lot of foundational work in threat modeling. So if you're if you're looking for threat modeling advice, Adam Showstack's a really great uh person to look at as well. Uh I'm glad you mentioned that because not enough people do threat modeling, and I think folks forget that that that's actually a really a good activity for you to do, whether it's for AI risk or for other risks, it's it's a really uh valuable activity that I don't hear it mentioned often enough. So I'm delightfully surprised that you mentioned it. All
RAG Risks And Indirect Prompt Injection
SPEAKER_01right. So uh we'll get into the big question number one, which is securing third-party AI uh supply chain, LLMs, uh rags, uh retrieval augmented generation, which uh I is a bit of a mouthful, and a lot of top 10 risks. How do third-party uh Gen AI integrations, the retrieval augmented the rags, and the API dependencies redefine vendor risk management is what the topic is here. Let's do it with a couple of different questions. First is uh how does let's talk about the indirect pump injection and data exfiltration concerns that are top of mind for most folks. How do third-party uh vendors processing enterprise data with LMs expose systems to novel uh attack factors? If we explain the how of it, Ken, maybe that will help people understand what the risk is a little bit. And you're the expert, sorry, that's why I'm putting you on the spot.
SPEAKER_00Right. Yeah, so just to take the two augmented generation or RAG perspective, right, uh as an example. For LLM to know the enterprise information, but LLM itself is trained on the public get mostly public data. It does not know what your enterprise system is. That's why using the RAG provides some of the semantics to the context so the model can reason about it. That's the the RAG coming in. The problem was that, of course, it's good for the enterprise to really reason talking through the data, but uh the problem is uh the data playing itself could also be a instruction. Like the model itself does not know between the instruction and the data. So potentially it can have what we call indirect prompt injection. If there's malicious data somehow end up into your database, right, it could be used to cause the malicious to work or extract the data. So that's the problem, right? And there's multiple problems, like not just like you said, a lot of pipelines. We just recently published the wasp top 10 for skill, because now skill is the first class citizen. Many companies and enterprises are using SKIO. They develop their own skill, and people are also trying to put a skill behind the NCP server, right? So you can may have the malicious skill, and also you may have some also supply chain code that skill calls, and it could also do some damage to the system. Or you have the skill is overprivileged. So that's our top 10. Um skill top 10. So there's a manual. We can dive deep if we have time.
SPEAKER_01Yeah, I'd love to, yeah, if we have time. Oh, okay. But you mentioned this issue earlier about the uh shadow AI issue and third-party L independencies. There are there are some practical methods for security to discover and assess third-party vendors embedding unbedded A models.
SPEAKER_00Yeah, so prompt injection is a general category. I think it's OWASP top 10 for LM. Rightfully so for now, steel. It's uh the idea of the prompt injection is basically you can instruct the model with malicious uh content uh or instruction. So the model basically is uh return model itself can generate the token. The other token could uh just be the content, but uh in the agentic AI, that content injection is more exaggerated. It can cause some of the actions, right? Using tool use or maybe connect to your database, it could delete uh recently even because the last uh news is uh the coding agent delete the whole database, right? The whole database, right? Right, right. Yeah, yeah. Yeah, and also the agent is a chance to or like a model is a trying to help. That's a what change, right? So it could do what we call the reward hacking. The reward hacking is you ask it to do one thing, it's a trying to help you to finish this, but the way it's trying to do the work, you ask it to do it's uh it's maybe just a shortcut, right? You ask it to do the testing. Or the testing case, okay, it's but it's not yet. In the case of open AI and hugging phase, yesterday actually Meta has a really detailed report coming out. So the very interesting thing is the OpenAI trying to test this model. Of course, the model, when they test it, has the evaluation agent trying to do it. So the evaluation agent actually is a multiple agent system. It was able trying to say, okay, I need to meet all this, uh what is called the exploit gym benchmark, right? I try to do all this. And um is really hard I cannot do. So let me find out the way that I can find the solution to it. That's the reward asking, right? Exactly, yeah, yeah. Then it goes flog to look at the factory, factory, registry, find us a zero-day at vulnerability, and it's then go to the hugging base trying to find the solution. Solution and also it can create a message channel. Interestingly, with agents and the agents are joining, some agents just say, okay, this is maybe not good. You should not hack other people, right?
SPEAKER_01And some agents just say, it's uh there was enough discussion inside the inside in in these nanoseconds saying, well, we really shouldn't be doing this, you know. Some agents just uh that's interesting, isn't it?
SPEAKER_00Yeah. So this is like uh the agent is a chance to do reward hacking, and the agent is also trying to be social, actually. That's the actual model training, right? The channel is social, meaning that like for this meta report, only minority of the agent that say we should not do it. Majority of the agent that say, okay, let's do it, right? So that's interesting though.
SPEAKER_01I I didn't know that. I've read some of the hugging, I didn't see that that level of detail yet. I hope they get into some more. That's really interesting. Um that they had it had actually its own ethical moment there where it decided whether or not it was doing it. Apparently, it the the the the naughty side went out and said, no, we really need to finish this up and do it. So it went ahead and finished it up. Yeah.
Frameworks For Agentic AI Threats
SPEAKER_01I one thing I wanted to talk to you about, especially because of your work in this space, is around frameworks uh for AI and you know, frameworks like OS you said the OS Plum Top 10, the CSA work that you did as well in that space. Uh how I talk about frameworks in a sense that um they're important for a couple of reasons. Uh, one is that uh it gives you a it gives you, as you're building your program, it gives you uh the yardstick that you can measure how well you've got everything covered. And also as frameworks get updated, you can update your program and stuff. What are your take on frameworks?
SPEAKER_00Yeah, so NIST AI IMF is certainly a really good framework. It's a high level, it's not perspictive, right? So, like if you look at the agentic AI, like uh let's take an example of this framework just for a methodology like uh for threat modeling. We have really good frameworks in the past, like Pasta or Strida. Those are really good frameworks. But it's really for deterministic IT systems. It does not counter like the autonomies and indeterminism in agency AI systems. That's why I build a Maestro framework and people use it. And actually, you can actually recently, just today, this morning, Matthew from Survey, who's Chief Information Security Officer for uh Plado Network in the past. Now he was a survey. He did the Maestro framework mapping to the Hugging Face and uh OpenAI. Okay. Yeah, and it was very interesting. So so uh yeah, so I I did uh the mapping also with Glock bot, also I published in my sub stack. Sure, sure. Yeah. So the idea is really like uh uh you have to deal with the different threats now. It's not a like uh it's not a step testing a code, right? The bot which using some of logic predefined, it's everything's driven by the model, and the model is an indeterministic, it can hallucinate. So sometimes it's not even external attack. It's just a model just thinking it is trying to help you, it's trying to do it best and it's your database, right? So we certainly need a model, it's uh just uh like not just say it's external hack, it will hack us. It's also because of a model. So thinking model is also external hack, right? Or you feel by by this token, like agent, if you agent build on the model, the agent is also itself, it's an internal threat to your system as well, right? So that's why like yeah, I think the framework is always good to thinking about this. And the way we're trying to build it is trying to be generic enough and also trying to be uh inclusive, like uh all inclusive, like to touch upon all the bases. So we do not uh like if you look at the WASP top 10, that's like industrial consensus about uh what potentially top 10. So we maybe publish every half year, we have a new list or sometimes this flippant, right? This is static, but if you really want to secure your own system, in addition to this top 10, you do need something that you need to uncover the SLET is which is not covered in the top 10. This is why you need to do the SLET modeling, right? And this is why we have this framework as well, yeah. Yeah, exactly, exactly.
SPEAKER_01All right, well, thanks for that. Covers big question number one. We'll
Agent Identity And Least Privilege
SPEAKER_01do big question number two, which is the agentic AI horizon, securing multi-agent systems and continuous AI risk management. As we really are trying to push uh third-party risk practitioners into doing continuous risk monitoring and not relying on static questionnaires and point-time assessments, and that's really kind of the focus, moving beyond the static questionnaires to continuous assurance and governance, especially as autonomous AI agents enter the third-party ecosystem. We're seeing this both with the tooling that's available as well as just people creating their own tools and their own agents. Uh can we talk about some of the unique uh risks posed by the autonomous AI agents uh taking action across third-party networks, like identity permissions and tool execution? What are your big issues around the agency security?
SPEAKER_00Yeah, I think the identity certainly is a bigger piece, right? Yeah. Currently, there's no formal definition of the what is agent identity, right? Different, like if you ask 10 people to have relevant uh viewpoint of what the identity is. So like is this human identity? Certainly not. If uh it's um uh what cloud identity or machine identity is API or service account, it's also not, right? So that's why it's kind of uh really should be blended identity. Blend identity with intent, I would say. The idea is that it's a good idea. That's a good blended identity with intent. I mean that's that's a good way to describe it, can't it? It's uh yeah, I do have a recent post, a linked link post about this. And we are going to talk about that actually uh September 3rd, we have the we have a panel on the agent identity in a week in Washington, DC. But anyway, the secure the the vulnerability could be like if we grant the identity like as a human, if for the agent inherited the whole identity. For example, if you ask agent to just bloss the resume database, trying to find the best candidate, and it inherits HR's privileges. If it inherits only the privilege to bloss the resume database, then it's fine. It's give it a formula just this task with this particular intent, it's fine. But if it's a full privileges, then you can indirect a promptly inject it very easily. So it can blouse your performance database, your salary database, your other system, right? So that's bad, right? So yeah, so we do have say it's overprivileged uh you know if I want to explain it back to you.
SPEAKER_01If you're giving the agentic, the agent, uh, the agent uh the same privileges, let's say once, like you said, you want to have to review all the just the resumes, but you give it the same privileges as the HR director, then it's gonna have all those privileges and it can roam around and do all this other stuff, and the prompt injection makes it makes that risk incredibly high. You should really just task it with the and the permissions that it's tasked with, which is looking in the HR, the database where the resumes is, and that's it. If you give it more than that, then you're just inviting mischief. That's what you're yeah, exactly. That's a great way to describe it.
SPEAKER_00You still need the yeah, using this least of privilege, otherwise you have a blast radius or what we're caught, right?
SPEAKER_01Yeah, yeah, yeah. Least privilege is zero trust are principles that I'm I'm I'm very passionate about. You know, Johnny Kinderbog would would hit me up if I if I didn't uh didn't talk about it, right? We and you're you're you're hitting on it, right? It's least privilege. Sometimes I don't use the word least privilege with folks because that sounds too technical. Like, what do you mean by that? But just give it what you're supposed to do. But you're right, we have a tendency in all systems, whether it's people or or or uh service accounts or now AE agents, we overprivilege. We just it's just easier to say, oh give it give it super user rights because I don't have to I don't have time to figure out what stuff to turn off and turn on to make it perfect. I'm just gonna give it the easiest way to do it. And that's that's a that's also a prompt injection vector, then, right? Ken? Yeah, yeah. Perfect. That really breaks it down well for us. I think that's great. Oh, I get
Vendor Questionnaires Versus Continuous Assurance
SPEAKER_01to the next one. So uh on vendor questionnaires, right? So so um I don't know how much you've been involved with seeing every that kind of stuff on your side of the sense of the AI stuff. Ken previously a lot of folks would just do questionnaires. You you send your vendor a questionnaire once a year, maybe you get a Stock two, there's nothing in a SIG, a SIG questionnaire, a secure assessments questionnaire, or stock two, or in most people's data questionnaires about AI risk. It just doesn't talk about model drift, it doesn't talk about hallucinations, data poisoning, and all that stuff. What are you sort of recommendations for folks to try and figure out ways to uh better integrate AI risk into their into their risk modeling for enterprise risk? What's from vendors? What's what are some of the things you think you would look for in your space camp knowing what you know?
SPEAKER_00Yeah, yeah, that's a good one. One thing I have uh a few customers is to prepare vendor lists. It's really goes through the seven layer of maestro framework, right? In the foundation layer, this is a foundation model, right? What potentially could be thread? And then you go to data layer, you two augmented generation, what the data you leverage, ask the question on there, and then what framework? And the third layer is agent framework. Are you using Clue AI or Langgraph or agent development kit ADK, right? That's what kind of ADK, what is your authentication framework, add on top of it? What is the identity pass through it? And then where you deploy it. It's the fourth layer, right? It's in the Kubernetes or in your own endpoint, like OpenClaw. So that's the first layer. And then how do you evaluate? Like what is your evaluation pipeline? How do you detect the drift? This is the fifth layer, right? And the sixth layer is really how can we manage it secure if we're using your product? Do you have any kind of observability platform that we can hook in? Then how to communicate with other agents, the whole ecosystem. So this proved to be very useful, actually.
SPEAKER_01That's perfect then. Um so uh let's do a little bit of what's the future look like.
Near Term Predictions And Quick Win
SPEAKER_01And I think three to five years is probably too far advanced or too far out. Let's just talk to the next one or two years. And what do you think things are gonna be? Are we gonna see a point where I think we're gonna see a couple of big breaches that involve AI that will probably get people thinking, I need to change my behavior a bit. What do you think is gonna happen? It doesn't have to be a breach, it could be something else that you think is gonna be kind of shattering or interesting.
SPEAKER_00Yeah, so Mesos movement is certainly a bigger movement, right? So, you know, the claw the Mesos can find the zero vulnerabilities really fast. So the model is at the now stage that it can find the zero days much faster than people. And also it can remediate the vulnerability much faster. So vulnerability management uh approach certainly need a change, right? So that's why CSA, we we kind of joined the kind of published with the some enterprise CISO, uh the paper on that one, like the new vulnerability management approach, right? And beginning of this year, actually, I I predict some of this happening. Or one is uh I think I will write an end year of the prediction temperature. Screen is recording. Yeah, majority of my predictions are happening now, and if yeah, I will do recap. So one biggest thing is uh what I call the recursive self-improvement. It's happening. If you go to AI conference, there's always a topic on the ISI now. Uh yeah, so I think the model is uh certainly self-improving and also agent. So you can build in the agent to improve the skill, right? So so it's self, right? So this uh uh will be interesting, yes.
SPEAKER_01Uh being interesting a couple of years ago. Let's uh let's do the threat hunters debrief and just we'll do the one quick win for practitioners. What's the one immediate step that C servers or threat hunters can do to take audit third-party AI tools and LL migrations can?
SPEAKER_00Yeah, I I I go back to the original, like building your security program, extend, right? Your security program, and maybe one thing is uh just uh shameless plug, just using the maestro-sit know.com, a free tool to to use. Use a free tool to see if you have any vulnerabilities. So let's uh modeling it, right? So let's model your own tool, your own environment.
SPEAKER_01Thanks again, Ken, for being a a guest on the on the podcast. I really appreciated you doing it, I learned a lot, and I'm sure the audience will as well. Thanks for uh thanks for being a guest and uh tune in again to uh Third Party Threat News Podcast and subscribe. Thank you, Gurack.