Check Point's Shahar Tal on Exploiting AI Agent Frameworks

Check Point Head of Agentic Security Innovation Shahar Tal joins the Nexus Podcast to discuss research he and colleague Yarden Porat conducted that uncovered a dozen vulnerabilities in popular AI agent frameworks such as LangChain, LangGraph, CrewAI, AutoGen, Microsoft Agent Framework, and Google ADK. The vulnerabilities allow an attacker to exploit the inherent trust given to AI agents and can pave the way to remote code execution and other malicious outcomes. 

Subscribe and listen to the Nexus Podcast here. 

Transcript

Copy Transcript Copied

Speaker 0:14

All right, welcome back to the Nexus podcast. Check Point researcher Shahar Tal joins me to talk about some significant Social Security issues in popular AI agent frameworks. The Check Point research was presented at Black Hat recently, and that's what we're going to spend some time talking about today. Before we do, my usual reminder to subscribe to the podcast. I've got a bunch of episodes coming up that were recorded at Black Hat with researchers, speakers, and a few other folks. So if you weren't able to make it to Vegas or just miss being in the desert, please subscribe to the podcast and uh we'll definitely bring you back there. So let's get started. How are you? Good to see you.

Speaker 1 0:58

Good to see you too. Um great to be here. Thanks for having me.

Speaker 1:01

Why don't you tell uh the listeners a little bit about your role and kind of your day-to-day?

Speaker 1 1:06

Yes. No, absolutely. Uh so my name is uh Shachaltal. Uh quick two decades in the cybersecurity fronts, um, starting out with one decade in the army. So that was Air Force 10 Unit A200. Um uh after after which I joined Checkpoint, actually, that was 10 years ago, and I led malware and vulnerability research back at the time. Uh, then I joined Celebrite. I led the company's research for about eight years doing digital forensics work. Uh super interesting work, but maybe that's for for another episode. Um, then uh together with uh two great uh friends of mine, um, Baruch and Drohr, we co-founded uh uh Sayata, uh, where I was the CEO. And that um uh just six months ago, we were acquired uh into Checkpoint, so kind of a homecoming uh for me. Um, but uh you know, an amazing ride. Um, and uh we're here now. So today I am the head of agentic security innovation right here at Checkpoint. Uh, and this research was done uh by Jarden Porat, who's been as a security researcher, um, who has been with Sayata from the very early days as well, uh, has been doing some really, really interesting research, and we worked on this uh for many months together.

Speaker 2:30

Security is just a big circle. Eventually, you work with everybody and work for a lot of different companies.

Speaker 1 2:36

Absolutely. It's a big uh it's a big pond.

Speaker 2:38

So let's let's talk about your research. So I attended the session um that you did with with Yardan and on post-injection exploitation across these AI agent frameworks. Um maybe let's just start at the beginning. Just kind of take me inside some of these frameworks. I think you looked at four or five specifically. Um what are they used for? And and you know, just kind of bring the listeners in into that.

Speaker 1 3:03

Yeah, no, absolutely. Um, you know, over the past year, um, we we audited some of the frameworks that the industry is building its agents or AI agents on. And these are Langchain, Langgraph, uh, Microsoft's agent framework, Google's, you know, uh ADK, uh, Cloudflare's code mode, crew AI. And with with these frameworks are used when you know you as a developer would want to write your own agent, right? So you need to build it on top of something. We've seen a lot of AI agent exploitation in the last you know couple of years, lots of prompt injections trying to persuade the agent to do something else than it was that it intended, and then we've seen a lot of um kind of topical tool abuses where you know you find a vulnerable tool, a specific, you know, misconfigured or something that can be injected into, and then you use that tool you know to gain something as an attacker. Um but we thought about this you know layer of infrastructure that we are all growing to rely upon. Um, and we said, well, what if we found problems with the infrastructure itself? Right? That would mean that you know, if if there was a problem there, it affects every agent, you know, no preconditions, no, no required tools to be installed. No, it's it just comes out of the box. That was you know what we set out to do, and we we kind of had the feeling that this was underserved, right? We we saw very little research into those frameworks, and eventually that was true. So uh, you know, we let's say we cared enough to report 21 findings, um, and uh were assigned 12 CVEs by the end, you know, the end of that research. Um, many of them critical. Um pretty much every framework we hit was was you know had a very significant finding. Um and yeah, so there were a lot of kind of intricate technical stories around each of them. Sure. But I think the overall the overall understanding is something that that we think we should you know put more light on is the fact that we are building these AI frameworks faster than we know how to defend them right now. Okay, we're building this very fast and we we're trusting it with a lot, trusting it with access to everything that we know. We're trusting it sometimes whenever we're making financial decisions and in the enterprise as well. Um, but we haven't given it kind of the three decades it took us to trust browsers or some of these other web frameworks. So I, you know, I think it's for us it was very surprising.

Speaker 6:08

You mentioned you found 21 issues, 12 CVEs were assigned. Take me through some of the vulnerabilities in terms of criticality, in terms of ease of exploit, who would go after some of these?

Speaker 1 6:22

Uh yeah, so I think interestingly, what we found is a lot of I'd say classic vulnerabilities, right? The bug bug passes that we know of, that we've known how to deal with for 20 years, and like memory corruption vulnerabilities, right? Like years after freeze, we we found insecure DCRization in like repeatedly, actually, across a couple of frameworks. Um and and so that was uh again something that we were surprised to find because we thought you know everyone's looking at how to attack those models and how to extract their deep thinking, um, you know, reasoning traces, and and and and we're not looking at the the basics. Um now if we were to look at one of them, you know, we we looked into Langchain, and Langchain is by far the most popular um open source AI agent uh framework with like 300 million monthly downloads. Um and so this one, you know, we we to understand it, we need to kind of you know uh understand what a what a uh how it serializes its state and stores uh you know kind of JSON blobs, which means what it remembers, what what tool calls did it make, etc. And as we were doing that, we were looking very carefully at how it's constructing and reconstructing those classes. And kind of a classic issue that you know ended up surfacing is what we call insecure deserialization. So what we were able to do is tell it to store some piece of data, something that's very kind of uh I'd say simple and and and non-malicious. When it deserializes, it's going to instantiate an arbitrary Python class that we as an attacker can control. Um, and so what that means is any agent relying on Langchain, right, could have been attacked using this technique, regardless of any tools being installed or not. All it took was for it to serialize and unserize its memory, right? It doesn't matter if it's on disk and SQLite or or many many other kinds of uh formats. Um where would you as an attacker uh exploit this? Well, you know, I I think the industry kind of collectively understands this is a a grand security problem because agents have all the permissions in the world. We want them to be useful. We give them all the permissions in the world, we we give them access to data, and so anything that the AI agent could potentially access and and uh perform, the attacker gets access to that directly. Um and in fact, because this was a remote code execution rather than you know something just internal, we could use it to laterally move about the victim's network or you know, take over other other agents or other enterprise software that are sharing that infrastructure. Right.

Speaker 9:56

So is there a a level of trust that's being exploited here? Like I've if how are we going from injection or exploit to the the trusted logic, I guess is what I'm asking?

Speaker 1 10:08

Yeah, so in in in this case, you know, it's we're using kind of a simple trick. We're just asking the agent to you know to call a tool um and give it the parameters, which isn't something malicious per se, right? You couldn't say, yeah, well, it doesn't necessarily get blocked by a prompt injection classifier. It's something that's red relatively innocent. Um but the way uh that that tool call is you know constructed or deconstructed is what makes it what makes it malicious. The trust we're exploiting is inherent trust in AI agents that everyone is you know is is is sharing right now. Um it's just something that uh it's it's not this is not specifically new to our research, right? We all know there are these problems. We just need more time to fix, to patch those vulnerabilities, to build the resilience into those pieces of infrastructure we all rely upon.

Speaker 11:21

And so the the payloads, if I recall from your session, yes. Um they were being embedded in either PDFs or images or something like that, trusted that would be injected into the prompt. Am I oversimplifying?

Speaker 1 11:36

Yeah, no, that's the yeah, that's another one where we found something so simple and classic, which is you know, which also affects the indirection prompt, the indirect prompt injection um route, which means just pointing an agent at uh you know at any document to read or to sort through um would cause this document to get parsed by a vulnerable PDF parser um that that we found there you know laying around since you know 2005. I think it was interesting for us also to see how you know the even the developers for crew AI, which is the specific one, um I don't think they realize that when they're using this library and and and calling the function get text, that it's actually going about you know, parsing images, fonts, decompression, actions, all sorts of things, uh, even though they just wanted to get or to extract the text out of the PDF, right? So that's something that again, kind of a classic, right? We know those seams between now the calling, some like you know, older uh functions, insecure parsers, um they've they've been they've been there uh throughout, um, but still we're writing them there very fast, they don't get enough scrutiny, and we're trusting them too fast. Um, so that's yeah, that's that's part of the problem. Right now, now I've been actually this then and I think this is an interesting point. Um, you know, when I said this, I was discussing with a couple of my colleagues, and one of them said, but wait, but what you know, there's all these AI models are finding vulnerabilities in code bases, um, and fixing them uh very very quickly. And that's true, and that's the optimistic part in all of this. Like looking forward, I'd say, yes, this is going to happen, but it's still very far from happening in all codebases, right? So you need a lot of human maintainer attention to enable this, to enable this amazing vulnerability, finding and patch management cycle uh in a codebase. And and uh I think it's gonna take like one and two and three years until the majority of significant code bases has embraced and adopted such cycles. Um, with pretty much all of the codebases that we've audited, we haven't seen that yet.

Speaker 14:24

Interesting, because just the s the speed of the last six to nine months has just been ridiculous, and I'm sure you're much more aware of it than than we are.

Speaker 1 14:34

I am I am living the dream. Listen, it's you know, you don't you uh you don't get to live through a lot of revolutions, but sometimes you know you you feel it and and you see it happening. Uh and the rate of change is is very high. Um, yeah, you know, it's but it's exactly what's again causing some of these exposures is is is that exact exactly this rate where you know we need a few years to get oriented around what's happening and make sure that it's secure, but we're using it. Uh, you know, I think like 18 months ago we didn't really know what this is going to look like. Sure. And today all of us are kind of you know asking Chat GPT in our pocket to do things for us or or you know, asking them things about our health, um, our finances, and again, this uh you know I I don't want to be an alarmist because I think it's a very good direction uh to go. You know, that's that's the way technology should advance too. Um but in many environments we still need to get more resilient.

Speaker 15:44

In the grand scheme of things, where do these vulnerabilities and these frameworks stand compared to other AI-related vulnerabilities and security issues?

Speaker 1 15:54

Yeah. You know, I think uh we we're seeing a lot of um a lot of headlines. There's there's a lot of uh of good coverage around the newer fancy AI attacks, attacking a model, make it you know, bypassing some prompt injection filter. Um and I think you know, we're gonna continue reading about a lot of agent failures this year, but very few of them will be the model's fault. Um, and and and I think I think models are getting to a point where some of the internal defenses are are quite reliable. I wouldn't say 100%, but they are getting much better. And specifically under the after the whole you know, anthropic fable kind of you know incident, they got it to a point where it's very difficult to get it to you know to do something, it's cybersecurity related at least. Um and and you know, the self-hosted models are out of the question. So at least in that area, I think we're gonna see more of the failures happen, not because of hallucinations, not because of a you know, a problematic uh sass side guardrail, but because of these vulnerabilities and because of you know people um failing to place the the right defense in depth. Okay, because guardrails have been debated to death by now. Uh and I I don't think still, I don't think people know exactly how they what they should look like because there's a lot of guardrails. Um but everyone understands you need them and you need um a few levels of them. Because you know, once you bypass one, sure, that's gonna happen, but what about the the other two guardrails that are there? Um and and and so you know it's it's it's a defense problem really rather than a model problem.

Speaker 18:03

We should probably talk about the risk uh involved here and uh around the vulnerabilities and the in the frameworks that you guys found. Excuse me, just how useful are they for an attacker? What what would an attack look like? Explain what you know an attacker could control, etc.

Speaker 1 18:20

Yeah, so in in many of the cases, it was just remote code execution. That that's the holy grail, right? Sure. If you are an enterprise running your agents, you know, and you think you're you're doing the right thing by self-hosting it and making sure you use the latest and greatest um, you know, AI frameworks. If someone were to exploit one of your agents just by you know a user speaking normally to an agent, you don't need um, you know, that's another interesting difference. In in history, an attacker needed like a sometimes like a non-standard attack surface, right? Like an API through some means to be open or some port that they could speak to. With AA agents, this attack surface is just speaking to it, speaking to a chat bot, which is exactly what we're building the agent for. So that's it it gives you this concentrated funnel where attackers get to try what they have. And so if an attacker can exploit this, then they are in quite literally. One, everything this agent ever did, any conversation that anyone has ever had with this agent, but any, but also anything this agent could potentially do. Um, and then also the lateral kind of infrastructure move that that you know I said before. Um and and so that's that's what one was interesting with with another case where we found kind of vulnerabilities and cloud flares infrastructure. Um, and and I think that was the other talk because you attended maybe the the post-injection talk, but we also had a Cloudflare code mode um uh presentation. We were able to break the basic promise of kind of Cloudflare workers, where a worker is supposed to be secluded from other tenants on the same machine. And we were able to break through that and pretty much get any secret from any tenant running Cloudflare code, right? That was and that was uh actually much wider than the code mode element. We started out by researching code mode, but we ended up with Cloudflare workers, which is kind of the underlying runtime that that code mode relies on, but it affects much more than than the than the agents. But anyway, that one was very simple for an attacker to exploit, just send this arbitrary JavaScript to a worker where that's something very normal that you're supposed to do. Um and that would exploit the server side for for that Cloudflare uh instance.

Speaker 21:12

To go back to the previous discussion in terms of the fixes, when you hear architectural problems, you think of kind of code overhaul, you know, huge problems uh to a huge project to fix these things. How difficult uh were the updates, etc. What can you tell us about?

Speaker 1 21:31

I I I will say that most of the issues were simple to fix in that, well, there was a missing check, or you know, they they had a wrong calculation for something, which is kind of the classics. Um in some cases, like in Microsoft's agent framework, they actually decided to re haul the entire serialization and move away from whatever this realization, you know. logic they were doing to something to pickle or something like that. And that was interesting for us as well, but it was also like a pre-GA version. So things are going to move around and sure it happens. The rollout itself is also not very complicated. Once you know once people know there is a vulnerable version, you roll out the version, you need to deploy it just like your every piece of software where you do your patch management cycles. One interesting uh exception to that was the Cloudflare instance where um you know we got uh two criticals uh they confirmed it they paid respectable generous bounties for the findings so like in recognition for for the work um but at the end that's a good thing what that's a good thing oh yeah no you know we we we actually got that from all of the vendors um and and and that's nice and they're showing you know the again the recognition but the thing was with those findings they they did assign them criticals and but they chose not to assign any CVE and that was kind of you know a bit puzzling for us we kind of went in a back and forth eventually um they said that it was a project policy not to issue CVEs or advisories which I would say it's I mean it's not unprecedented seen maybe a couple of projects maybe like more when it's more of an internal project um but we think it's it's the wrong call here uh specifically because you know worker D or this runtime that powers workers is also distributed for self-hosting it's distributed in npm and so if someone you know downloads a package and they kind of pin the January version before our fixes they will never know unless they hear the news that there were you know critical vulnerabilities in this uh you know in this version and that were fixed in you know the the version where all of our vulnerabilities were fixed is 2026 um 0616 which is like you know the June 16th one um or June 19th but uh again if you look at the release notes there is nothing there's nothing there to no advisory or no no credit um so again there are yes some some of the some of the areas here you know we saw we saw a bit of friction but um that was it that's unusual though that sounds a bit 2002 kind of I mean yeah it it is unusual we think it's again thought it was the wrong call it's not like we need the numbers or you know we don't count CVEs but uh you know forget the CVEs let's have an advisory in place to make sure people know something for those the bug bots the trivies of the world to pick up on and let you know in this CI C D cycle that hey you should update that library that's all yeah

Speaker 25:29

What uh what about enterprise security teams what should they be doing about uh for example the the issues in these frameworks um or is it just a matter of kind of relying on the vendors and the maintainers of these open source projects to adequately communicate and fix etc?

Speaker 1 25:47

I I don't think anything changed in the threat model um and you know everything that was correct yesterday is correct today um make sure you you know try to use the latest versions make sure you employ defense in depth list privilege any like everything is very uh you know take your take your old model point it here apply it carefully um we we we just have to uh you know we like looking at prompt injections and then preventing prompt injections and that's great it's important and that's a critical runtime guardrail but at the same time we have to assume the injection is going to happen and then let's budget for that world right so don't don't you know laser focus all of your energies and you know and and defenses at at one layer there's a defense plane to be you know to be layered here okay

Speaker 26:58

So just to to wrap up and just it's going back to a comment that you mentioned earlier we're kind of in a revolution here and obviously there's a lot of new innovation there's a lot of speed here does it need to slow down and or you know will the whole nature of of AI and these models allow it to slow down I mean what's uh are we okay ?

Speaker 1 27:21

My answer is twofold I mean one it doesn't need to slow down and two even if I thought it should have or even if anyone in the world thinks it should have we are not at you know at a position to make that you know to make that call the world is gonna be the world is going to adopt faster than any of us can you know think about it um or control and so uh you know this is happening make sure you know you put your best defenses security thought critical thinking into what permissions you're giving what actions are you allowing ai to take um and if you need if your risk tolerance is is lower then you you know uh don't take all that risk uh right I think it's it's really comes down to basics um I am so excited to be living in these times and uh I just want to see you know where what happens in two years.

Speaker 28:26

All right Shahar thank you so much for coming on the podcast I really appreciate it it's been really uh very entertaining very good stuff here so I'm sure the listeners are gonna love it.

Speaker 1 28:35

Thank you so much all of the credit here is really to Yardin who's done a lot of amazing work. We're actually we're working on some new stuff uh hopefully we get to record another episode sometime you know in in a few months.

Speaker 28:48

Yeah let's stay in touch thank you so much all right thanks again have a great