Detection Dispatch (Alex's Version)

Sysmon for AI Visibility feat. Anton Ovrutksy

• Alex Hurtado • Season 1 • Episode 10

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 55:50

Anton Ovrutsky from Huntress joins Detection Dispatch to talk about ATEN, his latest open-source project to do what Sysmon did for win event telemetry to AI agents. The idea came from a pretty simple realization: as security practitioners, we’re comfortable saying “we don’t have the telemetry for that”...And there’s a LOT missing.

In this episode we get into:

  • The “what the hell did I just give Claude access to?” problem — credentials, packages, skills, and systems pile up fast when you’re just trying to get the agent to work 
  • Why process trees tell you what happened, but not what the agent was asked to do
  • The huge blind spot around credentials living on endpoints — especially when an agent and an infostealer can touch the same credential store 
  • How ATEN connects the prompt that started an action to the endpoint activity that followed, including a session ID to tie it together 
  • Why we may need to start thinking about agents as something like an entirely new OS running on top of the endpoint
  • The problem with trying to reconstruct an incident across endpoint + prompt transcript + network telemetry when half of that evidence was never logged 
  • How a compromised skill or dependency can turn a seemingly normal agent workflow into a supply-chain problem
  • Why intent vs. action might become one of the most important detection primitives for agentic systems 
  • And the uncomfortable reality that we started giving AI agents autonomy before we had good visibility into what they were actually doing

The question goes from “What did the agent execute?”  to “What did I ask it to do, what did it actually do, and what did it have access to along the way?”

That feels a lot more like the telemetry problem we should have been solving from the start.

Detection Dispatch (Alex's Version) is an independent detection engineering & threat hunting podcast. Rebuilt. Community-first. Featuring a lineup of the real and active projects pushing the limits of detection engineering, threat hunting, and everything in between.

SPEAKER_00

Welcome back to Detection Dispatch, the show where we go find the people actually pushing the limits of detection engineering and threat hunting and all things in between. Today we're talking about a blind spot that most of us don't even know we even have yet. Coding agents, claude code, codex cursor, whatever you're running now, it seems to be living at the endpoint level, spawning processes, uh in touching credential stores, installing packages, reading files, and your EDR can see the process tree, but not really have an idea of what the agent was actually asked to do or why. The gap is super real, very new on everybody's mind. And today's guest built a tool specifically to address that. Joining me today is Anton Ovritsky on the on the pod. He's been in the space literally since the Sysmon days and quite literally built the Sysmon for AI telemetry. Anton, thank you so much for coming on the show. How are you?

SPEAKER_02

I'm good. I'm good. How are you doing? How's everyone uh hanging out?

SPEAKER_00

Uh good. I I gotta say, I have I'm still buzzing from this new gig that I'm finally able to share. I've been keeping it for quite uh on a low, low kilo profile for quite some time. And finally it's it's F FB official, LinkedIn official.

SPEAKER_02

Congratulations, yeah. That's awesome. I'm I'm so honored to be like on the episode where it gets announced, and it wasn't planned or anything. So yeah.

SPEAKER_00

It totally wasn't. We've been, I feel like we've been rescheduling this pod. You you just went on vacation, which absolutely take every to everyone listening, take this as your reminder to take your PTO. Take your PTO.

SPEAKER_02

Yeah, yeah. Take care of yourselves.

SPEAKER_00

Uh-huh. Take care of yourself. Did you go anywhere fun?

SPEAKER_02

No, I I'm I'm a homebody. I don't I don't enjoy leaving the house. You know, I just like hanging out at home with my wife and cats. We love a staycation.

SPEAKER_00

We love a staycation.

SPEAKER_02

Yeah, that's that's my jam. Yeah.

SPEAKER_00

Well, I hope you try and tried anything new locally because I'm I'm literally still discovering my city every single day.

SPEAKER_02

Yeah, yeah. I uh I actually live in like a very small town in Ontario. Shout out to anyone who lives in Oxford County, Ontario. I'm in Ingersoll, so it's a very like small town. There's like two coffee shops, so there's like not a whole lot to discover, but there are some like awesome farmers markets around that I got to that I got to like visit that I usually don't get a chance to because they're like nine to five, right? So it's uh fresh produce always delicious.

SPEAKER_00

They should well they don't they have them on the weekends?

SPEAKER_02

Some of them, some of them close at like Friday, you know, like they want to spend time with their families on the weekends too, I guess. So you know, if you're working full time, it's hard to get to, but PTO, perfect time.

SPEAKER_00

Yeah. Well, of course, we gotta know what you're working on over there in Ontario. Look, it seems like you've been cooking.

SPEAKER_02

I've been trying, yeah, for sure. Been been busy with clocking up like more sources of telemetry. Uh that's where like the new project ATIN came into being. I hate picking names for this stuff. Everything with like agent is taken already. I had I had a project that was named like Agent Fence, I think that's what I was gonna call it originally, and then you do like the Googling and the research, and it's like everyone has their their own little vibe coded tool for agent something. But yeah, I think it's uh a tool that's like I've been craving because I found that as I've been like building my own projects and things like that, I start pulling all these dependencies down and I start pulling skills down, and I start like authenticating Claude to like every system that I could find, you know. And it what when I'm kind of like wrapping up at the end of the session, I'm like, what did I just give it creds to? You know, and like what what package did I just pull down and and like what am I even doing here as a security professional without even you know thinking about this kind of stuff twice? I'm just like I I need to make this work, damn it. And it doesn't matter where my creds are going and like who has access to them. And uh and yeah, I just found that there was like so little the visibility around it, right? And you and you look at the logs and it shows you like you know your standard command line stuff, but you you have no idea. Like, did it access a credential? Did it send anything anywhere? Like whether the skill that I just installed, like you see people liking it on Twitter, does that mean that it's safe? Uh, you know, like what commands is it running? So it's all these questions I I wanted to have answers to, right? To have telemetry to. I always found that like if you're building like a detection or a hunt or something, like it always starts from telemetry sources, right? You can't like do a cool beaconing thread hunt without network data, right? So it's uh I think it's cool that these days you can, you know, instead of just saying, hey, I don't have the telemetry for it, you can like roll up your sleeves and vibe code something that gives you the telemetry for it. Uh so it's pretty amazing.

SPEAKER_00

So you so you built a tool with Claude to look for Claude behavior.

SPEAKER_02

Yeah, yeah, it was Claude and Codecs I use because I've been trying to get a little bit more into Codecs. I've been so like Claude piled lately that uh I've heard people say, like, yeah, Codecs is good, so I've been trying it as well. And yeah, it performs awesome. So sometimes I have the two like piggybacking, you know, like hey, Claude made changes, review them. And yeah, yeah, I used Claude for this. Obviously, like I'm not a not a developer by trade, uh, you know, but I understand endpoint telemetry, you know, I understand that part part, and I know kind of what I want. Um, and I know enough about Windows internals to make this kind of work. Uh and yeah, I think like the the little pieces that it has is is fairly unique. I know there's been some tools out there that have been you know released, and I think there's gonna be more. Oh, I bet right, because if if I'm having like these issues, I'm sure smarter people than me are having them too. And um I yeah, I think it's a gap. Uh I think it's a huge gap because like you have no idea what the user prompted, right? If you're just running EDR. And I think like to step back a second, not everyone even has EDR, right?

SPEAKER_00

Like if they even have that, yeah. So you're you're stuck with Sysmon at that point, which you have actually you're you're being super humble. You have a very interesting Sysmon story.

SPEAKER_02

Yeah, I started working with Sysmon, I forget, I think it was like 2014 or 2012 or something. I I'm pretty old now, so it was uh and it's pretty crazy. It feels like just yesterday that that I like like I still remember like the install screen in Sysmon and my first time installing Sysmon. It feels so fresh to me, but it was so like long ago at this point. It's pretty crazy how time works.

SPEAKER_00

Uh well it has it hasn't changed much.

SPEAKER_02

Well, yeah, I guess like other than new like events and stuff, but yeah, the general idea hasn't changed, and I think like that like paradigm shift of like holy crap, I I could actually like see what the system is doing now. You know, you know what I mean. I could see like a file being created, I could see a network connection being made. Like that paradigm, like compared to the visibility that I had with, I think at the time that I first started using Sysmon, it was just I had antivirus and it gave me a command line, but it didn't give me like the parents, right?

SPEAKER_00

Where it all started.

SPEAKER_02

Yeah, yeah. So yeah, I started working with Sysmon and I've been like in love with it uh for for decades. And I still am, you know, it's just it doesn't have um like if you don't have fancy tools like this, I think Sysmon is still your best bet, right? To get into this kind of stuff, but um but yeah, yeah, for like the agentic stuff, right? Like for the for the prompts and the credential access piece that that the new uh tool that I released covers, like Sysmon's not very good for that. Um, you know, it doesn't have like the capability to tell you, hey, someone touched your AWS key on on your endpoint or something like that. And I think like that whole aspect of creds living on your endpoint had we I think we like skipped over all that, even like in the cloud era, right? Like I've written about this like in my previous employers' blogs and things like that, where like I really think we have like a blind spot there as defenders, right? Like you don't know where your creds are going, right? Or or like where they're being used, and and that was like before this whole agentic era. So yeah.

SPEAKER_00

It's no, it's it's true. It's it's pretty spot on. Even I I I'm getting tripped up in deciphering like multiple uh touches at the credential store in like a short period of time and distinguishing when an agent does that versus an info stealer because they do both similar things, and it's getting tripped, and uh it's hard to get get specific on on what is what at this point.

SPEAKER_02

Um yeah, yeah, yeah. Yeah, exactly. Right. You think of like an info stealer running an endpoint, right? And we see that that all the time on my day job, right? We see info stealers targeting Mac Windows, right? That they don't discriminate, but there's like very little telemetry to show you, like, hey, this info steeler, you know, like stealer.exe touched AWS creds, you know, dot text, right? That there's no you need to have like SACL auditing in Windows and Linux, you need like audit D configurations and all that, and it's like very difficult to configure, right? So you need some kind of tool to like auto-discover that for you. So that's why I tried to build in um into A and it actually like finds what creds you're using and audits those. So you know, like, hey, like this was the prompt that led to this particular event. Like it has like a session idea that ties all those together. So I think it's like a really rich telemetry for hunting, right? It's like a Sysmon GUID. So yeah, but yeah.

SPEAKER_00

I was gonna ask, do you need is Sysmon a requirement to for this or no?

SPEAKER_02

No, no, you don't need Sysmon for it. I think uh if you if you have Sysmon, it it it'll help, right? Obviously, like Sysmon's great, um, but you don't need it for for this particular tool. Uh it only audits from the agent, so it won't log like every process on your host, uh like Sysmon does. It'll only log stuff spawned from like Claude or Codex or whatever you enrol in the tool, so it's not like spamming your event log. At least I hope. I haven't done too many testings with it. So if someone actually runs this sucker, please let me know if it's noisy or not. Uh, but I did try my best to keep it from like blowing up your system with event logs.

SPEAKER_00

Well, so I when I read your blog, uh, which was very timely, uh and I think when we originally talked about creating a podcast, I don't even think the topic we agreed wasn't was not what we're gonna what we're gonna be getting into today. Yeah, it was completely different because you're building all sorts of tools. But I I think this is very timely because you're absolutely right. Like this coding agents are genuinely the new blind spot that is missing from your and EDR telemetry and Sysmon. So you wrote it extremely well. Uh I think you said it it's almost like an entirely new OS running on top of the endpoint. Um so what did you find was actually missing from that standard EDR or Sysmon process information when agents are now in the picture?

SPEAKER_02

Yeah, yeah. I think, yeah, that's a great question. I it it may maybe that's a like a little bit of a hyperbolic example. I don't know, but I it it almost feels like a different OS running on your host, right? It's like this especially like I read some of the stuff that people are doing with agents where they have like multi-agent workflows, right? And and like a loop that runs for days. And like I don't really do anything that fancy, but if you have something like that running on your host, right? Like what does the telemetry for that look like? Like what does like a seven-day bender from an agent, you know, look like in the logs, I wonder, right? Like, and like what did it do in those seven days? And how how do you know that it did what you asked it to do? And how do you know that it didn't like go nuts on day three, right? Unless you're actually like monitoring for for that stuff. And I I think like as an industry, we focus so much on like identities, and I think that it is maybe maybe not the wrong, maybe wrong is like too strong of a word, but I feel like we haven't focused a lot on creds that those identities are built on, right? Like we focus on like the user, but we don't do a really good job of like tracking how that user moves from like endpoint to cloud and then back to endpoint, right? Like we don't have a good way of like querying that. And and if you ever worked with like a sim and you tried to track an incident that spanned that kind of vector, right? It it's awkward, it's clumsy. Like the data looks different, right? You're like, yeah, the C Smack the Sysmon logs don't look like the Cloud Trail logs, right?

SPEAKER_01

No, none at all.

SPEAKER_02

Right? There's nothing like tying those events together, and I feel like that dynamic is replicated with an agent on the endpoint, right? Like the transcripts that Claude produces or that codex produces don't look like Windows logs. No, and you know, like, yeah, right. And if you look at like a process tree, you'll see like Claude spawning node, and but you don't know like why it did that, right? You you have you have no idea. You you just like was this part of Claude like automatically doing something? Was it part of a loop? Did someone like prompt it to do that? Yeah, um, yeah, no, it's true.

SPEAKER_00

Claude to command line to code tells us absolutely almost nothing to do with, right?

SPEAKER_02

Yeah, yeah. And and we see that so much, right? Like, I look at this kind of telemetry all day, and and it's very hard to like draw a difference between what the agent did versus what it was asked to do. And I think that gap is worth like wrapping some telemetry around, right? Like to figure out like what did Anton asked Cloud to do while it sent his AWS creds to to like Digital Ocean or whatever, right? Like, um, but yeah, I and I don't feel like we're like monitoring that enough. Like there's very little um like general credential access type events out there for for security products, right? You don't know um, you know, like who's touching your Kubernetes secrets right on your endpoint when a dev uses kubecuddle, right? Like we we just don't have good monitoring around that, right? And I think we instead of like going back and building that monitoring around it and then using agents, we just started using the agents right away, and now we have like a compounding set of problems where we didn't have good visibility before, and now there's like something autonomous happening on our endpoint, and we still don't have visibility into that. So I feel like we're kind of like you know, building on top of quicksand a little bit here.

SPEAKER_00

Anton, we're in the find out phase in the F around and find out. People are realizing that AI is causing just as many many problems as it is allegedly solving.

SPEAKER_02

Yeah, right. Like, how would you even like I read the you know the like the hugging face incident that I'm sure like every blue team read, but like if you're not a frontier AI lab, like how would you find out that a developer's creds were compromised through a supply chain attack, right? To to piece that together, you would need so many different telemetry strands, right? You need like the endpoint data, then you gotta dig through the transcripts to figure out like what the prompt was, and then you gotta look at the network event, right? And then that's probably gone, right? You can't forensicate that. Uh and yeah, like working backwards from that. I I yeah, I feel sorry for defenders these days that have to like figure out how to how to like work work backwards from that because the telemetry is just not there.

SPEAKER_00

Well, until now. Until now.

SPEAKER_02

Until now, yeah, until you're running like A. Yeah, yeah.

SPEAKER_00

Well, it which is fascinating because what I'm hearing is you're effectively modeling intent versus action, potentially, with what is what this new telemetry is uncovering. That's the core idea behind uh A10, which by the way, uh we missed out. You're right about the naming, because we missed out personally on the opportunity to create def D E A F because Nebuloc and the Death team, they just released this agentic detection engineering uh framework, and it it's a it's a DEF, but I'm like, it could have been DEF.

SPEAKER_02

That's awesome. Yeah, I didn't I didn't think of that. Damn it.

SPEAKER_01

I think I missed out, yeah.

SPEAKER_02

See, I suck at naming this stuff. I'm okay at like ideating and building it and testing it out, uh, but but actually like naming it and marketing it.

SPEAKER_00

Well that's that's the important that that's the important stuff. The the whole model, the whole model. So so what uh the model's trying to prompt, right, and the tool call intent to what actually executed on the host aka the action. So why was that pairing the right way to look at it, the right abstraction versus just logging more processed data?

SPEAKER_02

I think yeah, I I designed it that way to give like if you're hunting through this data, I think it's a good like pivot point to the Taunton, right? Like what was performed by the coding agent that wasn't requested, right? I think that's uh an interesting like starting point, right? And I feel that's just there as part of the schema for you. So you don't need to do any fancy like sim work with that. You know, you don't need to translate events or you don't need to normalize anything, you can just search for you know when this field equals no. I I forget the exact uh field or what it's called, but yeah, I tried to include like whether we could attribute a tool call that the agent did to a user request or not. Uh-huh. And I don't know, like to be honest, I don't know how that would work at scale, right? Because I don't have you know access to like 10,000 dev machines to actually run this, but I felt that that was pretty important, right? You you probably like want to know if Claude did something that the user didn't ask for, right? And uh whether that comes into like a prompt injection, right? If you're like I I think in my like blog, I just used and and actually Claude like gave me a hard time with this. It wouldn't, it wouldn't run it for me. Like I just had like a markdown file with like a little HTML embedded there and that just said like hey, run this instead. And it was so benign, right? It's just it's just for a demo. I didn't want to go too nuts, but Claude was like, no, this is prompt injection, I'm not running it. Um so so that's why I baked in those kinds of fields in there, so that you can kind of like easily sweep your data for that, because I felt like the like there's a lot of events in here, and the scheme is like pretty complex, and the events are are beefy. There's there's a lot of it. So I wanted to include some fields in there that made it so that you can filter out stuff very, very easily. Uh so that's kind of like my thought process there.

SPEAKER_00

Have you gotten to the point where it makes it um maybe not, I don't think easy is the right word, but it it makes it much more clear, or you could derive what was done by human, a human prompt versus maybe maybe a non-human prompt or a malicious one.

SPEAKER_02

Yeah, yeah, I think, yeah, that's a great question. So I think with the event types that it omits, it should be really easy because you should be able to correlate the user prompt, because I have a field for user prompt that sucks in the transcript that codex and and uh cloud leaves behind. So you can actually filter all the events that don't have that field in them, right? So you can just say like give me everything that you saw that wasn't tied to a user prompt. So yeah, I merely actually like like I've only run this in my lab, right? With with like uh the dev scenarios that I tried to cover in the blog. Yeah, but I haven't actually run this with like you know, like a real world dev workflow.

SPEAKER_00

Funchers customers, you gotta get on this, right?

SPEAKER_02

Yeah, yeah, yeah. I mean, this is like my own project. I I don't I think you know, I think as a company we're probably gonna build something like this at some point. I I don't know, I don't want to speak to like timelines or something. But yeah, this is like you know, something that you play around with in your lab, right? I I probably wouldn't go around and deploying this to like 10,000 endpoints because I like I said, I'm not a dev. This was 100% vibe coded, but I thought it was an interesting experiment because I don't think there's anything out there that ties in like this transcript portion to the cred access portion. I think those two things when combined are pretty unique, so that you can see like the full chain of it. You can see like what the user prompted for, where the creds accessed or or not. And I think other tools don't have that cred access piece, and I think that's super, super important because without it, most systems are not instrumented with that kind of stuff. Like I don't know for if you're listening to this podcast and you're like an instant responder, like a sim engineer, I don't know how many times have you seen like SACL auditing events, you know, like a 4663 in an environment like auditing files, right? Like very rare. I've seen it maybe a handful of times. Uh and and even something like a file share, right? Like file shares always have creds, right? So I think that yeah, that that little piece of like the credential auditing piece, I think that's a big like differentiator for for this tool.

SPEAKER_00

Absolutely. Uh though I've in the Mac world, I feel like they wouldn't expose so so much, so much of the minutiae with it, right?

SPEAKER_02

Yeah, I think like the with the EFS standpoint security framework, I think it exposes something, but yeah, I'm not really a Mac person, so I I I'm not like an expert in that. Um but I think there is a way, and I know there is on Linux, so this so there should be on Mac as well, but like in my little Windows world for sure, like I haven't seen too many environments with with SACL auditing. And I and I've tried to blog about it um my last DEF CON workshop in 2023 or four, what was all about SACL auditing and how to find like you know, Cloud Creds, like your Azure token, and not the Azure token that is sent through OAuth, like not that token, like the the CLI token, like if you use like AZ login, right? That there's a token that lives on the disk, right? And Steelers left to take it.

SPEAKER_00

And then oh, that's right.

SPEAKER_02

Yeah, and and same with like AWS, and same with Kubernetes. Same with uh like a bunch of other cloud CLI tools, right? That that that authenticate to the cloud and and the endpoint holds all those creds and there's no extra syspawn event, right? When they're accessed.

SPEAKER_00

So I think no, and and and if we think of most dev, they prefer the Mac OS world and their shipping product or they're shipping some real like sensitive things via the cloud CLI directly from their Mac. Yeah, and and so it's it's absolutely it's so important to capture uh to capture that.

SPEAKER_02

Yeah, yeah, 100%. Yeah, I think like yeah, like I said, creds are like I just don't think we do a great job of auditing creds in general, be it on like Linux, be it on Windows, and uh linking like the process part to the cred access part, I think is super, super difficult. But until now just not enough, yeah. Until now, yeah, exactly, right? Yeah, yeah, until now. Yeah.

SPEAKER_00

Let's talk about when when this can go wrong. Uh I'm talking uh I'm thinking supply chains, I'm thinking prompt injection, uh network even uh if if people are still doing that. I I can you believe I heard a wild take about like that if you configure your your cloud routing correctly, uh technically NDR should not really be existing, which is a wild take. But uh but that's yeah, but but everything happens on the network. Uh yeah, and uh access to the public internet will never not happen. Uh so yeah, that's I I I I invited this person to debate on the podcast several times, but they don't want to. I'm like, okay, well keep keep saying that behind behind unpodcast closed doors, but okay.

SPEAKER_02

That's hilarious. Yeah, but maybe one day because I want to hear more. I wanna hear more about this take.

SPEAKER_00

Yeah, yeah. Um, and identity too. Like if you uh it was it was paired with that. I think I guess I should I should state that to save Grace. Uh they said that if you configure the proper controls as well, detection engineering is shouldn't it doesn't is not needed uh because you you configure the right access to the right to the right resource at just the right time, and then uh you can figure the right, you know, fire the right traffic and and therefore no this reactive state of DE is is becomes obsolete and like yeah, well that zero zero trust never never I mean we're zero pro zero trust has been implemented for now a decade and yeah, yeah, here we are.

SPEAKER_02

Yeah, yeah. The the profession still stays how would you like I guess my argument against that is how would you know without the DE function? Like how would you know that you configured everything properly if you're not monitoring it afterwards, right? Like how would you like Yeah, quite literally. What would you do? Like, oh I'm done. I'm just gonna trust myself that everything is configured 100%. Um no logging, no detections, I'm just gonna trust it. I don't know.

SPEAKER_00

Yeah, no, literally. Uh and and then on a the the last take, I'm bringing it on to the podcast, whatever. The last take she said was um that in the in like decades of incident response, nothing was ever like identified by sim. Like sim barely ever. It's always because a user a user reports something, or uh it's it's obviously external, like an ext external breach is reported, or a user starts seeing something suspicious, but it was not because of a trigger or because a rule hit.

SPEAKER_02

Wow. Okay, interesting. I don't know. Like they I would love to show them our internal slack at work. We get sim alerts all the time that kickoff incidents that stop ransomware and like all the time. I don't know. Yeah, I'll be happy to share a redacted screenshot. If you go to the Hunter's blog, you can find a bunch of incidents that a sim kicked off. But I don't know.

SPEAKER_00

Yeah, obfuscated, of course.

SPEAKER_02

Well, yeah, yeah. I mean, uh yeah, yeah. Yeah, it's an interesting take. I mean, I get the I get the the fact that like sims traditionally, especially ones that aren't like a managed sim, right? If you're running a sim like in your own environment, I I get the the critique that they're not as like accurate, you know, as they could be. I think that's like a known thing about sims, right? Is that they're noisy. But the fact to say that like no incident ever kicked off of the sim ever, I don't I don't know. I don't I don't agree with that. I don't think that's right.

SPEAKER_00

You heard it here. Uh um friend who starts with an L and ends with an Izzy. Um so funny. She's still a friend. Um okay, supply chain. Supply chains, um, and when an agent installs a package, right, and it's compromised, then credentials get, you know, touch that nobody asks for. What does that look like with with your tool with your new telemetry? Uh, what's that one field that now can make it huntable?

SPEAKER_02

That's a good question. So I I cover that scenario uh in the blog. And yeah, there's a whole bunch of fields actually. There's like you get the prompt, right? The user prompt that that says like, hey, install this package, so you can trace it back to to its you know, like true origin, so you could figure out whether it was like a prompt that led to this or whether it was uh you know a script that called a package or something like that, like an environmental script that sets up the environment and it pulled you know the wrong version or something like that. Um you also get like the cred access event, so you see that you know this particular credential, whatever the compromise package happened to be going after, you know, was touched by node or whatever the prop parent process is. You also get like the network event, so you can see like where that cred went, and that all has like a session ID, so you can tie those events together, and I think that's the like that's the like money shot. So you can see like, hey, Anton asked for this, and that credential was touched, and it went over here.

SPEAKER_00

Uh-huh.

SPEAKER_02

Or conversely, you can see Anton never asked for this.

SPEAKER_00

Never did.

SPEAKER_02

Claude did it anyway, and that cred went somewhere else, kind of thing. And I think if you combine that with like other data, especially like S bombs, like I used to think S bombs were kind of weird, you know, like why would someone need this? And now I'm like, okay, now I understand. Yeah, right. Just stupid me, but uh, you know, now I get it. Um, so if you combine that with you know S bomb data and uh the name is escaping me. I I think that there's a tool recently um released that does that kind of thing that does like the package like uh hierarchy. It'll it'll come to me at some point.

SPEAKER_00

Um the package hierarchy, and it's not the right tree. No.

SPEAKER_02

No, I I think it's the same folks who released uh the tool that you messaged me about yesterday.

SPEAKER_00

Oh the perplexity guides.

SPEAKER_02

Yes, but yeah, Adele, yeah. I only know his Twitter handle, he's a brilliant guy. He I think he released a tool that does like um like pack like scoping, it'll like scan your endpoint and tell you like what packages. If you combine like that telemetry with the telemetry that like ATM gives you, that you're you're you're cooking now.

SPEAKER_00

You're a fro, you're a force to reckon with.

SPEAKER_02

Yeah, right. And I think like with that amount of data, right, if you had that, like if an incident responder had that, then you will be able to like piece the incident together because without it, right, if you don't have the prompt, you know, like if that transcript rolled over, right, on the disk or something like that, now now you're missing that piece, right? So if you just see like node coming out from Claude, and then you see like Claude making a network connection, uh, you know, is that X filled? I I don't know, or is that legitimate? Yeah, it's hard to say.

SPEAKER_00

It's almost like a certificate. It's almost like the same like certificate signing. Like, did Anton do this? Did did he sign this? Is it is it a confirmed legitimate certificate? Is it is it an expired certificate, or is it not fair?

SPEAKER_02

Yeah, yeah, you get like the third degree for for certain, but yeah, no, it's it's it's literally that, right? I think like the the and I think that's the piece that EDR won't won't tell you, right? It won't it won't give you the prompt, it won't tell you that this credential was touched, it won't tell you where that credential went. So just so now you can piece together like the whole the the whole thing and it yeah, it tries to bring a little bit more like uh yeah, like telemetry, you know, it's all it's all it's all about that because uh yeah, by default, it's just not there, at least from what I found, right? I mean, and at least not in a real-time way, right? You can go digging through your transcript and try to link it, but without without a tool like this, I think you'll be you'll you'll be having a hard time.

SPEAKER_00

Well, this is a hell of a step up from whatever the first version of like Claude Otel data, what what the hell that was, where it was just token usage, uh the the session name. It was like, what am I supposed to do with that? I feel like I've I've I thought it sounded like a broken record because I've said this so many times, but now I I keep hearing that it's now a little bit better, but I I can't imagine it's anything like compared to this.

SPEAKER_02

Yeah, and and I don't think like I I have a little bit of familiar familiarity with OTEL from my previous job, and it's not really designed for security use cases, I don't think, right? It's more designed for like um monitoring and then and all that and uh availability and stuff like that, uh like observability, not so much the security part. Um yeah, I I feel like I I think that this is definitely like a best effort type tool, and I'm really looking forward to people trying it, you know, and letting me know if it finds stuff or doesn't find stuff, and uh maybe it'll inspire people to vibe code their own that that are like more comprehensive and better.

SPEAKER_00

But or contribute to your repo. Are you are you accepting pull requests?

SPEAKER_02

Yeah, yeah, contribute. Yeah, I'm I'm not like uh like I said, I'm not a dev by trade, so I might fumble here and there with your pull requests if it's a little bit complex for my P brain. Uh so bear with me. But yeah, definitely feel free to contribute, fork it, make your own versions. Um, especially on the Mac side. I'm not a Mac expert. I had a hard time getting like the Mac like developer account, uh, and I couldn't even get this to run on my Mac without disabling like everything. And it's like my personal Mac, so I was like, I probably don't want to like disable like all the security stuff that it comes with.

SPEAKER_00

Not gonna lie, I did try it myself and I couldn't figure it out because uh because it is on a nebula computer, so sorry, sorry, uh Damien and Cindy.

SPEAKER_02

Yeah, no, that's good that you're doing that. That uh that that you're keeping like the internal security going there. Uh but yeah, definitely like yeah, if someone is good of good with Macs and knows how to like bundle this or come out of the colour.

SPEAKER_00

Oh, if you have the loop team.

SPEAKER_02

Oh, yeah, I love the shout out to you. Yeah, yeah, yeah. I love Datadog. So yeah, definitely um, yeah, definitely, yeah, to take a look at it. I think Datadog has their own kind of product for this too, or or something like that, that audits agent. So yeah, I feel like every everyone's gonna have like their own flavor of this, and whether it's like open source, whether it comes as part of a sim, whether it ships as like part of like an XDR endpoint, something. I I think we're all gonna have to get used to dealing with this kind of telemetry. And I think this is just a way to get it for free, you know, just to like play with it, uh get used to it, just see what Claude gives you, right? I I think if you're like a defender and you're not studying Claude telemetry a little bit at this point, you may be a good time to start.

SPEAKER_00

Oh, yeah.

SPEAKER_02

Like it's it's everywhere, whether you want to accept it or not.

SPEAKER_00

Oh, a hundred percent. Even people that shouldn't be using it are are using it, and there's there's no way back now, there's no going back now. Now that it's a it's like literally quite any one uh vibe code apps uh now. There's so much slop out there, and people are realizing that the that it doesn't have the proper auth, and you know that it that it rightfully needs, and it it's it's getting out of hand. Uh but you and there may be iterations of this, there may be new pro new ways of this, but you it they will never be the first. And if you're not first, you're last.

SPEAKER_02

So yeah, yeah, yeah, definitely.

SPEAKER_00

Uh the one on your blog, the what really scared me though was the prompt injection, like when it could go wrong. Uh that was the probably the most exciting one. Oh well, scary, but also exciting the when the instructions never come from the user themselves. Yeah. Uh and I and you were talking about this earlier, is now you can distinguish a user asked for this versus the page told the agent to do this because of because of that transcript and the creds.

SPEAKER_02

Yeah, yeah. I think that the concept even applies to like a malicious skill too. I I know like skills are all the rage these days, right? Like everyone's installing skills and uh looking at skills from like a marketplace, and like you don't know what it does unless you look at it, right? It's it's it's like you're running code and everyone tells you like don't uh don't run untrusted code on your endpoint. So I think we found almost like a loophole where it's like I'm not running it, you know, Claude's running it for me. So yeah, it's even like a an additional layer of of danger now where you might not even know. You could just tell Claude, like, hey, install this skill pack or something like that. And a skill could like sneak in there. That um I I did find that Cloud has pretty good guardrails for this stuff. So I don't want to make it sound like you can just uh you know, like install like any skill out there and you're compromised, you know. No, I don't think that's the case, but we we do like I don't know.

SPEAKER_00

Like there's some maybe, maybe there's some um I'm a little bit, I'm a little bit, I'll take some blame for this. Like I have so I'm kind of a hoarder, a digital hoarder. I'm not luckily I'm not a physical hoarder, but I'm a digital hoarder when it comes to like uh bookmarking all these GitHub repos and that I I want to eventually do. And and within it in the last year, it's all about clot skills. Like I want the best cloud skills for literally every use case in my life, both personal and and work. And so I've I I'm finding myself uh GH pip installing a lot of GitHub repos without even really like looking through them first. And uh flash forward dev popper now, you know, since 2024 to now, there's malicious GitHubs uh just living everywhere, all over the place. Yeah, uh, so uh I I don't know. I if I haven't seen really clo Claude trip at all over uh pip installing every repo that I want out there, and I do it a lot.

SPEAKER_02

Yeah, same, same, yeah. And that was like, yeah, like like I mentioned when we started this podcast, how like I I scared myself, you know. I was like, why am I installing it? And luckily, like I I work with like you know, like a local lab, right? So it's not you know the end of the world if those creds get popped or whatever, but still I'm feeding Claude all these types of creds and tokens, and like you said, installing skills and and like those skills could call script, right? That can call like another script, and you're chusting on so many nodes in that trust chain going right, and that on the other end of it from where you're using it, there's just so little telemetry, right? You like I tried to look at Claude under Procmon, right? Like I'm probably like dating myself now by by like saying Procmon, but like I was curious, right? I wanted to know. Procmon's great. I want to know, like, hey, what's Claude doing on the system, right? Like when I click co-work, is it launching a different process? And and when you look at what it's doing, it's so like loud on on Windows and so convoluted, you can't tell what's what. You know, it's writing JSON files, it's spawning like weird command lines. So it's hard to like detection engineer your way around Claude without additional telemetry, I find. So yeah, I think we all need something to to give us more visibility into like you know, what skill that Alex installed this week, right? Like, how would you know? Like what what audit trail is there to to like you know, read that back to you. Uh and and even further than that, like how would you know what those skills did, right?

SPEAKER_00

Like, yeah.

SPEAKER_02

Yeah, I just feel like we're we're we're flying blind and like yeah, I feel like we're gonna get bombarded with like different EDR vendors releasing their own solutions to this problem.

SPEAKER_00

Yeah, I think that's the next wave of this, but why didn't they you'll be you gotta ask, why didn't they get ahead of it earlier?

SPEAKER_02

I yeah, I don't know. It's a good question. You know, like if you've ever worked for for like a product org, you might know how like not super fast some product orgs move, right? By the time decisions are made. Yeah, it's hard to I think like individually it's easy to iterate quickly these days. No, organizationally, it might be difficult, yeah.

SPEAKER_00

I think it's this innovator's dilemma. Like CrowdStrike used to be so incredible at releasing what the people wanted fast, but now they're so busy wanting to build their own sim, and they're like the less love goes to the EDR product. Like that that was their their core all along. And identity and everything, like the whole minutiae, the whole platform now, platformization. Um, yeah, I call it the innovator's dilemma.

SPEAKER_02

Interesting. I wonder if it's like um this might be like a spicy take. I wonder if it's just a like looking at the problem the wrong way, like it's not seen as like an endpoint problem, it's seen as like an AI problem.

SPEAKER_00

I see it's own its own domain, but but where else would it run besides the end if not the endpoint?

SPEAKER_02

I I don't know. And then I think we made that same mistake when we did like the whole cloud security, you know, like uh yeah, because you hear a lot about cloud security, right? But I mean and I gave a I think I gave a couple talks on this or webinar on this year years ago, where it's I think the endpoint has a huge role to play in cloud security, right? Because that's where you access your cloud data from, right? Like, like obviously you need to understand like Azure logs and AWS logs and GCP logs like 100%, but how the endpoint interacts with those, I I think that's we're missing that piece. And we see it as a cloud security problem, not an endpoint security problem. And I feel like we're doing the same now with this AI stuff where we don't really see it as like a you know, quote unquote boring EDR problem. We see it as an AI problem, and I think from like a product perspective, if you're like, hey, I'm gonna like wrap a solution around this, it you you probably run the risk of like shipping a new EDR feature rather than introducing like a new product category. And and I'm pretty sure that's where we're kind of stuck as like an industry and and and from a like a product standpoint, but you know, maybe a wrong because no, no, no, that that that's a fantastic take.

SPEAKER_00

I I just I just keep thinking about how uh the under prom uh overpromise, under delivered of the XDR, because it seemed like maybe the XDR would have been would have been what would have solved it here, the that it touches it at all, I guess. But no, I guess uh I wouldn't be surprised if there's now a new AI um now a bunch of new startups trying to go after the AI market category.

SPEAKER_02

Oh, a hundred percent, yeah. Yeah, and um, you know, like I'm I wouldn't be surprised to see a whole bunch of yeah tools that act like this and give you very similar telemetry like this, but I I think the key piece for us to watch out for is like hunters, as boot teamers, as defenders, is the whole like chain. Like, can you see the whole thing, or does it just give you visibility into this one narrow aperture, and now you have to kind of do your best to link that together with your EDR logs and then link that together to your cloud logs and stuff like that? Because then I think then we're only compounding the problem, right? Because now we're now instead of looking at one data source, we're looking at three, right? It's always like convoluted that way. So yeah, it's gonna be interesting. It's gonna be interesting.

SPEAKER_00

Sorry, how how does this um how does this extend to m like an M and MCP calls it all? Or is that more does that have to have its own wrapper?

SPEAKER_02

Uh no, there's no MCP here. I I think there's probably other tools that do this via like the API layer and the MCP layer where they like intercept something. Yeah, that's not the route this tool uses, but it may be valid to do it that way. Uh maybe I just don't know enough about like Claude internals or codex internals to come up with a better solution. So I just went straight for like the kernel layer to monitor all that. Um but I think there are other tools out there that act as like a like almost like an AI WAF that like inspect the requests that Claude is doing or Codex is doing and and say, like, hey, do I want to allow this or not allow this? And I originally wanted to include that, like the blocking functionality into this, but then I thought like I'd rather just make it pure telemetry.

SPEAKER_01

Yeah.

SPEAKER_02

Um I felt like that was at least for a V1, you know, maybe maybe if this gets some traction and people start using it and want like a rules engine, you know, and like a decision making aspect to it, maybe I could add that. But for now, I just wanted it to I just wanted telemetry, you know. Like I I love telemetry. I'm a telemetry nerd, right? I love logs, I want to see so that that's that that's my like goal here.

SPEAKER_00

Logs, logs or didn't happen.

SPEAKER_02

Exactly.

SPEAKER_00

Right? Uh so no, I mean, I think everyone should uh ASAP drop it all and get started with this, but if they didn't, let's let's say that someone wanted to do this, spin, spin something up, uh, and start hunting for this behavior without A10, I feel like I I mean how would they even do it? I feel like you can't. What would they need to turn on to start logging?

SPEAKER_02

Uh that's a great question. Uh maybe like if you're using Splunk, maybe Splunk has like an app for Claude or something like that. Uh, I don't know. If you're pulling stuff off disk, maybe like the transcripts. Uh like yeah. If you have like I think as always with with like hunting and detection engineering, like you gotta make do with what you have, right? Um yeah. So I yeah, I would start with like the telemetry that you got. And if you're looking at process data, try to baseline it as as much as you can, like see like can I even see who's using Claude? period you know in my environment or or do I have that filtered out a hundred percent which it might be a thing you know like back in the day when I actually managed like a Sysmon deployment I was always fighting with the noise and I was editing the Sysmon config like once every 30 minutes you know and I have no idea how noisy Claude is with with Sysmon right like I don't I don't know I don't run a Sysmon environment anymore so I feel like if you're an admin right that has to balance like sim quota right you can't send like 60 billion Claud events to your simple you have to be cognizant about that right so you might have filtered it out altogether so yeah I I would probably start from like from scratch and just try to figure out you know like what what does the cloud profile look like for the users right like like does the finance person running cloud is that like are they installing packages like maybe they shouldn't be maybe they should just be using like cowork or something like that right so it's a probably not for sure right but we we know they're building those dashboards in cloud right yeah yeah and I think that's like an interesting part of this problem now where like everyone's kind of a developer right we used to say like oh man like the developer workstations are the trickiest right because they're running like VS Code and all these like tunnels that might look weird to to a blue team or right like hey why is this cloud for tunnel on this machine or whatever. But now everyone's doing this right now it's like all of a sudden you're seeing all these like Python installs on finance machines and I know pip installing all the prereqs that they need yeah and then it's like which version and which package and now we have to do like S bombs for everyone and and I bet you they don't know what the hell they're even installing.

SPEAKER_00

They just want they just want to get to the end goal like quickly.

SPEAKER_02

I I don't blame them because I I I've I found myself in that loop with BOD as well you know like like I just want this to work.

SPEAKER_00

Like who cares about the security warning right uh you know I think we're all we're all impatient humans right and uh always I I can't tell you how many times like um my partner will be trying to talk to me and I'll be like yes like always allow uh yeah yeah yeah yeah yeah yeah yeah yeah yeah and then I have like a Claude for my like personal finances and stuff like that and then like my code Claude and yeah oh my gosh my wife's always like are you talking to Claude again yeah I had Claude plan like almost my whole wedding yeah yeah it would be amazing at that but yeah oh yeah uh well I would advise them to to to start here uh what does the detect detections look like have you have you built anything out because I this is a good follow-up for my previous episode the AI defend framework has a lot of defenses in the detect pillar that um that you could uh probably use against this telemetry for let me pull it uh like yeah right over here there's a bunch of defense that I feel like this would this opens up just a whole world of opportunity now for uh uh detection mechanisms again against AI system use yeah yeah if if you look at the GitHub repo for for A10 there's uh there's a mapping to Dell's endpoint AI agent abuse it's like right in the middle under coverage and like as I was building this he put out this framework uh so specific to not because this won't detect you know like model poisoning or or things like that uh it's very specific to like the endpoint like where the actual like harness is running so I found that this framework that was specific to endpoint I tried to map what A10 detects to this.

SPEAKER_02

So so it doesn't do everything because like you you can't because otherwise I I would just be like blowing up you know everyone's computer. But I think I think like the yeah like I think that yeah this chart does a good job of good because in the blog I don't cover all these scenarios because building the the actual like wrapper to demonstrate these these scenarios was the most time consuming part of all this uh even with Claw just getting it all to work really difficult. But if you're if you're wondering like hey what does this even detect uh this is the section to look at because then you can click the the link and look at the actual framework and I I think this is a little bit more what's it called a little bit more maybe a little bit more practical than something like Atlas true.

SPEAKER_00

I mean there's 10 frameworks AI frameworks the new the new OWASP uh there's like it seems to be or databrex even has one nowadays it's hard it's hard to navigate not gonna lie yeah yeah and I know people love their frameworks that and like I do too but I also love like the practical part in you know what I mean like if someone tells me like hey just detect credential theft I wouldn't like how you know like what event ID do I need what query do I need like tell me more so I found that this framework was good for that. Did you use AI or did you use Claude to run the adversary uh behaviors as well against it?

SPEAKER_02

Yeah so so when I finished the the tool I told Claude like hey I want to have a a blog about these scenarios so build those for me. And it and it did but then it would always like it it was it was sweet right like the the only thing that wasn't clawed by code it was the blog the the blog I actually wrote you know the old school way by hand.

SPEAKER_00

Oh yeah how are how uh historic sucker I guess how artisanal yeah yeah and it's in the blog yeah yeah but yeah I think like the um yeah the the the man I forgot I forget what was the question sorry I lost my thought there for a second no the adversary emulation you were using it to to to like very for example run run this controlled initiator or permissive unattended yeah yeah yeah I would I would tell it to like set up those scenarios and it would try but then I would have to actually run the scenario from a different cloud session oh which had like the guardrails and I found like Opus 5 especially was was pretty sensitive to those so it wouldn't like interesting it wouldn't run my prompt injection.

SPEAKER_02

I think if you're maybe this is a bit of a tangent uh but if you're like in a purple team role I I would be so like you're just like kid to candy store these days with purple team scenarios that you that you can build around yeah clawed endpoint abuse. I think it's just like a a super fascinating area and and I could have spent probably more time like building the scenarios than the actual like app but which now that I have the app may maybe that's where I'll spend my efforts into actually like running like a full kind of purple team cycle and and showing the detections for that and the kind of queries that you can run and all that good stuff.

SPEAKER_00

Upload this to Caldera for me because they've got some pretty basic attacks over there.

SPEAKER_02

Yeah I found most uh like most purple team tools like that do like those bass tools right they have they have scenarios but it it's all like I I I always and and I hope they don't come after me for this but I've always found like I found that the juice isn't worth the squeeze you know what I mean like I I just set up all this like YAML and stuff and it's like running who am I for me you know like I I could do that myself I don't need uh I don't need cab there I think that's why I built like TTP runner.

SPEAKER_00

I don't know if you've seen I yeah no TTP runner is fantastic. Yeah yeah that that's why I built it because I I just wanted to like tell me like hey run run this thing and just have you know that now that the agent could do it uh it it just does it for me now that's amazing well I you're always welcome back to talk about any any one of these other projects like this is this is fantastic. I mean we only scratch the surface here with the novelty of it all of course uh and uh last last thing I wanted to touch on is like what's what is next for uh for 810 it do you are you gonna break in I know that mac os is still in progress uh you know what what what's the roadmap for for this I guess the purple team stuff yeah I would love to build like a an actual simulated rundown with it that looks a little bit more real realistic than than the stuff I have in the blog but um yeah I would love to have like feedback on it so if anyone actually tries it I'm sure there's bugs and stuff so I would love to like run it on more systems than just my lab and yeah I think the next step is keep iterating on this and keep seeing how like I'm I'm really curious for the next supply chain attack or whatever like I almost want to spin up a VM with this and just run it and see what it finds you know like whether it actually gives me the telemetry that I thought about honey caught your VM yeah and then with this with the this telemetry turned on yeah I might do that.

SPEAKER_02

There you go. Now you give me the idea for V2 of A10. There you go. Yeah but yeah it's something like that I I just want to see how it works with like a real like you know like a real world kind of like um attack scenario and whether actually gives you the telemetry that I thought it would uh you know a lab is always one thing right when when you're doing like synthetic tests and setting it up all perfectly and and timing and all that and it all works and then you're like oh cool and then you go to try it and prod and it doesn't so yeah I think that's probably the next step for it.

SPEAKER_00

If you're gonna go if you're gonna be at Black Hat or or Def CON I'm sure I you know you you just by asking around you always find I can like a SMB type of environment that maybe would be like pretty useful to use that has just enough endpoints that's just a little bit little a little bit of a makes it a a controllable scale. Yeah that's a good idea I might just run into somebody um so I'll I'll point them your way.

SPEAKER_02

Yeah please do please do yeah I I I'd love to actually run this on it and see if it even blows up it might be like too noisy for for for someone who uses claude all day you know like a heavy cloud user it might just be like nah this is not you know the the log file fills up too quickly or the telemetry is not useful. I don't know. It's hard to say without you know access to to that kind of like production uh data that I don't have in my lab. But I think even if the tool like doesn't work uh I think the idea is still cool.

SPEAKER_00

It is that like it gets picked up on yeah oh my gosh it absolutely has a place for it in the community and I can't thank you enough for bringing this out to us and uh we want to we want to see we want to continue seeing obviously more of it so absolutely keep doing what you do.

SPEAKER_02

Thank you thank you so much. Yeah definitely uh yeah hit me up if you try it if it doesn't work if it works yeah hit me up either way uh I'd love to talk shop about it.

SPEAKER_00

Let us know and of course all of this will be available in the show notes. Thank you so much Anton for coming on the pod I know I've been after you for thank you for quite a while. Thank you you said no if you remember you were like no I don't know well you were like I don't know maybe maybe in a few months like tell them get come back to me in a few months and but I'm glad we waited because now we've we've got this incredible tool. If you're listening please download subscribe where can we follow your newsletter blog content? Uh just Antonlovesdmb.com excellent uh and if you are and if your EDR can't see uh the process or the activity that your codex agents are running you're only obviously only getting half the story good check out a 10 and until next time keep engineering