Detection Dispatch (Alex's Version)

Words are Cheap. Sense Making is Not..feat. Diego Perez

• Season 1 • Episode 5

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 1:08:15

What happens when a philosopher walks into a SOC? Apparently, he builds one from the ground up, spends a decade making sense of detection engineering across financial services, global IR teams, and now Canva. 

Diego Perez is a detection engineer who studied philosophy, taught himself security at 2am with a newborn in the other room, and has been quietly writing some of the sharpest unsloppy takes on the internet about what detection engineering actually is versus what we pretend it is. His blog Quasarops lives by one rule: words are cheap, sense making is not.

We hit on:

  • Why "garbage in, garbage out" is a heuristic that stops short of actually helping anyone
  • The Cynefin framework and why knowing which detections you need lives in the complex domain, not the complicated one
  • Detection as code: is it overrated now that coding agents exist, or are we asking the wrong question entirely
  • The Red Queen effect, Jevons' paradox, and why you do actually need AI in your SOC whether you like it or not
  • Agentic threat hunting: whose tokens do you trust, yours or a vendor's black box
  • Why the human element is more important than ever, and who exactly gets blamed when the model gets it wrong

    Follow Diego's substack: https://quasarops.com 

Detection Dispatch (Alex's Version) is an independent detection engineering & threat hunting podcast. Rebuilt. Community-first. Featuring a lineup of the real and active projects pushing the limits of detection engineering, threat hunting, and everything in between.

SPEAKER_02

This episode is brought to you by Detections AI, the DE community's favorite share detection repo that just shipped a new Intel exchange. Create, share, and discover threat intel tied directly to community detections, adding more context and better intelligence to the same mission. Welcome back to Detection Dispatch and to today's episode that is half in the present, half in the future, because of today's guest. He is joining me from one of the most gorgeous places to be doing detection engineering in the world. Diego Perez from Australia is here on Dispatch to talk about detection engineering and automation work, which he's been doing for a very, very long time. Currently works at Canva, which is one of my favorite apps ever. And he's going to be getting into it with me on an Agentix Worms and what it all means for the modern day DE practitioner. So, Diego, thanks so much for coming on the pod. I've been looking forward for this one, really.

SPEAKER_00

Thank you, Alex. Thank you for inviting me. I mean, I hope we can make it a great episode together. Maybe it has some things to say that are gonna be relevant. Uh maybe not. We'll see. But uh yes, indeed, I'm in Australia. It is a beautiful place. And I'm not in one of the big, big, big cities. So I'm not in Sydney or in Melbourne. I live in Brisbane. Maybe you don't have a lot of guests from Brisbane normally, but uh we are a Brisbane is a beautiful place as well. It's very lush. There's a lot of green here. Uh currently it's a little bit humid, uh, but that's that's Queensland and that's Brisbane for you guys. And we do have the Brisbane River here, which I really, really love. I really like rivers. I like the sea, but I do like rivers as well.

SPEAKER_02

So your does your typical day consist of shipping a rule, uh, tuning a broken one, vibe coding an app, and then going out to the river? Or my thing.

SPEAKER_00

My typical day consists of some of that, including dealing with a bunch of things that are annoying as well in life in general, admin work and bureaucracy, and also some some respite, I would say, some breaks in the middle, which is playing with my dog, taking my dog out for a walk. Her name is Panda. I have a I wanted to wear the I I have a little t-shirt that my wife gave me, which says has her face kind of like tattooed on the t-shirt. And uh, but yeah, today it was super wet and I couldn't get it. So, yes, playing with my dog, that it's in the middle of everything. Training with my son as well, just doing some soccer practice. I call it football, but uh you're gonna you're gonna call it soccer over there.

SPEAKER_02

That's the right way to say it. I I I grew up, I grew up watching.

SPEAKER_00

I can test that. I can test that. I I like football more, also because I'm Argentinian as well, right? So I was born in Argentina, I lived there the most most part of my life, and uh, you know, football is you're entering um enemy territory because I am a Ronaldo fan. Oh, you're a Ronaldo fan. Okay, okay, okay. Look, I've got nothing against Ronaldo, right? It's just uh of course I I have to put Messi up there first.

SPEAKER_02

Of course. I mean, there they are two legends. You you gotta you gotta give them their flowers. I I will say I we I talk about this a lot. I've talked about how what it what what is better? Someone who really works really, really hard and practices and really respects the craft, like Ronaldo does, or someone who's just naturally God like talented and he doesn't like I it's it's hard, it's hard to say.

SPEAKER_00

It is hard to say. I think everything is life is a is is a mix of um skill and opportunity.

SPEAKER_02

Yeah.

SPEAKER_00

And so so you maybe you're born with some natural gifts, you know, to be really good at a particular sport or a particular thing, but that can only get you so far. And then you have to actually apply a lot of work to get far if you want to get far. If you want to be at the cutting, cutting edge of things, you have to apply a level of discipline that that not always comes naturally, right? So discipline to the point of almost obsession with something. And I think Ronaldo has that, but I also think there's like this distorted image that Messi is just naturally good. And I don't think so. No, the guy, the guy trained a lot and you know, suffered a lot of things in his life as well. But you know, what do I know? I don't really know what how these people live their lives or what they've done.

SPEAKER_02

That's true. I guess it's only what they want to show. I mean, you you see that you see Ronaldo posting, like making his son work out with him like since five in the morning. I'm like, I'm like, Jesus, I feel bad for the son. Like, like, like let's go go play outside and have a childhood, not be benching like 20-pound weights at five in the morning.

SPEAKER_00

Yeah, yeah. Well, he's gonna be a strong, strong guy.

SPEAKER_02

He's gonna be a very strong kid. But well, thank you for sharing that. That gives us a little bit of of insight in into that mindset of yours, which really I really wanted to tap into to really set the the intention or the the really the framework of what we're gonna be talking about today. So, can you tell us a little bit about sort of how you ended up in in DE out of out of all places? Was it sort of the typical kind of arc where you started off at at as a service at IT desk?

SPEAKER_00

Yeah, yeah, yeah. Okay. That's a good origin story. So, first of all, a little disclaimer, I'm not here on behalf of Canva at all. I'm just uh here personally as you know, Diego, the guy who has done cybersecurity for over 10 years. And that leads me to how I started. So it's pretty funny, really. And uh though it was really hard as well. So um I used to be a primary school teacher, believe it or not. Well, I I I was I wasn't the primary school teacher, I was just the IT teacher, right? Like the IT guy.

SPEAKER_02

Okay.

SPEAKER_00

So so, and that's something I did for a while, but at the same time, I was studying philosophy. So I studied philosophy out of all things, and I spent many years studying philosophy, writing about philosophy, uh, you know, publishing things, essays, and stuff like that. But then I met my now wife, and she's Australian. And so, you know, a trip to Australia was in the cards. Of course, she always jokes that one of the first things I said to her is like, I'm never gonna move to Australia, you know that. And uh, you know, lo and behold, here I am. And 13 uh 13 years after that, I'm still here. So uh I've been in Australia for like 13 years now. And this is a beautiful country, and I definitely love and like everything I've done here. It's been it's been both beautiful and also hard, you know, because that's real life. So the way it started, uh, when I moved to Australia 13 years ago, I started trying, you know, I looked for work in IT and I couldn't find any. So I probably applied to about a hundred or more jobs, and I was getting rejected, rejected, rejected.

SPEAKER_01

Oh, man.

SPEAKER_00

Uh, until some guys called me and said, Hey, it looks like you have some IT experience there. I think you're gonna be great for this job because you know, service desk, you're gonna have to talk to people who don't know how to send an email, or maybe uh a server might break down. And you, you know, we have playbooks, you just follow the playbook, you know, escalate if you need to escalate. You'll be great. You know, boom, go go and do it. Six months contract. Start with a six month contract. And I was like, yes, because I was pretty much doing anything around that time, you know, picking fruits, mowing loans. Uh, I was uh, you know, like a barista as well, working in restaurants and whatever I could do. So that was the start. My English was pretty bad back then, and you know, one of the first things on my first day, they sat me right next to the boss. Not the boss, it was the boss's boss, right? And I was with my really bad English first day in service desk, and I was sitting right there, and I couldn't believe I was like, out of all places, out of in this massive office, you sit me right, you sat me right next to the boss. So that was a bit nerve-wracking, but I would say that was probably one of the probably the one of the least, the biggest, not the biggest challenge I had to face since then, right? So from service desk, I went to security because I was hired. I was hired as one of the first two security analysts in this place where they had the project to create a SOC from the ground up.

SPEAKER_01

Okay.

SPEAKER_00

And so servicing clients locally and in Australia. So that was probably the way I that I broke into security. I prepared a lot for the for the interview. I read a lot, I was already doing a lot of those things anyway, because every time I can, I just spend time learning.

SPEAKER_02

Yeah.

SPEAKER_00

So that led to a few other gigs after that, right? I became, after a couple of years, I became an IR lead for APAC here in Australia as part of a global SOC. And you know, this global SOC had IR leads in Europe and in America as well, and in the APAC zone. And I would say the people there were brilliant. I was working with people in the US, Canada, uh, Sweden, France, England, everybody, everybody there was brilliant. And that's probably one of the most intense growth moments of my life because I learned lots from these people. I was just amazed by how brilliant they were, all the things they knew that I didn't know. And how do you do that? And how you do how you do you, you know, dump RAM and do uh volatile memory analysis. And uh, how do you reverse engineer binary? You know, they made it look so so easy. And I was I was like, I need to learn how to do this. And I learned, I learned from them. I'm I learned from them from doing courses as well, but also from staying up at night until 2 a.m. 3 a.m. with a with a you know recently born child trying to you know trying to figure out things, trying to complete capture the flag exercises, learning about assembly code, you know, all that stuff happened together at the same time.

SPEAKER_02

And eventually that's I love hearing stories like that because that's so true. I some of the most impactful and work and learnings I've ever had has not been through a course or a book or a certification. It's been through other people and learning from other people.

SPEAKER_00

Yeah, 100%. And up until today, you know, like I I I feel I've been blessed with being surrounded by brilliant people across my career and learning from them, uh, whether it's in leadership or whether it's technical work or whether it's how to do uh how to translate technical things to business different business layers.

SPEAKER_01

Yeah.

SPEAKER_00

I've learned lots uh from people have the ha who have surrounded me. And I think I really, really appreciate everybody, everybody in my journey has been important one way or another. Good experiences and bad experiences both were equally important because they teach you a lot.

SPEAKER_02

It's uh it's true. It's absolutely true. The and they're also great for figuring out what's what's your next step in your career path. And you've been in financial services and you've been a creative, and you touched a couple different areas of of the SOC, and now you're you're at Canva. What are what are you what are you working on primarily uh these days, would you say?

SPEAKER_00

Okay. So lots of things, how to enumerate them. So I've been doing, I've done over the last 10 or so years, I've done a lot of icy technical work. But at some point in my career, I started mixing that with leadership as well. And getting into a different problem because that that's always something that happens to me is sometimes I've hit a ceiling and in where in my growth, and uh, I start to feel stagnated or I feel you know, some unease, some some something inside that says, we need to you need to challenge this a little more. You need an you need another challenge. And so at some point my career pivoted a little bit into how can I be a technical leader? Instead of just doing technical work because I was told to, it was more how can I design the the way forward? How can I help people build paved roads? So there is a blessed path for deploying something that is more secure than the alternative. And that's happening a lot right now with AI, for example. So what do I do? Like nowadays, a lot of my time is spent in keeping the operational tempo going. So making sure that we don't drift from the plans, the things we have to do, making sure that that detection that needs tuning gets tuned and assigned and prioritized. Detection ideas are triaged, prioritized, and assigned, projects are evolving in the right direction, that people are being coordinated accordingly. If somebody is not making it because of X, Y, or Z, then how do we replace that person temporarily? Making sure redundancies, uh not redundant, not redundancies in so redundancies, making sure that there is backup for everything we do, right? So that turns into a lot of feedback loops that end up coming back to me. And as part of those feedback loops, I need to make sure that I distribute the and I broadcast the right information at the right time. Because otherwise, that operational tempo starts to to decay and it starts to drift away from the from the milestone or where you want to go, actually. So a lot of the time uh people call it glue work. It's it's that keeping keeping the tempo in a way that you will achieve the the crazy big objectives that you want to achieve. You know, being there, like uh right now I work with a group of brilliant engineers, again, like brilliant people surrounding me. And I learned a lot from juniors up to principals. Uh, so I learned a lot from how they think, uh, the way they break down problems. And what I find again and again is that I know nothing. Is that's the feeling over the over 10 years of career? It's like at every point, every new challenge I've had, every new role I had, every new engagement I've been in, is that feeling that I have so much more to learn.

SPEAKER_02

So it's about it's very Aristotle of you, isn't it? Oh, then I know nothing. I see, I see where your mindset is coming in now from.

SPEAKER_00

Yeah, I you know, it's just about being a bit humble as well. I don't uh the reality is that the complexity of the problems any business faces nowadays is too complex. So you don't break down complexity in an easy way. You have to you have to pay attention, you have to keep your finger on the poles. Yeah, you have to you have to keep uh different tempos in mind as well and play them all. So yeah, I do a lot of that, a lot of some technical work as well, you know, deploying data pipelines, troubleshooting schema issues, uh making sure that we have the right data generating the right alerts, migration work, coordinating migration work. So there's a lot of that, uh setting up standards, etc.

SPEAKER_02

GRC. I really like how you framed the the glueness of the operational tempo. I mean, that's like detection and engineering as a core, and there's a real science behind all of it. And everyone always says, you know, garbage when it comes to right finding the right data for your for your program, everyone always says garbage in and garbage out, right? But really the core of DE is uh what does that even actually mean in in plain English here? Like, is there a height requirement? What's the the quality, the schema consistency, the entity resolution of it all? Because I'm so tired of hearing garbage in, garbage out. It's like everybody knows that. Is there anything more concrete that we can actually go by? Uh, because now with agents in the mix, the pipeline needs to be much cleaner now more than ever.

SPEAKER_00

Yeah, that's true. I think people uh we have we have sayings like garbage garbage in, garbage out because they help us simplify the world. So there is this saying that goes, all models are wrong. Some models are useful.

SPEAKER_01

Yeah.

SPEAKER_00

And there is the the reason for that saying is any model is a misrepresentation of reality because you cannot capture all the complexity of reality in a model. But does it mean that the model is not useful just because of that? Well, not really. Some models are actually useful. For example, I used to I gave a talk as a guest lecturer in the uh Queensland University of Technology where I introduced the engineers there with uh the origins of the London tube map. It's the the if you look at the you know 19th century uh pictures of the London tube map is really messy because there was a lot of topographical information in there, and what you know nowadays as a tube map was actually created by an electrical engineering drafter in in London, right? At the beginning of the 19 something. So the what's what was happening there, and the reason that it was simplified is because the original model of the world, so the original model of how the tube lines intersected and which suburbs interconnected was too messy because back then that was the way that uh maps were being drafted. So when that was simplified, suddenly it made more sense. And the the maps that you know of today are all based on that map, on that improvement from the London tube. So models of reality are useful, not they don't represent exactly reality one-to-one, but that's the point. It's there, it's a model. And so garbage in, garbage out, I feel it's one of those things where it's a heuristic because it just makes you understand something easily. But when you really inspect it, what does it really mean? So I think in detection engineering, the data problems don't go away just because you have AI, for example, or just because you have an automation here and there. I think what's important to understand in detection engineering above all, but in detection engineering is a it's a data discipline, really. It is yes, it is embedded in the world of cybersecurity, but when you start peeking underneath those layers, it's really a data problem, almost a data science problem, is a data in data engineering problem. How do you extract and mine data at the source that is relevant for the threats that you want to detect, and how that is transmitted in a pipeline and gets to the point where it becomes signal? Because signal is the the clean the clean information that you need to create even higher level signal, like detections. So, like a detection, in my view, a detection is a higher level signal that is based on lower level signal. So you create a detection based on logs, but those logs already come structured somehow, so they conform to some schema, right? So where does this schema come from? Well, if you don't if you don't uh create a schema yourself, then the whatever schema you receive at the source, yeah, that's what that's what's uh being propagated across the pipeline until it gets to your scene or your data lake or wherever you store your logs, right? So what I think is important is that any we can we can we could say that anything can be garbage really. The only way that you don't have garbage, and the reason for that I think is probably informational decay. Some sort of entropy, right? I'm not a physicist or a mathematician, but entropy, this concept that you know the order decays into disorder. I think that applies to the data world as well. Sometimes nowadays you have so many data sources and they come, they produce data in different schemas. Normally, sometimes they are vendor locked in, sometimes they are open source. But the important, the task of a data of a detection engineering team is to make sure that you build that API, that contract, that translation layer. So that translation layer can be a schema like OCSF or ECS that you, you know, the elastic schema that you conform to, and you say, from now on, uh at this point in the pipeline, we will apply a type of normalization so that logs coming from anywhere will be normalized to this unified schema. And that is the contract I will establish for any downstream processes. So anything, anything downstream from that, like building detections based on that data, you already established a contract for that. So your detection engineers know what data they are getting, what shape that data has.

SPEAKER_02

No, that's true. I I really like that framing. And everything is an API with a contract. I think that's a very good way to frame that pipeline. But if data is not created equal, that can very much break the pipeline if you don't have this uniform normalization of it. Which oftentimes when you have, when you are, I don't know if it if Canva does this, but many financial institutions they like to build their own software and applications. And most of the time they're not going to have your SIM's not going to have a custom log source for that. So it's almost up to you know the DE to really ensure that you're you're you're making that making that API translation connectivity tissue there.

SPEAKER_00

Yeah, for sure. And I like to say um I normally tell uh tell my teams that everything is an API. And not only in the technical sense, also in the operational sense. Yeah. So so as a team and as a as a function, as an engineering function, let's say detection engineering or threat hunting or instant response, you need to establish an operational API as well. So how can the rest of the business talk to you? And how can you talk and communicate with the rest of the business? So when you establish that API, it's easier for you to do things that otherwise would be really hard. For example, you mentioned uh you mentioned that some companies may build their own applications. Yeah. Cool. Yes, there's a lot of software companies out there, a lot of companies building their own in-house uh software. Perfect. Now, normally, I mean, I say normally, but I don't really know. I would assume that when you build your own software, you also have an application security function that does a little bit of threat modeling. Initially, it may be a mix-up of everything, but you have somebody somewhere that does some level of threat modeling on that internal application you're developing. Yeah. So that you know what are the trust boundaries, what are the risks here. Now, when that's happening, is detection and logging considered as part of the threat modeling? If it, if it is, excellent. If it isn't, you need to call that detection engineering body, you know, in the other team to come in and be there in those meetings with you, right? And that's part of the API. So what is what when the business creates or develops a new application, any business out there, what do they think is happening downstream from that? If you know, if they think, okay, this is being somebody is thinking about the security controls. It's not my problem, is the next hops problem in the pipeline. All right. Fine. Maybe the assumption is that somehow, magically, your application surfers that you may be exposing to the world will be will be logged. And so logs and telemetry will be generating for will be generated for that application. And somebody will be creating security detections based off of that. So if there is an abuse of your application in any way or an attempt to a vulnerability that leads to remote code execution, there is a bunch of processes happening somewhere that will secure your app. And maybe there is, maybe there are some processes and or maybe there aren't. And uh is detection considered as part of that? Is detection engineering engaged as part of that process? Those are some things that are important.

SPEAKER_02

Or maybe, or maybe they think that mythos is the end-all be-all solution to scanning their custom application.

SPEAKER_00

It's gonna be really expensive. It's gonna be really expensive. Imagine. I mean, I would love to just point mythos as as at everything, you know, like, okay, mythos, run wild uh within within constraints, but run wild and figure out, figure stuff out and just let me know when there is something critical I should know about. I think that's the that's the idealistic model that some people have in their mind. Yeah, they're operating from that point of view.

SPEAKER_02

I I really do think so. Spicy question for you. Do you think that detection, the the question of which detections do my or does like my organization actually need, is that a bigger, unsolved problem than how to author detections? Like, I aren't like have we moved way beyond using AI to build detections at this point? Like, isn't isn't it a much bigger fish to fry? Like figuring out which detections your org uniquely needs.

SPEAKER_00

That's a good question, Alex. That's a really good question.

SPEAKER_02

It's it's so I feel like LinkedIn is pretty split with I would never trust AI to write my detections. Like it does SBL like super shit.

SPEAKER_00

Right. I feel there's so much slop in LinkedIn, by the way. So there's so much things that just in the comments. Yeah, that's right. Yeah, I've seen it in the comments as well. And I'm thinking, is people build bots to order post slop comments in you know LinkedIn profiles or whatever, LinkedIn posts. I don't know, but yes, there are I I I used to follow a lot of people, and and now there's only some voices that where you can really see uh some artistry, some real tradecraft and artistry behind it, right? I think it's okay if you write it, some with AI, but you need to add your touch to it. Otherwise, it just sounds too robotic. But coming back to your question, I think if I think authoring detections and knowing which detections you need, it's always gonna be one of those conundrums and those big problems. Authoring detections, when you have AI, it starts to become really easy. So even you have even if you have detections as code, for example, even better is like that's even better and makes it even more easy, right? Or easier. Uh that's my English there. It lowers the bar, it lowers the bar of entry because suddenly you can point Claude or you can point Codex or OpenCode, whatever agentic CLI you want to use. You can point it at the source code of your detections. And you know, detections are really a query language, can be anything KQL, EQL, SQL, whatever the query language is, but it's logical and uh it's been embedded already in the training neurons of all the large language models. So it comes really naturally easy for some of these models to produce something that looks logically sound and that it seems like it will it will detect what you think it will detect. So developing detections, yes, I think that is easier than knowing which detections you should develop. And I think that's also why a lot of the the some of the limitations of the AI sock or of a fully automated detection engineering pipeline. There's a lot of gains from that though. Um I'm gonna go into that later. But the problem is always context. And I think context, engineering the right context so that your downstream processes have the information they need. It's part of this keeping the pulse on, keeping the finger on the finger on the poles, right? And making sure that you are extracting the information that you really need.

SPEAKER_02

Yeah.

SPEAKER_00

So knowing which detections you need to create, that's that's a lot of work. That's a lot of human work. Because we're talking that you need to be, you need to be talking to multiple teams. So this is cross-functional. You need to be talking to the business in different layers of the business to understand and anticipate what is have what is coming, what is happening next quarter. Is there a big release of something? Are we acquiring a company? Is uh is there a merger that I should be aware of? Merging companies normally brings attack surface, you know, unquantified risk that is there sitting somewhere that you need to start looking at, right? And then when you under when you start getting into the nuances and the details of what really means to understand the threats that may impact your business, and then once you understand those threats, it is a question of are you producing the right data to detect those threats? Is that data in a pipeline that is cleaning it and normalizing it and you know fixing it into a schema before it gets to the point where you can even build a detection for that? That's a that's a much bigger question that lives in the complexity domain and not so much in the complicated domain where I think detections live. So this is, I don't know if you've ever heard of the Kinevin framework. It's created by a Welsh guy and it's written like C Y N E Kinevin, but it's pronounced Kinevin. I don't I don't speak Welsh, but I know that's how you you pronounce it. And he created this framework, which is a four quadrants, right? And you have the first quadrant, which is the simple domain, then you have the complicated domain, then you have the complex, and then the chaotic. And normally processes will flow between these domains, right? Simple, the simple domain is things that are well ordered and easy to achieve, simple tasks. And the complicated domain is things like assembling a car. Uh you can assemble and disassemble a car, but normally you need some, you know, you need some expertise and to be able to do that, right? Whereas the complex domain, that's where the problems that are hard to solve live because they're normally not easily solvable, or they have many solutions. There's many potential solutions for the same problem. So I think understanding which threats your company faces, you know, yeah, doing threat modeling, CTI, you know, threat intelligence there, uh pen testing, active defense. This is what I call active defense. That is complex.

SPEAKER_01

Yeah.

SPEAKER_00

Crafting a SQL detection, that's complicated. But, you know, a lot of people can't just write SQL. But how do you know that that detection that you're building really addresses a business risk? That's a complex question.

SPEAKER_02

Very, very, and one that's primarily focused on human-centered work. Because the AI, your Claude code is not going to know any upcoming, I don't know, audit you have to plan for or what's going to be on the news tomorrow. Like it as someone who builds and maintains a detection content library, what it's really bad at for me personally is whenever I build a new detection, sure, it'll build it. But what it doesn't know is it's not going to say, hey, this detection is very similar to, you know, your rule ID number 456. Why don't you just add in an extra criteria to that detection instead of just make a new one? Right. And it's just more efficient.

SPEAKER_00

That's right.

SPEAKER_02

It's just, it doesn't, it doesn't know stuff like that. Uh but maybe and and maybe maybe if I pointed it at the GitHub, maybe it will know. But uh on on that note, on the GitHub note, uh version control of your detections, thinking of your detections like CI C D, detection as code by definition. Do you think it's overrated?

SPEAKER_00

Oh you're asking, you're asking all the the difficult questions.

SPEAKER_02

I gotta know. I gotta know. There's a recent uh uh not too long ago from Harrison, Harrison Pomeroy at PayPal. And shout out to Harrison. He listens to the pod. He's an awesome super awesome DE. And uh he he recently wrote a piece about how it it's it's it's probably more overrated than than you realize. Like, sure, it's great, it's it's nice to have, but is it really moving the needle in the DE practice? Who knows?

SPEAKER_00

Yeah. So I think kind of like we can connect this with the with the previous question, which is, you know, we can probably say detections as code lowers the bar to create detections. Mostly, most of all nowadays, where you can point your coding agent to a repo, and if all your detections leave there in the same format, then it's easy for the coding agent to pick up the context in the repo and just build something, or to understand how your CI pipeline works and the way that is deployed. I think probably people are calling it overrated potentially because of that, because you have coding agents. And so it's like is the maintainability of a CI CD pipeline that is centralized too heavy versus pointing equipping your coding agent with an MCP tool that can directly talk to your scene and deploy a detection. So that that would be one way to say, you know, what's which one is easier, which one is is harder. I I don't think I think we definitely need detections as code, but we shouldn't equate that to detection engineering. They're different things. Detections as code is a way to, or a CI CD pipeline, let's just say a CI CD pipeline to deploy your detections is a method, it's a vehicle to deploy detections. And it is good in the sense that it builds a paved road. So because you can add linting, you can add validation steps, you can add a lot of check marks and controls in the middle, it means that the end result will be more consistent, less prone to errors and false positives, and it will you are securing yourself a unified way to deploy the same detection versus having to do it via click ops. So you gain in consistency and you gain in speed. That's what I think that a CICD pipeline properly architected or detection Says Code delivers for you, right? Now, can you achieve the same result any other way? I mean, yeah, maybe you can point your agents to it, but now you have a new problem. So previously all of the checks and balances were centralized in a single pipeline. But if you want to offload that to MCP tools and agents living in your pool of detection engineers, then how are you controlling that? Suddenly, you also are in the need to build a centralized repository where all your agents.md files are, where all your harnesses live, where the instruction set lives, that has to be also kept and maintained centrally so that the pool of detection engineers or security engineers all clones the same setup and operates from the same setup, because otherwise you're going to introduce drift again. So maybe you can decentralize. What I mean is centralization versus decentralization is it you need to think about what are the trade-offs there. And why not having both? Why not both? Why not having your, you know, eventually I think that's where organizations will land. It's a virtuous feedback loop because you can have a centralized pipeline, you can have your agents distributed across the fleet, and they can both work together. But again, that's that establishes a paid road to develop and deploy detections. It doesn't mean that you suddenly have a detection engineering function. Because for that, you need all of the operational API that I was talking about that is also important. You know, who is who is looking at prioritizing uh detection, the detection backlog, who is talking to the business about the threats that impact them and whether we're collecting the right data in the right ways, who is uh building the data pipelines that make the data end up in your uh in your C-more data lakes? Who is so all of those, who is prioritizing detection tuning, who is measuring the false positive ratio, who is measuring which detections uh you know wake up people at 2 a.m. in the morning. Yeah. So those things, those things are part of that's what I that's what I would call a more holistic detection engineering function.

SPEAKER_02

Absolutely. You're right. They're not one in the same. DAC does not equal D DE. It is has so many layers around the systematic design of it all. And you're right, lowering the barrier also to deploy doesn't mean that the surface you're maintaining also is reduced, is what I got from that. Now, of course, we have to talk about threat hunting. Now that we've got agent swarms both on the vendor side and also within our clawed code agents. I mean, I've I've heard on a podcast recently uh there was a CISO that said, I can either, which is not not a positive thing to say, but he's but he said, I can either hire more headcount or I can kind of force multiply some of my best people and and just kind of give them unlimited tokens. Uh and and that's like spinning up clawed code and letting it letting almost anybody spin up claw to code or maybe even open claw and just going ham on that. And so we obviously are hearing a lot about agentic threat hunting, taking a lot of airtime right now. And there's this even split between vendors saying, oh, just trust our agents. Like we'll build agents inside whatever product or our AI SOC product, and then the other way, right? Kind of build your own agents, bring your own agents. You you set up your own MD file around your protocols and your policies and your frameworks and your procedures and and your markdowns, all of that. Do you feel more one way or another? Would you rather use your own tokens or would you actually trust, you know, maybe an AI SOC or a threat-hunting agentic vendor to uh and use the agents that they come with?

SPEAKER_00

Okay. Big question.

SPEAKER_02

Yeah.

SPEAKER_00

You're you're asking all the the not the easy questions, right? Like what's the weather today? What's the weather like today? So uh right. I think let me let me think about this. So somebody posted a few weeks ago, somebody posted something along the lines of, and this is not is it I've seen a lot of people already talking in these terms. What what the what I've seen as the most evolved, let's say evolved opinion on the relationship between security and AI is that AI will not replace the fundamentals. So the fundamentals like again, doing your threat modeling, understanding the threat landscape, having at least a basic CTI function that can advise you on that, having a security program in place, uh thinking about the security controls, and you know, you can go to NIST for that, and etc. etc. etc. The fundamentals. So the fundamental things of security don't change because suddenly there is AI. So that is, I would say that's the most um the the most evolved opinion so far, right? So if AI SOC comes here, but and you know AI SOC suddenly is an option, and using you know Gen AI for threat hunting is an option, fantastic, but it's not gonna change the fundamentals. Now yes, I that's have you heard that as well?

SPEAKER_02

Yes, yes, that I do there was a common trend out leaving RSA that everyone is going back to the fundamentals, and that it's AI is not a way moving forward in that direction if you and it's actually moving forward faster in the wrong direction if you don't have the fundamentals right.

SPEAKER_00

That's right, that's right. Now I have a problem with that, and I'll tell you what the problem is. I do agree with that, with that point of view, but the issue is what I I normally go back to these two concepts of the Gonz paradox and the Red Queen effect. And the Red Queen effect is I don't know if you've read, you probably read Alice through the looking glass or yes. So the name in Spanish is different. So I think that's the name in English, but in Spanish it's a different name, so that's why. But anyway, so in Alice, there is this there is this moment of the Red Queen where you need to run as fast as you can just to keep up being in the same place. So you're not advancing, you're not regressing, you're just staying in the same place, right? So that's the the Red Queen effect is an effect of evolutionary pressures. So when the environment around you is starts to evolve, they become more efficient. Let's say attackers become more efficient because they introduce Gen AI and better automated workflows to attack organizations. When they do that, they can launch more campaigns per day or per man hour. They can launch more realistic-looking phishing, they can create custom tooling faster than ever. You can find new exploits faster than ever. So they will adapt to attack you using the new tools, the new shiny tools. And they provide some real leverage and some real advantages. Because that happened, that's happening, it's a it's an evolutionary pressure on companies that are defending. Suddenly, yes, you have to keep up with the fundamentals, but the way you stick to fundamentals, that's what changed. So you need to stick to fundamentals, but also running at a higher speed just to keep up with the other side, right? So the offensive side. So on the defensive side, you need to employ the same tools because you need to assume that on the offensive side of things, they will be employing as many new technologies as they can for more for higher efficiency. So eventually what that translates into is that you do need AI and you do need AI somewhere one way or another to make your business more efficient, right? And protect it more. That's right. To move faster. Eventually you hit diminishing returns, though, because red queen effect, everybody will start moving at the same pace as you. So the sooner you pick up on these, the better. But eventually, that is the red queen effect. It's an evolutionary pressure. And this leads to also another concomitant effect, which is the G Von's paradox. And the G Von's paradox is what it's a he was an economist, and I'm not an economist, but the the gist of it is that efficiency gains in how you consume a resource lead paradoxically, lead to more consumption of that resource, right? So, for example, if you say, you know, um let's say cars, right? So suddenly you invent an engine that is twice as efficient as the previous engine. So it consumes less petrol to uh and and less less uh yeah, less energy to do the same amount of work.

SPEAKER_01

Yeah.

SPEAKER_00

Which means normally it translates into the car being cheaper. So the efficiency you can you you have efficiency gains in many aspects, like in energy, in engine, in the chassis, and then that makes the car cheaper. Does it mean that suddenly you're gonna have less cars in the economy? No, you're likely gonna have more because it's cheaper now, it's more accessible, it means more of the population will be consuming the resource. So suddenly you will need, you will start eventually needing more petrol and more oil extracted from the ground and et cetera, right? Even if you gain an efficiency, it turns into higher consumption of that resource.

SPEAKER_02

But that's a good thing, is it not? In the in the case of cars, but in the case of the product, like the workload itself, isn't it? Aren't we going to reach maybe Moore's law at that point, where it's it's going to not be efficient anymore? It's gonna hit a certain point where we can't be more and more efficient in the way that we actually use compute and workloads.

SPEAKER_00

Yes, I think as as compute is more efficient and you know uh machine learning models become more efficient, then uh you will have higher consumption of that. And intelligence will become commodity.

SPEAKER_02

Yeah.

SPEAKER_00

And I would I would dare say swarms of agents, swarm intelligence will become commodity, will become an infrastructure that you build on top of, right? So where I'm going with this is do you need AI in your socks today? Yes, you do. You you do because as the as intelligence becomes a commodity and an easy-to-consume resource at lower and lower prices, that will that will mean that that level of intelligence is at the hands of more and more people. So more script kiddies, more attackers, more threats will be um, you know, threat actors will have access to that resource and they will use it. They will not hesitate to use it, which will drive evolutionary pressures over the defenders. So you're gonna have to also leverage the same mechanisms to be faster. Now, the problem is I think people assume, or companies assume, that just because you have AI, you can embed it everywhere, or that Gen AI is the right solution. Maybe there is a machine learning or an artificial intelligence method that is a solution, but it doesn't necessarily need to be Gen AI. Is that kind of thing where you know for a hammer everything looks like a nail?

SPEAKER_02

I see. It's like can you use LLM or can you just use rejects?

SPEAKER_00

Yeah. Can you just use rejects or can you use a machine learning algorithm that, for example, may look at your alerts and learn from them and understand the likelihood of an alert being a false positive, right?

SPEAKER_01

Yeah.

SPEAKER_00

So some people are building, are building towards that kind of environment where instead of tuning, instead of tuning alerts, because when you tune an alert, you also think you are excluding noise. And a lot of the time that might be true, but you're also detecting signal against less of your surface as well. You're reducing the signal that you're interested in. And maybe in that signal, there is some hidden, there are some hidden threats that you're not picking up. So maybe the answer is not to tune so much, but the answer is to teach a machine learning algorithm how to statistically read your alerts so that they can uh anticipate whether the likelihood of it being a false positive or a true positive. So you don't lose signal, but at the same time, what is presented to the analyst or the engineer is a uh you know, a simpler artifact, let's say.

SPEAKER_02

So the question not only becomes should you use AI in the SOC, but it also becomes like whose tokens do we trust? Because I mean, now there's there's there's somewhere north of 97 vendors in the market claiming to do AI in your sock across across a bunch of areas, like a Gentex SOC, autonomous triage, autonomous threat hunting, autonomous, even tuning, like you said, with that the exact use case of figuring out which which value is the most like faulty, uh, and and suggesting recommendations for how to tune the I've heard of an AI socket that doing all these different flavors of use cases with AI. So whose tokens do you trust? Do you trust your own or do you trust, do they come with something special that they they think they know your environment more than you do?

SPEAKER_00

Well, I'm pretty sure that people will love not to have to pay for tokens. If you can do with somebody else's tokens, uh they'll probably prefer that. But you know, that's on that's not uh the reality of the world. So you're gonna have to pay for tokens. And then the question is, what do you do with those tokens? That that compute, that slice of intelligence that you're renting from the the the wire internet, the no sphere, uh, that has to, you know, somebody pays for that.

SPEAKER_02

Yeah.

SPEAKER_00

That might be so I guess there is a lot of there is a lot of different, there are different problems and different layers, right? So one problem is token efficiency and cost, and another problem is security. So how do you run those agents? If you want to let them lose so that they can access many, many tools in your environment and be more autonomic. So be more like an agent and less like a chatbot. You have to equip them with tools and you have to allow them permissions to do stuff. But the way they do that may be different, or maybe that's where the real question is. So do you install open claw on your corporate machine and let it run loose? Likely not. Do you install open claw in inside a I think Nvidia uh released recently a repo where you know they use K3S, which is like a small version of Kubernetes, where they secure, they can run open claw inside this self-contained pod. And so with uh an embedded uh sidecar gateway, so then you filter out what kind of HTTP endpoints that agent can access. And so you maybe you can give them all the permissions you want because it will be filtered out by many layers of controls, right? So at the at the permission level on the host, on the pod, and then the any controls that the gateway establishes. And so securing the way the agents run is one question. And there are many solutions to that nowadays. I think since the explosion of OpenClaw, that you know kicked off a lot of a lot of innovation in that aspect. How do you secure the agents? And uh on the other side, there is the efficiency and the token use. And that comes down to being able to combine the right models for the right tools. Not everything is a hammer, not everything has to be an ale. So it's so for some tasks that are more repetitive or less complex, you may be using, you know, more simpler, simpler models for that.

SPEAKER_02

I have I have you're you're right. I have not seen or come across yet, yet, I won't say there aren't any out there. AI sock vendor that lets you have the granularity of the agent control and what it can and can't do because it's mostly kind of with you know, locked behind a very black box, similar to a lot of their query languages as well, and their own um and their own proprietary rule sets as well. Or ML models. There's some that expose the models, and you can maybe build your own models with with notebook, you know, with notebook LM or Jupyter Notebook.

SPEAKER_00

Yeah, right.

SPEAKER_02

Not it's not so much that you see that. Uh, but I won't say it isn't out there. It may very well exist, but that's where I think that's where if you do bring it more in-house, you you do have more of that control, but a lot more to do.

SPEAKER_00

Yeah. And you know, a lot of those things, Alex, are like the classic build versus buy conundrum. So you don't have to build everything. Sometimes buying the right capacity or the right solution is what accelerates you. And sometimes that's the wrong answer as well. Sometimes you have to build and you shouldn't be buying. And those decisions are driven by um what is the sensitivity of the data that you're exposing? What are the assurances of the vendors uh of what they are doing with the data? Uh, is it staying within a VP, you know, a virtual private cloud? Is it um how many hops is it going through? So it has to do with a lot of business decisions as well. But sometimes, even sometimes buying the capacity instead of reinventing the wheel might be the right solution. But it really comes down to listing the trade-offs and the needs and the risk and having an informed decision rather than jumping straight into the into the train and going, ooh-hoo, yes, this is gonna solve all my problems. Unlikely it will solve all our problems, it will solve some and it will bring new problems you didn't have before.

SPEAKER_02

That's true. That's true. But you've inspired maybe my next next episode or a few episodes from now. I I do want to invite someone that has seen benefits of an AI sock vendor that isn't maybe my negative, pessimistic point of view on them. I I just I happen to think that all the ones I've seen is just a well-designed interface with interesting prompting on top of whatever detection platform they had before. They just consume the sim alert that you already have. They just consume like and it's like, are you making it better if you're just reading the data of your similar? You're not really improving anything out outside of just consuming the JSON of your similar. Like, I don't, I don't know. I'm willing to be, I can change my mind on it, but yeah, has inspired my the next episode.

SPEAKER_00

I think probably the vendors that will come on top will be the ones that realize that within your AI SOC solution, uh, some of them are really good, you know. But I do know, I do share some of the cynical view on things that you you just mentioned. But I think the the leverages in realizing that Gen AI is not the only AI that should you should be bringing into the AI SOC. So you should you should think of other types of artificial intelligence that you can bring in. You should think of different types of machine learning algorithms that can do the work without being Gen AI. And if you embed the right AI in the right places, that will compound to higher value than just bringing a summarizer, an AI that just summarizes something. Although I have to say, I have to say that having an AI summarize your alert saves you a lot of time. So the so you know there are little time savings here and there, but but it really depends on what parts of your pipeline you want to optimize for. And maybe some vendors will target that market segment where they're flooded with alerts. They don't have enough people, they cannot afford to have a detection engineering function properly tuning, bringing down the noise, understanding business risk from the data perspective. And maybe those places will benefit from a vendor that offers that kind of solution, right?

SPEAKER_02

Or maybe not, because a lot of these AI stock vendors are pricing per alert. Like you got to pay for every alert you have.

SPEAKER_00

Oh, right.

SPEAKER_02

If you have millions of alerts, then you're kind of fucked.

SPEAKER_00

Yeah, you're gonna be in a in a difficult situation.

SPEAKER_02

Yeah, but no, but but back to what you said at the very beginning of the episode, like AI SOC is not going to clean up the hard infrastructure, gluing, keeping and maintaining of your operational tempo, understanding what's coming up in your business across all the layers of your business. Like it's not going to fix that if they're just using, you know, the model that Anthropic is using and the data underneath is still what it was before. And I don't know. It's it's it's it all comes back down to the human focus work. And if they can wrangle, you know, whatever tool, insert tool here that they can't.

SPEAKER_00

Yeah. I don't think the the a lot of people are afraid that AI will take their jobs. I would say the human element remains as important, if not more, than ever.

SPEAKER_01

Yeah.

SPEAKER_00

Simply, simply from the perspective of even if you just think about AI, right? So let's say that you want to bring a lot of AI capacity into your business. Who's gonna run that? Who is going to do the risk assessments? Who's going to do the threat modeling? Who is going to maintain the infrastructure pipelines that make your unattended AI world work? Who is going to be thinking about the tools, the type of tools that your AI should have? Who is thinking about the uh failure domains? Who is thinking about the coupling and decoupling? All of these things I normally lump under system design. So I think system design is the edge, and that's not going anywhere. Uh, it doesn't matter what technology you bring in, there is always an operational cost to keeping it running and uh an expenditure cost. So there it's there is always a human element because whatever you bring into your business, it introduces a risk vector. And yeah, that risk may be accepted or not or mitigated, but it's people who do that, right? And where things go wrong, when things, because things eventually do go wrong at some point, when things go wrong, who do you think they want uh to have on the other side being accountable for it? A machine or a person? So the there's always going to be a human being accountable for something. If things go wrong, you can't just blame the model.

SPEAKER_02

But no, that was a very positive way, a human-focused way to end the podcast. Diego, I I'm super, super happy that we had this conversation. Uh, this is exactly actually how I wanted it to go and the kind of conversation I had in mind when I imagined it and when I relaunched the show. So this is exactly why I do this. So thank you so much for for getting into it with me. A traditional closing question, I like to leave the folks with uh a lot of valuable assets or resources or projects or a person in the community that you that you like, that you follow, that that you know of that maybe isn't getting enough attention right now. Because I I would kind of say you.

SPEAKER_00

Oh, right. Yeah, yeah, okay. Oh, there's there's a lot of people, and I just don't know, don't remember their names. So because LinkedIn is LinkedIn is full of amazing people, right? And uh I follow people in India, like Asia, and people in the Netherlands and Germany, and you know, uh different parts of Europe and the US and even people in Latin America. A lot of people do valuable things. So I I would definitely pay attention to anybody out there that is putting some artistry, some real tradecraft into what they're doing. And uh the signals for that, I think, is you will see less AI slop in the way they talk, in the way they express themselves, how they write, how they post. Uh, there are some there are some people who are really taking the time to think about what they will be, you know, posting out there.

SPEAKER_01

Yeah.

SPEAKER_00

And you can be you can use AI as a companion to think about or even write part of what whatever you want to say, right? That's fine. Uh, we all do that because, again, um, Red Queen effect. You you have to keep up with the evolutionary pressures around you.

SPEAKER_02

Yeah.

SPEAKER_00

Now, there are people who who bring their artistry into it. They they they their tradecraft is important. And um I do value those voices. And uh all of the slop AI slop voices, I quickly, those quickly fade out from my from my feed, right? Quaserops, that's my blog, and that's where I write a lot of my very opinionated things. You will see, I you know, you will see that that the my phrase or my saying there is uh words are cheap, sense making is not. And and that's how I how I approach my writing, right? So like words are cheap because anybody can say whatever, anybody can say anything. But doing the real work of sense making, it's it's experiential, it requires your whole body, you need to bring your body to it, you need to engage with problems. Uh, you can't just talk from you know high up as if you know your words will just change things. It's uh there is effort and work. And so follow the people who you think are sense makers. That's what I would say, that's what I would recommend. Follow the people who help you make sense of the world of this chaotic world, right? And revalue wisdom. You know, wisdom is important uh because wisdom is lived experience, and there is a lot of wisdom out there, and you just need to find the right, the right wizards to to express this wisdom, right? And be part of that wisdom uh that is shared around the world in different spaces, not just social media. So yeah, and uh you know, my blog is I don't think a lot of so I I don't have a lot of followers anyway. It's just there's just a handful of people who read that, and I think that's because my way of writing is not for everybody. It's just there's some people who can read between the lines and and kind of like engage with what I'm trying to say, you know, the words behind the words. I think there's a term for that which is anti mimetic. And I think I like just think of myself as anti memetic, right? So like less viral, less like a meme, more like a read between the lines kind of kind of situation. That's that's my That's the way I normally write. And yeah, so uh I don't know. I I wish I had like a person I could I could tell you, I could point to, but there's so much information in my head every time, every day, every hour, that I I have to I'm really bad at remembering names.

SPEAKER_02

No, I think I no, I think your your blog is great. I first came across it uh when you wrote it was it was a year ago when you you wrote something you wrote the triangle, the unfolding the AI narrative.

SPEAKER_00

That's when I was your content.

SPEAKER_02

And it was circulating on LinkedIn. And you're right. Initially, it's not a like a viral meme catching thin thing, but it is so well thought out. And I think that's that's where that's truly where things really start flowing. And I I I do appreciate the way you write. So I think uh I'm going to obviously spotlight you and um and and link and link that that in the show notes. That's awesome. I I I envision a world where hopefully it doesn't turn this way, but we don't build detections anymore. But every time like a session is activated, there's an agent that kind of sits monitoring it and then it turns down when the session is is done with. And then the the agent monitoring goes back and reports or or or just blocks anything that's not allowed. But then what will we do? I don't know.

SPEAKER_00

Oh, there's gonna be there's gonna be somebody maintaining that for sure. You know, somebody laying out, laying down the pipelines, maintaining the infrastructure for it, monitoring for errors and bugs. So you'll probably be doing that part of that aspect of things, engineering, engineering the paved road.

SPEAKER_02

That's true. That's true. That I think that that's gonna be the next DE in the next 10 years for sure. If not already.

SPEAKER_00

It changes so much, Alex.

SPEAKER_02

This has been an amazing conversation. Thank you so much. I am your host, Alex Hurtado. This has been Detection Dispatch. We'll see you all on the next one.