The Former Google Cloud CISO's Take on AI, Agents, and What Comes Next

Some guests have seen the field from one seat. Phil Venables has seen it from most of them: CISO at Goldman Sachs, the first CISO for Google Cloud, and now a partner at Ballistic Ventures. So when he says what's worth paying attention to in AI security, it's worth a listen. We get into why he's more worried about attackers scaling up than about the flood of new vulnerabilities, what changes when the actor in your environment is an agent instead of a person, and why the smartest move is still going back to the basics you already trust. 

🔗 Podcast: https://alice.io/podcast

Follow the show so you don’t miss the next episode.
New episodes every two weeks. Stay curious.

Transcript

SPEAKER_00 0:00

The dirty secret of the entire security industry is there's been way more vulnerabilities that have gone unexploited than have been exploited. And so most organizations, targets of opportunity that have not actually been targeted, it means they can just be relentless. They can scale the monetization of those attacks so there's just no place to hide anymore. AI-driven attackers, they're going to be relentless in finding that one misconfigured system, otherwise wonderfully configured cyber hygiene.

SPEAKER_02 0:29

If AI has ever made you stop and think, wait, what is happening? You're not alone. I'm Mo, and I'm a security researcher asking the same questions. On Curiouser and Curiouser, we're having open conversations with experts, researchers, and leaders working at the edge of this space, talking through how AI is taking shape, what's shifting, and how people inside the work are thinking about it as it happens.

SPEAKER_01 0:52

So join us and listen in as the conversation takes shape.

SPEAKER_02 0:56

Hello, hello, and welcome back to Curiouser and Curiouser. Uh, really excited for today's guest. Uh he's got a really cool background, um, but as usual, I do not want to skewer it, so I'll let him introduce himself. But today we've got Phil Venables, who is a partner at Ballistic Ventures. So, Phil, thank you so much for being on today.

SPEAKER_00 1:18

Yeah, pleasure to be here. So, yeah, just a bit of background about me. So I've um uh been doing cybersecurity for a long time. I initially started as a software engineer many decades ago, but got into doing security in various forms. Um was a longtime chief information security officer at Goldman Sachs, then uh chief operational risk officer and a board director. Spent the past five years at Google as the first CISO for Google Cloud and uh and ran security engineering for Google's technical infrastructure. And yeah, now as you said, I'm a partner here at Ballistic Ventures. So we uh uh we invest in Seed and Series A and other early stage cybersecurity companies. We've got a great portfolio across uh across many different segments of cybersecurity. So looking forward to the discussion today.

AI's Real Impact on Security

SPEAKER_02 2:04

Yeah, no, I mean you guys definitely have a very cool portfolio. I mean, I've been on one of your podcasts, and again, all the topics that uh y'all cover at ballistics are super cool. Um, and I think you're probably one of the best people suited to like, I guess, talk about this space as well. When we think about AI and like everything that AI is doing, especially in the security space, uh, there's just like so much noise, and it's really hard to just cut through it. So that's like kind of how I want to start because you've seen this from a lot of places and angles where most people haven't, um, from Google infrastructure, um, even like some of the advisory boards you've been on. So, what's kind of like the vantage point that you're seeing for um AI security from like what is all the noise that people are really just like getting bogged down by um versus what is actually helpful?

SPEAKER_00 2:51

Yeah, so there's as with any massive technology shift, just like we've seen in prior shifts of internet, mobile, cloud, AI is probably an even more pervasive shift. And when you look at how it impacts security, you've got to unpick it a little bit. So there's the security of AI, and so that's all of the things that security and risk teams are doing to make sure that their organizations, when they deploy AI for business purposes, that they're doing that in safe, secure, regulatory, compliant, managed ways that that manage all the risks, not just the security risks of those AI AI deployments. Um, and then you've got the you know security uh as delivered by AI, so AI for security, and this you see across everything from software security, security operations, a whole range of different things. And then finally, you've got this broader impact of how AI is affecting the entire security landscape. And I think, as you know, that's what's dominated the headlines for the past few months with the so-called mythos moment. Although that's a little bit of a false moment in time because you know, the quarters and year before that, everybody in the security community was seeing and realizing the profound effect that AI models, particularly models trained for coding, could have on finding vulnerabilities and chaining vulnerabilities together for a tech. So, you know, one element of this is we've got this tidal wave of vulnerabilities that is hitting us because the models are so good at finding vulnerabilities. Um, but that's kind of a short-term thing and a long-term thing. So the short-term thing is quite worrying. Long-term, though, everybody is applying those models to their own code code base to find and fix things at rates that we've not seen before. So I think ultimately that could be a good outcome. Uh secondly, though, which I think is largely going underreported, but for me is more worrying is the extent to which attackers are now using AI to industrialize what they're doing. So they're using attackers, they have always been resource constrained, and so there's always been more targets that they've not exploited than they have exploited. Now in AI, just like everybody else, they can industrialize and scale and 10x

How Attackers Industrialize at Scale

SPEAKER_00 5:11

or 100x of volumes of attacks they can put together. So that's going to be the really worrying thing. There's kind of no room and no place for organizations with weak security to hide anymore. And then finally, you've got what you might describe as a quest for authenticity. So you've got um fakes, fake workers, fake content, fake brands, all of this other stuff that's driving us as consumers and businesses and governments to want to know what is authentic versus not. So that's kind of a kind of the grand tour of everything's going on. So you can see why everything seems to be changing all at once, because it because it actually is.

SPEAKER_02 5:50

Yeah. And you touched on a lot of like really, really big things too, from like the attackers being able to catch up in these capabilities really fast. I mean, if you just look very recently, right? Fable five came out, and the only difference between Fable V and the most recent Mythos release is a couple of safeguards and and uh you know protections, right? So it's like the the very thin line in the sand between uh an attacker's capabilities and what is actually frontier uh continues to get smaller and smaller. Um, even as we continue to see like open source obliterated models um start to get used across open source attack platforms. So really like just the ability and the scale that uh attackers are able to just go and productionize is really, really extreme.

SPEAKER_00 6:38

On the other side, well, and and and as well, you know, when you look at you know, mythos is not the only model with these capabilities. You've got Codix, you've got Gemini, you've got others. And as you point out, over time, more and more of the open models are going to have more of these capabilities. Certainly, when you look at many of the mythos discovered vulnerabilities, it's been clear after the fact that other lower-end models can and did also discover those when you apply them. And so I I think anybody that's basing a sense of security on the restrictions in advanced models or the restricted availability of the advanced model is going to be disappointed because ultimately the cat's out of the bag, everybody's gonna have this capability, to your point.

SPEAKER_02 7:23

Yeah. Um, there was like early research done last year, and it continues to be done every time a new model is released, that um researchers continuously show that with open source models and the right type of reasoning, right? Being able to like actually walk a model through how to go and do this, um, you eventually train these open source models to go and perform these mythos type um, you know, mythos level uh exploits and these chain of thought and chain of reasoning. So really it seems like the reasoning layer is the big differentiator that is continuously getting smaller and smaller as these models just get better. And the cost of that is really, again, it's becoming lower and lower. Um, when I looked over the weekend and I saw like Fable was out, my Opus too um limits were now doubled, right? Because they were like, oh, well, Opus is now gonna get cheaper. Um, but that's because Fable's now out and that's the more expensive one. Um but a couple months ago we saw that again, you know, Sonnet is now like the cheapest one, where at one time it was the most expensive thing to run. So these costs, um especially from producing code, um, doing reasoning on these really complex problems, it's getting a lot cheaper and easier. Um, but the vulnerability density isn't really going with it. So um, we're not only just seeing like more code, but we're seeing way more surface area that attackers have to like kind of pick at. And it's just like there's so much to attack now. Uh, I'm wondering like, uh like our practices, they already don't cover most of these things. Like, um, I've always said that AI is challenging the foundations of the organization uh security posture, where if your organization did not have that great of a security posture or was not ready in most cases, this is really going to test a lot of those foundational practices. So I guess does that concern you at all with how fast the threat landscape is growing, how cheap it is to perform these attacks, and um how easy

Is Your Security Posture Ready?

SPEAKER_02 9:16

it is to scale them?

SPEAKER_00 9:16

So, on the final point you made, so I I agree with that. So, Google's developer operations research analysis, uh, this thing called Dora, not to be confused with the European Digital Operational Resiliency Act. So the Google's DORA team did a lot of research on this. And and the conclusion is kind of obvious in hindsight, but it's good for it to be founded on research that if you take AI-driven software production into an organization that's got quite unmanaged and chaotic software development lifecycles, you're going to get chaos amplified. If you deploy AI-driven or agency-driven software production into an organization that's got quite a well-controlled software production pipeline and controlled build and testing processes, you get productivity amplified. And so this really does shine a light on the organizations that have yet to get their software production under control with testing and security and quality assurance and all the things that we expect. I think again, when you come back to the notion of we're going to generate more code with all of this, absolutely the the amount of software and the amount of backlog most organizations have in their desire to produce software and new systems is now being met and if not exceeded by the capability of some of the models. So we're going to get an enormous amount more software. I think what is not entirely clear yet is whether in that software there's going to be a greater or lower density of vulnerabilities. So some models are better at producing secure code than others. Most organizations, back to that point of the software development process, are getting better at post-training or prompting models to generate secure code using the libraries they expect and then not bringing in any unauthorized or extraneous um third-party libraries. So that whole process is getting better all the time. And I think, you know, if you if you'd put me on the spot now with a prediction, I I think ultimately the models are going to produce a lesser density of vulnerabilities. There's still going to be vulnerabilities, but I don't think we're going to see a uh I don't think we're going to see a higher density of vulnerabilities. I think so ultimately software is going to keep getting better. But to your point, there's going to be a massive amount of software, and so we're going to see bigger attack surfaces. The thing though, and you're correct to point it out, that we really do need to worry about how attackers use this. I mean, the dirty secret of the entire security industry, as I mentioned before, is that there's been way more vulnerabilities that have gone unexploited than have been exploited. And so most organizations are kind of targets of opportunity that have not actually been targeted. What AI does in terms of the industrialization of attackers, it means they can just be relentless. And so even if they don't get an immediate exploit from even a new vulnerability they've discovered, they can just at low cost just keep hammering away at organizations. They can scale the amount of organizations they target, they can scale the amount of organizations that have a vulnerability that can be subsequently exploited in various ways. They can scale the monetization of those attacks. And so I think that's the real issue here is not necessarily that we'll discover more vulnerabilities, although that is an issue. The real bigger issue is that this enables attackers to be relentlessly operating at much more significant scale. So there's just no place to hide anymore.

SPEAKER_02 12:45

You know, from a security standpoint, it's probably a bad thing, but it's pretty exciting, right, to see like how much this is growing. I think we have to see attack innovation to innovate on defense as well. So the exciting part, I think, about the entirety of AI is um while we do have this attack surface that is now, as we said, getting hammered relentlessly, um, there is the opportunities for teams to innovate in really interesting ways. So, for example, um, when this first happened, open source repos

The Defender's Opportunity

SPEAKER_02 13:14

were getting pounded by um PRs and bug fixes that were coming up all the time. And I remember in some cases, these open source programs would close their um their bug programs and they would just say, no, we can't handle all this volume. Well, now we're actually seeing a lot of these maybe go to like auto vulnerability fixes, right? We're seeing some cases uh where teams are now leveraging AI to help them reduce some of that, like both the cognitive load of so much more noise, but also streamline some of those operational uh inefficiencies that existed just because uh it required so much human touch. So I guess what are like some of those exciting parts of programs and maybe how is AI kind of like uh lifting up those teams that either didn't have the budget, didn't have the people, um, maybe didn't have the time to prioritize. How do you feel like it's gonna kind of change the the landscape for them?

AI Levels the Playing Field

SPEAKER_00 14:11

Yeah, so AI is a great kind of democratizer of capability, and so we see this quite a bit where organizations, and I I've always thought this that organizations want to be more secure, and they either can't afford to be or they don't know how to be. And you know, when they don't know how to be, they can't afford to hire the security team that does help them understand what to do, and then they can't afford to pay for all of the controls. And you know, there's some very high-end organizations and even some medium organizations that do this really well, but for most organizations, they're in this kind of permanent debt of not being able to deploy enough security capability, whether it's basic cyber hygiene all the way through to other types of more advanced things like continuous red teaming. And now what you see is organizations um applying technology from particularly new startups that have built technology based around AI, using AI agents to provide almost infinitely scalable security capabilities. So I mean, you've probably seen some of the announcements that we did recently. So, for example, Kevin Mandia's new company, Armadin, is basically AI agents for full spectrum red teaming. So most organizations don't have the resources or capability to continuously red team themselves. And now you can now you can do that. Um, another company we just invested in, a company called Above Security, is the same thesis, but for um but for insider threats, most companies would love to have a world-class insider threat program. They generally can't afford to do that in the way they would like. Now you can augment a small team with a large amount of agents to deliver you a world-class program. Another one, exact same thesis, a company we invested in called BreachRX, which is AI and workflow support for managing incidents of various forms and coordinating incident response and incident disclosure. Uh again, same thing. Everybody would love a world-class uh multi-domain incident response team. Not everybody can afford that, and not everybody can afford to scale that. Now you can get agents to augment a team to do that. And the same pattern, uh, the same pattern repeats everywhere. But can stepping back a little bit though, it's worth reminding ourselves that while AI is going to be a massive boost

Why the Basics Still Win

SPEAKER_00 16:30

to security, whether it's software security, operation security, or all the things I just talked about, we've also got to remember that good old-fashioned, basic high levels of cyber um cyber hygiene type defenses, strong multi-factor authentication, network segmentation, binary authorization, all of those least privilege access controls, many other things implemented relentlessly and implemented well, provide a defense to even AI-assisted attackers. The main thing though is we've got to implement those at much higher rates of consistency, because back to that point of AI-driven attackers, they're gonna be relentless in finding that one misconfigured system in your otherwise wonderfully configured cyber hygiene, uh, and that they'll use that as a launching pad. So um, so yeah, and I think AI is a tremendous boost, but you don't need AI, you don't only need AI to defend against AI-driven attackers. You can you have to do all the basic stuff as well.

SPEAKER_02 17:31

Yeah, no, that's again a great point. And it's always like back to basics, making sure you have your foundational practices in place. Um, it's just security tech debt, right? If you don't have that fixed, how do you move on with getting a better world-class program? Um, you've kind of got to dot all your eyes. Um, but on that, I'd actually like to go back to agents because you brought up the keyword. Um, everybody's really talking about them, even though it's been around or the concept's been around for a while. It feels like right now agentic is just like taking flame. And um, you mentioned augmenting staff with agents, which is the natural progression of how this technology is going and how I think uh we are envisioning it. So when we look at agents and bringing those into uh the environment, I think it introduces uh a lot of unsolved problems that uh I think we've tried to apply traditional security mechanisms to um around like IM access management and role-based access controls, data loss prevention solutions. But all of these were really designed around the humans, right? So the threat model for an agent is totally different. So when we think about the actor within our organization as an agent, what do you think are kind of the new ways we need to think about that trust layer? Um, and maybe how we need to start reasoning about trust uh for AI systems that are working autonomously within our environment.

SPEAKER_00 18:52

Yeah,

Trust and Identity for AI Agents

SPEAKER_00 18:53

so I I there's a number of ways of looking at this. And I and it's interesting, we're at the very early days of this, and uh yeah, I, you know, some of your listeners may be old enough, I suspect not many will be, to remember the early days of the internet. So in the kind of the late 90s, early 2000s, we had a set of technologies for the nascent commercial internet, browsers, web servers, load balancers, all of this type firewalls, intrusion detection systems. But there wasn't there wasn't really a set of fixed design patterns of how all those things should be plugged together. And so there was a lot of you know uncertainty around how the security models should work, and then over a period of years, the design patterns got locked in, and then we overlaid security onto those design patterns. This feels like exactly the same moment where almost every day somebody's inventing a new design pattern for how agents should communicate or how agents should interact with resources or how agents should drive a business workflow or a technology workflow. So I think we're not really going to see a stability in how we think about security until we start seeing some more coalescence of common agentic design patterns for particular problems. And an example of that is exactly on identity. So, you know, should an agent uh operate under a delegated identity and a delegated set of permissions from a human? Yeah, probably for certain use cases where an agent is acting on your behalf. Uh, but for other use cases, an agent should probably have its own permissions in the context of a business workflow. And then maybe for other cases, it should only have an ephemeral identity that's spun up and spun down to deliver a particular subtask in a workflow coordinated by other agents. And every one of those different use cases will have different properties of how you think about the identity, the permissions, the observability. And so all of that is going to be different in every use case. Then overlaid on top of that, we all know agents using models are by definition non-deterministic. So you can you can ask a model a question ten times and you might not always get the right answer ten times. But for deploying agents into business processes, particularly financial transactions, health transactions, other critical infrastructure, those need absolute determinism. And so putting deterministic controls around these non-deterministic agents that aren't in the model itself or in the agent itself, but is in surrounding guardrails that enforce business policies, just like you would enforce business policies around a human's non-deterministic behavior, is what we still have to architect. And there's, you know, again, there's many, there's many companies and many solutions and much work from the foundation model companies to look at how you augment agent activity with agent guardrails. And it it's kind of very reminiscent. Some of your listeners may be aware of this, very reminiscent of what banks do in high frequency trading systems. You know, they have Um, you know, they have algorithms, you know, some machine learning derived, some not, but then they always have independent checks like circuit breakers to detect if the agent in in modern language is going off the rails than to kind of block its activity. And so we're gonna need the same type of agentic enterprise control plane that we've built for other purposes. And again, this is evolving as we speak because the design patterns are evolving.

SPEAKER_02 22:27

Yeah. Um, you know, like one of the big things that like we think about um at Alice is a lot about um how do you get an environment where agents can kind of play and you can understand. There's this concept of a gym or like an RL gym where we have agents kind of like go at scale, they kind of just like run their business scenarios and we just observe them and we watch kind of just like an Antill farm. Um and they go, they do their thing, and we recognize the behaviors, we see where it's going out of alignment. But since it's a simulated environment, we're not seeing those same risks happen in a production environment. But we can catch all those like really bad behaviors before it actually goes into production. Unfortunately, this is not something I think every organization has. Um, not at any fault for the organization, but because a lot of organizations are actually getting agents from another place, right? So they may be going to a vendor and bringing agents from outside, kind of like uh almost like getting a bringing in a subcontractor or a contractor to work on one of your teams or a project. So um as an app sec guy, I've always been trained to kind of shift left and move more left, but in this case, uh there is no left, right? You kind of just have to um depend on processes that you have in GRC and your SOC to hope that like this new external party in your environment is kind of you know playing playing well with everything else. So I'm wondering these controls that get embedded for agents that uh that are bringing in, are they really like holding up? Are there maybe something something else that we need to think about in in this like runtime layer for agents that we're kind of consuming, not just creating?

SPEAKER_00 24:01

Well, it's interesting you kind of use the shift left because you know, in other spaces, I'd advocate, and including this, that you're exactly right, we need to move from shift left, but instead of shift left, we need to shift down into the platform. And so this could be you know where agents and in fact any other piece of software should inherit a set of security and other risk mitigating controls from the platforms that they operate within. And that includes the runtime environment, the interfaces to other systems, the libraries, the frameworks that they pick up controls from. Um, and so having that kind of shift down helps you with not just the agents you've developed, but the agents that turn up in your environment from third parties, from you know, potentially unexpected sources. Because essentially, as a as an enterprise, you want to you want to be able to reason about to say, for example, let's say I've got a payments gateway or a stock ordering system or some other critical system. Um, I don't want to build the controls only into the agents that may be interacting with that, just in the same way that organizations don't depend on correct human behavior. You have a combination of trained humans, or in this case, trained and well-controlled agents, but you still build tremendous amounts of access control, transactional policies, auditing, observability, and many controls into the resources and systems that are being manipulated by those. And then reasoning about that collective enterprise control plane is what's critical. And that has to be in the you know, shifting down into the substrate of the organization in the runtime environment. Now, the big question is what happens to that third-party agent that's running in your environment when your controls stop it doing something. How do you signal back to that vendor, hey, I've blocked you because you were doing something crazy? Um, all of these things are yet to be defined, and you know, and that's that's why this space is so exciting.

SPEAKER_02 26:29

Yeah, exactly. And I really like how you put it in shift down, you know, getting lower into the stack where agents are operating. This is kind of fundamental, or it's it has to happen, especially when, like you said, you have agents that are likely agents within your organization from different vendors all kind of interacting with each other, right? You don't necessarily control any of those interactions or have much observability on that shift down layer as to like what they're doing, right? So if I bring in an agent from one platform and an agent to another, they happen to interact on a project, then we've got a problem where there's no human, it's just agents, and we don't understand the gravitational risk that one agent has on the other. So being able to report that back, you know, I've said that guardrails are really value adding when you can take the signal from guardrails and turn it back into a flywheel process. Um, because again, something gets caught by a guardrail, it's not necessarily training the model to get better, it's not training the agent to get better, it's just stopping a bad result from happening. Um, but from there, turning that into signal where you can go and retrain or provide feedback, I think that's really important to be able to say, hey, we caught something bad happening. This is how you um we would like to see this going forward, this is how you fix it. So I think having that feedback loop or that flywheel cycle embedded within like uh when you work with other vendors or outside is gonna be really important in moving forward with shifting down.

SPEAKER_00 27:50

No, I absolutely and I think you know it's gonna be just like any, just like all of the other security we've um invented and deployed in the past, it's all about how each layer interacts with each other in that kind of feedback loop. So if you've got a resource that is continuously rejecting the attempted behaviors of an agent, then you can't just keep rejecting that. You've got to think what went wrong in the identity and access management process for that agent that I've not appropriately permissioned it, or what went wrong in the enforcement layer that has an agent trying to do something that is against the policy that you want as detected by the resource that for which that kind of rogue access is being attempted, and how all of that ties together. We've spent decades doing that in the kind of human to application to back-end system to re- other resources, and we're gonna have to do the same thing. But the I think while it's gonna be conceptually similar, I think the nature and the scale, and to your point about kind of anthills, the extent to which we'll see emergent behavior from agent interactions that weren't quite predicted in our policy management systems, are gonna create some unusual behaviors that we also need to manage and monitor for.

SPEAKER_02 29:05

There is this other piece that comes with um kind of like all these agents at scale. The platforms that they're originating from or being developed on. We call them foundational models. Uh, I think Andreas had called them like a God model, right? Like we have these really big models that do everything. So now we're moving towards a place where um we have a couple of these big winners in the in the model space for enterprise. And a lot of vendors and solutions providers are relying on these models to create these solutions, uh, whether it's providing an agent, providing an MCP layer for this particular platform, um, or so on. So this concentration kind of reminds me of like these hidden dependencies or like these third-party dependencies that everyone kind of shares, but you don't see it at like uh at the high level because of where they are embedded. So I may be interacting with an agent that may be using open AI's technology and it has the same kind of issues, regardless of what use case I put it in, right? It's just like a basic benchmarking kind of thing. But now we're seeing this at scale with multiple providers, multiple agents. I mean, maybe I'm just like thinking about it wrong or I'm seeing it wrong, but does this problem like, do you think that this kind of exists at scale or how um AI has been rolling out?

SPEAKER_00 30:15

Yeah, well, I I think in general, nobody quite knows how it's all gonna land, and that's just the nature of, you know,

The Hidden Risk of Foundation Models

SPEAKER_00 30:22

that's the nature of the beast we're in at the moment. But I think when you look at the first of all on concentration risk. So, you know, we've had concentration risks in multiple industries and technologies for years, and people figure out ways of managing it. You look at the hyperscale cloud providers, never mind the foundation model providers that depend on the hyperscalers to run these things, you know, that's a degree of concentration risk that's managed in various ways by the hyperscalers or by companies figuring out ways to be able to deploy redundantly across multiple cloud and even on-premise infrastructure. So there's there's ways to deal with that. Then when you think about model dependency, most organizations I've seen over the past few years have been quite careful not to be wholly dependent on one model, mainly because they're always upgrading models anyway, just because they're trying to find the optimally priced model for the problem that they're trying to solve so that they're not solving every trivial problem with the highest model, and they're making sure they can refer up to a more sophisticated model when they need to. Now, this does point to a dynamic where you see in multiple spaces in in all of these different vendors that sometimes get accused of just being an LLM wrapper is I think, you know, there are some vendors like that, but there's a lot that don't that they provide a layer to enable companies to have a lot of routing or portability across models. And they pick up a lot of the work of model validation, model testing. And so you see these companies like Rogo AI and investment banking or Harvey and Legal Services, all these other companies. And essentially what they they provide a set of context, domain-specific knowledge, connectors, but also that layer that lets customers dynamically choose which model they're using and then leave it to that service provider to assess and validate whether that model is fit for purpose of what they're doing. And I think a lot more organizations over time are gonna, you know, they'll have plenty of UK use cases where they directly use the models, but they'll also have plenty of use cases where they just say, look, I value that layer of kind of context connection, knowledge, model validation so much it's worth me paying a premium to not directly use the underlying models. And I think you'll see various, various flavors of that. How that shapes out in software security is gonna be gonna be interesting. So you see plenty of harnesses around the models that help companies do software security. And then you also see plenty of companies that are helping organizations deal with the output of that. So I think there'll be multiple different frameworks of how you do this, uh, but not all organizations are gonna want to directly use the models for all things because they they just don't want to deal with all of the testing, the model routing, the updates, all of that kind of stuff.

SPEAKER_02 33:18

I I completely uh in agreement. I did recently see a post from Harvey where they um enabled uh Fable V within their environment as soon as it was released. However, they released it with a benchmark that showed exactly how well it was performing on their stat. So I think, like like you said, these providers, they're not always um just a wrapper for AI. They're a context layer, and they provide um a layer of context that these foundation model companies don't. So they would rather provide that context layer, allow you to figure out how you want to interact with it, but the consistency of results is kind of what you care about. And that's why you choose the model that makes most sense to you. Obviously, they figure out pricing on their end, and yeah, this one costs more because it costs us more. But otherwise, it's really on you to figure out, okay, well, this is how we want to go for it. And I think that's kind of uh that context piece, the cost, the value for the context, and then um the provider, all these three things makes it really difficult for leadership to kind of make choices on exactly what they need to be doing. Um, all of these systems get more complex, especially as they get more interconnected, especially when I think even the conversation that you and I just had specifically around having a model, having the context, and figuring out which solution fits best based on all of this. I think there's a lot of uh the it's almost like the answer is to get everything versus um get one thing. And that's not very cost effective. So I'm wondering if there's an easier way to provide leadership and engineering teams overall a more useful picture of like both the solutions that they can pick and the risks that are coming with all of these different things.

SPEAKER_00 34:58

Yeah, well, I I I think I think in each layer of security, you'll see some companies acting as that that context layer over and above the models or what the models deliver to do that. You know, you use the phrase kind of context. Do you know one of the key differences you see in many companies is they they build and deliver a context graph for an organization that builds its knowledge up that is then used in the use of the model to make better grounded decisions in the context of that organization. Um, you see this in plenty of places. Uh, for example, we've got a company called Armacode, um, that they basically help companies do vulnerability operations. And so they take in all of this information about vulnerabilities that have been discovered by AI models or from traditional sources and build this big graph of what your organization looks like to help you prioritize how to fix things and how to make sense of that. We see similar things in in many other spaces. So that that context graph is and the uh over and above the models is absolutely key. Because ultimately, again, most organizations they don't just want to use the models for the model's sake, they're using it to solve a business problem. And often the business problem is best solved by augmenting the model, not just using the raw model.

SPEAKER_02 36:17

Yeah. So based on this whole conversation, I guess if you had to take away one thing or give something to an organization to take away today, maybe an action that they could take or uh something that they should think about when they're either making their next purchase or building their next solution. What's like the one key takeaway that you'd want them to have?

SPEAKER_00 36:35

Yeah, I think it's to remember that while all this is new, it's not necessarily new from a

Back to First Principles

SPEAKER_00 36:42

first principles perspective. So managing AI risk is about software lifecycle risk, it's about data governance, it's about the operational risk of putting the guardrails and hardrails around behaviors. It's it's understanding who's got what identity, what resources being now, all of that is we've done that for decades in good and bad ways. It's somewhat different and more pressured in an AI and agentic environment, but it's the same basic principles. And I think sometimes in massive moments of change like this, we forget to go back to first principles, or we get, you know, get get dragged into the hype that this is all new and the first principles aren't relevant, which is just not true. So just I would say to everybody, just remember, trust your instincts on going back to first principles. And when you think like, why should I let an agent do all these things? The answer is you shouldn't. You should not just rely on the agent behaving problem, but you should put like controls around it, you should craft its privileges, you should put you know, access control enforcement in resource games. You've got to do all of that just like we've done for years. So trust your instincts and uh and keep sticking with first principles and everything will be fine.

SPEAKER_02 37:53

Cool. As a tech enthusiast, as a longtime software developer and everything, uh, what are you most personally excited about? What would you most be excited to see in the next couple of years with AI? What would you love to see emerge? What's like the thing that gets used?

SPEAKER_00 38:07

Yeah, it's not so it's kind of excited, but not necessarily in a good way, in a more of a curious way. So I think we've not done enough

The Second-Order Effects Nobody Sees

SPEAKER_00 38:15

yet to think about the second order effects. So again, I I go back to these kind of previous waves of technology. So, you know, when the smartphone really took off in the late 2000s, there was lots of discussions about the risks. But nobody imagined, and they couldn't imagine at that point what the second order risks were. So all of the things that came from what we built on mobile infrastructure, whether it's social media or kind of gig work or all this other stuff, there was loads of risks that came from that. I think we've yet to really develop an understanding of what the second order effects are going to be from this first wave of AI deployment. And this, you know, again, back to that Ant Hill's comment is what does it mean? What does a world look like of billions of agents, all with different models and reward functions, interacting in different ways under competitive pressures? Like, who knows what risks are going to emerge from that? I mean, there are some things that could be quite predictable, like when is the first agentic flash crash going to happen when some website posts up an incorrect price and a billion agents descend on it to try and buy and lock in that contract. You know, we're very close to that, I would think. And then what's these other second order effects? So I think what's going to be fascinating over the next few years that I'm uh excited to see how we figure out how to manage is just the the second order effects of emergent properties coming from a world of trillions of agents wired together in unpredictable ways. It's uh it's gonna be wild.

SPEAKER_02 39:49

Yeah, it is also gonna be very fun. I know like I'm in the middle of organizing an agent-only conference. So I'm excited to see what kind of uh what kind of talks they put together and stuff, right? So cool, Phil. Thank you so much. Uh again, it was a real pleasure to have you today. Where can people find you? What do you do you got going on? What's next?

SPEAKER_00 40:07

Yeah, yeah. So um, you know, my blog, you know, content out every two weeks is philvenables.com and uh I'm on X at Philvenables and uh and uh then beginning of next year, there's uh publishing a book on uh how to scale security uh for uh for organizations in a pre and post uh AI world. So uh look out for that. That'll be announced on uh all the uh all of the social channels in the in the coming quarters. Sweet.

SPEAKER_02 40:35

Thank you again so much. I'll be looking out for that book for sure, and maybe we'll have you on again to talk about it. Yeah, that'd be great. Thank you, Phil. Thanks for your time. And yeah, uh to everybody stay curious and have a great day.

SPEAKER_01 40:47

If this episode helped cut through the noise, like or subscribe so you don't miss what's next. Thanks for spending time with us. Until next time, stay curious.