The Current State of AI: Security, Cognition, and Enterprise

his week, we examine the cutting edge of cybersecurity innovation, where new startups are focused on securing AI agents and models. We also dive into the enterprise adoption gap, the rise of "shadow AI," and a fascinating MIT study revealing the cognitive toll that continuous reliance on large language models might be taking on the human brain.

Transcript

Welcome back to AI Weekly, I’m your host, Michael Hoosch, and  we have a jam-packed agenda today covering everything from the seed stage innovators looking to secure our AI future, to major enterprise platform releases, and even a new study that forces us to question what our relentless pursuit of AI productivity is doing to our minds.

We begin in the world of venture capital and cybersecurity innovation, as AI took center stage at DataTribe’s Cyber Innovation Day. DataTribe, for those unfamiliar, is a seed stage venture foundry focused on early-stage cybersecurity firms. Their annual DataTribe Challenge is a pitch competition where the most promising startups present their products to an audience of security experts and investors.

The importance of AI in the security landscape cannot be overstated, as four out of the five finalists this year directly involve AI. DataTribe MD and Chief Innovation Officer, Leo Scott, explains that the goal of DataTribe is to jump in and co-build with founders to help accelerate growth and de-risk the process, leveraging the experience of their team who are all former startup operators.

The five firms chosen are Ackuity, Cytadel, Evercoast, Starseer, and Tensor Machines. Let’s briefly break down what these innovators are building, because this is truly a glimpse into the next frontier of cybersecurity.

First up, Ackuity. They are focused on security for AI agents. With agentic systems becoming more prevalent—and we’ll touch more on agents later—Ackuity collects telemetry from all interactions within an agentic system, whether it’s between the user and the agent, different agents, or the agent and other tools. Scott notes that they are doing a kind of pre-SIEM processing, providing real-time threat detection and response capabilities because existing SIEMs don't have the context for this new kind of agent telemetry.

Next, we have Cytadel, based out of London. This firm offers AI-driven autonomous red teaming. Their platform leverages AI for decision making to build complex attack paths, achieving a level of complexity historically only possible with human red teamers. This means organizations can validate defenses, quantify resistance to ransomware, and generate reports on weaknesses and possible solutions.

Starseer, the fifth finalist, provides security for new AI models themselves. The rapid adoption of AI is creating a little-understood and often ignored brand new threat surface. Starseer's platform helps AI system builders reduce system risk and improve resilience before deployment. They treat AI systems as probabilistic, analyzing the models, training data, refinement steps, and surrounding architectures to surface exploitable weaknesses and operational risks.

And while most are AI-focused, we must mention Tensor Machines, the only finalist not directly focused on AI. They provide a hardware security solution that allows printed circuit boards, sensors, and other components to continuously self-certify authenticity while in use.

Finally, there’s Evercoast. They are described as cyber adjacent, focusing on providing the data used in embodied AI—the AI that controls robots. They are adapting their process, which began in the entertainment industry, to train wheeled robots how to move independently. The goal is to provide clean data for the AI to learn from, making it more difficult to hijack the robot for nefarious purposes.

 This focus on agents and new AI systems aligns perfectly with a major theme we’re seeing in the corporate world: the drive toward Agentic AI.

Google Cloud just launched Gemini Enterprise, which they call "the new front door for AI in the workplace," aiming to put an AI agent on every desk. This platform bundles Google’s entire AI stack, including the powerful Gemini models and the core technology formerly known as Google Agentspace, into a cohesive experience. The goal is to democratize the creation and use of AI-powered agents for automating complex workflows.

One of the key components is a central governance framework that allows organizations to monitor, secure, and audit all agents from one place, incorporating protections like Model Armor. Early adopters like Virgin Voyages have deployed a fleet of over 50 specialized AI agents, seeing impressive results like a 40% boost in content production speed and significant cost reductions. They emphasize that AI is meant to unleash human potential, not replace people.

However, the path to AI adoption is definitely not smooth. A new report from Red Hat found that despite substantial investment, 89 percent of businesses are yet to see any customer value from their AI endeavors. The primary barrier, cited by 34% of respondents, is the high cost of implementation and maintenance, closely followed by data privacy and security issues at 30%.

And here is a finding that should raise alarms for every Chief Information Security Officer: 83 percent of organizations reported the unauthorized use of AI tools by employees, a phenomenon now widely known as "shadow AI". This highlights a major disconnect between IT strategy and the daily practices of the workforce, inevitably introducing security risks and inefficiencies.

This leads us right into the critical security flaws we’re already seeing in developer tools.

Earlier this month, Legit Security detailed a major vulnerability in GitHub Copilot Chat that allowed for the leakage of sensitive data from private repositories. The flaw combined a Content Security Policy, or CSP, bypass with remote prompt injection. Legit Security’s researcher, Omer Mayraz, was able to influence Copilot's responses and successfully leak things like AWS keys and zero-day bugs from private repositories.

The attack leveraged a hidden comments feature, which allowed instructions to be injected into other users' context

The exploit was sophisticated, involving creating a dictionary of letters and symbols and pre-generating corresponding Camo proxy URLs for each, ultimately bypassing GitHub’s restrictive CSP blocks and exfiltrating data when a user clicked the resulting URL. GitHub has since addressed the issue by disallowing the use of Camo to leak sensitive user information.

But AI isn’t just a target; it’s rapidly becoming our most powerful defender.

Google DeepMind recently deployed CodeMender, an AI agent designed to autonomously find and fix critical security vulnerabilities in software code. This system has already contributed 72 security fixes to established open-source projects in the last six months.

CodeMender uses the advanced reasoning capabilities of Google’s Gemini Deep Think models. It’s not just reactive, patching newly discovered issues; it’s proactive, able to rewrite existing code to eliminate entire classes of security flaws, like applying bounds-safety annotations to prevent buffer overflows. Crucially, it has an automatic validation framework to ensure changes fix the root cause and don't introduce new problems. The DeepMind team is currently taking a cautious approach, with human researchers reviewing every patch before submission.

Now, let's pivot to perhaps the most fascinating and concerning piece of research this week—a study that touches on how all this technology is impacting us, the users.

A study from MIT found that using Large Language Models not only makes the human brain work less hard, but its negative effects continue and affect mental activity in future work. Researchers monitored brain activity using EEG during essay writing, comparing groups that used LLMs (like ChatGPT), groups that used Google Search, and a ‘brain only’ group.

The results were stark: the unaided group exhibited the most active grey matter, followed by the search engine group, with the least neural activity found among the AI users.

The study also tracked 'ownership,' finding that few students using an LLM were able to reliably quote or summarize what they had written afterwards. Furthermore, the LLM-using group produced statistically homogeneous essays.

The longer-term effects are perhaps the most concerning for society. Over four months, participants in the LLM group performed worse than their counterparts in the brain-only group at neural, linguistic, and scoring levels. Those who used AI from the outset showed reduced brain activity over time, and were less able to perform cognitive tasks when asked to go ChatGPT-free.

The researchers suggest that AI can be beneficial, but only if used after a person has fully explored their thoughts, experience, and knowledge without technology. Using AI as a replacement for human thought, considering, and summarizing skills seems likely to cause the ability to think effectively to diminish into the longer term, highlighting a "pressing matter" of a "likely decrease in learning skills". The research group states that more study is required before LLMs are recognized as being net positive for humans.

The innovation wave is strong, from DataTribe's finalists securing agents to Google DeepMind fixing our code. But as we democratize powerful AI with tools like Gemini Enterprise, we must simultaneously wrestle with the security implications, like the GitHub Copilot vulnerability, and the cognitive toll that continuous reliance may impose.

That’s all the time we have for this week. Thank you for tuning into AI Weekly. I’m Michael Hoosch, and we’ll catch you next time.