Maximizing capability gains for AI agents

An AI system built software in 14 hours for $251, a task normally taking a human team months. This is AI agents in education.

In this episode:

• An AI system, Opus 4.7, accomplished a complex software development task in 14 hours that typically requires human teams months, showcasing dramatic AI capability gains.
• The way we interact with AI is evolving from co-intelligence chatbots to autonomous AI agents that complete complex tasks with minimal human oversight, shifting the focus to managing AI in schools.
• Ethan Mollick emphasizes that domain expertise is crucial for maximizing the effectiveness of AI agents, as experts achieve better results when assigning AI for complex tasks.
• Educators must adapt their approach to AI literacy, moving beyond basic prompting to teaching students how to critically manage AI agents, thereby preparing them for the future of AI work.
• The exponential growth of AI means schools must implement continuous professional development for teachers to strategically leverage AI for complex tasks and navigate constant technological shifts.

Chapters:

• 00:00 — Cold open & welcome
• 00:30 — Opus 4.7: Staggering AI capability gains
• 01:00 — Measuring AI's ability to do real work
• 01:45 — Near-frontier AI models and their exponential growth
• 02:30 — Implications for AI agents for educators and workflows
• 03:15 — From chatbots to AI agents: A changing interaction model
• 04:30 — Managing AI in schools: The manager's role in the future of AI work
• 05:45 — Rethinking AI literacy and collaborative reasoning
• 06:45 — The impact of exponential AI capability gains on institutions
• 07:30 — Designing professional development for continuous adaptation

What are AI agents and how do they differ from chatbots for educators?
AI agents are long-running, smart, self-correcting AI systems that tackle complex tasks with less human intervention than traditional chatbots, requiring educators to shift from interactive prompting to managing AI workflows.

How can teachers use AI agents for complex tasks in the classroom?
Teachers can assign AI agents to handle time-consuming administrative tasks, content generation, and aspects of differentiation, freeing up human capacity for unique human skills like judgment and relationship-building.

What is the future of AI work for students and how should schools prepare them?
The future of AI work involves managing AI agents rather than just using chatbots, so schools should teach students to critically evaluate AI outputs, identify biases, and apply human judgment to AI-generated insights.

Featuring: Dan Fitzpatrick, Ethan Mollick, Claude Fable, GPT-5.6, METR, UK’s official government AI Security Institute, GDPval, Epoch, Opus 4.7.

Read the original source

Follow AI in Education with Dan Fitzpatrick for more on AI in education.

Transcript

SPEAKER_00 0:00

If this episode makes you think, please let us know in the comments and support us by subscribing and leaving a review. Thank you. Today we are exploring the rapid advancement of AI capabilities and what that means for how we'll work with these tools. Drawing insights from an article by Ethan Mollock titled The Twilight of the Chatbots, what the article really drives home is how quickly things are accelerating in AI right now. In fact, one of the most striking findings is that an AI system called Opus 4.7, working on its own for just 14 hours, was able to build a software package that would have taken a human engineering team between 2 and 17 weeks to complete. That's a staggering capability gain, and it cost only about 251 in tokens. Now Ethan Mollock, who you might know from his incredible work at Wharton, really makes the case that this isn't just about faster model releases, though those are happening too. We're seeing accelerating capability gains, especially when we look at AI's ability to do real work. He points to several assessments that try to measure how much human work AIs can actually accomplish. For instance, organizations like METR and the UK's official government AI Security Institute estimate the amount of human programmer hours an AI can do with a single prompt. Another one, GDP Val, compares human experts in many fields to AI performance using professional judges. What they're all finding is that these capabilities are increasing at a better than exponential rate. As I mentioned, the recent findings from Epoch on Opus 4.7 are just mind blowing. Think about that. A task that could take a human team months being done by an AI in less than a day. And in his own experiments, Mollock found that a model called Claude Fable was able to work autonomously for nine hours on very complex software projects that would have taken a team well over a week to complete. Now he's quick to point out that AI systems can't pass Goritz every test, and they aren't always cheap to run. But the rate of improvement is truly phenomenal. Now it's not just the frontier models, the ones from Anthropic, OpenAI, and Google that are showing these leaps. There's also a second set of what he calls near frontier AI models, mostly from China, which are open weights models. That means anyone can use or modify them, making them really cheap to operate. And guess what? They're also climbing up their own exponential improvement curve, just a bit behind the closed US models. He even shows this in a graph of AI performance in a test called AA Briefcase, which simulates a complex multi-week consulting engagement. So here's what this means for us in education. This dramatic increase in AI capability gains isn't just a technical curiosity. It profoundly impacts the landscape students are preparing for, and frankly, the landscape we as educators are working in right now. It means we have to really start thinking about where we apply AI agents for educators within our workflows. If an AI can tackle tasks that take weeks of human effort, what does that free up for teachers and school leaders? It's the ultimate expression of outsourcing the doing, not the thinking. We should be identifying those administrative tasks, those content generation tasks, even some aspects of differentiation that are consuming precious teacher time and seeing how these powerful agents can handle them so we can redirect our human capacity towards the things only humans can do wonder, care, judgment, relationship, and imagination. The second thing teachers should know is that the way we use AI is fundamentally changing. Until very recently, most of us were interacting with AI as a co-intelligence. You'd ask a chatbot to do something, you'd check the results, you'd give it the next step. It was a constant back and forth, with the human firmly in the loop at every single turn. And that approach is still incredibly useful, but the article points out that for valuable work it's becoming less and less the dominant way AI is being used. Instead, we're moving towards AI agents. These are long-running, smart, self-correcting AI systems that don't need constant human intervention. They come with extra machinery, things Mollyck calls harnesses, which give the AI access to tools and an environment to act in, and specific apps built for agents like Claude Code or OpenAI's codex. This means the already increasing ability of the core AI models can be improved even further by a well-designed harness or app. It's like giving the AI a toolbox and a workshop to operate in, rather than just a conversation window. So work is increasingly about assigning work to these agents, rather than working interactively with chatbots. The article shares a fascinating joint study by OpenAI and academic economists that shows how quickly this is happening within OpenAI itself. And here's the kicker. It's not just coders who are using agents. Legal, HR, and other non-tech functions have adopted agents at nearly the same rate. This makes OpenAI a kind of canary in the coal mine for what will happen in the broader world of work. Increasingly, work at OpenAI looks like managing AI. A quarter of their workers are running at least four agents every single week. This means other roles are starting to become coders of a sort, not in the traditional sense of writing code line by line, but in orchestrating these AI agents effectively, and they're good at it. A separate study of clawed code users found that software engineers had a similar success rate to other professions when using clawed code on coding tasks. What truly mattered, according to the article, wasn't the user's profession but their expertise. The more domain experience someone had, the more successful they were in using clawed code within that domain. And even more interestingly, the more useful output they got from clawed from each prompt. This is a crucial distinction. We are moving from a world where non-experts use chatbots to fill in gaps, to one where experts use agents to get work done. And the best way to use these agents is to think of yourself as a manager. This insight has massive implications for managing AI in schools. We've often focused on teaching students how to prompt, how to interact with a chatbot. But the future and frankly the present is moving towards teaching them how to manage AI agents. Think about a year eight geography lesson where students are tasked with analysing climate data for a specific region. Instead of just asking a chatbot to summarize information, an agent could be assigned to pull data from multiple sources, run some initial statistical analysis, and even visualize trends all before the student even engages. The student's role then becomes less about data collection and more about critically evaluating the agent's output, identifying biases, interpreting the implications, and using their unique human judgment to tell the story of that climate data. This is about teaching students not to outsmart machines, but to outthink them. If you're finding this conversation about AI and education thought-provoking, please consider following or subscribing to the podcast. We regularly explore these shifts and what they mean for our schools. So for school leaders, this means rethinking what AI literacy truly is. It's not just about memorizing tool features or accepting outputs uncritically, it's about developing that collaborative reasoning ability, understanding AI limitations, managing complex AI conversations with precision, and having a reflective awareness of AI's influence. It's about understanding what the AI knows and crucially what it doesn't know. The article's finding that domain expertise is key for effective agent use underscores that our role as educators is to deepen students' subject matter knowledge and critical thinking, not just to teach them how to use tools. The third big takeaway from Ethan Mollock's article is simply the nature of being on an exponential curve. It means that each change over a fixed window of time is larger than the one before it. We keep experiencing what is a steady doubling of capability as a series of sudden, almost shocking leaps. He argues that this explains the turbulence around AI far better than the usual stories about hype. AI isn't capable of being a real cybersecurity threat until suddenly it is, causing improvised policy changes. Markets discount whether AI might undermine a business model until suddenly it can, leading to massive swings in stocks. These lurches get read as signs of an immature field that will eventually settle down. Mollock doesn't think it's going to settle anytime soon. The instability, he suggests, is what happens when institutions that move at the speed of people, or worse, committees, try to track a capability curve that is very much not human in nature. And as long as we're on some sort of exponential, and for as long as it lasts, the gap between human and machine speed only widens. This is critical for how we approach AI for complex tasks and managing AI in schools. For department heads planning CPD, this isn't about a one-off training session on a specific tool. It's about building a culture of continuous evolution, not revolution, where teachers are given the time and space to experiment and adapt. We need to be designing professional development that empowers teachers to understand these exponential shifts, to develop the skills to manage AI agents, and to think strategically about how to leverage AI capability gains to enhance learning and reduce workload. It means fostering an environment where teachers aren't labeled resistant, but are supported as the best drivers of innovation when given the resources they need. The core message here is that the rapid evolution of AI agents means we're moving beyond simple tool use. We're entering an era where our most valuable skill as humans and what we must cultivate in our students is the ability to manage increasingly powerful, autonomous digital colleagues with wisdom, judgment, and purpose. That's all for today. Thanks for listening.