AI, good intentions & falling math scores

A new study reveals a statistically significant drop in adolescent math scores after using an AI tutor for exam prep, despite students' good intentions.

In this episode:

• A study from the University of Tübingen revealed a significant 10-point drop in adolescent math scores after using an AI tutor for exam prep, indicating challenges in self-regulated learning with AI.
• The research identified a large gap between students' good intentions for learning and their actual, often superficial, help-seeking GenAI interactions, with monitoring and evaluation being nearly absent.
• Higher extraneous cognitive load, caused by the demands of navigating AI tutor adolescent learning, predicted lower math scores, highlighting how AI can inadvertently hinder deep learning.
• Effective AI math education requires explicitly teaching students metacognitive skills like epistemic vigilance and agency over the AI, not just providing access to the technology.
• Educators should design tasks that embed the process of AI interaction, such as annotating chat logs, to foster crucial self-regulated learning AI behaviors.

Chapters:

• 00:00 — Cold open & welcome
• 00:25 — AI tutor adolescent learning: The shocking math score drop
• 00:55 — Understanding self-regulated learning AI challenges
• 01:30 — Intentions vs. enactment: The gap in student AI use
• 02:30 — The impact of extraneous cognitive load on AI math education
• 03:40 — Explicitly teaching help-seeking GenAI strategies
• 04:30 — Cultivating epistemic vigilance and agency over the AI
• 05:25 — School leader implications: Purpose over technology
• 06:05 — Designing for thinking and reflective AI engagement

How does using an AI tutor affect adolescent math scores?
A study found that adolescent students experienced a statistically significant drop in their math performance after using an AI tutor for exam preparation, despite having good intentions for learning.

What is self-regulated learning AI and why is it important?
Self-regulated learning AI refers to students' ability to monitor and evaluate their own comprehension and the AI's responses, which is crucial for preventing passive learning and ensuring the AI truly supports deeper engagement.

How can teachers minimize AI cognitive load in math education?
Teachers can minimize AI cognitive load by explicitly teaching students how to formulate effective prompts, manage AI conversations, and design tasks that scaffold metacognitive skills like monitoring and evaluating AI outputs, rather than simply giving access to the tool.

Featuring: Dan Fitzpatrick, Rania Abdelghani, Peter Kaiser, Kou Murayama, University of Tübingen, Mistral-Large, Zimmerman's cyclical model, Gemini 2.5 Pro, Baden-Württemberg.

Follow AI in Education with Dan Fitzpatrick for more on AI in education.

Transcript

SPEAKER_00 0:00

If this episode makes you think, please let us know in the comments and support us by subscribing and leaving a review. Thank you. Today we are exploring some really thought-provoking and frankly quite sobering preliminary findings from a new work in progress paper titled Regulating the AI Tutor Intentions, Help Seeking and Self-Regulated Learning in Adolescent Gen AI Use. It's by Rania, Abdelghani, Peter Kaiser, and Ku Murayama from the Hector Institute of Education Sciences and Psychology at the University of Tubingen in Germany. What these researchers found in a study with 98 grade nine students using an arranged AI, i.e. tutor adolescent learning, was a statistically significant drop in their math performance after using the AI for exam preparation, going from an average of 67.5% on a pretest down to 56.9% on the post-test. That's a pretty stark number, isn't it? Now this isn't just a simple story of AI is bad for learning. It's far more nuanced, and it really gets to the heart of what we mean by effective learning in an AI-powered world. The authors highlight that while generative AI tools are becoming common learning companions, we still don't fully understand how adolescents truly regulate their use during authentic learning tasks. We often talk about self-regulated learning, or SRL, and high-level help seeking as crucial safeguards against students just passively copying answers, but this study offers a process sensitive look at what actually happens moment to moment when students are in the driver's seat with an AI. The researchers recruited nearly a hundred grade nine students from three public gymnasium schools in the German state of Bathropun Baden Wurttemberg. These students were given a curriculum aligned mathematics task to prepare for an upcoming exam, and they used a web based Sunku Mistral Lodge AI tutor. Before starting the chat, students actually selected their learning goals. And get this, their stated intentions were overwhelmingly learning oriented. 83% wanted step by step examples. 80% looked for tips and strategies. Only a tiny eleven point eight percent said they just wanted the final solutions, and just over a third wanted to finish as quickly as possible. So their intentions were really good, right? They wanted to learn deeply, but here's where we hit the first significant finding, the enormous gap between intention and enactment. What the students said they wanted to do with the AI didn't often translate into what they actually did. The interactions captured across 10016 chat turns were dominated by what the paper calls instrumental requests. This means students were asking for hints, explanations, or procedures, which sounds good on the surface. However, the critical elements of self-regulated learning, like monitoring their own comprehension or evaluating the AI's responses, were nearly absent. The median for both monitoring and evaluation turns was 0%. Students were rarely making their comprehension needs explicit, and they weren't really checking if the AI's answer actually met their goal or asking for clarification. It was a lot of asking, but very little critical thinking about the answers they were getting. This points to a really interesting challenge for us as educators. We talk about one self-regulated learning AI, and we know that humans need to stay in the loop, but this study suggests that students might not have the metacognitive skills to actually be in that loop, even when they want to. They might know that asking for explanations is good, but they lack the deeper interactional strategies to genuinely engage with the AI in a way that truly supports their learning. The paper highlights that the only intention that did strongly predict student behavior was the just give me the final answers option. Those students were significantly more likely to engage in executive help seeking, meaning they just got completed solutions without the productive struggle. This brings me to the second major takeaway, and it's a big one for any school leader or curriculum designer. The negative impact of extraneous cognitive load. The study found that higher extraneous cognitive load predicted lower post-test scores, even when they controlled for students' prior math knowledge. Now, extraneous cognitive load essentially refers to mental effort that isn't directly contributing to learning. It's the noise or the unnecessary difficulty. The authors suggest that interacting with an I in few AI tutor adolescent learning might be introducing new demands, like figuring out how to formulate prompts effectively or managing the conversation itself. This means that instead of freeing up cognitive capacity for deeper learning, the AI interaction might actually be consuming shifts, that capacity, leading to cognitive debt rather than cognitive growth. Think about that for a moment in your own context. We often talk about AI helping us hold the complexity, so we have capacity for creativity. But here, the complexity of using the AI seems to be hindering learning. It's a powerful reminder of one of our core pillars. Outsource your doing not your thinking. If the very act of doing the interaction with the AI becomes a burden, if it adds significant cognitive load, then it's not truly outsourcing repetitive tasks. It's just shifting the burden of complexity. This really challenges the assumption that simply having an AI available automatically leads to improved outcomes, especially in EHI feed AHI Math Education. So what does this mean for us in schools and how can we translate these findings into practical action? I think there are a few key areas that really jump out. First, for a year eight math teacher, for example, this study underscores the need to explicitly teach and scaffold help seeking Gen AI ecru strategies. It's not enough to tell students use AI responsibly. We need to design tasks that demand monitoring and evaluation. How about asking students to submit their AI chat logs as part of their work? But not just the raw log, ask them to annotate it. Where did they ask for clarification? Where did they check if the AI's response was accurate? Where did they challenge it? This moves beyond just the final product and brings in the process which aligns perfectly with our three Psession model product, process and performance. The process of how they interacted with the AI becomes part of the learns and part of the assessment. Secondly, for department heads planning CPD, this research highlights the vital role of teaching epistemic vigilance and agency over the AI. These are two really interesting terms the researchers inductive coded using mock Gemini 2.5 probes for their analysis. Epistemic vigilance means a student's ability to react to an inaccurate or mismatched AI turn. Agency over the AI captures attempts to shape its role, format, or scaffolding, like asking for hints instead of direct answers, or requesting self-testing. These aren't just technical skills, they are deeply metacognitive. We need to train our teachers to model this kind of interaction, to show students how to critically evaluate what the AI has given them, to understand its limitations, and to manage the conversation with precision. We have to teach students not to outsmart machines, but to outthink them, and that requires nurturing these critical interaction skills. If you're finding these ideas helpful for navigating AI in education, please do consider following and subscribing to the podcast for more insights. Finally, for school leaders, this paper authored by Rainier Abdelghani, Aradho Siam, Peter Kaiser, Chao, and Chowunt Ku Sunim Rayama isn't a call to ban AI. It's a powerful call to rethink our approach. It reinforces our pillar of purpose over technology. We must start with the educational purpose, genuine learning, critical thinking, self-regulation, and then design the AI interaction around all that. We need interfaces, or even just classroom protocols that actively scaffold those missing monitoring and evaluation behaviors. It might mean starting with simpler prompts, designing tasks where the AI is intentionally limited in its outputs, or building in mandatory reflection points. It's about designing learning that cannot be faked because it demands depth, care, and imagination from the student. The real value is not in what the machine produces but in how the student responds, how they think with hewit, and how they grow through that productive struggle. This is all part of an evolution, not a revolution. We're learning how these powerful tools truly impact learning, and sometimes those impacts are surprising. The key is to pay attention to the data, like this from the Utai Torrange TB University of Tubing and Shifrekun team, and continuously adapt our pedagogy to ensure that AI truly enhances human capability, rather than inadvertently undermining it. We need to design for thinking, for careful questioning and for reflective engagement, because machines can compute but they cannot wonder and they cannot truly care. That's all for today. Thanks for listening.