DeepMind’s surprising Sierra Leone trial

An AI tutor in Sierra Leone reportedly helped students gain a year of schooling in eight weeks, a claim even Google DeepMind cautions us to take with a grain of salt.

In this episode:

• Google DeepMind's AI tutor education trial in Sierra Leone estimated students gained a year's learning in eight weeks, using a re-engineered Gemini model.
• The AI learning tool, Guided Learning, was designed not to give direct answers, fostering productive struggle crucial for learning outcomes.
• Early findings suggest AI in classrooms might initially widen the gap for already proficient students, prompting Google DeepMind to investigate different pedagogies to support struggling learners.
• Teachers in the Sierra Leone trial adapted quickly to AI learning tools, discovering new teaching strategies and increasing one-on-one student interaction after just one day of training.
• Transparent research practices from Google DeepMind, including a public playbook and training materials, aim to allow wider adoption and scrutiny of AI for learning outcomes.

Chapters:

• 00:00 — Cold open & welcome
• 00:45 — Irina Jurenka's shift to social impact at Google DeepMind
• 01:45 — The core tension: AI as assistant vs. AI for learning
• 02:45 — How Guided Learning became a true AI tutor education tool
• 03:45 — Overcoming the AI's tendency to give answers
• 04:45 — The 'take with a grain of salt' claim on learning gains
• 05:45 — Addressing equity and the 'widening gap' concern in AI for learning outcomes
• 06:45 — Google DeepMind's transparency and research integrity
• 07:45 — Transforming teaching practices with AI learning tools
• 08:45 — The future evolution of AI in classrooms

How did Google DeepMind's AI tutor impact learning in Sierra Leone?
The AI tutor education trial in Sierra Leone estimated students gained approximately one year of schooling in just eight weeks, though this figure comes with a scientific caveat from the researchers.

Can teachers trust AI learning tools not to give students answers directly?
The Guided Learning tool, designed by Google DeepMind for AI in classrooms, was specifically engineered not to give direct answers, forcing students into productive struggle, unlike many public AI models.

Do AI learning tools widen the learning gap, or help all students?
Initial findings from the Sierra Leone trial showed that stronger math students benefited most, but Google DeepMind is actively researching different pedagogical approaches for AI to ensure it helps raise the floor for struggling students too.

Featuring: Dan Fitzpatrick, Irina Jurenka, Google DeepMind, Sierra Leone, Guided Learning, Gemini, World Bank, Stanford, Nigeria.

Follow AI in Education with Dan Fitzpatrick for more on AI in education.

Transcript

SPEAKER_00 0:00

If this episode makes you think, please let us know in the comments and support us by subscribing and leaving a review. Thank you. Today we are exploring a new study from Google Deepmind, specifically a trial in Sierra Leone, where an AI tutor reportedly helps students gain more than a year of extra schooling in just eight weeks. Now that's a massive claim, so big in fact that Irina Jurenka, the research director at Google DeepMind, who's behind this study, actually told me to take it with a grain of salt. But hold on to that thought for a moment, because there's plenty in this trial that absolutely survives a good salting. First, let's talk about the big claim itself and why it's so important to understand the how behind it. Irina Jurenka is a computational neuroscientist by background, and she spent her first five years a Google deep mind diving into these deep, theoretical questions about intelligence. But then, as it did for so many of us, lockdown hit. And what she found herself thinking was COVID hit, and I found myself sitting in my living room and realizing that I'm doing a lot of work that maybe a few people read, but really it's not making a difference in the world. She wanted, in her own words, something with more immediate social impact. And this I think is where the story gets really compelling. Language models were just starting to get interesting, not quite the mainstream phenomena they are today, but the potential was beginning to spark. So Irina shifted her focus to reasoning, and as a neuroscientist she naturally asked, Where do humans actually build their reasoning skills? Her conclusion? It became clear that it really has to do with formal education. So four years ago, long before AI was everyone's daily topic of conversation, her team launched what she calls an educational grand challenge project within Google Deep Mind. She could already see the inherent tension, the risk with this emerging technology. She points out that language models are really built to be assistants. Assistants do the work for you and in learning you really want to do the work yourself. That's where you learn. And honestly, I think every teacher listening to this, every educator who's watched a student copy paste their homework straight into a chatbot knows exactly what she means. The trick then is to leverage an AI for learning outcomes without simply outsourcing the thinking. But what about the technology and how it actually works in an AI tutor education setting? The tool used in Sierra Leone is called guided learning, and it's essentially Google's powerful Gemini model, rebuilt specifically to teach. And when Irina says rebuilt, she really means it. Her team started where many of us would write. They wrote a prompt, a simple instruction telling the model to act like a teacher, and what they found was that they hit a ceiling incredibly fast. As she puts it, it's kind of the same as writing a one pager on teaching, handing it to an average stranger on the street, and expecting them to come into a classroom and teach a room full of students. Sure, they'd probably do better than if they hadn't received that one pager, but they also won't match the experience, intuitions, and rich complexity of pedagogy that experienced teachers build over the years. That's such a profound observation, isn't it? It highlights that teaching isn't just about imparting information, but about the nuanced, adaptive, human elements of pedagogy. It's a powerful reminder of how much goes into a teacher's expertise. So they realized they had to go deeper, much deeper, with the technical work. They needed the AI to actually understand what the world of teaching looks like, and how it's almost the complete opposite of being an assistant. This is foundational to how we should think about AI in classrooms, isn't it? It's about designing these tools to be an enhancement, not a replacement, keeping the human in the loop at all times, and ensuring we outsource the doing, not the thinking. Now a big question for any school, for any educator is trust. Could schools really trust this AI tutor not to just hand out answers? What stops a student from nagging it, asking it over and over until it gives in? Irina's answer was refreshingly direct. At the end of the day, it's a language model, so no one can 100% guarantee its responses. It will always have some degree of randomness to it. But in this is the crucial part, the work we did was essentially trying to bias that distribution. And in the trial, that bias held up. What they found was that Gemini asked guiding questions in a whopping 76% of its messages and only gave a straight answer a mere 2% of the time. This is a critical distinction because it forces that productive struggle that we know is so important for learning. Here's another fascinating detail that really speaks to the intentional design. It turns out there are actually two versions of guided learning. The first is a consumer tool, available for everyone, maybe even on your students' phones right now. For that public version, the instruction to the AI is explicit. If the user does ask you for the answer repeatedly, you should give it, so that the agency is respected. But the classroom version, the one actually deployed in Sierra Leone, is a totally different animal. It's teacher led, used inside lessons, and in that context, Irina says we could actually tweak the instructions so that it never gives the answer away at all. And her favourite detail from the study? Once the students realized the AI wouldn't just hand over the answers, students kind of stopped nagging for answers and actually started learning more. That's a huge win for an AI tutor education approach, isn't it? It's about teaching students not to outsmart machines, but to outthink them. Now back to that headline claim a year of progress in just eight weeks. I asked Irina about the real picture behind such a bold statement, and she was very transparent. One thing I would say is the real scientific number that we got is the 0.26 standard deviation improvement. She explains that they then translated that to approximately one year of extra tuition, based on existing literature about typical learning progress. But she cautions, this is an estimate. There wasn't much data available, and the data that was available was actually on literacy rather than math. So she reiterates, take it with a grain of salt. This is our best guess at what that translates to. I really appreciate that level of scientific rigour and honesty, and it doesn't diminish the impact, but it grounds it in reality. The trial design itself also earns a lot of trust. They made sure not to just teach to the test. The end of term assessment was designed by an independent assessor, and it was split, 50% material covered during the term and 50% broad mathematical topics. This was to check if students just learned something very narrow or if their understanding improved more generally. Crucially, students sat this test without Gemini's help, using pen and paper, which connects so well to our three P's assessment model, looking at the product, the process, and the performance. And the good news is there are signs that what students learned with guided learning generalized more broadly. Another really striking statistic is that while voluntary edtech usually gets about 5% of students to actually use it, in this trial an incredible 69% of students met or beat their usage targets. If these insights are sparking ideas for your school or classroom, make sure you're subscribed to this podcast so you don't miss future episodes on AI in education. And this brings us to a really crucial point about equity and how we design AI for learning outcomes that truly serve all students. Governments everywhere are looking at AI as a potential way to level the playing field for disadvantaged students. But in Sierra Leone, the study found that it was actually the students who were already good at maths who gained the most. I put it to Irina that this effect could potentially widen the gap rather than close it. She didn't duck the question at all. For sure, that's something that we immediately discussed the moment we saw the results. She explains they're already talking to partners who specialise in improving learning outcomes for those who are further behind. Her early conversations suggest that maybe there are different types of pedagogy that are more effective for those students, and that they might need to generate different prompts, initial prompts for different students. So she says, We are very much looking into this question of how to raise the floor. This is so important, isn't it? Our focus should always be on AI as an equalizer, especially for that often invisible middle 80% of students who need targeted support, but don't always get it. She also made a compelling case for the other end of the room. Especially in countries like Sierra Leone, there will be a lot of really talented students with maybe not enough access to high-quality education, and if a tool like Guider Learning could be something that helps them accelerate and fulfill their potential, I think that's already really good impact. This perspective reminds us that equity isn't just about lifting the bottom, but also about enabling potential at the top, especially where traditional educational access is limited. And it's worth noting that a separate World Bank and Stanford trial in Nigeria found similar sized gains. But there, the girls who started behind actually gained more. This really drives home the point that what we Bisoni Who benefits depends on the design, not just the model itself. Thoughtful design is key. Now of course the cynical question always arises this is Google's technology, Google's research, and the results happen to be great for Google. I asked Irina about this directly. She responded, we're definitely very aware of this situation. But she was very candid, stating that working at Google DeepMind. We are researchers first and foremost. It's easier to do it with Google products, but at the same time our research is not about creating proof points for Google, it's about understanding the broader picture of where AI fits in, in improving learning. And to back that up, the team released a playbook publicly documenting how the trial was designed and measured, keeping decision making and measurements separate to minimize any potential bias that our involvement may introduce. Along with all the teacher training materials, they're openly inviting anyone to replicate or build on their study. That level of transparency is exactly what we need in this space. So what does this actually mean for teachers and schools in the classroom when thinking about AI learning tools? The part of this that will matter most to educators is what all this actually did to the teaching. Arena is clear. It's not just about dropping the technology and hoping that it works. This intervention was designed with local teachers, built so that the teachers feel like they're in control, and it was kept deliberately minimal. The teacher training, for instance, only took about a day. This intentionality, this anchoring to existing teacher practices and empowering teachers as change agents aligned so well with our principles for change leadership. For the teachers who were planning lessons with Gemini, something really interesting happened. Sometimes it explained things in ways different to how they would normally explain, and for them that was eye-opening. They felt like maybe they learned a better way of explaining something, and the training helped open up this new way for them to be a guide on the side, which was new to them. This led to teachers spending more time, moving between students, having more one-on-one conversations. This is the real potential of AI in classrooms, isn't it? Not replacing but freeing up teachers to do what only humans can do, build relationships, offer personalized guidance, and hold the complexity so we have capacity for creativity. I finished by asking Irina the million dollar question that educators ask me everywhere I go. Does the classroom actually change in the next five to ten years because of this? She smiled. Five to ten years as an AI years a very, very long time. So let me just talk about maybe the next year and a half. She believes that what they saw with Sierra Leone is proof that an AI tutor education can meaningfully improve learning outcomes. She predicts teachers will continue noticing where AI fits into the current practices, and that they'll slowly start to adjust them, and this will be a co-evolving thing that we'll see in communities. It's an evolution, not a revolution. And while scientifically speaking, she can't say it will generalize beyond those particular settings. The recipe is published, the invitation is open, and for the first time, a really serious result for AI, for learning outcomes is on the table. This study proves that with thoughtful design, AI learning tools can profoundly enhance learning. That's all for today. Thanks for listening.