S1 E6 - Data, Data Everywhere: But Learning Still Won't Stick

In this episode, Jocelyn explores how to close the gap between assessment data and classroom instruction. She argues that diagnostic and formative assessment need to be built directly into school-wide systems, not treated as separate from planning, and offers three practical questions and simple tools to help leaders and teachers make data actually drive what happens in classrooms.

Transcript

Speaker 0:00

Welcome to the Leading Learning Podcast, I'm Jocelyn Seamer. Today I'd like to discuss something I've been watching play out in coaching sessions over the past few weeks. Our team has been working alongside leadership and teaching teams who are carrying two big pieces of work at the same time. They are finalising their assessment for reporting, which is, as we know, the marks and grades that go home to families at the end of semester. And in those same weeks, they're beginning to plan how they will strengthen instruction in semester two. That's two enormous jobs sitting on the same desk. And what I've noticed again and again is that those two jobs seem to be running on separate tracks. So the reporting gets done, the data is gathered, the grades are settled, everything goes home, and then it's finished. And then the semester two planning starts. It starts from a curriculum document or a scope and sequence, from where the units say the class should be by a certain date. As though the reporting that just happened and everything that they learned from that process hadn't actually taken place at all. And here's what I find myself saying to these teams. When our summative results disappoint us, the natural instinct is to look harder at that summative end, to refine the assessment tasks, moderate more carefully, and tighten the rubric. And all of that has a place. But if we want to think about student outcomes, I think that's the wrong place to start the conversation. The grade at the end of the semester is a result, it's the output of everything that has happened. By the time we're sitting down to assign the grade, the teaching is done and learning has either happened or it hasn't. If we want to lift those summative results, the work has to focus on what happens upstream from that reporting moment. The answers to our summative challenges live in the diagnostic and formative assessment that shapes instruction across the whole semester, long before we sit down to write reports.

Speaker 2:17

So sorting out that upstream work is what actually moves the grades. And many schools, without realising it, are doing reporting on one track and the planning on another. When the thing that would help us the most to help students is to bring the two things together. That is where our work is focused, and that's what this episode is about today; hHow we close the gap between the data we collect and the data we actually use, and why doing that is a leadership job before it's a teaching one.

Speaker 2:54

Every teacher in your school learned about assessment in their initial teacher education. They can tell you the difference between diagnostic, formative, and summative assessment. They sat an exam, wrote an assignment years ago, and none of this vocabulary is new. The gap that we're trying to address is not knowing what the three kinds of assessment are. The gap is whether those three kinds of assessment are actually baked in to the system of instruction we are asking our teachers to be a part of. Because simply providing professional learning as an offering and hoping that it will help is a recipe for underwhelming outcomes. If we are serious about creating what you will probably have heard me call a minimum viable system of instruction, which is the simplest set of conditions that produces success for all students, then we have to get serious about all the kinds of assessment. We have to build the diagnostic and formative work directly into our universal school-wide processes rather than leaving them to the intervention team or to a once-a-term data drop.

Speaker 4:12

Let's start with the diagnostic side of this discussion. Now, diagnostic data has a really comfortable home in the intervention space. That's the nature of the work. So before we work with a student who is behind, we find out precisely where the gaps are and we teach into them. But in the everyday classroom, diagnostic assessment is unfortunately pretty rare, and I think the reason is structural. Our systems of instruction are so focused on teaching this year's content, the work for this grade, this term, this unit, that there is almost no space left to check whether students hold the prerequisite knowledge they need to access that content in the first place.

Speaker 4:59

So picture a secondary maths classroom beginning a measurement unit with a focus on surface area and volume of three-dimensional shapes. On paper, it looks pretty straightforward. The unit plan is there, the textbook chapter is there, and the final assessment is probably already written. So the teacher begins with point number one. But some of the students in that room are not yet secure in the difference between area and volume. Some cannot reliably name the 3D shapes they'll be working with or list their properties. This is critical background knowledge that is assumed by whoever created the unit of work and wrote the textbook. When a student doesn't have that critical knowledge, those missing pieces do not just simply sit to the side, they become an additional load on the student's working memory. Compounded by the gaps in knowledge, the new task, the new learning, can become so difficult that things for the student get out of control and they've passed the point where learning is possible. Without meaning to, we've set some students up for failure before the unit has even begun.

Speaker 6:16

Here's another way to put it. Picture two buckets, one bucket represents short-term memory, the other is long-term memory. If learning is a permanent change to long-term memory, then the student's job is to transfer water from one bucket to another. Some students come to this process with a nice sturdy ladle. That's their existing schema, including all the things they've learned up until this point. Other students' vessels, say a plastic cup, are cracked, and each time they attempt to transfer water, they spill some. Now water transfers, but the process is less effective and it's a lot slower. A third group of students are attempting this task with a sieve. Every time they try to move the water from one bucket to another, it just gushes through. They end the allotted time having worked incredibly hard, but only having a few drops of water that has actually been transferred. When students are trying to access grade level content with significant gaps in their prior learning, it's what it's like to be them.

Speaker 7:24

This is not only a maths problem, and it's not only a secondary problem. It is the same in phonics, where this week's words depend on the code that has come before. It's the same in science, where a student can't reason about a fair test because they don't know what a variable is. It's the same in writing, where a student cannot build a persuasive piece because they're not sure what the structure is. Right across the country, in every subject and every year level, we keep colliding with one idea that we haven't properly considered. You cannot build strong learning on shaky foundations.

Speaker 8:06

So the question that sits closest to the classroom in all of this is: are we teaching the right things to the right students at the right time? This is a diagnostic question, and answering it doesn't require another commercial tool or column in a spreadsheet. It's the kind of thing that our team helps schools build into their systems of instruction, and it's almost embarrassingly simple, which is really nice because it means that it should be pretty simple to implement.

Speaker 8:39

When a teacher is about to start a new topic, they give students five minutes to do a brain dump. They ask the students to write down or draw and annotate everything they already know about the topic. After all, our system of curriculum is set up to build on what has come before. And this piece of paper and a pencil is the whole instrument. The way that we frame it for students matters, so let me give you the script. You can say something like,

Speaker 9:10

"I'm asking you to do this so that I can get the teaching right for you. If you do not know anything about this topic yet, or you can't remember anything, just put a big question mark in the middle of the page, and that is completely fine."

Speaker 9:26

Say you're starting a unit on ecosystems with a focus on coral reefs. You give the five minutes, you collect the pages, and then you have a look. You're not marking against a rubric, you're not writing feedback, you're not entering anything into a spreadsheet. This is a brief assessment for learning and for nothing else. In a few minutes of looking, you can see where the class already has a foothold, where you will need to firm up prerequisite knowledge before you move on, and which students will need stretching because they have arrived knowing a good deal of the content already. Naturally, different subject areas will lend themselves to different formats of diagnostic assessment. The point isn't that a brain dump is the cure all, it's that checking what students already know before you start teaching is a great idea.

Speaker 10:20

Before we move on to formative assessment and its place in this whole piece, let's have a quick look at some research examining the practice. A 2013 study across secondary schools in two Nigerian states worked with 427 approximately Year 11 students and 12 biology teachers. Schools were split into three groups. In the first, teachers used a diagnostic remedial approach, identifying the specific errors and weaknesses students had, and then teaching directly into the gaps. In the second, teachers taught the same content in the usual way without that diagnostic step. The third was a comparison group that didn't receive the lessons at all. Students were tested before and after, and then again later to check retention. The students whose gaps were diagnosed and addressed outperformed both other groups on achievement, and the gap widened on the later retention test. So they not only scored higher, they held on to more of what they had learned. Effectively, building schema allows students to build better knowledge and skills for the long term. Now, this is one study and one paper, but it's certainly not the only example pointing to the benefit of diagnostic assessment as part of our teaching cycle.

Speaker 11:50

Importantly, across the various papers reviewed for this episode, the diagnostic test itself was not the impactful factor. It was the response to that assessment and how instruction was adjusted that really mattered. So while the diagnostic assessment is easy to implement, responding to it is the trickier bit, and that's where instructional coaching comes in.

Speaker 12:17

Now let's turn to formative assessment. In most schools our teams work with, the summative end is reasonably well established. There's a system for collecting data that feeds grades and reports, linked, of course, to the curriculum. The in -the -moment check for understanding is also better understood than it used to be, although there is a long way to go for this to be embedded across the board. Many schools have done good work with checking for understanding during lessons, using mini whiteboards, exit tickets, and close observation during daily review. This is the heartbeat of that in-the-moment responsive teaching, noticing when a student's understanding breaks down and acting on it before the lesson moves on. That work matters, and I don't want to diminish it in any way. But there is a gap, and it's a big one.

Speaker 13:12

What happens between the lesson and the summative assessment? What do we actually know about whether last week's teaching stuck while there is still time to do something about it? For the most part, we don't check. We have a scope and sequence or a yearly overview and we teach to that timing. We move on because the plan says move on, not because we have evidence that the class is ready. And then months later we sit down to do the reporting and discover that some things did or did not take hold at the precise moment when it's too late to do anything about it. And there's our two tracks again. The summative data arrives after the teaching it should have informed. Now, daily review and retrieval is supposed to help us do this work, but really, if we look around, most teachers are still only finding their feet, just including a daily review time in their classrooms. We are nowhere near making that daily review responsive and effective in the way we target it and use what we find. And I know that's a sweeping generalisation, and some schools are leaning into this more than others and getting great results, but if we look broadly, this is where we're at.

Speaker 14:30

One of the simplest things that our team helps schools to do is this work of a weekly check-in built around a simple question: How much of what I taught in this area last week has stuck? It might be a handful of quick questions at the start of the week. It could be a short retrieval task, it could be a few problems on the board, it could be a brain dump. It takes very little time and almost no preparation. This weekly check-in does two jobs for us at once, which is why we value it so highly. It evaluates the impact of last week's instruction and tells us honestly whether the teaching has landed. It's also, in the same breath, a piece of diagnostic data for what comes next because it shows us exactly what we need to reteach or firm up before building that next layer. We get an at-a-glance picture of how things are really going week by week, instead of finding out at report time that the whole term of teaching didn't stick.

Speaker 15:38

Now, it's really easy for me to sit here and tell you all of this as though the only thing that's standing in the way of it is that someone hasn't explained it clearly. There are real constraints to making this work happen, and the biggest one, particularly in secondary, is time. The most common thing secondary teachers say to our team is that there is simply too much of the curriculum and they could not possibly slow down delivery. We'd never fit it all in, they tell us. And the difficult truth is that within the system as it currently runs, they're very often right. Let's just sit with how strange that is for a moment. That we, as educators and the education system, have built a system of instruction whose main job, really in practice, is to get through the content, covering what the document says must be covered in the time allowed. We have done that so thoroughly that stopping to find out whether students have actually learned anything feels like a luxury we cannot afford. And I want to be really careful because this is not about teachers not caring or not working hard enough. The teachers we meet are dedicated individuals. It's about the status quo of the system within which we all work. We prioritise coverage, so coverage is what we get. If we strip back teaching to its essentials, it's actually pretty simple. You find out what students don't yet know, you teach them what they don't yet know, you check that they have learned it. That's the job. There's something really wrong when that simple version of teaching sits in tension with what so many teachers feel the job has become, which is to look at the curriculum, deliver what it is, as much as you can fit in, and then reward the students for whom it happened to work. I get how hard that tension is to live with, and how little of it is a teacher's fault. And this is why this work is not the work of the teachers first. It's the work of the leadership team to define what teaching and learning looks like in our school. We don't want to bolt on another assessment into an already overloaded system. We want to help you to redesign the system of instruction itself so that finding out, teaching, and checking are built into the minimum viable system that you are seeking to create. All of the work that we do is designed to build the decision-making capability of leaders and teachers so that this work continues into the long term.

Speaker 18:35

If you take something from this episode into your next leadership meeting, I'd really love it to be three questions. They are not only questions worth asking about data, but they are the three that sit the closest to the daily work of the classroom teacher, and each one maps onto a practice that I've mentioned in this episode. The first is whether we are teaching the right things to the right students at the right time. That's a diagnostic question, and brain dump is one simple way to answer it before a unit begins. The second is whether students are retaining what we have hopefully taught them across a term. This is what that weekly check-in is built for. How much of last week's teaching has stuck, and what do we need to revisit before we move on? The third is whether students are holding on to their comprehension through the lesson itself. That's the check for understanding, the responsive teaching work of noticing and adjusting in the moment.

Speaker 19:39

Let me take you back to those coaching sessions and the teams doing their reporting on one track and the semester two planning on the other. The shift that we're working towards with them is not a heavier workload. It's the opposite. It's the recognition that the information surfacing through assessment is the very thing that should shape what they plan to teach next and how. The data they collect starts to change what happens in classrooms, which is really the only good reason to collect it at all. Collecting data is an activity. Using it creates impact. None of the three practices that I've described in today's episode are difficult on their own. The brain dump takes five minutes, the weekly check-in maybe just a little longer. Checking for understanding is increasingly woven into lessons themselves these days. What is hard and what is genuinely difficult in our work as leaders is building these structures into the system of instruction we are creating so reliably that every student benefits, no matter which classroom they happen to be sitting in. That is what a minimum viable system for all students looks like in practice.

Speaker 21:06

I hope that this episode has affirmed and challenged, in some way, that you have been able to take some elements from it to discuss with your team. Don't try and do everything at once. Just start with one thing and do it really well. I'll see you next time on the Leading Learning Podcast. Bye.