Academic Distinctions: A Podcast to Make Sense of American Education
Hosted by Stephanie Melville and Zac Chase, "Academic Distinctions" is a podcast for educators that tackles the reading and research teachers often don't have time for. With experience as classroom teachers, district administrators, and federal policy wonks, the hosts bring a unique perspective to discussions on education's "greatest hits" and current events. The podcast is committed to delivering engaging, informative, and actionable content that is relevant and responsive to the needs of educators.
Academic Distinctions: A Podcast to Make Sense of American Education
006: Why our brains like order and how John Hattie's rankings impacted education
In this episode Stephanie and Zac talk to Dr. Ji Son from Cal State Los Angeles to get a better understanding of why it's so hard to add complexity to information once we've got our minds made up as well as some helpful tricks to make new information stickier. Then, they dive in to the work of John Hattie and begin to examine how his attempts to share what's "effective" in education spread like wildfire.
Hello, it's good to have you back. Welcome to this episode of Academic Distinctions. Today we're going to be talking with Professor G. Sun on why our brains like things to be in order and what makes those things sticky. And we're going to be discussing a guy named John Hattie and the impact he had on education when he published a book that sold a bunch of copies. So stick around. I think you'll like it.
UNKNOWN:Music
SPEAKER_01:Before we talk about education today, we wanted to stop and talk about something we've noticed as a trend as we prepare these episodes for you. Each time we see someone's magnum opus of education research full of citations and complexity, we also see that the history shows the most basic pieces of that research that make it into the field.
SPEAKER_03:See Bloom's Triangle and the fact that he said it was the most often quoted book about education that no one has ever read. So today we're going to start by helping all of us understand our brains a little bit better. To do that, we're happy to welcome Dr. Ji Sun from Cal State LA. Hi, Ji. Hi. Great to be on the pod. We're so excited to have you.
SPEAKER_01:So I think we're going to have a pretty interesting conversation today. We are struggling with a lot of hierarchies and a lot of lists. And as we read these books that people... love but haven't read, we noticed that there's a lot that's getting left out. And so we were hoping you can help us think through and understand maybe how brains are working. And so here are two important things we noticed. In our episode about Bloom's taxonomy, that he and his colleagues never organized their hierarchy into, or their taxonomy, into a hierarchy or a triangle. But the way that our brains and the way that it gets passed on culturally is that that taxonomy is a hierarchy and that it's in a triangle. And then the other piece that we're noticing, and these are just kind of the roots of us wanting to talk to you are that when Anderson and Marzano and others tried to improve upon or add complexity to bloom at all's original work, the thing that stuck was the, hierarchy. So the thing that is sticking around is the idea that is not the idea that was put forth in the first place. And when we try to course correct, culture and brains stick to the stuff that's there. Why does that happen?
SPEAKER_00:Well, first, I want to say the fact that the triangle was sticky, it tells us something about humans. So it's not so much like, oh, why do humans love triangles? It's really, why did we love this triangle, right? But I love triangles, G.
SPEAKER_01:More of an octagon man myself, but please carry
SPEAKER_00:on. There was something about it that struck a nerve. And I would argue, I mean, I don't really know why the triangle had a moment, right? But I would argue that it probably hit a nerve. It probably mimicked some part of people's experience where they felt like, hey, I look at that and I recognize something in it. Now, the question is, what is the experience that kind of got triggered by this triangle such that it's the thing that made it into the cultural lexicon?
SPEAKER_01:Is it? So we also noticed, or from kind of our own anecdotal experience, is that people know the bottom level and they know the top level. Yeah. But if you asked them to fill in all of the different pieces in between, their minds are kind of a blank and they're like, well, this one is somewhere in
SPEAKER_00:there. It's like a rainbow in between, right? Yeah. And I would argue... And maybe Jim Stigler and many others would argue, we probably really only remember three. So Bloom's triangle, if I could call it that, the problem with it is that it has more than three things. So it doesn't really stick. So we remember it almost as like a continuum, right? Like there's the worst memorization and there's the best creating, applying, Whatever. But they don't really even memorize the last one. They just remember it as a continuum. Like one is the worst one and one is like the place we're trying to get to. And then there's a lot of variation in between. But they could kind of be a little loosey goosey about the stuff in the middle because they think of it like a grayscale. You're getting like better and better and better.
SPEAKER_01:Okay, but see, this is really frustrating. Because we're talking about educating human beings and helping them improve their lives and their understanding and their knowledge. And these are really complex and interesting ideas. And they're all getting lost because this is what our brains are holding.
SPEAKER_03:Yeah. So maybe the question is, you know, how do we get folks to come to understand, like a new complex idea?
SPEAKER_00:Well, understanding complex ideas are a whole shebang, right? Because it's like, are humans really made for complexity? Are
SPEAKER_01:they?
SPEAKER_00:And so here I want to introduce a metaphor that I think will be helpful for us going forward. So Jonathan Haidt talks about human minds, and it's not just Jonathan Haidt, but he has this wonderful metaphor that I think will be helpful for us. Imagine a human mind as a man, a little rider, riding on top of of an elephant, right? In cognitive science, we often call the little man system one and the elephant system two, but I like Jonathan Haidt's imagery of this little man riding on top of an elephant much better. The little man is the logical part. It's the part that learns complex ideas and nuance. It learns wonderful, complex, interesting, structural, relational things. But it's small for a reason. The elephant has a lot of power. And the elephant is everything else in our minds that define and determine what we actually end up doing. So a small part of us is logic and reasoning and complex ideas. But the big elephant part of us is all that other stuff. perception, how we feel in the moment, all our stereotypes, how we've done things before, what's easiest to do in the moment, what the world is set up to do. That's the kind of stuff that we also have to think about. So I love talking to you guys because you're so thoughtful. And I, as an academic, would love to just spend all our time thinking about how to get people to learn complex ideas. And Honestly, we have a paper about that. It's called the Practicing Connections Framework. And we really think a lot about that. But one of the things that you have to realize is how little of an effect complex learning has on actual behavior.
SPEAKER_01:But then...
SPEAKER_00:You're stunned.
SPEAKER_01:Well...
SPEAKER_00:Are you sad? Doesn't that make you sad?
SPEAKER_01:Yes. There's so much to learn and understand. And not even from an achievement side of things. But as we are more connected in a world, our need to understand the complexity of other humans is so great that what you're talking about makes me worried about about our ability just to connect to other people, not just to understand like trigonometry.
SPEAKER_00:So here's, you know, I'm not all doom and gloom. Thank goodness. We've had enough of those episodes.
SPEAKER_01:Did not know what we were getting into. We
SPEAKER_00:still as humans act in very complex ways. I'm just saying that It's not because we just think complex thoughts that we're able to act in complex ways. We act in complex ways for many, many reasons. And a small part of that is due to our complex thoughts. Like, can I give you an example that's relevant to teaching?
SPEAKER_01:Yes.
SPEAKER_00:Like, you know, it's... It's interesting. Many teachers have this idea, hey, we should get more students to talk in class. I want to hear from a broader diversity of students. That feels like a great goal. Often, that's the rider saying, hey, I value the diversity of voices. I want to hear from many different students. And so that's a great goal. But if you're a teacher who's strapped for time, who has to move on, you might really be like, oh, my God, I'm just going to call on the kid who I know knows this thing, who could kind of say it, and we could call on him, and then we can move on, right? And that's the rider. However, if you change the teaching practice, like, for example– The thing that they do in this classroom is cold calling. Then the teacher doesn't have to do it because of the complex thought of like, hey, I value diversity, nor do they have to default to, oh, I'm just going to call in the kid who knows the answer. They just default to cold calling because that's the culture of their class that they've set up. You see what I mean?
SPEAKER_01:So today we're going to talk about John Hattie's work.
UNKNOWN:Okay.
SPEAKER_01:It is big and it is detailed and it is complex and it has been oft misunderstood. But it's a big old list of things. And that's the appendix. In the original, 138 things. And that's the appendix. But that's what people remember. Yeah. So how do you communicate a big complex... If there is an idea that needs communicating, that needs to be complexly understood... What are some ways that we can, I mean, what you're saying is like, let's replace this practice with this practice. Kind of like training it out, making it very simple, saying we're going to do this now.
SPEAKER_00:Not only. I mean, the thing is, I'm just saying that you need to get the elephant and the rider to align. If you're just planning on making the little rider do all the work, that means you're going to go wherever the elephant wants to go that day.
UNKNOWN:Yeah.
SPEAKER_00:And so whatever your values are, you got to make sure that that really is in the classroom practice in a way that when a teacher is tired, when the class is dragging, that they could still do it. Because you know what? That's 90% of our experience, right? And so what I'm saying with even like Hattie's work is that if you have 180 things, people are going to remember everything. A random three at best and zero for the vast majority of
SPEAKER_03:time. Sure, sure. Too much is too much.
SPEAKER_00:Too much is too much. And it's because think about the writer. If you're a writer, if you're a person in the classroom and you're tired, you haven't had lunch yet, half your students came late, you're trying to drag them through this topic that nobody's particularly interested in, right? You are an elephant teaching other elephants.
SPEAKER_02:That
SPEAKER_00:is the situation we are all stuck in. And when you're doing that, you can't think of the 180 things and think, which of those 180 things is relevant for this moment at hand? That's just not something that you're going to do. But if you have some bigger thing, if you have a priority that you're going to be trying to really do, boom. That's something you could commit to. And I would argue that's why the Bloom's thing took off too. It gave people a priority structure. It said, look, this is the least priority. This is the worst priority. It gave people a decision tree. If you don't have time, if you don't have resources, just spend your energy doing this. And I would argue that happened for the top of the pyramid and the bottom of the pyramid.
SPEAKER_02:Because
SPEAKER_00:it's like, if you have one thing you want all your students to do, maybe you want them just to do the best thing. And if there's only one thing your students could do, maybe you just all want to get them at level on the bottom rung, right? But it gave people a way to make decisions.
SPEAKER_03:I really like that. But okay, so yes, I like that. And We know that, you know, Blooms, Marzano, Hattie, they all put out revisions and refinements that did not stick. So when new ideas come down the pipeline, you know, those refinements and the revisions, how do we get that to stick? Like that never sticks. We're only ever connecting to that first impression, that first read through.
SPEAKER_00:Well, and that's if that first impression was any good. For every Hattie, there's 10,000- You think it wasn't good. That's what sticks. Right. There's 10,000 learning scientists that you don't know their names, right? So for whatever reason, there are some people who stick. And then those people, whatever their theory was, some crazy part of it stuck. And it tells us more about us than we realize, right? And- I would argue that especially in American culture, it's very hard to revise anything. Americans love an innovation, right? AI is going to change the world. We love a silver bullet. Everyone likes to say there's no silver bullet, but secretly we're really wishing for a silver bullet. Oh, yeah. Right? No one's really that interested in a system that incrementally improves. So you actually have to hide it, you know? I'm old enough where I remember when YouTube didn't have a lot of videos on it. But YouTube was really different. But it didn't say, we're rebranding to a new website where we're going to do this other thing. They just made incremental improvements where we still think the whole thing is still YouTube, right?
SPEAKER_01:But every feature was tagged as like new.
SPEAKER_00:But a lot of times people didn't even know that it was happening, you know? Yeah. There was a point where in YouTube, you would have to click next to get another 15 or 20 videos. But they just changed to infinite scroll where try scrolling down YouTube. You never get to the
SPEAKER_01:end. Right.
SPEAKER_00:That's true. And they just switched it without telling anybody about it.
SPEAKER_01:So then if we want to add complexity to somebody's understanding, rather than saying we're adding complexity to your understanding or to our... discussion about this, we say, we've got this new thing we're going to try.
SPEAKER_00:Yes. I would say if you want to really change the framework, you need to brand it as new and sexy and good luck in getting that to pop because 10,000 people have tried to get their thing to pop. Well,
SPEAKER_01:initiative fatigue is a real thing, right?
SPEAKER_00:There's
SPEAKER_01:that line of like, oh, another new thing? No, thanks. I'm going to go ahead and close my door.
SPEAKER_00:Exactly. I mean, you know, I love Common Core, but I was around the math reform game for a while. And when you read it, it is beautiful, but is it that different from the previous frameworks? Which is why it's so funny how much backlash there was because it's like, all right. It's always the thing. It's like, is it the branding or is it the real truth of the thing? The truth of the thing only changes by 10%, but they branded it differently, right?
UNKNOWN:Yeah.
SPEAKER_00:which has its pros and cons. And that's always the thing. You could brand it as a new thing, but it's gonna have a different set of pros and cons. Or you could do the YouTube angle and just secretly build it in. There are innovations, but guess what? You don't really have to learn anything new. It just stretches you a little bit from what you used to do.
SPEAKER_03:So I have a related but unrelated question.
SPEAKER_00:Of course.
SPEAKER_03:We love a segue. Confirmation bias, right? For our listeners at home, it's how people tend to look for, interpret, recall information that confirms their existing beliefs or opinions, right? They also tend to discredit or ignore contradictory evidence, which I kind of feel like is in line to some extent of what you're talking about a little bit in terms of like why our first bit of information sticks so firmly to, to what we work with, right? Like revisions and refinements don't really work. So what, We know about this idea of confirmation bias, but so what can folks do to kind of subvert that in their own thinking and in the thinking of others? I don't feel like it's a unique thing to American education. Maybe it is, but, you know, so what can we
SPEAKER_00:do? Americans don't particularly like the word innovation and new, you know? Yeah. They love it more than most. But you're right. Confirmation bias is a human thing. Everybody has it. It's not something that you're necessarily going to overcome. But here's where that elephant and rider thing comes in handy. Like, often people think when you feel happy, you smile. Right?
SPEAKER_03:For those of us who just exist with RBF pretty consistently, you wouldn't know that.
SPEAKER_00:Except for those. But... There's a fair amount of research that also shows the other direction is true too. When you smile, you feel happier, right? So confirmation bias goes in this way. Like you have these experiences and that changes how you think, right? And we wish how you think changed too. what you experienced and vice versa, right? We wish that was more true. The way that I would say it is you need to give people new experiences that don't quite fit into their framework. And then those experiences have a chance of doing something besides confirmation bias, because there's no theory that those new experiences have yet to tag onto. And that's what we've seen in our work with teachers, where it's like they're doing data science, they're doing these new things, they're doing coding in classes. And because they're doing these new things in their classes, it's almost like an opportunity for them to kind of shake up their concepts because they don't have a concept for this stuff yet, but we're seeding them with experiences. And then once they have those experiences, we tell them the concept and they're like, oh my God, that really... with my experiences. We're almost using confirmation bias by seeding experiences and later presenting them the concept that's going to confirm that they're going to confirm with their experiences. It's a very long game. Yes.
SPEAKER_01:Christopher Nolan was trying to get us to understand that with Inception. And
SPEAKER_00:then we just watched the top
SPEAKER_01:spin and we never got it.
SPEAKER_00:But we decided to do it with teachers and education and math classes.
UNKNOWN:Yes.
SPEAKER_01:This has me thinking about the human library out of Denmark. I don't know. Do you both know how nerdy are we? So the whole idea there is that you can check out a person. Oh,
SPEAKER_03:that gives me feelings.
SPEAKER_01:No, in a really cool way, right? So people sign up if you have a– if you're an older person or if you're somebody who's queer or trans or of a different ethnicity, you can put yourself into the library. And then people can come and check you out to have a conversation with you.
SPEAKER_02:Wow.
SPEAKER_01:Oh,
SPEAKER_02:that's beautiful.
SPEAKER_01:Right. And it is a really fascinating idea. But what you're saying and what their whole theory here is around kind of contact hypothesis is that once I've talked to somebody who is that difference that I do not own,
SPEAKER_02:that
SPEAKER_01:that's probably going to shift my thinking at a foundational level about other people who have that difference that is not my own.
SPEAKER_00:Right then is the right time to give them a new concept is what I'm saying. So that's where it brings it because it's wonderful to go out and have new experiences. That's great. But if you want people to glom on to a new complex concept, what you want to do is pair it with those new experiences.
SPEAKER_01:Okay. So novelty is key and important. And
SPEAKER_00:Cause it gives you a shot.
SPEAKER_01:Cause it gives you a shot. It opens the door. And once you're there, give like the easiest form of the solution that you would like the most basic. We want you to do this at the most basic level. We're going to give you the complexity, but
SPEAKER_00:I'm not that cynical. That's
SPEAKER_01:what it sounds like.
SPEAKER_00:I still think right at that moment is when you want to introduce the complex idea of
SPEAKER_01:Okay. I guess I'm more thinking about your elephant isn't like when you're tired and cranky and all those things, you're not going to think through the complexity of the idea. So you want to make sure that first and foremost, that they get the most basic or key piece of information. No.
SPEAKER_00:No? I would not say that at all. Here's the way we think about it. Okay. First, you give people real experiences. That's really important. That connects to the elephant. That gives them this like meaningfulness basis, right? But after that, you do want to connect to complex concepts because you don't want the elephant to do things that are unmoored from the rider. But the thing is there's an ordering. The elephant comes first and then the rider comes second because ideally the elephant and rider want to move in the same direction together. Okay. The reason that I advocate for that is there are going to be times when the elephant wants to move in one direction. And what you want to do is be okay with that direction. The rider wants to be like, hey, that's a great direction. I love it for you. I have no power in the situation, but I'm happy to go along for the ride. You don't want the rider to be unhappy because ultimately people are going to burn out. If it's just elephant stuff without the higher order writer stuff, giving them that sense of inspiration and coherence that people really love. That's why we stay in hard jobs like teaching. You need that.
SPEAKER_01:So this would be where like scripted curriculum tried to do the thing that I was describing.
SPEAKER_00:Correct. It's just the elephant.
SPEAKER_01:You're all elephant.
SPEAKER_00:Stop thinking. Leave the rider to do whatever he wants. We know if the elephant goes in this direction,
SPEAKER_01:we'll be fine.
SPEAKER_00:Right.
SPEAKER_01:Yeah.
SPEAKER_00:And I don't think those are the right things because you know what? Even those scripted curricula, they always get something wrong and you need a great teacher in there for their writer to be like, you know what? I know I'm doing this, but I don't think that's quite the right way. But I think if we tweak it, we could get the elephant and writer to go in the right direction. And I really believe that that kind of information from researchers and teachers and curriculum designers working together is what's really going to help both the elephant and writer have coherence.
SPEAKER_01:One of the interesting pieces as we were reading that book and thinking, I wish other people read this book. There's a part when they're talking about the taxonomy that says, of course, this should be used, and I'm paraphrasing here, in concert with a school's theory of learning and philosophy of education. To say that the ethos of your school is should also guide the use of these tools as we are giving them to you. And I think about all the times where we talk about the tools and we never ask, how does this align with who we are or who we want to be as an institution or an organization or a collective? And I think what you're talking about right there is, again, hitting on that like, tools are great. Tools minus what do we believe in, who are we, what are we about, doesn't get us where we want to go.
SPEAKER_00:Because you need them to be connected. That's the thing. And a lot of times, the tools and the thinking are disconnected. The most relevant paper I have for you guys is actually we have this theory paper that we just submitted revisions on about how embodied experiences in the classroom, like hands-on sort of activities– I think there's this nice analogy where people are like, do hands-on activities. And I would argue that's elephant. It's like, oh yeah, just do this, play with these manipulatives. But a lot of times people do weird things with manipulatives because it's unconnected to what you're supposed to do with these manipulatives. And one of the things we argue in this paper is there's a reason why in the literature, there are lots of times when manipulatives or whatever hands-on activities don't lead to better learning. And it's because you need to think about that alignment between hands-on activities and more abstract, complex learning. That's why my ultimate message is don't give up hope.
SPEAKER_03:Oh, my goodness. Okay. So let me see if I can make sense of some of this. To understand a new and complex idea, we got to– oh, dang. Okay. How do I want to say this? Where
SPEAKER_00:do you start? Do you start with the complex idea? Nope.
SPEAKER_03:No, you piecemeal it.
SPEAKER_00:You do start with real concrete experiences.
SPEAKER_01:Okay. And when their brains are open
SPEAKER_00:and when they're ready,
SPEAKER_01:you say, here's a complex idea to
SPEAKER_00:the complex abstraction.
SPEAKER_01:So that later when I want to use it, my brain's got it all filed away nicely. And it's
SPEAKER_00:interconnected. Where when you think of the abstract thing, it reminds you of the concrete stuff. And when you do the concrete stuff, it also activates the abstract thing.
SPEAKER_03:Yeah.
SPEAKER_00:Isn't it beautiful?
SPEAKER_01:Yes.
SPEAKER_03:It is. I love it.
SPEAKER_01:G, you're real fun.
SPEAKER_03:All right. Thank you, G.
SPEAKER_01:Thank
SPEAKER_00:you. It really has been an honor. Thank you for letting me just pontificate. It's
SPEAKER_01:our favorite kind of conversation.
SPEAKER_00:Yeah.
SPEAKER_03:When we come back, we'll dive into the work of John Hattie and examine if what works works, even when the math might not.
SPEAKER_01:Hey, Stephanie.
SPEAKER_03:Hey,
SPEAKER_01:Zach. I shared that conversation with G, and it gave me an idea. I want to tell you a story.
SPEAKER_03:Ooh, I love story time. Okay, I'm ready.
SPEAKER_01:Once upon a time in the late 20th century, it's a more specific time, I suppose, a man named John Hattie. And one day, he stepped outside his metaphorical door, looked across the world, or maybe the parts of the world that were accessible to him through journal articles and databases. And he did a meta analysis of more than 800 meta analyses in education comprised of more than 50,000 separate studies and applied some statistical mumbo jumbo that I sort of understand. He wrote a book called Visible Learning. a synthesis of over 800 meta-analyses relating to achievement. It rolls off the tongue.
SPEAKER_03:Zach?
SPEAKER_01:Yes, Stephanie?
SPEAKER_03:I feel like we're jargoning right now. What?
SPEAKER_01:No, I was just telling the story.
SPEAKER_03:Yeah, I feel like we need to break down what a meta-analysis actually is. So meta-analysis is when you take a bunch of studies about the same thing and combine their results to try to find a general pattern. We can sort of think of it as finding the average of a bunch of different results and interpreting those averages as actions we can take. Sort of. It's a good enough interpretation for this episode anyway. So Hattie found more than 800 meta-analyses that looked at lots of smaller studies on education. These meta-analyses had put similar studies together and tried to figure out what usually happens when you have a specific thing in education or when you change a specific thing in education, like having students set goals for a class period or tracking whether living in poverty impacts a student's learning. And so what Hattie did is he took those meta-analyses and did another round of meta-analysis on them. He did a meta-meta-analysis, averaging the averages, if you will.
SPEAKER_01:Thank you, Stephanie.
SPEAKER_03:Oh, you're so welcome.
SPEAKER_01:So, back to the story. Yes. Hattie took this book around the world. It was renowned. In fact, this book and its successors have sold more than 2 million copies. I've never sold 2 million of anything. And they've been translated into 29 languages. Now, it is hard to undersell... Why? Why? Because it gave a list. And as G pointed out, it was new and sticky. It satiated our need for a silver bullet, even when we know there are no silver bullets.
SPEAKER_03:We know our brains love a list.
SPEAKER_01:Now, for those of you not in education, or who maybe just have seen the effects of Hattie's work, but never actually looked at Hattie's work, visible learning, deep in Appendix B... included a listing of 138 influences on student achievement covered by Hattie's meta-analyses. And there are hundreds of pages, about 300 pages almost, in this original book. But it was that list of 138 different things that were studied that made it sticky. And that is what people noticed. And that is what people remembered. If you remember back to our conversation with G not too long ago, and if you remember our conversation about Bloom's taxonomy, you know that folks love when that information is presented in the hierarchy. And so when they looked at Hattie's information, it was really easy to say these things at the top of the list look like they're really going to improve student learning. And these things at the bottom of the list, maybe I don't need to pay as much attention to. So schools... Then took that and they said, you know what, we're going to do these top 10 things or we're going to do these top 20 things. That's going to be what we do to influence learning. And they maybe let some other things go. In fact, sometimes they discouraged teachers from using practices lower on the list in favor of those at the top.
SPEAKER_03:Yeah, Hattie was all the rage when I started teaching. And let me tell you, as a first year teacher, I was very, very ready to have someone tell me what was most likely to give me the biggest bang for my buck. I drank the Kool-Aid. I did. Yeah, I did the jigsaw method. It was number seven on the list. I brought up prior learning number nine on the list. I told them they could do it. It was number three on the list. I was able to reference the moves I made from the list, and my principal was so proud of me, little tiny first-year teacher Stephanie. But, you know, whether it was because I was a newbie or because the moves weren't right for my kids, I didn't actually move the needle for my students. But, you know, the moves made sense in my brain. Like, logically and from a practitioner standpoint. So I kept at it. And as my teaching practice improved, the relationships I built with my students improved. And, you know, eventually the needle did move. But I don't know if it was from the moves from Hattie's list or me figuring out my vibe and rhythm as an educator, you know? It
SPEAKER_01:sounds like you need a study of your use of things from studies about studies to help your students study.
SPEAKER_03:Yeah, maybe.
UNKNOWN:Yeah.
SPEAKER_01:I have a story similar to yours. In one district where I worked, there was an administrator who frequently cited the importance of reciprocal teaching and the effect size that it had shown in Hattie's work. Now we'll get to effect size later, but it just is the number that made it higher up on the list. But none of us knew anything about math or science. Well, we knew about math, but we didn't know about statistics. We didn't know about effect sizes. So we didn't challenge anything. We didn't ask any questions. And this was largely how the book was received, is educators who mostly don't have any background in understanding research methodology or statistics kind of took things at face value. Or they said, this isn't quite what I think happens in my school or with my kids.
UNKNOWN:Right.
SPEAKER_01:eventually, not too long after it was published, statisticians started to look at the math Hattie presented. And they had some issues. Not only was each of these meta-analyses not necessarily comparing apples and oranges, it was more like comparing apples and orangutans. And that, if you are a statistician, is not good.
SPEAKER_03:No. I'd say even if you're a mathematician, it's not good.
SPEAKER_01:If you... are some sort of, you know, orange grower, also probably not good because then you've got a lot of primates and oranges. And you don't want to juice both of those. Anyway, in our next episode, that's right, this is our very first two-parter. We're going to dive into what was wrong that kept the math from mathing in Hattie's work. And we're going to ask a question of our guest of whether or not it matters. Was the impact net good, net bad? What was the effect size? I know I'm using that completely wrong of Hattie's work on education.
SPEAKER_03:Thanks for listening to Academic Distinctions, friends. Like, subscribe, and call your parents every once in a while. Follow us on Instagram at academicdistinctionspod. Find us on Blue Sky at fixingschools or find us on Facebook at And as always, in these unprecedented times, remember to wear your sunscreen and stay hydrated. If you've got a question or a topic you'd like us to tackle on the show, send them to mail at academicdistinctions.com. Until next time, I'm Stephanie Melville.
SPEAKER_01:And I'm Zach Chase.
SPEAKER_03:And you just got a little smarter. Bye for now. This podcast is underwritten by the Federation of American Scientists. Find out more at fas.org.
