Science of Reading Essentials: Assessment

In our latest Science of Reading: Essentials episode, Susan Lambert, Ed.D., draws on expert voices from across the podcast to show how assessment can be one of the most exciting and empowering tools an educator has at their disposal. Featuring Jan Hasbrouck, Ph.D.; Stephanie Stollar, Ph.D.; Kate Winn; Kymyona Burk, Ed.D.; Kajal Patel Below; Gina Biancarosa, Ed.D.; and Melissa Farrall, Ph.D., this episode covers the three types of assessment every educator needs—universal screening, diagnostic assessment, and progress monitoring—and how they work together; what the Oral Reading Fluency measure actually measures, why it matters, and how it's often misunderstood; and why assessing comprehension is so complex, and what teachers can do about it. 

Show notes: 

• Check out all of our Science of Reading Essentials episodes.
• Sign up for our upcoming Assessment Webinar Series. 
• Listen to Season 3 of Amplify’s Beyond My Years podcast.
• Listen to the full episodes:
  • The right assessment and the right data with Dr. Jan Hasbrouck
  • Assessment as your best friend, with Kate Winn and Stephanie Stollar, Ph.D.
  • Language is always an asset, with Kajal Patel Below
  • Understanding assessment, with Melissa Farrall, Ph.D.
  • Reimagining comprehension assessment, with Gina Biancarosa, Ed.D.
  • A proactive response to the adolescent literacy crisis, with Kymyona Burk, Ed.D.
• Subscribe to Susan’s new Science of Reading Substack.
• Join our Science of Reading community Facebook group.
• Connect with Susan Lambert.

Quotes:
“If you don’t have a question about your students, you don’t need to do more assessment. This should not be a compliance activity.” —Stephanie Stollar

“We never want to do one iota, one second, one minute of extra assessment that we don’t have to do unless it answers a key question that makes us better teachers.” —Jan Hasbrouck

“About 90% of all reading challenges are due to difficulty recognizing the words.” —Melissa Farrall

“We don’t want more testing. We want the right kind of testing.” —Kymyona Burk

“Once you ask a question, you can change what it is that someone has understood from what they read.” —Gina Biancarosa

“When we love the kids, we want to do the best for them.” —Kate Winn

“Ideally, kids would be assessed in their home language.” —Kajal Patel Below

Timestamps*:
0:00 Science of Reading Essentials: Assessment
3:00 Why educators have complicated feelings about assessment
4:00 The three key questions every assessment should answer
6:00 Universal screening
8:00 Screening multilingual and English learners
11:00 Diagnostic assessment
12:00 Progress monitoring
15:00 Why alignment across all three tools matters
16:00 Teacher-administered and curriculum-based assessments
18:00 Oral Reading Fluency (ORF): what it is and why it matters
24:00 Why ORF is widely misunderstood
28:00 The challenges of assessing comprehension
38:00 Key takeaways and next steps
*Timestamps are approximate

Transcript

Copy

[00:00:00] Stephanie Stollar: If you don't have a question about your students, you don't need to do more assessment. This should not be a compliance activity.

[00:00:10] Susan Lambert: This is Susan Lambert, and welcome to another edition of Science of Reading Essentials, a special series of Science of Reading: The Podcast from Amplify. On these essential episodes, we distill the information from our expert guests on a single topic so that you come away with the critical information you need to support students and educators in your community.

[00:00:33] And today, we're focusing on assessment. When assessment gets talked about in the media, we often think about reports like these.

[00:00:42] CBS Texas: Just hours ago, the nation dropped its newest report card, and there's...

[00:00:46] NBC: A national exam administered by the Department of Education confirmed that...

[00:00:49] CBS: Only 35% of 12th graders are proficient in reading, the lowest...

[00:00:54] Susan Lambert: These sorts of reports and stories about test scores can give the idea that assessment is just an evaluation tool, something separate from instruction.

[00:01:03] And I think one of the misconceptions about assessment is that it's another thing that has to be done to students, when what we need to see it as is an important part of the instructional process.

[00:01:17] And so I think the assessment we're going to talk about today is not that high-level assessment about how is our country doing or how are their states doing, but what we want to understand is how are our students doing and how can we use that information to make instructional changes.

[00:01:38] So now get ready for a deep dive into assessment.

[00:01:41] Kymyona Burk: There are these blanket statements like, "We don't want more testing." I understand, but we want the right kind of testing.

[00:01:48] Kate Winn: Believe it or not, reading assessment can be so exciting. It can be so empowering.

[00:01:54] Reid Lyon: Assessment is a great friend.

[00:01:59] Susan Lambert: Over the course of this episode, I think we're going to demonstrate that assessment is a great friend. But first, we want to acknowledge that that might sound far-fetched to a lot of people.

[00:02:11] When she joined us back in 2022 for Season 5, Dr. Jan Hasbrouck gave us a lot of great reasons for why educators might not immediately be in love with assessment.

[00:02:23] Jan Hasbrouck: There is a lot of anxiety around assessment. I do think, in my experience, my many decades now of experience, I think most people who go into teaching, they...

[00:02:33] First of all, you don't go into teaching because you love assessments. That's why, that's why I want to go into teaching. Certainly, that's not the reason. I mean, that's laughable. So I think a lot of us who become teachers have an aversion, a fear, a dislike, a distrust of assessments for a lot of reasons.

[00:02:53] Then there's the issue of, in teaching, we never have enough time. We never have enough time or money or people to do the work that we want to do. And so we can feel annoyed about assessments there, too, because it's absolutely right. The most important thing we do is our teaching, and we don't have enough time to do it.

[00:03:12] Susan Lambert: So Dr. Hasbrouck and many guests have emphasized that assessments should have a clear purpose.

[00:03:19] Dr. Hasbrouck explained on Season 5 that there are three key questions that assessment should be answering. We're going to explore each in greater depth, but I think this is a great overview.

[00:03:31] The first question is one that Dr. Hasbrouck described as a threshold question for all students that an educator serves.

[00:03:38] Jan Hasbrouck: Which of these students might need some extra help? Who needs some extra help? Then the next one is I found a student who needs, or probably a few students who need some extra help.

[00:03:50] The next question is, what kind of help do they need? What are their skill strengths? What are their skill deficits? How am I going to use my precious time and resources to give them what they need, not what I think they need or I wish they needed, but what do they really need?

[00:04:07] And then, the third driving question is, after instruction starts, it's is the work working? So we have categories of assessments that answer these questions.

[00:04:20] That first question, which kids might need some extra help, that's those benchmark screening assessments where we give to all kids, and the reason we give them to all kids is we're making sure that nobody falls between the cracks. We're finding those kids who might need help.

[00:04:35] When we've found them, we want to answer that question, what kind of help do they need? We have a category of assessments called diagnostic skills assessments. Do they need help with phoneme awareness? Do they need help with phonics? Do they need help with reading text fluency? Do they have all that down, but they just need help with compre... just need help with comprehension, or is that the focus?

[00:05:00] And then once I've got that information, I can use that to plan my instruction. And then I should bravely pull out the assessments designed to answer that question, is the work working?

[00:05:12] I do think that's the bravest question that a professional educator can ask. I want to evaluate my own teaching, and that category of assessments is called progress monitoring assessment.

[00:05:24] Susan Lambert: So let's now take each of those one by one.

[00:05:27] Let's start with screening. Dr. Stephanie Stollar co-authored the book Reading Assessment Done Right with Kate Winn, and she laid out this criteria for a good screener.

[00:05:38] Stephanie Stollar: Screening tools really need to be constructed so that they are brief, reliable, valid, standardized indicators of those essential skills, and most importantly, that they're predictive of important reading outcomes in the future.

[00:05:55] Susan Lambert: And just a couple weeks ago, Dr. Kymyona Burk gave us a great overview of the purposes of screeners.

[00:06:02] Kymyona Burk: A universal screener really does one of two things. It identifies those students who are at risk for reading failure versus those who are not.

[00:06:10] Now you can provide that type of targeted support that those students need before they get the report card or the parent comes to say, "Well, what's happening?" You can already begin your interventions.

[00:06:22] Susan Lambert: So that's one major purpose of a universal screener.

[00:06:25] The second big purpose of a screener is to evaluate the health of the system as a whole. Dr. Stollar elaborated on this additional purpose.

[00:06:35] Stephanie Stollar: The biggest misconception is that we're only doing screening so we find the low-performing students. And yes, that's an essential task because the earlier we find them and the earlier we intervene, we can impact the course of their lives.

[00:06:52] But there's a second reason we do universal screening, and that's to evaluate our instructional systems. When you want to know which students and which systems within your school are at risk or on track, that's a universal screening tool that answers that question.

[00:07:12] Susan Lambert: And when it comes to screening, there's something important to keep in mind regarding multilingual and English learners. Whenever possible, we want to screen them in their home language and in English.

[00:07:24] Here's my colleague Kajal Patel Below, who joined the podcast in 2024.

[00:07:30] Kajal Patel Below: It is so important to be screening for dyslexia and for other reading difficulties.

[00:07:36] And so, on a positive note, because of that, many states are working on legislation to require dyslexia screening. That sounds like a very positive thing.

[00:07:49] Susan Lambert: Yeah.

[00:07:50] Kajal Patel Below: An unintended consequence of that is, well, what do you think happens if you screen students in English, which is what the majority of districts are doing is they're screening in English.

[00:08:03] What happens when a student is still on their journey of acquiring English?

[00:08:08] Susan Lambert: Mm-hmm.

[00:08:08] Kajal Patel Below: And if that screening is mandated, and then if the results of that screening have consequences, so, well, what happens is, is that you have an over-identification of students that are then misidentified as having potentially a reading difficulty, including dyslexia.

[00:08:25] That is what is a major concern in the advocacy community for multilingual and English learners because these assessments are most often done in English and therefore we can have that misidentification, and it's really damaging to students to be labeled in that way when it's inaccurate. It drives the instruction that they're getting, and then maybe that is the incorrect instruction.

[00:08:48] And just to put some numbers on it, our own data shows that when students whose home language is Spanish were screened in English, when we looked at a particular dataset, kindergarten students whose home language is Spanish were screened in English, 72% of them were identified as needing Tier 2 or Tier 3 instruction. So that's a very high number.

[00:09:11] However, when those same children whose home language is Spanish were assessed in Spanish, only 33% were identified as needing Tier 2 or Tier 3 instruction, which shows the impact of language on that assessment. And so, ideally, kids would be assessed in their home language.

[00:09:35] Sometimes that's not possible. Spanish is the home language of about 75% of multilingual and English learners, and there are thankfully some Spanish assessments out there that can be used.

[00:09:50] It's a challenge though with other languages, and there are many other languages in this country. So if kids are being assessed in English, that data just has to be interpreted accordingly.

[00:10:00] There's deep concern that it wouldn't be, particularly when the screening is mandated, and so advocates just want to see that addressed.

[00:10:07] Susan Lambert: So a quick recap on universal screening, the first of those three assessments.

[00:10:12] As Stephanie said, screeners need to be brief, reliable, valid, standardized indicators of essential skills, and they need to be predictive of reading outcomes in the future.

[00:10:24] As Kymyona noted, screeners have two big purposes. They identify individual students who are at risk of reading difficulties, and they also identify the health of the overall system.

[00:10:38] And as we just heard from Kajal, students should ideally be screened in their home language. And if that's not possible, the data of their assessments in English should be interpreted accordingly.

[00:10:49] So we've talked about screening as the first critical tool in an educator's assessment toolkit. Next, let's talk about how diagnostic assessments fit into this. Here's Stephanie Stollar again.

[00:11:02] Stephanie Stollar: Screening tells you which students and systems need support. Diagnostic assessments tell you exactly what skill to teach next, so they're more fine-grained.

[00:11:12] They are not standardized, so they are not about comparing students' scores or predicting performance in the future. They are not brief. They're usually in depth and take quite a while to give.

[00:11:26] Susan Lambert: After we've administered a diagnostic, this information is super useful because it will pinpoint the kind of instruction we need to deliver so that the student can close gaps.

[00:11:39] The diagnostic assessment will tell me exactly what to teach that student. And when I know exactly what to teach the student, then we can employ progress monitoring. So after screening and diagnostic assessment, the third tool we need to focus on is progress monitoring.

[00:11:57] Here's my exchange with Stephanie Stollar on progress monitoring.

[00:12:01] Progress monitoring implies that we've taken some kind of action, and now we're going to monitor or see how well that action has produced results. Is that right?

[00:12:15] Stephanie Stollar: Yes. It's like the GPS for educators. It is the feedback to the educator so that you can make decisions with ongoing, a series of data, like data points on a graph, instead of making decisions at a single point in time.

[00:12:32] What you're looking for with progress monitoring is a trend. I think it's important for educators to understand it's not about the data point you just collected compared to the one last week, but it's about visually looking at the student's overall trend of progress over time.

[00:12:52] What's important for selecting progress monitoring tools is that every single measurement opportunity is at the same level of difficulty. This is what's behind the scenes in good progress monitoring tools that are curriculum-based measures, is that the authors have controlled for each one of those progress monitoring probes, if you will, being the same level of difficulty.

[00:13:18] If you see a student's score go up on the graph, you can be confident it's because they're learning, they've acquired more skills, not because that one is easier than the one that came before it.

[00:13:30] It's a fairly formal technology. Teachers will often say, "I'm monitoring this student's progress." We, in the book, talk about this idea of, I don't know, I would think of it as lowercase progress monitor. You're monitoring a student's progress.

[00:13:47] That's not the same as capital P, capital M, progress monitoring. It's a more formal act of checking in repeatedly with the same level of difficulty of task so that you can see if the student is trending towards the goal that you set with the instruction and interventions that you've been providing.

[00:14:07] And it's not just about documenting failure. The whole idea behind progress monitoring is to trigger a change to your instruction. If the trend is not going up towards the goal that you set, then you should take some sort of action. You should make a change to instruction.

[00:14:25] Susan Lambert: And ideally, all three of these tools, screeners, diagnostic assessment, and progress monitoring, are aligned.

[00:14:34] Here's Stephanie's co-author, Kate Winn, with more on that.

[00:14:38] Kate Winn: Having the tool, like the progress monitoring tool that aligns with your screener and being able to track those data points, there's just something, I know I used the word empowering before, but I keep coming back to it because it gives you direction.

[00:14:51] And then like Stefanie said, you see if it's working, keep doing what you're doing. If it's not working, there are things you can consider and change. It's not this big mystery about what to do.

[00:15:02] Susan Lambert: Essentially what aligned means is that they're all assessing similar skills or the same skill. If they were misaligned, we would be progress monitoring something that we weren't actually assessing in the screener, and that isn't helpful.

[00:15:18] That doesn't help us understand how well the student is responding to that intervention instruction. But when the screener, the diagnostic, the progress monitoring are all aligned, we should be able to see then how well the student is responding to that instruction.

[00:15:35] And there's another type of assessment I'd like to talk about too: teacher-administered assessments that are drawn directly from the curricular tool or the program that you're using.

[00:15:47] First of all, a caveat. This discussion assumes that you have a high-quality literacy program that you're actually teaching in your context.

[00:15:58] And if you have that high-quality instructional program or curriculum, then you'll want to use the assessments within that program, both the formative daily assessments, meaning the activities you have students do, along with maybe end-of-unit assessments. You want to use those because it gives you valuable information about how well students are learning the information, the topics, the concepts, the skills that you're teaching.

[00:16:31] And if you're not seeing your students progress or being successful on those daily formative assessments and end-of-unit assessments, you need to think about, "Do I need to change how I'm delivering my instruction? Do my students need more practice with this? Do I need to pull a small group of students to give them reteaching or also additional practice?"

[00:16:54] And you can use those assessments to really understand what's happening day to day to day within your core program or within the Tier 1 instruction that you're delivering.

[00:17:07] Again, that assumes that you have a high-quality instructional program and also that that high-quality program is also teaching and assessing similar skills that your screener is assessing and that you're progress monitoring when it comes to intervention.

[00:17:26] So you can hear the importance of all that alignment.

[00:17:29] And if I can give a little tease to our upcoming season, Season 11, we're going to be talking about what that looks like within this Multi-Tiered System of Supports and why it's so critical to have alignment.

[00:17:46] So at this point, we're going to make a little bit of a transition, and we're going to talk about some specific types of assessments that can really make a big difference.

[00:17:56] And I want to start with the Oral Reading Fluency Measure, or ORF. It is fair to call it a kind of screener. We typically don't administer Oral Reading Fluency until first grade, but as students start to get a little bit older, it's a key indicator of how well they may be comprehending or what next steps for intervention instruction might need to look like.

[00:18:22] The Oral Reading Fluency Measure, or ORF, will tell us how well a student is reading in terms of their reading rate. We actually administer it, we time them for a minute, and we count the words.

[00:18:36] Now, we know we don't want students to be reading too fast or too slow, and we look at this measure because it'll give us an indication of how automatic students are with their word recognition, and in other words, how well then they have this sort of instant recognition.

[00:18:52] Back in 2022, during Season 5, Dr. Jan Hasbrouck spoke all about ORF. She explained she was initially skeptical decades ago when she was an experienced reading specialist who started some postgraduate-level coursework at the University of Oregon. A brand-new professor named Jerry Tindal was teaching the courses.

[00:19:13] It turned out he'd been working with other scholars at the University of Minnesota on creating ORF.

[00:19:19] Jan Hasbrouck: And what they were all studying together was this idea of creating very short, reliable, valid assessments that teachers could use to both quickly identify kids who might need some academic help and evaluate the effectiveness of that.

[00:19:40] So that's what Jerry had been studying as a doctoral student. One of those assessments that was created by this amazing team of people was this weird little assessment called Oral Reading Fluency.

[00:19:53] And I remember sitting in that class. It had just been invented.

[00:19:59] Susan Lambert: No.

[00:19:59] Jan Hasbrouck: Nobody heard about it. Nobody was writing about it. There were some research reports, but Jerry was talking about it all the time. This was the key to everything. And you're like, "I'm supposed to be learning how to go out into schools and help teachers, and all you're talking about is assessment." It didn't quite jive.

[00:20:17] And then when he said... He told us about ORF: "You have a child read aloud from an unpracticed passage. You have them read for 60 seconds. You score their errors. You calculate words correct per minute, and that's this amazing tool." And he said, "Then you take that score, words correct per minute... What do you do with it? You create building-level norms, and then you can find if your student is at the normative benchmark or not."

[00:20:53] And I just sat there thinking, first of all, "60 seconds? Are you kidding me? I'm a reading specialist. I have a master's degree. I've been doing this for a long time. I can't imagine that 60 seconds is going to give me what I need. And are you kidding me that we should create building-level norms?" My hand went up.

[00:21:15] Jerry and I, to this day, still recall that conversation. Arrogant little student that I was, who said, "Jerry, I've been teaching primarily my 15-year career in Title I schools. You get to be a Title I school by having low-performing, usually high-poverty kids. So I don't get that we should create building-level norms for these low-performing, at-risk kids, and then use those low norms to let us know anything about kids. That doesn't make any sense to me."

[00:21:52] He said, "Well, what should we do about that?" And I said, "Well, we need to create national norms." That was the spark of now 25 years of research that Jerry Tindal and I have done together to create those national norms.

[00:22:09] But I did take some convincing just to believe in the value of that ORF assessment. And so there could not be, Susan, a bigger skeptic. I know there are skeptics of ORF. When I share ORF, or now I'm very rarely sharing something new with people, they've heard of it. Words, oral reading fluency, DIBELs, Acadiance, whatever it is, they've heard about it, and they may use it, but people are as skeptical as I was about the value of it.

[00:22:41] And I get it. I sit there and listen to their skepticism and say, "Yeah. Yeah. It doesn't make any sense." But I've been using it now for, what would this be? 30-some years, with several research studies using it. I've used it as a practitioner. I've used it as a grandmother measuring my own grandson's progress. It works.

[00:23:06] That team back at the University of Minnesota essentially invented the tool that we, as reading professionals, can use as a thermometer. It has reliability. We can trust the score. It gives us consistent scores. It has validity. It gives us good, useful information. And just like a thermometer, it's super quick.

[00:23:30] But just like a thermometer, it only tells us essentially whether a student has a fever or not. Should we worry about you or not worry about you?

[00:23:41] If we can understand that this weird little assessment really is valuable when we understand what it is and what it isn't, how to use it, how not to use it, when to use it, when not to use it, I could not imagine being a professional without that tool, but it is widely misunderstood.

[00:24:02] Susan Lambert: Yeah. I think you mentioned to me that you think the name itself breeds some mis- misconceptions.

[00:24:09] Jan Hasbrouck: Oh, yes. Thank you for remembering that. It does. People have written about this, that it was very unfortunate to put the label of fluency on that assessment. What words correct per minute is measuring is accuracy. Words correct per minute, which is a rate measure.

[00:24:28] And in the general research, the science of learning, rate plus accuracy is considered fluency. In educational psychology and where we study skill acquisition, you start with accuracy, and then as you get more proficient, your rate increases, and so you get to a point of fluency or automaticity.

[00:24:50] So it is, in a way, a measure of fluency in the way the general world thinks about it. But we reading teachers think of fluency as something much, or we should think of fluency as something much more multifaceted and complex that at minimum includes prosody or expression, which has nothing to do with an ORF score.

[00:25:13] And in reality, when we're really looking at does a student have sufficient text reading fluency to allow comprehension, it is their accuracy, it is their rate, it is their expression, it is their metacognition, it is their background knowledge. It's all of this stuff that really experienced reading teachers think of as fluency.

[00:25:39] So they think of fluency that way, and then you come along and say, "Here's this little weird 60-second assessment that measures fluency," and they say, "Oh, no, it doesn't." And you're right. It doesn't. What it does measure is words correct per minute. We could have just called it words correct per minute. It is oral. It is reading, reading aloud, but we shouldn't have called it fluency.

[00:26:03] There are some colleagues of mine that suggested it should have been named Oral Passage Reading, OPR, Oral Passage Reading. That's probably a little bit better. John Hosp has suggested, "Let's just label it automaticity. It's a measure of automaticity." I like that too, 'cause I think that really is what it's measuring and why it's important.

[00:26:25] But yes, I think it's unfortunate, and I spend way too much time in my webinars and trainings talking about what ORF is and how wonderful it is and how it's weird, yes, but you should trust it, but it should never have been called fluency.

[00:26:40] Susan Lambert: So the message here is trust that ORF measure, but know it's a measure that needs... It's an indicator of something.

[00:26:48] Jan Hasbrouck: It's an indicator.

[00:26:50] Susan Lambert: I think the thing about ORF that is so important is that it is highly correlated to reading comprehension. And so it's often misunderstood, the importance of actually administering that ORF measure, what it doesn't always tell us. It doesn't always tell us what we need to do next in instruction.

[00:27:09] It's like this marker for, hey, if students have this great oral reading fluency, they seem to be reading at a benchmark score, then we can assume they're doing okay, and we may not need to do any sort of intervention.

[00:27:25] If we notice that they're reading very slow or their ORF reading passage is not at benchmark, we may need to do some next-step diagnostics to better understand what to do in instruction. But it's a really important measure that we need to pay attention to in our instruction.

[00:27:45] We just talked about ORF, and we know that ORF is highly correlated to reading comprehension, meaning that if kids get a strong benchmark score on ORF, it's likely that they're comprehending well.

[00:28:01] But let's transition to talk a little bit about assessing comprehension overall. Sounds easy, but it's really difficult.

[00:28:10] And we've spent lots of time on the podcast. We did a whole season talking about what comprehension entails, and we know that there's many factors that can impact comprehension.

[00:28:24] If students aren't recognizing instant words, it's going to impact their later comprehension. If students don't understand syntactical awareness, it's going to impact their later comprehension. So it's challenging.

[00:28:39] Let's talk about these challenges of trying to assess students' comprehension. On Season 10, Dr. Gina Biancarosa gave us an overview of how we've tried to assess comprehension and why it's actually so difficult.

[00:28:54] Gina Biancarosa: Historically, for over 100 years, the way that we've assessed reading comprehension is by asking questions after someone's done reading. The problem with that, it's twofold.

[00:29:06] One, you're not getting at that process. You're only getting at the end state. But even more importantly, once you ask a question, you can change what it is that someone has understood from what they read.

[00:29:20] Susan Lambert: Tell me more.

[00:29:21] Gina Biancarosa: Basically, we could ask questions about inferences that one might have made while they were reading, but you could have a reader who didn't make an inference while they were reading, and it's only instigated by the question itself.

[00:29:36] When we ask, "Why did Harry hide under the invisibility cloak?" they might not have been thinking about what was motivating Harry. Sorry, a little bit of a Harry Potter fan. But once we ask that question, suddenly you can think, "Hmm, I wonder why." So you may not have been wondering why while you were reading, but the question itself instigates that thought process.

[00:30:03] And so that's really the problem with asking questions, period. Even if you were to ask them during reading, which folks like Kristen McMaster have done, has done a lot of research on, do we improve comprehension by asking questions during reading versus after reading? And the issue with asking those questions is, again, you're changing that process, potentially changing that process.

[00:30:28] For some readers, they may have made those inferences naturally. But you have no way of knowing.

[00:30:33] Susan Lambert: Here's Dr. Melissa Farrell, who also joined us on Season 10, talking about the challenges of assessing comprehension.

[00:30:40] Melissa Farrall: Sometimes we ask students to respond to our questions after a single read, where they read the passage, and we take it away, and then they then have to answer the questions.

[00:30:51] Some folks feel that's kind of unfair, that there's too much memory involved, and that we're assessing more memory than actual comprehension.

[00:30:59] Some tests permit students to reread the passage so that they can reread the passage the first time, look at the questions, and go back and read strategically the second time. Is that really what we want to know?

[00:31:14] There's an interesting thing here. Do we want to know about a student's ability to learn when reading? Is that truly reading comprehension, meaning that the texts we give them are somewhat maybe exotic in their nature, unfamiliar content, so we're really asking them to grapple with new knowledge?

[00:31:34] Or should we be looking at comprehension with respect to their background knowledge and give texts that are more or less familiar? Is that what reading comprehension is, our ability to understand, given that we come to it with all this background knowledge? And I'm not sure there's any one right answer.

[00:31:54] Susan Lambert: Dr. Farrell is the author of Reading Assessment: Linking Language, Literacy, and Cognition, and there was a specific passage I wanted to run by her.

[00:32:04] So you wrote, "Reading comprehension is not easily quantified, and it can be difficult to separate out the factors that contribute to reading comprehension." You also say, "At the very heart of reading comprehension testing lies a problem."

[00:32:22] Why is this so hard?

[00:32:24] Melissa Farrall: Why is it so hard? I think because when we think about reading comprehension, we really want to think about the "making meaning" part. We want to think about how it is we make meaning in all our intellectual glory using the full range of our cognitive skills.

[00:32:41] But the problem is, is we test reading comprehension in print, and maybe that sounds somewhat, oh, I don't know what the word is, somewhat surprising.

[00:32:52] But when we are assessing the skills of individuals who struggle with decoding, we just can't get to the thinking part. We are in a whole other domain, and what we really learn about is the need to have a deeper, better understanding of those decoding skills and phonological processing skills as a pathway to then making meaning for print.

[00:33:18] Susan Lambert: And I think the idea is, I'm thinking from a teacher's point of view right now and a teacher's probably like, "Oh, man. Are you saying that I should be giving listening comprehension assessments if my student is less than what I think is successful on a reading comprehension assessment?"

[00:33:36] Melissa Farrall: I think we should be doing both listening comprehension and decoding, because we really want to rule out challenges with oral language, as well as have that deeper understanding of decoding and phonological processing.

[00:33:49] Susan Lambert: Right now I'm thinking again as a classroom teacher. What are you telling me that I should be doing with some of my curriculum-based reading comprehension measures? I teach this content, then I give students a passage to read, and I have them answer questions about that passage.

[00:34:08] Is that helpful to me? Should I be doing that in the classroom?

[00:34:12] Melissa Farrall: I think there's every reason to do that. I think that we have to make sure, however, that all of our students have access to that rich content, and that means that for our students who are struggling with decoding, that they have access to audio text, because there's no reason why they can't participate in activities designed to promote vocabulary and verbal reasoning skill and background information, but they need to do it through their ears as opposed to trying to get it off the page.

[00:34:41] We talk about the assessment of reading comprehension. We see, for example, the scores on the National Assessment of Educational Progress. And they have a test that purports to measure the comprehension of narrative text and expository text.

[00:34:57] And when we see those low scores, there are always articles in the newspaper bemoaning the state of reading comprehension, and there's certainly, certainly reason for that. But the fact is, is that those low scores don't tell us a whole lot. They may speak to a particular level of comprehension. But they don't inform us as to the whys.

[00:35:20] We may know that a child is struggling with reading comprehension, but that singular test that we give doesn't answer any of our questions as to why that struggle is taking place, unless we are then crafting an evaluation that lets us look in greater depth at both decoding skills and at language skills.

[00:35:46] And we know that about 90% of all reading challenges are due to difficulty recognizing the words. So what is it we get when we see... Do we get a true measure of reading comprehension with a child who simply can't read the words?

[00:36:03] And I think that's a fascinating question, and for that reason, we need to supplement our testing with measures of oral language, with listening comprehension, so we have a true measure of the potential for reading comprehension.

[00:36:17] Susan Lambert: Hmm, I love how you said that.

[00:36:19] Melissa Farrall: And I will also say that oftentimes we are doing these evaluations because children are struggling, and there's a lot of bad news in our reports. We do our best to talk about both strengths and weaknesses, but the reality is, is that we're going to see these low scores.

[00:36:34] When we add that measure of listening comprehension into our battery, we now do several things. We get a measure of potential for understanding, and we can say, for example, to parents, "We believe that once we teach your child to decode, that she will be reading at this particular level."

[00:36:51] And that's something to anticipate. It's something to look forward to. It's a bit of good news because, yes, the intellectual power is there. The thinking skills are there, but what we have to do is train your child in phonological awareness and in decoding skill in order to get there.

[00:37:10] Susan Lambert: The big takeaway is comprehension is complex, and there's different elements of the comprehension process or the things that we do while we're reading that impact the comprehension product or the outcome of the comprehension.

[00:37:26] And the best way to really assess comprehension is, first of all, to understand all the elements included in the reading process so that we can better understand what's impacting students in their comprehension product.

[00:37:45] So as we enter the home stretch here, let's talk about some of the biggest takeaways and next steps.

[00:37:51] The big idea is assessment is a critical instructional tool, but it has to be done right. First of all, we want to only collect data that will inform instructional next steps.

[00:38:03] Here's Jan Hasbrouck again from 2022 with one last impassioned reminder of this.

[00:38:10] Jan Hasbrouck: We never want to do one iota, one second, one minute of extra assessment that we don't have to do unless it answers a key question that makes us better teachers.

[00:38:24] Susan Lambert: A second big idea that Kate Winn shared, assessment is not deficit-focused.

[00:38:30] Kate Winn: As teachers, we do love each and every one of these unique little children, and we do, in terms of a whole program, we do honor their strengths and the assets that they bring to the classroom, their gifts, their talents, all of that. That's true.

[00:38:46] It's also true that it's our job to teach them to read. So if we know that an assessment is the tool that is going to help them learn to read, I think it's okay that we are maybe using terminology like "below benchmark."

[00:38:59] If this is an essential skill and they are not meeting the benchmark to tell us that we can feel safe that they're going to have reading success, that's important to know because we want to get to them right away. We want to target their instruction. We want to get them to where they are at benchmark.

[00:39:13] This is not like a lifetime label that we're focusing on something the child can't do. But I don't think it's doing the kids... I think it's doing them a disservice if we only want to be focusing on the good qualities.

[00:39:26] And so I think when that's coming from a place of love, I think the messaging is when we love the kids, we want to do the best for them, and it will include some assessments like this.

[00:39:36] Susan Lambert: Third, all students need to be screened consistently. Here's Dr. Stephanie Stollar.

[00:39:41] Stephanie Stollar: First of all, all students need to grow. Even your high-performing students need to grow across the year.

[00:39:47] If you're on track at the beginning of the year, great, but you should be growing your skills, and you should be performing higher at the middle of year and higher still at the end of the year.

[00:39:58] The second reason is that in the primary grades with curriculum-based measures for universal screening, the measures change over time. So if you didn't screen kindergartners at the middle of the year because they were on track at the beginning of the year, you wouldn't have a chance to assess them on that skill.

[00:40:18] Susan Lambert: Also remember that universal screening isn't only about identifying students who need support. It's also about evaluating the health of the entire system.

[00:40:28] Here's Stephanie again.

[00:40:29] Stephanie Stollar: So if we don't include every student at the middle and end of the year, let's say we take all of the high-performing students out of the mix, then we've skewed the sample, and we can't use that information to give us feedback about our adult systems of instructional support.

[00:40:50] Susan Lambert: And then remember to be cognizant of best practices for screening multilingual and English learners.

[00:40:57] Again, here's my colleague Kajal.

[00:41:00] Kajal Patel Below: The assessment data in English is still of value, but it needs to be interpreted with a different lens. You can't just look at it and say, "Oh, okay, well, I think this child is at risk for reading difficulty. I'm going to put them in this group."

[00:41:16] Susan Lambert: As we wind down, some final thoughts on assessment.

[00:41:19] First of all, remember that assessment should be closely tied to instruction. It's part of the instructional process. You need to know what students know so that you know what to do next in your instruction, and this is especially true for students needing intervention.

[00:41:34] And the next thing is that assessment is a critical element of a Multi-Tiered System of Supports. And why would I bring that up at the end of the episode?

[00:41:45] Because Season 11 is all about MTSS, and you really need a foundational understanding of assessment, the purpose of assessment, how it can be used to adjust instruction, how it can be used to ensure that students are tracking and they're learning with the instruction that you're delivering.

[00:42:04] And so I'm really excited for Season 11, all about Multi-Tiered System of Supports and taking a deep dive into each of the elements. And remember, assessment is a critical element in that process.

[00:42:21] Thanks for listening to the latest edition of Science of Reading Essentials, a special series of Science of Reading: The Podcast from Amplify.

[00:42:29] We've already done four of these special episodes on writing, comprehension, dyslexia, and learning science. We have all these special episodes collected in one place on our resource site, amplify.com/soressentials. That's where you'll also find free resources especially curated to complement these episodes.

[00:42:54] If you'd like to go even deeper into assessment, we have an upcoming free webinar series on this topic. Find out more information and save your spot now at amplify.com/assessmentandinterventionweek. In just one week, we'll be releasing our official Season 11 trailer, and then we'll be kicking off our MTSS season with Dr. Joan Schumann.

[00:43:18] Joan Schumann: When you're bringing in this model, it flies in the face of this fixed mindset where you're like, "Nope, this is what we do for all kids, and then there's special education programming for kids with long-term needs."

[00:43:30] And what MTSS does is it disrupts that and says, "Well, wait a minute. Hold on. Aren't there a lot of reasons why kids would struggle to learn beyond just having a disability?"

[00:43:42] Susan Lambert: That's coming up. Science of Reading: The Podcast is brought to you by Amplify. I'm Susan Lambert. Thank you so much for listening.