Science of Reading: The Podcast

S10 E6: Understanding assessment, with Melissa Farrall, Ph.D.

• Amplify Education • Season 10 • Episode 6

Share episode

In this episode of Science of Reading: The Podcast, Susan Lambert is joined by Affiliated Scholar at the Stern Center for Language and Learning, Melissa Farrall, Ph.D., to discuss understanding assessment. Melissa explains why it's beneficial for every educator to understand the fundamentals of assessment, especially comprehension assessment. Together, Melissa and Susan discuss the relationship between reading comprehension and language comprehension, why reading comprehension can be challenging to assess, and how, in a perfect world, educators would be trained both in the Science of Reading and assessment.

Show notes:

Quotes:

  • "My view of reading comprehension is that it is thinking guided by print." —Melissa Farrall, Ph.D.
  • "If we supplement our evaluation with measures of listening comprehension, we can then get a sense of an individual's ability to make meaning." —Melissa Farrall, Ph.D.
  • "In a perfect world, we would have not just evaluators, but educators who are trained both in the Science of Reading and in assessment so that we can all sit at the same table and participate." —Melissa Farrall, Ph.D.

Episode Timestamps:
00:00 Introduction: Exploring comprehension assessment, with Melissa Farrall
07:00 The legacy of Jean Chall's research on the developmental stages of reading
10:00 "Reading Assessment: Linking Language, Literacy, and Cognition"
17:00 Comprehension is thinking guided by print
21:00 Different ways of assessing reading comprehension
27:00 Kintsch's construction-integration model
30:00 Word recognition
33:00 Reading comprehension is not easily quantified
38:00 How background knowledge affect the meaning-making process
41:00 The two modalities of language comprehension
45:00 How today's educators might think differently about comprehension instruction
48:00 Closing thoughts

*Timestamps are approximate, rounded to nearest minute



[00:00:00] Melissa Farrall: When I think about reading comprehension, I think about it in terms of our cognition and all its glory.

[00:00:09] Susan Lambert: This is Susan Lambert and welcome to Science of Reading: The Podcast from Amplify. This season is all about comprehension, and today we're continuing our exploration of comprehension assessment. Last time around Dr. Gina Biancarosa spoke about her groundbreaking work on a new kind of assessment. If you haven't listened to that one yet, please check it out. We'll have a link right in the show notes too.

This time we're continuing to explore comprehension assessment with Dr. Melissa Farrall, author of Reading Assessment: Linking Language Literacy and Cognition, as well as the co-author of All About tests and Assessments. She recently retired as director for evaluation at the Stern Center for Language and Learning in Williston, Vermont.

In short, she's a real expert when it comes to assessment and I'm thrilled to bring her onto the show. And don't forget to submit your questions about comprehension at amplify.com/SORmailbag. And now here's Dr. Melissa Farrell.

Dr. Melissa Farrell. Thank you so much for joining us on today's episode.

[00:01:20] Melissa Farrall: I'm, I'm very happy to be here.

[00:01:23] Susan Lambert: Oh man, I just can't wait to get into this because, we'll talk about it in a minute, but, um, but you have created a resource that is absolutely fabulous. And like I said, we're going to get into it in a minute, but I'm really excited to have you here. Before we jump in, I'd love for you to tell us a little bit about how you became interested in literacy.

[00:01:45] Melissa Farrall: I actually have a rather unusual background. I came to literacy and instruction of children with learning disabilities from a background in Slavic linguistics.

[00:01:56] Susan Lambert: Oh wow. That's, that's a new one.

[00:01:59] Melissa Farrall: And it may sound really esoteric and it may sound like it's very disconnected, but that background gave me a strong, uh, strong command of phonology and phonology in different languages.

And I understood all about speech sounds and I understood what happened when you combine them together and how it is we group them into morphemes. Is this sounding very familiar to you?

[00:02:19] Susan Lambert: Yes, sure is.

[00:02:20] Melissa Farrall: Yes. So my first course, when I decided to become an educator, my first course in reading was a course in the stages of language and reading as described by Jeanne Chall. And my professor at the time was Sarah Brody, who was a disciple of Jeanne Chall. And this course made me look at reading a whole new way. Not that I had ever really thought about reading before, you know, because I learned to read easily, it wasn't a big deal. I never thought about that process because for me it was pretty much seamless.

[00:02:50] Susan Lambert: Yeah.

[00:02:51] Melissa Farrall: And uh, anyway, I, I had this fascination for phonology and a fascination for phonemic awareness, and this idea that phonemic awareness was the make-or-break skill. That it really determined whether or not you would become a reader and speller. And that sort of, um, subsumed me for a while, and I remember reading all kinds of articles and I took off from there.

[00:03:12] Susan Lambert: Oh, that's so, that's so incredible. Can we go back though? Tell me a little bit about Slavic. Like how did, where's the Slavic connection here?

[00:03:22] Melissa Farrall: Um. It's, well, my Slavic connection, I studied Russian as an undergraduate and I spent a semester living in Leningrad during the Cold War, which was a very interesting experience.

And then I went on and, and my Ph.D. is actually in Slavic linguistics, which required me to have a command both of the literature and Russian—19th century Russian literature is like the greatest century of, of literature ever created, you know. Um, it also required that I had a, a command of linguistics and the structure of language so that we could compare the development of Russian to Czech or to Serb or Croatian.

And so I spent many, many years sort of studying phenology and actually ended up with a specialty in medieval Russian literature. My job was to examine old Russian manuscripts to try to determine the language of the original source document. And, and once again, it sounds really disconnected and it sounds really kind of strange and it sounds kind of weird, but the fact is, as I spent my time analyzing spelling errors in old Russian manuscripts to look at what they would reveal to us, uh, about the scribe and about the language of the original document.

[00:04:41] Susan Lambert: Wow.

[00:04:42] Melissa Farrall: So, I, to this day, am fascinated by spelling errors and what they can tell us about how a student, um, perceives sound patterns and words, and to what degree they have internalized rules for sound symbol correspondence and structural analysis and all those good things.

[00:04:59] Susan Lambert: Wow. Well, I promise you that when we do a season on 19th century Russian literature, you'll be the first one on and we can actually talk about the content of that.

[00:05:13] Melissa Farrall: I would love to do that.

[00:05:15] Susan Lambert: That is so fascinating. And what's really fascinating about it, we're going to sort of get into this idea of assessment, is that you were doing assessment of language through that spelling analysis way back then.

[00:05:28] Melissa Farrall: Way back then, and little did I, I had no understanding, no idea at the time that that would essentially become my career path, you know, analyzing children's spelling errors.

[00:05:40] Susan Lambert: Sometimes you just don't know. You take the next step of something that's interesting in your work and ta-da, here you are. That's so fascinating. I'm going to ask you one more question too. You mentioned the name, Jeanne Chall. I don't know that we've talked specifically about who she is. I think her name has come up a few different times in some different episodes, but I would love if you could tell our listeners just briefly who she was and why she's important to this world of the Science of Reading.

[00:06:12] Melissa Farrall: I think that she belongs in that pantheon of reading researchers and, and it's interesting that we don't talk about Jeanne Chall very much these days, but she was in her heyday and glory, Professor Emeritus at Harvard University.

The reason she's important is that as a very young new professor, sort a newbie in the field, she was asked to research this question of why reading disorders were epidemic in the United States. And she had no idea what she was getting into at the time because she, you know, she wasn't a specialist in reading. And anyway, ended up doing, um, sort of developing a theory of language and reading development that was very similar in some ways to the developmental stages of Piaget. And what she said is that all children, you learn to read following the same path. And that path begins with a foundation and language development, that children then become aware of speech sounds, and they move through a series of stages.

The first one, sound symbol correspondence; the next one, automaticity and fluency, then to a stage of comprehension. And you know, that expression that we use when we talk about how children go from learning to read to reading to learn?

[00:07:29] Susan Lambert: Yeah.

[00:07:30] Melissa Farrall: That was Jeanne Chall.

[00:07:31] Susan Lambert: Yeah.

[00:07:32] Melissa Farrall: Anyway, she was sort of, um, monumental in the field at that time. She inspired a tremendous amount of controversy because she cited the importance of direct instruction for reading at a time when the focus was really on the making meaning part of reading and whole language instruction. And, um, at one point she had, um, even earned the label of phonicator.

[00:08:00] Susan Lambert: I hadn't heard that before.

[00:08:02] Melissa Farrall: Yeah, yeah, yeah. She was called the phonicator by those who were not similarly persuaded.

[00:08:09] Susan Lambert: Yeah, that's great. You know what we'll do is, um, there's a couple of texts of hers, for sure one is still available, we'll link it in the show notes for our listeners.

[00:08:18] Melissa Farrall: Oh good.

[00:08:18] Susan Lambert: In case they want to go back. I think it's really important as we're thinking about, what is the evidence-base show? What is, you know, what are we learning? That, um, that we have a foundation of understanding in some of the you know, this key research, and I think Jeanne Chall is one of those—she actually, uh, really opened my eyes a lot to better understanding how, how kids learn to read. So I appreciate you, uh, taking that with us.

[00:08:43] Melissa Farrall: Yeah. I think one of the reasons that we don't talk about Jeanne Chall so much anymore is that her view was largely sequential.

So she talked about how it developed. It was a stage theory, and nowadays we really think more about the simultaneity of all the different processes that we use when we read. So we don't just speak about of development of phenological awareness and decoding skills. At the same time, we are talking about the development of language skills.

[00:09:12] Susan Lambert: Yeah. Yeah, that makes sense. And we're going to dive into this a little bit, but it does speak to, um, an understanding of the complexity of reading comprehension, and thus the book. Okay. So let's make a transition to this book, this fabulous book. I have it in front of me and I have to tell you... So the title of the book is called Reading Assessment: Linking Language Literacy and Cognition.

I used this book in my doctoral work at Mount St. Joseph University and I had forgotten how rich a text this is and how comprehensive it is. So anybody that's listening right now that says, "Oh, I'm not really that interested in reading assessment." This book is bigger than just reading assessment. So can you just provide a little context of, of why you wrote it, how long it took? A long time, I hear.

[00:10:10] Melissa Farrall: Um, the book took about three years to write. And it was pretty much a 7-day-a-week endeavor. And I, I remember getting up early on weekends and leaving my husband, who was still comfortable in bed, and going off and saying to him, "Time to make the donuts." It was that kind of, you know, so it was a, uh, a full-time plus effort for several years.

I wrote the book out of frustration. At the time I had been teaching a course in, I think it was more broad, not just reading assessment, but academic assessment. And we would require our grad students to buy these very expensive textbooks.

[00:10:46] Susan Lambert: Hmm. Yeah.

[00:10:47] Melissa Farrall: And very few of those pages were actually devoted—I mean, we're talking, you know, maybe they were five pages out of this 300-page textbook—were devoted to the assessment of reading or the assessment of writing or the assessment of language. And by the time you had read through that text, you really had no better understanding of how it is to think about designing a comprehensive reading evaluation than you did prior to reading it. So I felt that students were spending this money on textbooks and they by and large were not particularly helpful in permitting students to understand that link between taking high-quality data...

[00:11:28] Susan Lambert: Yeah.

[00:11:29] Melissa Farrall: And good instruction. So that was number one. Number two was my experience sort of as a beginning educator at evaluation team meetings, where we would go to attend such a meeting and the school psychologist would pretty much dominate the meeting and talk through the whole thing and present his or her evaluation. Evaluations were full of jargon, were very difficult to understand, and the fact is, is that I always felt the, the playing field was not particularly level in that the other individuals are sitting around the table, the other educators, didn't have a sufficient background to really engage in an in-depth, helpful conversation about what should be included in a reading evaluation and how do we in the end interpret those results.

I had experienced at that time those moments where I would have a gut reaction to something that the school psychologist would say. It just wouldn't jive, or it didn't sound like the student that I had worked with. And what I kind of wanted to do was to empower educators to feel confident enough to act on those reactions, because sometimes in an assessment we don't fully capture a child's profile.

And as much as I love doing these evaluations and as helpful as I think they are, they are often conducted in a fairly artificial environment that's not particularly real, and that's free from distractions and other things going on. So my goal was to create a textbook that educators and evaluators would find helpful and that would really permit them to both read reports with a deeper understanding and/or craft reports making better decisions about what they should include, what test instruments they might want to consider, and then how to interpret their findings.

[00:13:27] Susan Lambert: It's really fabulous. And again, for our listeners, I'm just going to read through a few of the chapter titles: Reading Theory and the Stages of Reading Acquisition; Oral Language; Linguistic and Cultural Diversity.

It's the first four chapters before you even get into statistics and test development. So it's just this real comprehensive approach that, if you're going to assess something, there's some foundational elements you need to understand, including, what is reading and how it develops. So I just want to thank you for this.

[00:14:01] Melissa Farrall: Um. I, I'm presently teaching a course in reading assessment and we had class number five last night and we finally got to dyslexia, and I said to them, "I bet you thought I would never, ever get here. But we've been covering all of these, um, all of these foundational domains that are so critical to, uh, drawing good conclusions from an evaluation."

[00:14:23] Susan Lambert: It's amazing. And the, the copyright date, I think, is it 2007? Does that sound right?

[00:14:27] Melissa Farrall: No, I think...

[00:14:28] Susan Lambert: No, 2012.

[00:14:29] Melissa Farrall: 2012.

[00:14:30] Susan Lambert: Yeah. And still quite relevant.

[00:14:33] Melissa Farrall: Yeah. I, I hear tell every once in a while that's a little bit dated, but I, I think that there are many things that are sort of tried-and-true about the textbook. The, the faces may have changed, but the names are still the same. Kind of thing, in that the Science of Reading is well established with respect to, um, is settled with respect to phonological awareness, decoding fluency, all of those good things; work in statistics, that knowledge that we need to interpret, test scoring systems, for example, that is all settled. That hasn't changed. What has changed is that we do have some new additions of tests. So we now have, for example, a Woodcock-Johnson in its fifth edition. We have something called a Wechsler Individual Achievement Test in its fourth edition.

And we probably, and I will admit there's some things we now have a greater understanding of, so I do think we have a greater understanding and appreciation for how it is the reading brain develops, and what this whole orthographic processing thing is about.

[00:15:33] Susan Lambert: Yeah, yeah, yeah. We had a little conversation with Katie Pace Miles who just wrote a book on orthographic mapping, and that was a great conversation to better understand that process.

So, you know, our season is all about comprehension. And our listeners have been taken through a series of understandings of, you know, comprehension is pretty big. And when we talk about comprehension, we have to talk about things like word recognition and orthographic mapping, because those are the gateway into table stakes, if you will, for that comprehension process.

And we've been asking folks to provide their definition of comprehension. And so I would love it if you could provide your definition of comprehension and maybe include what educators need to know about comprehension.

[00:16:23] Melissa Farrall: So my, my view of reading comprehension is that it is thinking guided by print.

[00:16:31] Susan Lambert: Oh, I love that.

[00:16:32] Melissa Farrall: And what is it that I mean by thinking. So when we think about, for example, the, um, the Simple View of Reading, which talks about reading comprehension is the product of skills in two domains, one of which is decoding and the other is this thing called linguistic comprehension. You know, and, and you might think of, uh, a synonym for that might be listening comprehension.

[00:16:56] Susan Lambert: Mm-hmm.

[00:16:56] Melissa Farrall: And so we have come to this understanding that the skills that we have to process oral language are the same skills that we use to make meaning when we take language off the page of print. But I think about it in a much broader sense. So listening comprehension, linguistic comprehension to me, makes us think about those verbal skills. It makes us think about our command of vocabulary, our command of syntax, our ability to think inferentially, you know, the, the entire sort of structure of language. I think about that as augmented and enhanced by our other cognitive abilities, which include, for example, spatial thinking, fluid reasoning, the ability to detect patterns and solve novel problems like, pull that exact tool out of your cognitive toolbox to accomplish a given task.

I think about it with respect to our working memory, which is that cognitive workplace where it all happens like the, that's the room where it happens. We take in new knowledge, we combine and contrast it with what we already know, and hopefully that we do that with organization and structure so that we can access it permanently. And that organization and structure thing that comes to us from our spatial system.

So, yes, we have a lot of language skills, verbal knowledge, verbal reasoning skills, all those good things. But it's our spatial system that helps us to organize all that knowledge and to prioritize it. So when I think about reading comprehension, I think about it, like in terms of our cognition and all its glory: our verbal thinking skills, our spatial thinking skills, our memory, our ability to execute tasks with ease and proficiency. All those different things go into making meaning from text.

[00:18:54] Susan Lambert: Wow. That's a lot.

[00:18:57] Melissa Farrall: It's a lot, and it's helpful, and one of the reasons I included intellectual assessment in my book is that, not that I expect to turn people into evaluators of cognitive functioning, that really wasn't my goal, but I wanted people to have an understanding of what the different views of intelligence are, what those domains of intelligence are, so that when they are looking at these scores, they would have a better grasp of an individual's profile as a reader and the bigger picture as opposed to just zeroing in on just those language skills.

[00:19:36] Susan Lambert: Yeah. Yeah. It, it's a great reminder because, well, going back to, um, me as a grade three teacher, and thinking about a simple question like, did my students comprehend the text that I just asked them to read? That question is really sort of shallow compared to how, you know, you just described this whole comprehension process. It feels shallow.

[00:20:09] Melissa Farrall: Well, I think that we all, and one of the interesting things about being in this field is that no two people share the same definition of what reading comprehension is.

[00:20:18] Susan Lambert: Yeah.

[00:20:18] Melissa Farrall: And because we don't necessarily agree as to what reading comprehension is, there is certainly a lot of diversity in the way that we decide we want to assess it.

[00:20:28] Susan Lambert: That's a great connection. Yes.

[00:20:29] Melissa Farrall: So there are lots of different ways of assessing reading comprehension. Sometimes we ask students to respond to our questions after a single read, where they read the passage and we take it away and then then have to answer the questions.

Some folks feel that's kind of unfair, that there's too much memory involved and that we're assessing more memory than actual comprehension.

[00:20:51] Susan Lambert: Mm.

[00:20:52] Melissa Farrall: Some tests permit students to reread the passage so that they can read the passage the first time, look at the questions, and go back and read strategically the second time.

Is that really kind of what we want to know? There's an interesting thing here. Do we want to know about a student's ability to learn when reading? Is that truly reading comprehension? Meaning that the texts we give them are somewhat maybe exotic in their nature, unfamiliar content, so we're really asking them to grapple with new knowledge. Or should we be looking at comprehension with respect to their background knowledge and give texts that are more or less familiar? Is that what reading comprehension is? Our ability to understand, given that we come to it with all this background knowledge. And I'm not sure there's any one like right answer.

[00:21:47] Susan Lambert: Yeah.

[00:21:47] Melissa Farrall: There's no one point of view here. A lot of the way in which we would assess reading comprehension would ultimately depend on those referral questions and what it is people want to know.

[00:21:57] Susan Lambert: Yeah, that's a really good point. We've had so many guests on the, on the podcast and so many folks that I've talked to, like Hugh Catts was, "Well, reading comprehension depends on your purpose for what you're reading the text for," or Dan Willingham has said, "Well, some of these high-stakes assessments are really more about an assessment of what you already know or your background knowledge."

So, I love how you said that because it's a really good reminder that we have to connect our purpose with discovery, with the assessment that we're choosing to employ. I think I said that right. I don't know. Maybe that's more confusing than what the way that you said it.

[00:22:37] Melissa Farrall: No, I think, I think that was good. I think we ultimately really have to ask ourselves what it is that we want to learn from this assessment. And, you know, maybe we do have concerns about the role of memory in reading comprehension. Um. Maybe we, we are also really interested in that question of strategic reading and can, can students read for a specific information?

I think there's also, one of the things that's sort of challenging that we talk about... We talk about the assessment of reading comprehension. We see, for example, the scores on the national assessment of educational progress, right?

[00:23:14] Susan Lambert: Yep.

[00:23:14] Melissa Farrall: And they have a test that purports to measure the comprehension of narrative text and expository text.

And when we see those low scores, they're always articles in the newspaper bemoaning the state of reading comprehension. And there's certainly, certainly reason for that. But the fact is, is that those low scores don't tell us a whole lot. These reading comprehension tests, they may speak to a particular level of comprehension...

[00:23:42] Susan Lambert: mm-hmm.

[00:23:43] Melissa Farrall: But they don't inform us as to the why's. So we may know that a child is struggling with reading comprehension, but that singular test that we give doesn't answer any of our questions as to why that struggle is taking place, unless we are then crafting an evaluation that lets us look in greater depth at both decoding skills and at language skills. And we know that about 90% of all reading challenges are due to difficulty recognizing the words.

So what is it we get when we say, do we get a true measure of reading comprehension with a child who simply can't read the words.

[00:24:28] Susan Lambert: Yeah.

[00:24:29] Melissa Farrall: And I think that's sort of a fascinating question. And for that reason, we need to supplement our testing with measures of oral language, with listening comprehension, so we have a true measure of the potential for reading comprehension.

[00:24:44] Susan Lambert: Hmm. I love how you said that.

[00:24:46] Melissa Farrall: And you know, I will also say that oftentimes we are doing these evaluations because children are struggling and there's sort of a lot of bad news, and in our reports we may, we do our best to talk about both strengths and weaknesses, but the, the reality is, is that we're going to see these low scores.

When we add that measure of listening comprehension into our battery, we now do several things: we get a measure of potential for understanding, and we can say, for example, to parents, "We believe that once we teach your child to decode, that she will be reading at this particular level. And that's something to anticipate. It's something to look forward to. It's a bit of good news because yes, the intellectual power is there. The thinking skills are there, but what we have to do is train your child in phonological awareness and in decoding skill in order to get there."

[00:25:37] Susan Lambert: Yeah, that makes so much sense. Hey listeners, I just want to jump in here to remind you that in our very last episode we did talk about an innovative, new reading comprehension assessment that does get at the wise.

I really encourage you to go back and listen to that episode if you haven't yet. Okay. Back to the conversation with Dr. Ferrell.

I want to go back to your book. It is chapter 12 that you get to this issue of comprehension. And when I was rereading that chapter, I think you said, "Yes. We finally gotten to comprehension and maybe you're wondering why." But during, in the course of you know, that chapter, you provide a brilliant explanation of the construction-integration model, Kintsch's model, construction-integration.

I'd love for you to talk us through that a little bit, because I think it's something that we haven't talked to our listeners about a lot, but is actually super instructive as we're starting to unpack this idea of comprehension. So can you describe that model a little bit and why, why you think it's important?

[00:26:47] Melissa Farrall: I will do my best. So Kintsch speaks to the process by which we make meaning. By which our thinking is guided by print, right?

[00:26:55] Susan Lambert: Yeah. And it's a process model, right? So we're thinking about the process.

[00:27:00] Melissa Farrall: We are thinking about the process, and I think that his model helped. I look back over the years that the things I've learned, and I think that his model was sort of instrumental in helping to clarify for me this reading comprehension thing.

[00:27:14] Susan Lambert: Okay.

[00:27:14] Melissa Farrall: You know, that's one of the reasons that, that it's in the book and he talked about there being an author who creates a text that is the, the actual message sort of in print. And there's a lot of terminology associated in this particular model. He refers to it as the text base.

[00:27:33] Susan Lambert: Yeah.

[00:27:33] Melissa Farrall: And that text base consists of different layers, and that's kind of how I think about them. It consists of a microstructure and something called a macrostructure. And the microstructure is the language of the text. It's the vocabulary. It's the different kinds of sentences. It's the linguistic foundation that we have to have in order to then move into making meaning.

The macrostructure is the story grammar. It's the structure of the text, the fact that the text has a beginning, a middle, and an end. That it has a resolution at a certain part in the plot development. It has a setting, it has characters, all those different pieces. And the reader's job is to work with that text, is to confront that text and bring to it their own background knowledge to manage both the demands of the, the language of the text and its content, as well as the structure of the text, resulting in a highly individualized understanding so that, because my background knowledge may be very different from yours, we may arrive at very, very different conclusions regarding that author's message.

[00:28:46] Susan Lambert: Hmm. And can I ask a follow-up question on the microstructures of the text?

So we were just talking about if students' word recognition abilities were impacting their comprehension, the word level would be one of the microlevel sort of skills you need to have to actually engage with the text. Is that right?

[00:29:12] Melissa Farrall: Yes. So that you, you really couldn't meet the demands of either of these levels, the microstructure or the macrostructure if you weren't able to read the words.

[00:29:21] Susan Lambert: Okay, so, so the model assumes that you can read the words on the page first?

[00:29:25] Melissa Farrall: Yes, it does.

[00:29:26] Susan Lambert: Okay. Yep. And I think you just said in our previous conversation that there is some indication that 90% of the kids that struggle with comprehension, struggle with that word recognition piece of it. Did I get that one right?

[00:29:43] Melissa Farrall: That's absolutely correct. It's rather amazing, isn't it? Because we think about comprehension, reading comprehension, sort of disorders as being widespread. You know, that's what we think about challenges where the reading and, and most of us, and I think the public assumes that we are talking about the thinking part of reading, the making meaning part of reading. But the fact is, is that many of our students are not even reaching that level. They don't have access. They don't have, if there's no ramp onto that super highway of making meaning, it just totally blocked for them because they are in a world of, what does that word say?

And I often get this image in my head of a text where many of the words are just blacked out. So for a child trying to make meaning and they can't read the words, um, they don't have full access to the vocabulary, they can't certainly access sentence structures and pull words into meaningful groupings because they are just experiencing huge gaps in their command of what's written on the page.

[00:30:44] Susan Lambert: Yeah. I wanted to make a fine point on that because, um, I think it's often the assumption that a reading comprehension assessment is going to help us understand next steps instruction for students. But if we recognize that there is a lot, a lot of students that the reading comprehension assessment isn't going to work because they have that gap in, in word recognition skills, we can't even apply this construction-integration process model to understanding what those students are doing because their issue sits outside of the model.

[00:31:21] Melissa Farrall: Yes, that's very true. But once again, if we supplement our evaluation with measures of listening comprehension, we can then get a sense of an individual's ability to make meaning...

[00:31:34] Susan Lambert: Yep.

[00:31:34] Melissa Farrall: Through a micro structure and a macrostructure in their own individualized way, that's free. That's uncontaminated by print. It's understanding an individual's profile and the, the challenge they have mastering the alphabet and sound symbol correspondence, right? Their lack of fluency and how that helps us then make recommendations for instruction.

[00:32:01] Susan Lambert: All right. I'm going to read a couple quotes from your book, and have you respond to them.

[00:32:07] Melissa Farrall: Okay.

[00:32:07] Susan Lambert: So you wrote, "Reading comprehension is not easily quantified and it can be difficult to separate out the factors that contribute to reading comprehension." Continuing with, you also say, "At the very heart of reading comprehension testing lies a problem."

Why is this so hard?

[00:32:29] Melissa Farrall: Why is it so hard? I, I think because when we think about reading comprehension, we really want to think about the making meaning part. We want to think about, about how it is we make meaning in all our intellectual glory using the full range of our cognitive skills. That's what we want to think about.

But the problem is, is we test reading comprehension in print, and maybe that sounds somewhat, oh, I don't know what the word is, somewhat surprising. But when we are assessing the skills of individuals who struggle with decoding, we just can't get to the thinking part. We are in a whole other domain, and what we really learn about is the need to do, to have a deeper, better understanding of those decoding skills and phonological processing skills as a pathway to then making meaning for print.

[00:33:29] Susan Lambert: Hmm. And I think the idea is, I'm thinking from a teacher's point of view right now. And the teacher's probably like, "Oh man, are you saying that I should be giving listening comprehension assessments, um, if my student is less than what I think is successful on a reading comprehension assessment?"

[00:33:48] Melissa Farrall: I think we should be doing both listening comprehension and decoding. I mean, because we really want to rule out challenges with oral language.

[00:33:56] Susan Lambert: Yeah.

[00:33:56] Melissa Farrall: As well as have that deeper understanding of decoding and, and phonological processing.

[00:34:01] Susan Lambert: Right now I'm thinking again, as a classroom teacher, what are you telling me that I should be doing with some of my curriculum-based reading comprehension measures. So I teach this content, then I give students a passage to read and I have them answer questions about that passage. Is that helpful to me? Should I be doing that in the classroom?

[00:34:25] Melissa Farrall: I, I think there's every reason to do that. I think that we have to make sure, however, that all of our students have access to that rich content. And that means that for our students who are struggling with decoding, that they have access to audio text.

[00:34:40] Susan Lambert: Mm-hmm.

[00:34:41] Melissa Farrall: Right? Because there's no reason why they can't participate in activities designed to promote vocabulary and verbal reasoning skill and background information, but they need to do it through their ears as opposed to trying to get it off the page.

[00:34:54] Susan Lambert: Yeah. I'm also glad that you said that too, because it's such, it's, it's confusion sometimes for teachers. And they think, I've heard teachers say that it feels like the students are cheating, or "I am not providing the rigor that I need to when I'm having my students listen to text as opposed to reading that text."

[00:35:15] Melissa Farrall: Yeah, and it's, it's, it comes back to that understanding of what exactly reading comprehension is and the idea that it's different things to different people. So if your idea of ... I need to think about this for a second.

[00:35:31] Susan Lambert: Go ahead.

[00:35:33] Melissa Farrall: We have to really think about what our goal here is. If our goal is to teach critical thinking skills, then we can do that whether we do it off of a page of text or whether we do it through our ears.

[00:35:49] Susan Lambert: Mm-hmm. Yeah, that's a really great example. Thank you for taking the time, the time to think that one through. I, I love how you said that. I also have another question for you because, again, I've had quite an aha over time about this idea of reading comprehension. First of all, just this idea that you have to give kids access to the print, and if they can't do that through, like you said, through their eyes, then access to the print through their ears. But this idea that reading comprehension isn't a thing that you do. You don't just comprehend a text or not comprehend a text, right? There's different levels or layers or depth of comprehension or depth of understanding. You mentioned the fact that if you have more background knowledge about a topic than I do, you are going to come away with a different kind of meaning-making process than I do.

[00:36:44] Melissa Farrall: Mm-hmm.

[00:36:45] Susan Lambert: Can you respond to that?

[00:36:48] Melissa Farrall: I think it's a very broad question, and let me, and you can redirect me. Feel free to redirect me if you feel like I'm, I'm taking it in a different direction. So certainly we all come to print with different experiences. And you know, one of the reasons we talk about background knowledge being so critical, because this is not just the knowledge, it's the experiences that we have and the language that we develop to describe those experiences and to think about them and to process them. And it's that language that we use, right, to enhance our meaning of text.

So we all come to text with different background knowledge. We all come to text with different amounts of verbal reasoning skill. We differ in our vocabulary. We differ in our ability to categorize words and understand how word meanings, how, how words are similar and how they're different.

We differ in our ability to work with words with multiple meanings and to understand which particular meaning is appropriate at this particular moment. We differ in our commands of sentence structure and our ability to process a variety of different sentences, and that's important because we have like three main types of sentences.

Simple sentences permit us to make a statement effect or ask a question; a compound sentence, now, in terms of logical thinking, now lets us take two thoughts or ideas or facts and link them together somehow. It's a wonderful thing. It's, it's their first efforts at organizing our thoughts, right, in anticipation of organizing our thoughts into a paragraph.

[00:38:29] Susan Lambert: Right.

[00:38:29] Melissa Farrall: And then we get to complex sentences, which allow us to make all sorts of connections between those facts and concepts. And we can think in terms of causality and we can think in terms of, of timing, and we can think in notions of exclusion and, and link things with words like "although," or "even if despite."

So we have some individuals who are like highly skilled in their, in their vocabulary and in their command with sentence structure and their ability to think sort of on, to think both concretely and to think more abstractly. And we have some individuals who are more limited, right, in what they understand.

But wherever they are, we use that as a platform for taking them to that next level. So the cool thing about reading for meaning, once we've taught kids how to decode, is that, that language becomes permanent, it's stationary, they can read it, they can read it again, they can reflect upon it. As opposed to trying to take in all of that content through their ears.

And you know, if we struggle with our memory, if we're a little slow to process something, we may not fully grasp all those speech sounds that are coming our way. And then we are trying to make meaning, um, based upon this kind of fragmented input.

[00:39:52] Susan Lambert: Yeah. That makes a, that makes a lot of sense. And um, you know, it's a good reminder. I think it is so simplistic though, it's a good reminder that there are benefits to being able to read the printed word in terms of cognitive understanding, right, of being able to work with that information.

[00:40:11] Melissa Farrall: Well, absolutely. And I think that when we think in terms of cognitive development, language development in particular, when we look at typically developed students—children—we are really thinking in terms of learning through two modalities. We are thinking about the command of language that we develop by virtue of our experiences and what we take and through our ears. And then we think about what it is we are able to learn from print; content that is lengthier, that's more highly organized, that is more specialized in its content, and we really want individuals to learn through both channels so that they can fully develop their intellectual potential over time.

[00:40:57] Susan Lambert: Yeah, yeah, that's a great way to say it. And, and if, if they're only learning through the speaking and listening channel, words disappear into the ether and they're gone, and you can't go back and review them unless they're recorded. And that's not always possible.

[00:41:13] Melissa Farrall: Mm-hmm. And the content is very different. The vocabulary is different. And I know that we all think that in our homes, that when we have conversations, we are using relatively high level of vocabulary, all these sophisticated words. After all, we're really smart people. Of course we do that.

But for the most part, when we are chit chatting in our homes, we're using a very limited unspecialized vocabulary. And so that the, the children who don't learn to make meaning from print, who can't use reading as a tool for learning, don't have access to all those great words and all that great knowledge and experiences.

[00:41:50] Susan Lambert: Yeah. And just making a connection back to that Kintsch model. The whole idea is that through the reading process and these micro and macrostructures, we're coming away from a text, no matter how long that text is, we're coming away from that text with a mental model or a thing to hold onto, right?

[00:42:13] Melissa Farrall: Yes, we are. And we can only do that once again if we're able to read those words with ease and automaticity.

[00:42:23] Susan Lambert: Yeah. Yeah. For sure.

So you were talking a little bit about background knowledge, whether that's content background knowledge or other kinds of background knowledge that students bring, which plays a role in comprehension.

[00:42:36] Melissa Farrall: Mm-hmm.

[00:42:37] Susan Lambert: Is there ever a way that we can minimize or control for that to see like, are they making inferences? Are they understanding cause and effect? Like is that even possible?

[00:42:50] Melissa Farrall: And so once again, we can only go there presuming that our students are able to read the words, right? Because now we're talking really about that thinking part of making meaning.

Um, I think, as I noted before, some test publishers will use content that's really exotic so that the content is unfamiliar to students. And then we can really look at their ability to learn from particular content and to make those inferences to draw conclusions. We always hope that test authors design their questions with thought and consideration to different types of text, narrative and expository, as well as those questions that are literal in nature and more abstract. So we can't, yes, we can get at that information. And once again, we can always supplement that with oral language testing, which we might be doing, or we might bring our friendly speech and language pathologist in. We should bring them in more often, right, to help us with understanding of those, um, language demands and those critical thinking abilities.

[00:43:59] Susan Lambert: I wonder with all of that, and we've sort of hit on some of this, how might educators now think differently than about comprehension instruction, which I'm putting in air quotes, and then comprehension assessment, given it's such a big construct here?

[00:44:17] Melissa Farrall: Um, how might they think differently? I think first of all, we want educators to understand that performance on our reading comprehension test, in and of itself, that reading comp, what we call reading comprehension subtest, right, can give us a level, but it's not explaining to us the whys and why is it this particular student is struggling?

We need educators to understand that the majority of difficulties with reading comprehension are actually due to an inability to read those words. Right? So that the performance you get on a reading comprehension test, the failure to answer a particular question, sometimes it's due to the fact that, "Well, maybe we're not good at drawing conclusions," But sometimes the failure to answer is, "I couldn't read all the words," and the problem is on the surface. It all looks the same.

[00:45:09] Susan Lambert: Yeah.

[00:45:10] Melissa Farrall: You know, so we really need to think about not just assessing reading comprehension, but we need to think about assessing both those domains, that decoding domain and that linguistic comprehension domain, because then we can isolate the individual strands that make reading comprehension possible, and then we know how to design our instruction.

[00:45:31] Susan Lambert: Yeah. That's a good summary of all the things we sort of talked about, um, and, and, and put a little more concisely, but I think this is just a really important conversation. Again, because comprehension is a complex process, that reading comprehension requires a level of automaticity at the word level, and you can't access it without that. And, one reading comprehension assessment cannot assess all of the things that you need to know in terms of how you're processing language. Does that seem right?

[00:46:12] Melissa Farrall: That's, that's very true. There is no one test that does it all, particularly those tests that are identified as being measures of reading comprehension.

And the other thing I guess I would add here is that, despite our best efforts, there's no such thing as a perfect test. Right? Some test measures, some things really well, other tests don't, and we really have to have a good understanding, both as educators and evaluators, of the instruments that we use, because that will be really helpful in permitting us to draw well-founded conclusions regarding our students.

[00:46:53] Susan Lambert: Hmm, that's great advice. As we wrap up and close, is there any closing thoughts that you might have?

[00:47:03] Melissa Farrall: Closing thoughts. Um. Reading comprehension is complex and I think that when we conceptualize doing an evaluation, we want to be sure that we, as the professionals involved in that evaluation, have the sufficient background knowledge in order to make good decisions about what domains of thinking, all the different skills that permit automatic word recognition, so that our evaluation is comprehensive, so that we've ruled out things that are not concerns, so that we are left looking at areas that we need to work to improve. We have sort of, in the evaluation business, we have many people who are highly skilled at administering tests.

[00:47:54] Susan Lambert: Mm.

[00:47:55] Melissa Farrall: They're not all trained in reading. In a perfect world, we would have, not just evaluators, but we would have educators who are trained both in the Science of Reading and also that training in assessment so that we can all sit at the same table and participate as we review evaluation results. I don't know if that came out very well. You can tell me if that was okay.

[00:48:22] Susan Lambert: It was perfect, and it brought us full circle to why you wrote this book in the first place and what some of your concerns and frustrations were. So again, for our listeners, we'll link in the show notes to this book. It's fabulous. It's fabulous. Um, this conversation was so insightful and um, I appreciate your time and thank you for the work that you're doing.

[00:48:47] Melissa Farrall: Oh, you are very welcome. And thank you. It was just an honor to be here. Thank you so much.

[00:48:53] Susan Lambert: That was Dr. Melissa Farrell. Check out the show notes for a link to her outstanding book, Reading Assessment: Linking Language Literacy, and Cognition. For even more on comprehension assessment, make sure to go back and listen to our conversation with Dr. Gina Biancarosa. Gina describes an innovative approach to reading comprehension assessment that addresses some of the challenges we talked about on today's episode. Next up on our comprehension season, we're diving into syntax and why it's such an important facet of comprehension.

[00:49:25] Julie Van Dyke: We need to move the needle on the nation's report card, right? We still have two-thirds of students who are unable to read what we call proficient.

[00:49:34] Susan Lambert: Mm-hmm.

[00:49:34] Julie Van Dyke: And I think the only way that we can really get that number to be closer to the 95% of what we really want to see, is to do something new and syntax is the new thing. That is the part that we need to really develop.

[00:49:47] Susan Lambert: That's next time. Two quick reminders. You can access a collection of free comprehension resources by visiting at.amplify.com/comprehension101 and submit your questions on comprehension at amplify.com/SORmailbag. We'll be tackling your questions on an upcoming episode. Meanwhile, check out the latest edition of Beyond My Years for a conversation about finding your unique teaching style.

[00:50:20] Neysa Olivares-Torres: I think it's important first for any teachers, especially new teachers, to just really understand, like, what are their strengths first?

[00:50:30] Susan Lambert: That's available now in the Beyond My Years feed. We'll have a link in the show notes. Science of Reading: The Podcast is brought to you by Amplify. I'm Susan Lambert. Thank you so much for listening.