TD&T117 Is this a good research project?

This week I'm critiquing some published work to give an insight into how anyone can look at research to hold it to account.
As teachers and school leaders are begin encouraged to be research informed I think we also need to be research literate, this involves being able to evaluate whether a piece of research is 'good'. In this episode I use four criteria to evaluate Pearson's published research to answer the question: is this a good research project?

The criteria used are:

1. What is the research question or aim?
2. What is the methodology and method?
3. Who are the participants?
4. Does the evidence justify the claims?

Episode transcript

Mentioned in this episode
Pearson unveils vision for future-proofed design education curriculum
The Future of Design Education
The Value of D&T on Zenodo
Redesigning D&T data on Zenodo

Ciaran Ellis posted a thought-provoking question on LinkedIn recently: Do design decisions involve value judgements?

What do you think? Join the conversation over on LinkedIn and let us know what you think. 

Transcript

0:56

So this week's podcast is a reflection and a critique of some work that's been put out by Pearson's talking about a new proposal for future proofing, design education. I thought it was the right time to do something about this. Because in the last week or so, myself, Matt McLean and Sarah Davis have had an article out critiquing whether private organisations should be leading on curriculum development. So that's, that's one angle. But in this podcast, I wanted to take a different angle looking at this research, which is one of the things that had been niggling away at the background for me around this research, since it was kind of put out for call for people to participate last year. And and what that is, it's about the quality of the research. So my question that I'm hoping to answer and explore in this podcast is, is this an example of a good research project. So here, I'm not so much critiquing, who's doing the research, there's a little bit of that in terms of its Pearson's, and it's an organisation, but I'm using it as a kind of a case study, to get people to think about when they look at a piece of research some of the questions that they might ask, and I'm going to come at this very much from the perspective of being a university tutor, I supervise masters and doctoral students, I review, colleagues, research as well before it's published and give anonymized feedback. So this isn't anonymized, because you can, you know, it's me that's talking about it. But I am a great proponent of people doing research and research that contributes to the conversations about how to teach what to teach, who should teach, the pedagogy, the curriculum, the assessment around design and technology education, I really do strongly believe that we need to have a growing community of people who are engaged in looking at research and using research as well as creating research. So this podcast is done in the context of that, but equally about a piece of research that is professionally causing me some disquiet. So I wanted to share that in an open forum. And I'm welcome for people to come back at me, and critique what I say Why else would I put a podcast out or any of my writing, or any of my thinking? So that's what I'm coming at it from this, I'm sure there's a couple of people listening, who might well have been students of mine, so they know exactly how I would approach critiquing their work. So I'm gonna, I'm gonna do that in a similar way. But obviously, in a way that I'm trying to represent what I think people need to ask about research when they read it, and also asked about research when they are designing it. And I'm going to use the Pearson's work as an example. So because that's where I'm sitting at the moment is I'm looking at this work. And I suppose I'm using that this in two ways, one to critique research, and people can think about what makes good research, and one from a perspective that some of the things that have caused me some disquiet about it. So there's a few things that I want to explore in this week's podcast, I want to explore when we set out a research project, what's the aim? Or the research question that we're trying to answer or resolve? Or find a way towards? What's the methodology and the methods that we're using to how we're going to try and answer this question or meet this aim? What's our approach going to be? How we're going to collect data, who we're going to collect data from? So And what's our particular stance within that? Do we come to this with our own perspectives, our own ideas? And is that going to affect the quality of the work that we're doing? And so within that, I want to spend some time thinking about participants who do we ask to get involved in research? Who do we collect data from some of the limitations of the approaches that we can take and which one I see that has been taken within this study. And then finally, I'm going to talk about whether the claims that are on the Pearson's website, so I'm just looking at public information, I'm not looking at anything private, I actually don't have access to anything private. And I'm looking at these claims. And I'm going to critique those and say, whether I think the claims that are being put forward are justified by the evidence that's provided. Okay, so that's something else that I would look at when I'm reviewing research when I'm discussing with my students, you're making this claim, but what's the evidence? Is the evidence strong enough for you to make such a strong claim or a weak label a generalisation and so forth? Okay, so while I'm talking, I'm gonna click through a couple of screens, if you hear me pausing or clicking, it's because I want to make sure that I'm referencing the work that I'm critiquing properly. Okay, so first of all, some I'm starting, I'll put the links in the show notes so people can see. So it's all open and transparent about about what I'm doing and where I'm coming at this from. Okay, So one of the aims of the study seems to be an attempt by Pearson's or a drive by Pearson's to raise the status of design education in schools and reverse the downward trend in student uptake. Okay, so that's the claim. So that's about the claim as the aim of the study that's been articulated on the website, okay. And they, they're going to try and resolve that they claim by with a new future focus design and technology curriculum. Okay, so they're saying, We've got this problem, we've got this decline, and we're going to do that. Okay. Our research shows that we're going to do that, that we can do that through a new curriculum. Okay, that's future focused. Alright, so that's, that's kind of the two parts, I'm going to kind of talk about the bit that happens in the middle between the aim reversing the decline, and increasing the status? And what they're saying is the answer, and we'll look at the research that kind of sits in the middle. Now, if, if this if a student had come to me and said, Allison, I'm going to do some research? Because I want to find out how can we? How can we raise the status of design education in schools, and reverse the downward trend in student uptake? Okay. I'd be asking questions like, well, you need to define what you mean by design education. So I find that quite interesting that on the website, it talks about design, education, not design and technology education, I find, you know, in research studies, we need to be clear about the focus, because design education potentially could include art and design, as well. And given the NSA ad, that's the subject association for art design a part of this and that, that kind of hints at this as well. And given that the designer Technology Association aren't listed in that kind of introductory paragraph that suggests that it's not actually about design and technology so that I would be critiquing one of my students say, there's some confusion here, in terms of this being setting this out as a, as a research study. Okay, so somebody might come back to me say, Oh, this isn't a research study, but but they talk about the research, okay, that they've done, to kind of support some of these things that they're wanting to address, a person's wanting to address, and reverse the downward trend in student uptake. And so I'll be asking for some specifics about what we mean by this downward trend, downward trending, in what given that we've got art and design organisations in this, and that design and technologies are a downward trend, where's the evidence? Now, we know if you're listening as a design and technology person in England, we know that there has been a significant decline in the number of pupils studying design and technology. And we would also say that this downward trend is quite complex, there's been quite a lot written about about how that's happened. So it's a very big ambition to do here, that that on one surface doesn't seem to be very clear about which part of design education, it's about, and it says, in schools, and I will then be saying, So what sorts of schools do you mean? Do you mean primary? Do you mean secondary? Do you mean, pupil referral? Do you mean independent? Or do you mean state really would rather say private than independent? That's a whole other ideology, ideological conversation I could go off on. So there needs to be some clarity to me, in a research project here about an aim like this, when we're talking about the status of design, education, what design education we're talking about in schools, what type of schools primary, secondary, private states, you know, pupil referral, we've got special educational needs, you know, schools that are out there as well that that also follow the national curriculum in, you know, so what are we talking? And reverse the downward trend? What what downward trend specifically, are we talking about it? The implication is, as we read on, that it's around GCSEs. Okay, so that's, that's a very long explanation just about my critique about the the first aim and not say that the ambition is ill founded, but the clarity isn't there. And then moving on to the second part of my kind of critique, I be critiquing the methodology and the methods. So one of the things I ask students and researchers to think about in their methodology is is thinking about their own bias or their own place in this work. So some people argue, for example, that we have to acknowledge where we who we are in, in the, in the study, as the researcher as the data collector is the designer of the study, that we have our own biases and our own beliefs and so, and that may well affect and colour, how we see the study. So for example, when I was doing my study about the value of designer technology, education, I had to first of all kind of Get out there, out of my system, what my own beliefs were about what the value of design and technology was. And people have listened to me talk about this study, know that some of the data I found quite uncomfortable, but I had to think about how I could become more objective about it so that I was presenting as much as I could a true interpretation of other people's values that may well have been in conflict with mine. So that's, that's kind of part of this methodology is thinking, who's collecting the data? And how might who they are affect the collection of that data. So you know, Sarah, Matt and I have written in the last week or so, about this, we've we've suggested that because Pearson's are a business and have shareholders, that their intentions towards reversing the downward trend may well be more related to their shareholders and wanting to increase the number of young people studying GCSE qualifications that are provided by Pearson's. And again, you know, they're answerable to their shareholders completely understandable. But that needs to be articulated and open and transparent. As I said, you know, me during my study about the value of design and technology, well, of course, I think it's valuable, of course, I believe there are particular values to design and technology. But if I, if I come to my study, with those at the forefront of my mind, then it makes it difficult for me to listen to other people's perspectives, and to see other people challenging what I'm doing in my study. So that's one of the methodology. And it's also about thinking about when when we then write questions in a questionnaire or an interview or a survey, how do we collect that data? Do we collect numerical data? Do we collect, you know, written responses, which we then have to interpret? Because sometimes if we have a question that, you know, we, we select an answer on a scale on a Likert scale, then that might restrict what we think. But it makes it easier to analyse. So are people actually, when they respond to a Likert scale as they are telling their truth, or they're telling their truth in a way that you're asking them to do it, because of the way you've set out those five points on that scale, for example. So that's a real skill, as a researcher to writing surveys, and being aware of your own bias and what you believe, to be true in terms of people's responses. So again, going back to my study about the value of Design and Technology Education, I was very much aware that as asking for people's perceptions, I wasn't actually actually asking for evidence that design and technology had those values, or it contributed to those things. I was asking for their perceptions. I was, for me, a my view of how I saw their responses. That was what they said and what they perceived. And I was not there to question whether their answers were true, or whether there was any, any evidence, other people who take a different response, and may well take a more numerical may look at numbers and see those in a much more clinical way than somebody who takes my position about research does. So that needs to be articulated, I believe. And, and to me, it's not it's not very clear in the person's I mean, it's not a research paper. So you know, it's a couple of web pages at the moment. So it's very difficult to dig around underneath that. And then in terms of the methods, so a number of methods appear to be in use, I think there's a survey that was out last year, which I think I, I know, I clicked on the link, and I know I read it, I can't remember that I actually responded or decided that it wasn't for me or I wasn't comfortable as a peer about the questions. I can't remember, I know there was an online survey to complete. And I know that because I must have signed up to that, because I didn't get an email with to have a look at the data. And I had some concerns at that point. Because I think it's always about the way the questions are. So I can't critique the questions because I don't have them in front of me. And I've gotta be careful that I'm not going off on a gut feeling or a misinterpretation from last year. But we've got to be careful about the questions that we ask that may well be accidentally leading. Okay. We don't intend for people to leave. And so for example, if I go back to the Pearson's site, it says calls for change comes from new research amongst 2200 Plus secondary school teachers and leaders show that around half believe modernising the subject will benefit the curriculum. Now, I don't know what question has led to that analysis. I mean, that could have been a straightforward question, as do you believe that modernising the subjects will benefit the curriculum? I don't know. Or it might be select from these different ways that could be used to benefit the curriculum. select as many as you want, or select three or your top two or one or rank them. Set from these five, six Different ways, for example, that could benefit the curriculum. Okay. And again, we're now talking about curriculum, whereas before, in the aims, we're talking about reversing the downward transition assumption, that curriculum is going to reverse that downward trend, where's the evidence that that will, okay, because I might put in there or somebody else might put in there. But some of those ways we might reverse the trend, giving them more or less than time, increasing the number of teachers, reducing class sizes, increasing school budgets. Only allowing pupils that are high achieving, to, to stay study the subject. There's a whole other ways modernising the curriculum is possibly only just one way so. So the way the data analysis is currently presented on the website, doesn't reveal how the data were collected, what what questions were asked, and whether those questions, unintentionally led people to a particular answer. Because it and again, going back to the methodology and the aims, because Pearson's who are an organisation who publish qualifications, you have qualifications that people can buy, then, and that those qualifications are also used then to define design curriculum in schools. They have control over that. So their questions are going to be around that. Whereas if you look at some of the writing by David Spangler, for example, in the debates book about why, why did design the technology fail, the curriculum is only a minor part of all of that. So again, there's there's a potential here for over claiming that this proposal is going to reverse the downward trend. In student uptake, when actually we know it's less than time. It's availability of teachers, we've got poor student recruitment of teachers for a whole variety of reasons. And under under resource because, you know, school budgets are decimated. It, the decline has been happening since 2008. And I can go on. Okay, so, so going back to my critique here of the method. There's an assumption here already, that curriculum is going to be the answer to reverse the downward trend. Okay. When there may be other reasons. But then, when we're looking at the data that's been presented, and trying to work out what the data collection, questions might have been. modernising the subject will benefit the curriculum. Okay, one answer. But then also, there's the answer questions there about or seems to be questions about design and technology addressing global challenges, like climate change? Well, I would argue that if you put that as a question, do you think the design or technology curriculum or a future design technology curriculum should address global challenges like climate climate change? I've you're struggling to find anybody who said, No, of course, I don't think it should. Of course, it shouldn't have anything to do with design and technology and what we do. Whereas we know that sustainability and climate changes, has always been a part of design and technology. And we can talk about that from David Leightons work back in the 1990s, around the different values that we use and explore as part of the curriculum or design the technology. So again, potentially there there was a leading question that that led to that. That kind of leads us to a point of whenever he's going to say, Yeah, of course, that's what we should do, and design and technology. And then the other part is equipping learners with the skills needed in a future workforce. Well, of course, education has got to do that, of course, education has got prepare young people to become part of future workforce. So nobody's going to dispute that. Now, we're not going to say, Well, really, I think what should be happening in the curriculum is that we should be teaching children, things like McRobbie, or how to make Welsh love spoons, because those are inherently important to our culture. Now, I'm not I'm not denying the importance of crafts and heritage, but what I'm saying is, of course, we're going to think about the future and about employment and employability of the many rather than the few. So that's kind of critiquing the method there is there's more that I could explore within that but I'm conscious that I've already been talking for about 20 minutes, okay, and I've even got on to the participants bit because this is where I have the biggest issue is, if what Pearson's are claiming is that a new curriculum is needed to address to reverse the decline of the sub I tend to give it more status, we have to remember that our curriculum in England is for primary and secondary. So to me if you're doing a study like this, where is your primary teachers? At the moment, it says 2200, plus secondary school teachers. And we're looking at the whole of education, I believe, when we're designing a curriculum. So that's where the aims, again, are clear. And we get down further down, you know, we start looking at a future curriculum that inspires children. Well, are we talking about primary school, because if we talk about primary school, we need to consult them, the teachers. And if we talk about primary school, then really we're also talking about early years, because early years feeds into primary. So we need to be getting representative from early years. And at the moment, the evidence is that they're not involved because it just talks about secondary school teachers. Okay, so let's turn it around and say that actually Pearson's are focusing on reversing the downward trend and student uptake of GCSEs in design and technology, or in any design, education related post 14 qualification. All right, so you can hear we're getting more specific that so yeah, of course, you're gonna talk to technically school teachers. But I want to know, which secondary school teachers you're talking about, and are they representative of the population? Now, what I mean by that is, isn't is balanced representative of the whole population of teachers, that's male and female, location, age, ethnicity, length of service, are you can you can you state, and it's very difficult to do this in a research project to get it representative, but you need to be, for me in a study transparent about that now. You know, patients are running this as a business, they're their business sorting out this. So. So putting that data out publicly becomes very difficult for them, because who owns that data that's been collected. And again, I think that then leads us on to ethics. But let me just don't pick this little bit more about about participants. You know, what I can say that what I've seen about the Pearson's approach is, I would say they've kind of used a snowballing approach, which is a highly rated, it's got its limitations, but it's a highly rated recruitment method for participants, which is we start with our own contacts, and then we say to our contacts, and I'd love you to take part in this study, please take part in the study. And then by taking part the study, I'd really like you to share this with other people and get other people to do it. So can like snowballs. So I might go, I filled it in and I might go to my mates, I think you should fill it. If I was in a department, I might say to my department, great if you could fill this in, because I think we need to get more voices in this. And then they might go, Oh, I thought this and so I might go back, they might go back to other schools that they used to work in or colleagues that they've got that teach design and technology and kind of that would snowball. And how, again, you can't guarantee that you get a representative sample of the population, but you should be able to analyse count your participants, demographic data, location, age, length of service, ethnicity, gender, and so on, because then that kind of gives you more of assurance when you're doing your analysis about what you can claim. Okay, so one of the challenges here is that when it's about secondary school teachers, it's very difficult to say that this is about second about teachers, thinking that the design and technology national curriculum needs to change that that's not what's been said in this study. But it does talk about curriculum, there is a lack of clarity about what we mean about curriculum in this study. It's very difficult to make that claim if your participants are, for example, predominantly white men from London, who are secondary school teachers who have less than 15 years of teaching. I'm only saying I've got no evidence, but that's where a transparency of the participants and the recruitment methods becomes good gives an assurance about the validity and confidence about about the study. So I've been saying to my students, you need to think about your participants and how you are recruiting them. And then we'll be looking at the data analysis. So I've kind of intimated this a little bit already. So 2200 Plus teachers, I don't know exactly how many around half believed modernising the subject will benefit the curriculum. Around half is that less than half is that a lot less than half? Is that more than half is that a little bit more than half is that on half now we can know if you're in England or if in Great Britain, that 51% is what took us down, you know, down to the Brexit roots. That means 49% weren't happy about it. And we've also heard about the the issues around data collection and the questions What happened around Brexit? Now I'm not I'm not liking Pearson's study to Brexit, but because there isn't that transparency about what half means or around half, okay? Means it's very difficult. And if it's only around half is that really sufficient to justify putting forward a new a new curriculum. And then the same number it goes on to say, also support both the move to design and technology addressing global challenges and equipping learners with the skills needed in the future workforce. So there's two things, it's claiming that the same number, I don't know what the same number is, let's say it's 50%. Let's say it's 51%. Support, addressing global challenges and equipping learners. But we don't know whether that same number. It's the same number, or it's the same people have said, Yes, modernise. And I also think those same people are saying, yes, include climate change. Yes, prepare young people for future workforces. So that's, that's kind of difficult to kind of work out what what the data analysis is, is claiming because it's kind of quite vague, around half around half same number, in what way we're saying same number. And then it goes on to say, strengthen the role of the subject is also backed by two thirds of Art and Design teachers, specifically, how can I aware of these words, these aren't designed technology teachers come from if a part of the 2200 Is that where the design education comes from, but all of the then post work? That the post work is in built on, which is around? Okay, we now need a new D and T curriculum is around design and technology. So why are we talking about art and design? So how many of those 2200 were actually on design, and it also talks about leaders, but it's a whole other spin off. But I'm hoping that kind of gives you a flavour of how I might look at a study and how I might review a study. And also where I'm coming from in terms of the disquiet and discomfort is too strong around the Pearson proposal. Because I think I think what persons have done by taking this action by leading these this research, as flawed as I think it is, in terms of what's been currently presented. And in terms of the ongoing conversations, I think what is what has been fantastic is it has stimulated discussion about what what does this look like, okay, but we need to be very careful when we're making claims based on data that isn't transparent, isn't shared isn't public, but it's also going to publish behind, it's behind a wall, because we are aware that, and this leads me to my final point, really, that some people have been asked to sign non disclosure agreements. Well, I could have never done that in my research, because I'm gonna come at this from a different angle is around ethics. So there's a call in universities, if you've got funded research or non funded research that you make your data available, your data collection methods, and your your data available, so that other people can actually use your data to do similar analysis, or to do different analysis or to use or to critique. And that's kind of something that we now have to do, we have to do a data management plan. So that means that when we collect data, we have to ask our participants permission for how we store the data, and what we can do with the data. And we have to make them aware, whether we're making it publicly available and whether it's going to be anonymized or not. So, you know, and I've got, I've got an invested interest in talking about proposals for a new curriculum and developing science technologies. I'm working with a group of teachers where we're exploring what a redesigned DNC curriculum might look like. And part of my data management plan is that I am making that data available on Zenodo. And it's the same that I've done with my values in science technology, research, that data is also available on to node it's not publicly available. It is actually people have to request it, but the redesigning DNC I've made publicly available, and I wrote that into the ethics at the beginning. So if you're doing research, you need to think about the ethics of anonymizing and pseudonyms. But you also have to think about the ethics of who has access to your data and whether you're making it public and whether you've got permission to do that. So there's a huge amount going on in just looking at this one study in two web pages to critique and I'm hoping that's given people an insight into two things. One, which is kind of a more minor point is been able to look at the person's proposal and go, well, where's the strength? You know, how do they make those claims? And you might have a greater insight into that than I do. But actually, I think what's more important is that we need to we do need to be engaged in research, creating research, and I'm hoping that what I've talked about today, and is a long podcast, least partly because I really do believe in, you know, I really do believe in this stuff. And I do think the more people we have in design and technology that are engaged in research, using research and creating research and sharing that research, the stronger it makes our community, which is why I feel that I need to critique research, which is why I do peer reviews why I support people, because it's the strength of the research, that is important, not just that we're doing the research, and whether the research is reliable and valid, and the claims that we're making are justified by the data. So I hope that's given you an insight into the way my brain kind of thinks sometimes when I read research, I am currently reading a lot of research about design and technology. So I'm kind of completely immersed in this at the moment, and I'll be talking after Easter in May and June about some of the research that I've been reading. But anyway, as usual, thanks for listening. If you've got ideas, thoughts, views, opinions about what I've said, and please do come back to me, I put my work out for critique. I've recorded this today. It's what I think today. If somebody gives me some different evidence, I might think different tomorrow, but I'll let you know. Thanks for listening