Beyond the abstract: a guide to understanding research papers with Dr Steve Kamper

In this episode, Steve uncovers some helpful tips on how to best interpret research including why it is important and how to be more nuanced in your appraisal. Steve covers how to find the real question the paper is asking and also how to understand the impact or effect size and why that matters more than black and white conclusions.

Dr Steve Kamper is Professor of Allied Health at the School of Health Sciences and Nepean Blue Mountains Local health District. He has been continuously supported by NHMRC fellowships since 2008, and is currently a NHMRC Emerging Leader Fellow.

If you like the podcast, it would mean the world if you're happy to leave us a rating or a review. It really helps!

Our host is Michael Rizk (@thatphysioguy)

👏 Become a better physiotherapist with online education from world-leading experts:

https://www.physio-network.com/

Transcript

SPEAKER_02 0:00

Reading articles, you know, it's almost a learning experience. The meta side of it, it's about understanding bias. It's about understanding what makes information more reliable and valid as opposed to not. And as I said, those same concerns apply to wherever you get information. Same concerns apply to reading the newspaper or

SPEAKER_01 0:27

Interpreting research is something we all try to do, but it can be tricky. It can be overwhelming at times. Today, we had Dr. Steve Kamper. Now, he is from the University of Sydney and he has been published on hundreds of research articles and he spends a lot of his time in the research space. He helped us break down when we're looking at a paper, what question is this paper actually asking us? And he also gave us some mental frameworks of questions we can ask ourselves before proceeding to interpret a paper. He then went into my favorite part of the episode, which was the effect relationship and the size of the impact that matters rather than looking at a paper and saying black and white or yes and no. How to interpret that was really important. This was a really practical episode and will take your interpretation of the literature to another level. Please enjoy it. My name is Michael Risk and this is Physio Explained. Welcome, Steve, and thank you for joining us. Thanks, Michael. Good to be here. We're talking about research, how to interpret research, how clinicians can do this better. From your perspective, why should we look at that? And what you'd said to me is, why would we bother doing that?

SPEAKER_02 1:38

Yeah. So, I think this is the first question because I think if clinicians want to use research as part of their practice and decision-making process, it's a conscious decision because you can practice without ever reading an article. You can do your job and whatever else. So, that's the first thing. It's making that decision that I want to use research and integrate it into my clinical decision making. And I think there's a couple of good reasons why you would do that. Firstly, it's a source of information which is clearly relevant to practice. You can have arguments and disagreements about how relevant and in what way and whatever else. But there's this massive body of evidence which speaks to the patients that you're seeing And so from my point of view, that's a no-brainer. Of course you want to use that information, right? So the other is there is some evidence to suggest that patients get better outcomes. And so that's fundamentally what the business of clinicians are in. A couple of other ones are the public expects it. So the people who come and see you as a practitioner, they expect that the decisions you make are based on scientific understanding of anatomy and physiology and treatment effects and all that sort of stuff. So there's an expectation that you're working there for the patients. A really quick story about that. Some years ago, I was teaching some evidence-based practice stuff in a hospital up the coast here in New South Wales. And on the way through, I stayed in with a mate of mine who I've known for a very long time through school, and he's a plumber. And we were having a beer, and he was saying, what are you doing? I said, I'm teaching to the allied health and the nurses and stuff like that about how to use research evidence to make their clinical decisions. And he said, as opposed to what? And I thought that was a really interesting perspective. So this guy doesn't have a background in health at all. It's a bummer. But he's asking the question, well, if it's not based on science and research evidence, what is it based on? And I think that's reflective of where patients are probably coming to clinicians. The final reason is, is your professional associations expected, right? So I have a background as a physio. I didn't treat for very long, but that's my background. And I think it's the same for just about any other profession that you work with. Those professions are saying we are a profession whose practice is based on scientific evidence. And so there's an inherent understanding as a health professional from your profession that that's what you're doing. They're the reasons that come to mind for me. There's some other ones as well, and some of them speak to funding and stuff like that as well. But for me, they're the really core reasons why I think it's important, why I think it's valuable. But that personal decision actually really matters. You have to be motivated to do it because

SPEAKER_01 4:15

it takes time. I love that conversation with the plumber, as opposed to what is almost like, well, what are you doing if you're not? A personal question I had for you, Steve, off of the list we spoke about was, I have a little bit of a concern because I see a lot of new grads in my clinic in Sydney and I talk to them a lot on the Facebook group. And the feeling I'm getting is that they're actually not reading journal articles. There's obviously some wonderful products like the Physio Network Research Review, which is good. It's a shortcut. But there's a lot of Instagram learning. There's a lot of social media learning, which is good and is valuable. But my feeling is, I don't know if I'm becoming a dinosaur, but my feeling is we'll never get that real in-depth knowledge. of how to apply the research if we're not actually reading the journals. And that was a bit of a concern coming up for me. Are you seeing that on ground? And maybe do you have one or two strategies that might help someone who isn't reading a paper or do we?

SPEAKER_02 5:09

Yeah, I mean, I think the first observation that I would have is this is not something which is confined to new grads. So I don't think an older generation on average is doing this to any greater extent than new grads are. So I just don't think it's... part of what most clinicians see as their day-to-day job. That's the first thing. As to strategy, you have to decide it's important and no one else can make that decision for you. So that's a personal thing. You have to decide it's important for me to know this. The second thing is researchers have done a really, really poor job of making their work applicable and accessible to clinicians. Research articles are essentially researchers writing for other researchers. There's a whole history as to why that happens and it's ridiculous, but it's the way it is. And hopefully that will shift over time and stuff on social media, stuff on Instagram, stuff on TikTok, physio network, all that sort of thing. These are attempts to make things more accessible and they're great. The challenge is maintaining the depth of information and the quality information to enable people who are using them to make sensible interpretation of it. I am hopeful people will get better at doing that, at sorting through from the stuff which is useful and higher quality. I think it is important to read articles and I think it's important because I think that the job of appraising the article, the fundamental part of that is the same as appraising an Instagram post or a tweet or something on TikTok. Yeah. But The job is the same, and it's about understanding sources of bias. And actually, research studies are set up in a way that if done well, they should do that. And so reading articles, it's almost a learning experience. The meta side of it, it's about understanding bias. It's about understanding what makes information more reliable and valid as opposed to not. And as I said, those same concerns apply to wherever you get information. Same concerns apply to reading the newspaper or clicking on an article through Facebook or Google or whatever else.

SPEAKER_00 7:18

Are you struggling to keep up to date with new research? Let our research reviews do the hard work for you. Our team of experts summarise the latest and most clinically relevant research for instant application in the clinic so you can save time and effort keeping up to date. Click the link in the show notes to try Physio Network's research reviews for free today. Is

SPEAKER_01 7:42

that a reflection that you share that in general, I like what you said, it's not a new grad thing. Do you think in general across the board there is less reading? And I'm mindful I'm in a very private practice bubble and you're in a research bubble, so maybe we see completely different things. I don't know

SPEAKER_02 7:59

the answer to that. To be honest, I mean, I suspect there's more engagement with what would you call it, social media side of things, stuff which is highly shareable or whatever else. There's probably more engagement with that than there was in the past. So at some level, that's a good thing. Yeah. I don't think there's ever been a time where lots and lots of clinicians have been spending lots and lots of time reading research articles. So it probably not changed all that much. Yeah. You know, the question is for people like me and for other researchers is how do we use the greater accessibility or the greater distributability, dissemination, capacity that social media and different platforms offer to get good quality information out there. So that's one thing. As I said, regardless, people have still got to learn good ways of filtering. And I'd argue that understanding something about research is a really nice, it's a sort of gateway into doing that. You can understand something about bias and research methods, It will help you understand those Instagram, you know, to appraise those Instagram posts and whatever, and see what's missing and what information do you need further to make that judgment as to whether to buy or not. Same with PD courses, you know, going whatever course you go on the weekend, it's exactly the same. You still have to make an assessment of whether you're being fed a bunch of horse poo or something which is useful and reliable. You've still got to make that judgment.

SPEAKER_01 9:18

Let's get into that. So we want to talk a bit more specifically about research and papers. And the question you had is, how do we understand what the research question is? So, how might we go about doing that? Yeah, look, this

SPEAKER_02 9:30

is fundamental to the point where if you pick up a paper and you read the introduction or you read a bit of the introduction, you read the methods. If you can't get the question and kind of hold it in your hands, if you can't do that, don't read the paper. And I'm 100% sure that's reasonable because if you can't do that, if you don't have a really, really strong understanding of what that question is, there's no way you can interpret the data in any sensible way. And so that's completely separate to the quality of the methods, to the risk of bias, to anything else. None of that matters until you've got the research question in your hand. They don't have to worry about whether the analysis is any good or you can interpret the tables or whatever, what the findings are, until you've got the question in your hand. And I would say the question has to be so specific that you can imagine in your head, I can kind of think of what data would need to answer this question. Yeah. So you should be able to do it at an intuitive level.

SPEAKER_01 10:32

Yeah.

SPEAKER_02 10:33

The other thing is you should be able to explain what that question is to someone who's not a professional. So to your... to your partner or your son or daughter or your mom or your cousin or your friend or whatever who's not in the business. Because if you can't do that, you probably don't understand what the question is. And look, that may be your lack of understanding of what the researcher has done. Just as likely, it's because it's not clear in the paper. So this actually offers a way to filter studies. You can get a hold of a study. If you can't understand the question with that degree of clarity, Don't read it. Move on. There's plenty of stuff out there. That's the first filter for me and it's really fundamental. When people think about not understanding research, they don't realize that that's way more important than anything else. People get worried about, I'm not sure what a p-value is and I don't understand analysis and I don't know the difference between a cohort study and a crossover or whatever else. None of that matters as much as do you understand what the question is. The way that I think about research questions is, They're always one of three types and only one of three types. They're either description, so they're describing what's going on, prediction, we've got some information now, can we work out what's going to happen in the future, or causation. So does A cause B? And they're of two types. One is there a factor or a variable which causes an outcome which is important, or they are about is one treatment cause and effect, to a different degree than another treatment. So again, with the question, you should be able to go, is it descriptive? Is it predictive? Is it causal? And again, if you can't do that, you don't understand the question. You're not going to be able to interpret the results, even if they're high

SPEAKER_01 12:24

quality. You're making very complex ideas quite simple. I'm building a scaffold in my mind. It's a wonderful mental exercise to do those two things you just suggested before you read a paper. I've never thought of it that way. So that's already really helpful. Yeah, good. Yeah, thank you. I wanted to touch on this one because it's interesting. It's the size that matters. How do we interpret the effect of the relationship?

SPEAKER_02 12:48

Researchers forever have got really stuck on p-values. And this is something that's come through to clinicians as well. What's the p-value? Is it less than 0.5, more than 0.5? There's a huge debate amongst statisticians at the moment in the sort of field that we're talking about. How much does this matter? You know, is this... important and some people fall on different sides of the fence. The side of the fence that I fall on, I'm a long way from a statistician and I don't understand the complexities of a lot of the debate that's going on. But this is an attempt to say something is or it isn't or something works or it doesn't. Actually, that doesn't matter. So that might be a difficult thing to get people to get their head around or understand. But I'll say to you, it doesn't matter where the treatment A, gets a different effect to treatment B. That's what a P-value tries to get you. There's problems with it, but that's what it tries to tell you. What matters is how different. Let's say we're talking about treatment effects. Okay, we've got treatment A, we've got treatment B. The clinician's got a patient standing in front of them. They're going to make a decision, do I give them treatment A or treatment B? Let's say it's very simple. We've got a study, looks at treatment A versus treatment B. Let's, for now, put aside all the quality sort of stuff. The question's very clear. It's this treatment A versus treatment B. As a clinician, what you do in that situation is you have a discussion with your patient and say, here are the two options. Treatment A, it involves this, this, this, this, and this, this many sessions, this much time, this is what you've got to do yourself, this is how much it costs, this is the risk of things going bad. Here's treatment B, this is what you've got to do, this is what it costs, this is how much time, this is the risk, blah, blah, blah. The only thing that matters is not whether they're different, it's how big a difference. Because if one of those things costs heaps more, if there's a lot more risk of adverse events, if it's a lot more time and effort, then probably you want a big difference before choosing treatment B, right? And vice versa. If they're more or less the same in terms of all those risks and costs and whatever else, then you'd be willing to accept it just being a little bit better. That's what matters. We call that the estimate of the effect. So how big a difference it is. So that might be one point on a pain scale. So one point on a pain scale, You might be going, okay, I'm willing to take that on if the alternative is more or less the same as the baseline. But if the alternative is surgery with a risk of something awful going on, you might go, you know what, I want more than one point difference. I want two points, three points, four points. So that's just one of the ways that size matters more than this sort of nebulous idea of is there a difference or not.

SPEAKER_01 15:28

I really like that because I feel like I'm seeing a lot of that doesn't work or that does work. And I feel like you're really applying the nuance. And when we're talking about to our patient, there's so much area there that's in between yes or no or does or doesn't work. So thank you, Steve. We've already run out of time, but you've provided some wonderful frameworks. And I feel like I'm more knowledgeable than 15 minutes ago. So thanks, mate. Excellent. My pleasure.

UNKNOWN 15:59

you