Module 5 Acquire evidence from the scientific literature

This episode accompanies Module 5 of the course, which focuses on gathering evidence from the scientific literature (academic studies) that will help us address the question we are trying to answer. This module aligns to chapter 6 of the Evidence-Based Management book.

Modules 5, 6 and 7 all focus on the scientific literature, so when you listen to their corresponding podcast episodes, the picture will hopefully become more complete. 

In this episode we discuss the process of searching for academic studies; the difference between everyday / managerial terms and academic constructs; how to find the right terms to search for; the importance of obtaining studies from the right sources, most especially peer reviewed academic journals; the difference between journals and “magazines”; and the importance of documenting the process you followed to identify your evidence base.

This episode was updated in 2025 to reflect the use of AI tools such as Consensus and Google Scholar for searching academic databases.

We also explore how the peer review process and the meta-analysis help us get to the best available evidence.

 Host: Karen Plum

Guests:

• Eric Barends, Managing Director, Center for Evidence-Based Management 
• Denise Rousseau, H J Heinz University Professor, Carnegie Mellon University
• Barbara Janssen, Fellow of the Center for Evidence-Based Management

Find out more about the course here:   https://cebma.org/resources-and-tools/course-modules/ 

Transcript

00:00:03 Karen Plum

Hello!. Welcome to the evidence based management podcast. This episode accompanies Module 5 of the course, which is the start of our exploration of the evidence from scientific literature - the studies carried out by academics in our field of interest. The rest of the journey is covered in modules six and seven, and their corresponding podcast episodes. 

Since this episode was first published, the online course has been updated to cover the ways that AI can help in our search for academic research, and so we've updated this episode to reflect those changes. 

So Module 5 looks at how we acquire the evidence from the scientific literature that we're going to use to explore and answer the question that we've identified earlier in the process. The course teaches a thorough method of searching for scientific studies and documenting the process used to find them, but depending on the situation, this isn't always necessary. 

Sometimes a quick search in Google Scholar or an AI search engine like Consensus is all you need, but the course equips students to use either approach. 

I'm Karen Plum, a fellow student of evidence-based management, and in this episode I'm joined by two podcast regulars, Eric Barends, Managing Director of the Center for Evidence-Based Management, and Professor Denise Rousseau from Carnegie Mellon University. My third guest is Barbara Janssen - experienced researcher and fellow of the Center for Evidence-Based Management. Let's get started. 

00:01:57 Karen Plum

Having had some exposure to the search for academic studies, I thought this module might be fairly straightforward. In my day job, I've been responsible for commissioning searches of the academic literature on several subjects of interest to my organization. Eric, Barbara and their colleagues conducted the searches and critically appraised the studies they found, presenting us with the best available evidence. 

But as with anything you don't fully understand, this seemed a straightforward process which delivered interesting, relevant studies on which I could take action and help others understand what we'd found. 

I now fully admit that I didn't pay huge attention to the process that lay behind the results, and having now done this module, I realize there's a lot more to it. 

So let's go back to basics and ask why we're searching for academic literature in the first place. Essentially, we have a problem that we're exploring and we're looking for the best sources of evidence to guide our decision making. 

Now we could do a Google search on the subject, see what comes up and look at what seems relevant or interesting, or what aligns with our assumptions about what's going on in our organization. 

At this point, alarm bells should be ringing in your head. What's going to happen if we do this? When we search for evidence, wherever we search, we don't know whether what we find is any good, if it's reliable, or whether it's trustworthy. And if we're basing an important decision on this evidence, then we need to have confidence in what we find and to be able to explain our methods to the decision makers. Here's Eric. 

00:03:37 Eric Barends

We want to have a number of studies that are on the topic that we're interested in or about the issue we're trying to solve. We want to do that in a way that we don't miss important relevant studies. That is actually what we're doing. You need to take steps in a specific order and that is because it needs to be reproducible and transparent. So when someone would ask me where did you find these studies, where do they come from? And you can say, well, they are published in the past ten years or twenty years and you can find them in these databases if you search in this specific way. 

And you also want to make sure that the way you search, the search terms you use, are indeed appropriate terms, relevant terms, and it's important that you don't miss important studies because you used the incorrect or inappropriate terms. 

00:04:38 Karen Plum

So if we just go to Google and put in some terms that we think are right, we'll get tens, hundreds or even thousands of results. It would be really difficult to identify how many studies there are, or even find the studies themselves in amongst the noise that is Google. 

On the other hand, searching in research databases allows us to be specific about the type of research we're interested in, so we can filter the studies that are the best fit for our circumstances. 

By being systematic and methodical and through documenting how we identified our sources, we're going to be able to show that our process is not only evidence based, but that it's transparent and reliable, as Eric mentioned. 

The purpose is to show that the best available evidence was used, rather than relying on opinions, personal experiences, or gut feelings. Module 5 explains in detail how to step through the process Eric talked about, but we don't always need to go through the whole process for every topic. Perhaps we just want to take a quick look at something. 

The options for doing this have changed in recent years. We now have tools that can interrogate the research databases, as Eric explains. 

00:05:55 Eric Barends

This all changed quite recently, two years ago with the development of AI ChatGPT and stuff like that were the first tools. However, now we got specialized tools specialized in the sense that these are search tools, AI driven search tools that search in these databases or online and look at all the databases at the same time and they are very friendly. You can just ask those tools a question like as we explain in the online course, Consensus AI, but there are many, many more so, but just as an example, Consensus AI, you could ask what are the most important drivers for people to leave the organization and that makes it so much easier because you just ask questions. 

And you get you get papers or actually an answer from the AI that had a look at several papers and based on the findings of these papers, the AI compiles an answer that pulls together all the findings and tells you what is there. It takes you 30 minutes at the most to ask your questions, look at the answers and you can get a pretty good idea of what is out there in terms of empirical papers relevant to your question. 

00:07:29 Karen Plum

Eric refers to this as the quick pragmatic search, in contrast to the rigorous search process that's detailed through Module 5. This is thorough, time consuming and can be hard for students and even academics to master. So when would we opt for this approach as opposed to the quick and pragmatic search?

00:07:49 Eric Barends

The moment it becomes an important decision in an organization, meaning it involves quite some resources, it affects a lot of people, it is important for the organization. Then just relying on Google Scholar and Consensus, just taking a quick and pragmatic approach is not OK. You need to look further. 

You really need to have a more systematic and rigorous search to answer the question because it involves a lot of people, it's more like it's a moral, ethical thing. We are responsible for these people, so we need to make sure we did our best and we really, really looked into this in a rigorous and systematic way. 

00:08:38 Karen Plum

So if there's a lot at stake financially or in terms of resources, or if there's a high level of risk, change or upheaval involved, then we need to do the extra work - to be thorough and rigorous about the evidence and its applicability to our situation. 

One of the things we spend some time on is identifying the right search terms to find the appropriate studies and during the course, the importance of language and clarity keep being reinforced. 

There's always a danger that we assume we know what a particular word or term means. And while many terms might be commonly used in management circles, they aren't necessarily ones that academics use when conducting their research. 

00:09:24 Eric Barends

The biggest challenge is finding the right search terms. The problem here is that academics and practitioners are living in separate worlds, or they live in different countries and therefore they speak a different language. That means that we managers, we practitioners use terms or as we refer to it - constructs - construct is the more scientific term for an idea or a variable or an insight or something you do. 

For instance employee engagement? That's an idea. That's a word we use, a term we use. But we're not completely sure what it means. Same counts for instance, organizational culture. We managers use the term all the time. But from an academic point, you will notice that it's not really clear what it means. 

00:10:32 Karen Plum

So we have to find the right academic search terms, not managerial search terms, so that we can search for the things that academics have researched and written about. We need to know what the construct is. So firstly, Denise explains what a construct is and how it differs from a simple term or phrase. 

00:10:53 Denise Rousseau

Construct in psychology is used to refer to an abstract idea or entity that can be observed. What the construct does is to specify the features that define the construct so that we know when we're observing phenomena, whether what we're looking at conforms to the definition of the construct. 

So a construct that in my own research is the construct of a psychological contract. OK, so it's defined as a system of beliefs regarding an exchange relationship a person has with another - could be an employer, it could be a team, et cetera. 

00:11:37 Karen Plum

Clearly there's more to this than a simple definition, and that maybe explains why some of the academic terms really aren't that guessable by those outside the field. So if we don't know what the academic term is that corresponds to the problem or topic that we're researching, or there's a lack of clarity about what the terms actually mean (the course refers to this as conceptual clarity), then what do we do?

00:12:02 Eric Barends

So this is where AI tools can really be helpful. So you can ask ChatGPT as a large language model what, for instance, are alternative terms for absenteeism? Or this or that. Or you could even ask ChatGPT about conceptual clarity, if there is high conceptual clarity regarding this term and then CHPT will understand what that means and say, well, actually not because also this term is being used or that term is being used. 

Google Scholar and Consensus are very helpful because they don't take your question as the boundary. They look broader. For instance, if you would ask Consensus. 

Does information sharing have a positive impact on team performance? It will show you papers with information sharing, but also the term knowledge sharing so they help you to find the right terms and think about conceptual clarity.

They will come up with alternative terms because they're built this way. They say hey, Eric is searching for information sharing. But actually yesterday we had Karen and she actually searched for knowledge sharing. And those two things are pretty much alike. 

So given the fact that Eric now search for information sharing, we'll also give him some findings from the term knowledge sharing and that is how you learn more about the correct search terms and search terms are maybe the most important ingredient of an efficient search because it's garbage in, garbage out. 

If you search with the wrong search terms, you get the wrong search paper. You say nothing is there and we have this. Often students search for a topic and say I can't find anything in this research database and then we say well, maybe your search term is actually incorrect or to limit it go to Consensus or go to Google Scholar or even ChatGPT and then it turns out there are more relevant alternative terms that you should use and when you put those in the research database, suddenly all the relevant research pops up. 

So even when the criteria tell you, you should do a systematic and rigorous search in real scientific research databases, you always start for that reason with Google Scholar or Consensus to make sure you have the correct search terms. 

00:15:00 Karen Plum

I asked CEBMa Fellow Barbara Janssen, for an example of a topic where the academic term wasn't obvious. She told me about a company that was interested in the impact on workplace performance when people have financial worries. 

00:15:16 Barbara Janssen

We looked into Google Scholar and then financial distress popped up, but equally other terms like financial well-being financial satisfaction, financial wellness and the difficulties sometimes also is that they aren't always, always defined very clearly. Terms are used interchangeably. This makes it kind of difficult. So in the end we noticed that there was a way to measure financial distress, and the mostly used measurement was called the financial distress slash financial well-being scale. 

So that led to some important terms and then we checked them into databases. The number of hits that would come up and in the end we were including more terms. So we included financial distress, financial wellbeing just because it was a difficult topic. There was a lot of research, but the search terms weren't always very distinctive. 

00:16:14 Karen Plum

Certainly I would have struggled to come up with financial distress and clearly it wasn't the only term used, but I think it serves to illustrate the point. 

Before we move on, I wanted to talk a little bit about the challenges of searching directly in the research databases. Given the number of papers that the databases hold, you're required to enter precise search terms and take a step by step approach to identify the most appropriate studies. 

During the online course, we get to practice and use Boolean operators - those keywords and, or, not - that are used to broaden, narrow or combine results when we're searching. 

I made the usual mistakes of mixing up and and or, but both Eric and Barbara admitted that these still regularly trip them up too, so I found that quite reassuring. 

00:17:09 Eric Barends

You will make that mistake again and again and again, and you will all need to be alert that you make that mistake. You need to check, and when something weird happens like you have suddenly zillion number of papers or the other way around, you end up with only three papers, but you, you know, the first search you had 10,000. What happened? Well, you probably used and where you should have used or, or the other way around. 

00:17:38 Karen Plum

So that we get the best available evidence, we're advised to search for peer reviewed studies which are found in peer reviewed journals. 

So it's firstly important to distinguish between these and non peer reviewed sources as Denise explains. 

00:17:55 Denise Rousseau

Most academic journals are peer reviewed. The non peer reviewed journals, we call magazines. Harvard Business Review is a magazine, it is not a journal. And that's an important distinction. But magazines are very important. They illuminate information and invite often very talented authors, but whether or not they provide peer reviewed evidence that has been critiqued for trustworthiness, magazines don't do that. That's not their job. 

00:18:26 Karen Plum

So magazines play an important role for sharing ideas, opinions and experience, providing inspiration and discussion. I've read plenty of great HBR articles. They get me thinking, and I share them with colleagues. But if we have a business critical decision to make, then we need good quality evidence from peer reviewed journals, rather than a single study written up in the Harvard Business Review.

I've also seen academic research written up in an article in HBR and it's not always clear how it was conducted. Even when I tracked down one particular study, there were many questions and gaps. Many of our clients were asking my colleagues about this article, and so I drafted a response, simply highlighting the gaps and the things that they didn't talk about, which caused me to be cautious about the claims that they made. 

To this day, people still quote the research to me as their last word on the topic. So it goes to show that these articles really have staying power. Anyway, I was curious to know more about the peer review process. 

I think it's important to be able to explain why you did or didn't include studies in your evidence base and being able to clearly explain this to others is part of being credible and knowing your subject. 

Academics obviously want to get their studies published so other academics can see their work and to contribute to the body of work on their topic. The authors submit their study to an appropriate peer review academic journal. Oh, and more on journals later. The journal editor will then invite 3 experts in the field to critically review the study. 

And make comments and suggestions. All of this is done anonymously, and I wondered why. 

00:20:06 Denise Rousseau

Oh, I think the anonymity of the peer review process is really important. Number one, the reviewer is a stand in for the field as a whole, not for himself or herself. And so it takes kind of issues of our own identity out of what we’re writing so when I give somebody feedback on a paper, I'm not giving them feedback, as Denise Rousseau researcher on organizational change or psychological contract; I'm coming at it from the perspective of I'm a psychologist with some years of experience, and I'm trying to help this person do better work. 

One of the issues I think with that developmental orientation that has become increasingly important in the peer review process in the last couple of decades has been that you're trying to help the author find the trustworthy part of his or her work and strengthen and show that off. 

That can often mean you ask them to peel away things that are more gratuitous or aren't really strong enough to stand up as a trustworthy claim to help them craft what is effective. 

00:21:14 Karen Plum

I guess if you knew who was commenting, you might be open to authority bias, or you might take what you know about that person and contextualize their feedback. Maybe you start to think, oh that person doesn't like this sort of research or whatever, whereas this way you have less opportunity to find ways to dismiss what they've said. 

I can imagine that sometimes you may get conflicting feedback from different reviewers, and Denise explained that it's then the editor’s role to help the author to navigate that journey. 

Given that Denise has been a journal editor and is also an active peer reviewer of other people's work, I wondered what sort of feedback reviewers typically provide. Denise identified three areas. 

00:21:57 Denise Rousseau

The first is the - how well framed is the question or the problem, and whether or not it comports with what's already known about the problem. Are they missing existing literature? You know, if you kept using the word morale over and over again and morale is not a scientific construct, but job satisfaction is, you know, going to say well, wait a minute, what do you really mean by that and what literatures are you trying to tap and framing of your question. So that the first is the setup and the logic of how the question’s being framed. 

Then there's that broad swath of how did you study your question? What research design? What samples? What? What surveys? What interventions? What background information have you presented it's often much easier to get consensus on the methodological quality of the study than it is on the logic and the grounding in the literature. So that's the second area that you'll get a lot of critique on. 

And I'd say the third area I call this basically it's the marketing of your ideas at the conclusion of the paper, it would be the implications, you know, are you really telling us much about how the theory should change, how the research practice should be different? Our reviewers often very good at developing the connections between the big ideas and the field and what you found here. And what the research implications are, and the practice implications. 

And that has the nice knock on effect of then threading back into the beginning framing of the article when you talk about your contributions. And in my experience, that's one of the areas where reviewers are particularly helpful is having the scholar think in terms of how does my research contribute to the field and framing upfront so that when you get to the red thread at the end, it's clear what impact you've had. 

00:23:45 Karen Plum

As a researcher, I guess it must be good to know that there are other academics out there that are prepared to provide you with support for your work. If you're working alone or in a small team and you're fully immersed in the process, it must be good to get fresh eyes on what you're doing. A bit like expanding the expertise in your team, if only remotely. 

As a support for learning, I think that must be great and by the same token, if you've documented your search process and you're sharing the results with others, there's always a likelihood that someone will mention a study you didn't find or didn't include. I asked Barbara how she reacts when this happens. 

00:24:24 Barbara Janssen

What you want to do with evidence based practice in general is to try to get everyone to make use of different sources of data and to get people to think about it. So the good thing is when someone comes up with an article, it's someone that's interested, had a look at it, thinks this might be interesting. Why didn't you include it?

And I think that's always a discussion that’s always worthwhile to have with someone because it makes you better, it makes someone else better. It makes the process clear about why we are looking in a certain way. In that sense, it's a gift to the process, I think. 

00:24:59 Karen Plum

And that's the thing with evidence based practice, there are always opportunities to learn, to improve and to share. 

Moving on and returning to the peer reviewed journals, all of which have electronic or printed pages to fill, there must be a sense of needing to feed the beast as well as to maintain their standing as respected and prized sources of research. But there are also others that we may not have heard of. 

That said, the standing of the journal isn't a guarantee for the quality of the research that they publish. Sometimes, studies published in highly ranked journals are later found to have spurious findings that can't be replicated. 

And in any case, what do we mean by highly ranked? Eric explains that this is about impact factor and relates to the number of times studies are cited or referenced by other researchers. 

00:25:52 Eric Barends

If your study is published in a journal with a high impact factor, you get a lot of points, and if you have a lot of points, that means you're doing very well that you make an impact. However, however, that is not an indication for the quality of the study. 

A lot of people think it is, but it's not. And again, it's often the other way around. Journals that are frequently cited are cited frequently because the studies they publish have very special or surprising findings.

However. surprising findings or new findings, or completely new insights often turn out to be wrong after a while, when we try to replicate them. So again, that is not an indication for the quality. We need to determine the quality of the study by critically appraising the study, not by looking at where it was published. 

00:27:01 Karen Plum

Even the fact that the studies are peer reviewed doesn't guarantee quality. And when we talk about quality, this relates to whether the study design was appropriate for answering the research question. The methodology may have been sound, and a correlation identified, but if we're looking for evidence for a causal relationship, the ultimate prize, then this doesn't advance our cause. As Denise explains. 

00:27:25 Denise Rousseau

So is a study published in a peer reviewed journal automatically trustworthy? No. It has made a cut - there's some odds that it can be trustworthy but critical issue in evidence based practice is the definition of trustworthiness of a scholarly article or publication is a function of the research question you're asking and the design the study used in order to answer the question. 

So if you as a practitioner are searching on an issue where you have a causal question - what are the success factors that lead to psychological safety? That's your question - it's a causal question - how do we get from here to there? You need studies that actually can test whether there is a causal relationship. They test whether or not there's been a change in psychological safety, as a function of the success factors that you're interested in, are looking at. You have to have measured that change or controlled for the change. 

So the study can't just be a survey. It can't be a case study, you know, because neither of those looks directly at change and rules out alternative explanations for why a variable moved the needle or a concept moved. You need a certain design and that means, therefore that us, as the evidence based practitioner needs to make a decision for himself or herself, whether the kind of study that the study that is reported in the journal, provides trustworthy evidence on the question you ask. 

So trustworthiness is a function of the question that you ask. It's not a function of the study where it appears. Shoe on the other foot, however, when that researcher did the study that is now published in a peer reviewed journal, they asked a certain kind of question and they asked it in a certain way. The job of the peer review process is to identify whether the question that the scholar asked has been responded to in a trustworthy way in the design that's used in the published study. 

And one of the things I think that is likely to be the case is that the peer reviewers have done a decent job of identifying the trustworthiness of the design and the methodology used for the question the author asked. 

00:29:57 Karen Plum

As with other aspects of the evidence based process, we take nothing for granted. It's all about being vigilant in terms of what's appropriate and relevant for what we're researching. 

Another way of getting a good body of evidence about your topic is the meta analysis. If your search identifies one or more of these, then that's a great result. 

Apart from anything, it will confirm that the search terms you're using are well established and will provide good explanations about what's being researched and identified so far. So what is a meta analysis?

00:30:31 Eric Barends

It's a study where the researchers have done a lot of the work that we intend to do. They searched for studies on a specific topic in a very thorough way. Combined, all these studies, had a look at the effect sizes and calculated the pooled as we say or mean average effect size based on all these studies, the aggregated effect size based on all these studies. 

And that's great because instead of having just one study and a second study and a third study, meta analyses are studies of studies, so there are often based on at least ten or twenty or sometimes even sixty or even more studies. 

So if you find a meta analysis, you should be happy, because then the researchers have already done the work for you. 

00:31:35 Karen Plum

Naturally, the date of the meta analysis is important. If it was done within the last few years, then it's likely it has strong and good evidence. Anything conducted longer ago may also be good, but as we're taught, we need to ensure we find out what has been researched more recently that may impact the results of that meta analysis. 

All of that said, Eric explains that there is something else we need to bear in mind about meta analyses. 

00:32:02 Eric Barends

A meta analysis often only provides you a correlation. We will explain in module six and seven exactly what that is. But as you probably already heard, a correlation is not the same as a causation. So when there is a relationship between two phenomena, between an intervention and an outcome, it doesn't mean that the outcome is caused by this phenomena or his intervention.

So that is something that is very important to take into account that most of these meta analyses are based on cross-sectional studies, correlational studies, so what they provide is a relationship. Is there a relationship between these two variables or between this construct or this intervention and the outcome.

And only a handful of meta analyses are based on controlled studies or even randomized controlled studies. Obviously, when you find meta analysis based on randomized controlled studies, that would be absolutely great. However, as you will see in module six and seven, where we explain what is a randomized controlled study and what is a controlled study or longitudinal study, there are not many out there. 

Again, most of these meta analyses give you information about whether there's a relationship between two variables or a construct and an outcome. Still, that can give you a lot of information. 

00:33:38 Karen Plum

So more to come in the next two episodes. And for now, here's a summary of the two principal options we've discussed - one using Google Scholar or Consensus, and the more in depth rigorous approach. 

00:33:50 Eric Barends

The way the module teaches you how to search is kind of the nuclear option rather than you would go about this in daily practice. You should take into account that if you want to see whether there's any relevant research on the specific topics on a daily basis because someone makes a claim during a meeting or whatsoever, you probably will just go to Google Scholar, just use one search term and see whether there are meta analysis and that's it. 

That'll be your search and that will give you a good idea of whether there are any studies out there, and if there are meta analysis that's already an indication that there are a lot of studies - otherwise there would not be a meta analysis. 

Start with a simple search with one term, see what's out there in Google Scholar and go from there. 

00:34:52 Karen Plum

So if you're simply looking into a comment or a claim made by a colleague and the conceptual clarity of the term used is clear, then Google Scholar and Consensus are your best approach. 

And finally, I've spent a lot of time in this episode discussing peer review and why it's important to always look for studies that have been through this process. It's easily done by checking the box on the research databases so that only these studies appear in your search results. 

If you'd like a succinct summary of why it's important, I think this explanation from Denise pretty well wraps it up. 

00:35:30 Denise Rousseau

Peer review is important because it is a way of debiasing the assumptions that the author has made about his or her work. It is a methodological check on the appropriateness of the research design and it is a logical check on the arguments and assumptions and framing that the author has engaged in, in positioning the study and interpreting the results.