Nutrition is based on science. And while the science may not always agree and even change over time, just like all scientific fields of research, without credible research to inform your views you are really just flying blind in the wind at the mercy of feelpinions and what your social media news feed shows you. The base of all scientific research is the communication of it through research papers published in peer-reviewed journals. In this podcast, I’ll show you how to delve beyond the title of a research paper and instead apply a critical filter to all parts of the research study. Developing this skill will allow you to form your own view of how much influence to give to a research study, rather than be led astray from those seeking to influence you.
Nutrition can be confusing at times. With a clash of competing voices and conflicting media headlines and a good dash of conflict of interest, how can you forge your own path in determining where the truth, or perhaps that should be, the view closest to the truth today lies? If there is one skill that would assist you in doing this above all others, it would be the ability to critically read and interpret a scientific study. Being critical here means to question the information to evaluate the worth of it. It is not the same as being negative and finding flaws because you don’t like the message.
To understand a study, as well as how it relates to previous research on the topic, you need to read more than just the title. So what I aim to do today is to draw on my 30 years of experience of direct involvement in scientific research to give you a guide for how to go about reading and critically appraising a scientific paper.
Abstract and introduction
So, you’ve come across an article reporting on a scientific study and you’ve managed to track down a copy of that paper from the journal website – well done, that can be a feat in itself. And spoiler alert for next week’s podcast, I’ll outline how to keep up with research and where are the best sources to do this.
So, what’s next? It is page 1 of the paper – that being the abstract and introduction. The abstract is a short summary that covers the main points of a study and should reflect the title of the paper. A good abstract will be clear and informative and reflect the content of the paper. With only a few paragraphs to do this, sometimes an abstract can be unintentionally misleading, either by omitting certain key findings or only focussing on certain areas of the research. Treat the abstract then as a guide to what you would expect to read when you delve into the paper in greater depth. Before citing a study as evidence, make sure to read the whole thing, because it might turn out to be weak evidence . Reading titles and abstracts are great to get a feel for a research field, and no one has time to read every paper in full they come across, even me, but if you want to cite research as evidence, you need to have a good handle at what is contained in it after the abstract.
After the abstract, we have the introduction which sets the stage for the research study. A good introduction should acknowledge previous literature in terms of arguments ‘for and against’ what the authors are presenting as their research question. Introductions are also a great place to find additional reading material since study authors will frequently reference other relevant, published studies on the topic. The introduction will also outline what the goals of the research are and why the research question is important. Sometimes a rationale is given that there is a literature gap which the work is attempting to address, but this necessarily isn’t a problem so the work may not be that novel. The end of the introduction leads into ‘what is unknown’ and makes it clear why the authors are doing the study and you will usually find this in the last paragraph of the introduction.
Methods
Normally straight after the introduction, we have the methods section. Understanding the methodology is vital for determining the strength of evidence and the validity of the conclusions.
I would argue that understanding the pros and cons of the research design chosen is the single most important part in evaluating a scientific study. For human research, there are two broad types of studies: observational and intervention studies. Observational studies are where people are simply ‘observed’ over time or data is collected from existing sources and here, the researchers have no influence on the outcome. For intervention studies, there is some form of external influence that is normally introduced by the researchers and a randomised-controlled trial is considered the ‘gold standard’. Here’s an example: a study where the dietary habits of a large group of people who report having IBS are collected is an example of an observational study. While a study evaluating how those same people fared in terms of their IBS symptoms when on a low-FODMAP diet compared to a regular diet is an example of an intervention study.
There is no one perfect study as they each have their own strengths and weaknesses, but knowing the merits and shortcomings of each type of study will go a long way in you being able to evaluate the conclusions of a study. So, I’ll spend a little time talking about those pros and cons.
There are three main types of observational studies and these are cross-sectional, case-control and cohort studies.
Cross-sectional studies are descriptive studies and provide a ‘snapshot’ of the health or nutritional status or people at one point in time. These studies measure the prevalence, but not the incidence of a factor of interest. The number of people who have T2DM in Australia today is an example of cross-sectional prevalence data. How many people develop T2DM each year is an example of incidence. Cross-sectional studies are very useful in providing information on the distribution of disease or for health planning and resource allocation. National Nutrition Surveys are examples of cross-sectional studies.
Some of the advantages of cross-sectional studies include:
Disadvantages
Here’s an example of what I mean by confounding. You collect data on sunburns and ice cream consumption. You find that higher ice cream consumption is associated with a higher probability of sunburn. Does that mean ice cream consumption causes sunburn? Of course, it doesn’t, but it may be possible to make that conclusion because of the association. Here, the confounding variable is temperature: hot temperatures cause people to both eat more ice cream and spend more time outdoors under the sun, resulting in more sunburns.
Another observational study desig n is known as a case-control study. This is where you start with the outcome of interest which is either a case (so someone with a disease like cancer for example) or a control (which is someone without the disease) and measure past exposure. So, what you are doing is comparing similar groups of people with and without a disease and looking back in time at their diet and lifestyle habits to see what influence this could have had.
Some of the advantages of case-control studies include:
Disadvantages
And the third type of observational study is the cohort study – this is the most powerful type of observational study. Here you have a group of people (called a cohort) who are followed over time and their diet and lifestyle habits are monitored. You then wait for health outcomes of interest to develop such as cancer, heart disease and so on. This study allows you to compare disease rates in those who were exposed to the factor of interest (such as eating a high fibre diet, exercising regularly and so on) and those who were not exposed before the disease outcome of interest develops. Importantly, this is still an observational study so there is no external influence by the researchers in shaping the diet and lifestyle habits of the people.
Some of the advantages of cohort studies include:
Disadvantages
Keep in mind that observational trials cannot show causation since the scientists conducting the study are not controlling any variables. But they are valuable in building a strong case.
And finally, we have intervention studies. The gold standard for a scientific study is a randomised-controlled trial. The most important concept here is that the exposure is assigned be it an exercise program, a specific type of diet, or an education program around healthy eating. If possible, both the participants and the researchers should be blinded to who received the intervention, but this is not always possible.
Some of the advantages of intervention studies include:
Disadvantages
In terms of the strength of conclusions you can make about an area, no one study is ever definitive, but as you move up the evidence tree starting at case studies then cross-sectional studies then onto case-control studies and then cohort studies and finally to RCTs, the evidence grows until you can combine it together into a systematic review and meta-analysis – types of studies you would have heard me refer to a lot in my previous podcasts when I provide sup porting evidence. You can put a case forward for any nutrient causing or preventing any disease you want if you selectively cherry-pick a single study from the research literature. What matters most is what the field narrative of research tells you and here, it is systematic reviews that reveal a much truer picture.
The area of study design can be complex, but there is an excellent online tutorial that I’ll link to in the show notes that gives a short overview of each type of study and how they are ranked in the levels of evidence and go on to inform scientific views – it is well worth doing this tutorial if you’re serious about getting your critical analysis skills up to speed http://himmelfarb.gwu.edu/tutorials/studydesign101/
Also in the method section of a paper, for studies involving humans you’ll find demographic information like age, sex, and lifestyles of the participants, how they were recruited, and details about the intervention itself. This is all very relevant to know as it affects the generalisability of the study. For example, if a study only recruits post-menopausal women for a supplement trial to improve bone health, then the results will have little relevance for males or adolescent females.
Note the inclusion and exclusion criteria for how people were recruited as this will always be given in the methods. It is important to know this for the applicability and generalisability of the findings.
There are several other aspects of the methodology to pay attention to. One is the sample size of the study which is the number of people who were studied. The larger the sample size in a study, the more reliable the results are. Look for a statement, usually in the statistics section of the methods, that the authors calculated the ‘power’ of the study. Power gives a guide to the minimum sample size needed to still give a good chance that statistical significance would be found. There is no ‘magic’ number for sample size, but the more the better. A study can be adequately powered with 20 participants in it so the results should be valid, while another study may be underpowered with 800 people. If a study is underpowered, it may mean that a study can’t answer the research question. Some ethics committees don’t allow studies to run unless they are sufficiently powered, unless it is a pilot study to explore a new research question.
Statistics
The methodology section usually concludes with a statistics discussion. This area is not for the faint-hearted and unless you have training in statistics, it will likely be undecipherable. Determining whether an appropriate statistical analysis was used is an entire field of study, so when reading statistics, try to focus on the big picture. Here take note that if something is ‘statistically significant’ it does not mean the same as ‘important’. Statistical significance may still be ‘clinically insignificant’ while ‘non-significance’ may just be a me asure of an under-powered small sample size. For example, if researchers find that taking a weight loss supplement helps people lose an extra 0.5 kilograms of fat every year and the result is statistically significant, you probably wouldn’t recommend people taking the supplement because its benefit is so small – and may come with some side effects and be expensive.
Results
Next comes the main results of the study. A question to ask yourself here is: do the results relate to the research question? If the study was an intervention study where one group of people received a treatment, such as taking a supplement to help with exercise endurance, and this was compared to a control group, look for reporting of intent-to-treat analysis or a statement on why people were lost or dropped out of the study. Almost every study has participants that don’t finish the trial or don’t follow the instructions. A lot of dropouts or non-compliant participants should raise red flags though, especially if one group of participants was more prone to dropping out. An imbalance in dropouts in one group is a big flag that something is askew with the intervention and this could bias the result.
Another thing to look for are studies that use a whole battery of endpoints or outcomes. If you measure enough things, you probably will find something of interest and this can get the focus of the study conclusion. That is fine if the key outcomes were what the study was about, but if they were well done the list as secondary measures then the researchers may just be dressing up their study.
In the same vein, take very special note of where an over-reliance is given to sub-group analysis. This is where the researchers ‘slice and dice’ up their results to look at a narrower snapshot of people, such as in a study involving a broad cross-section of the general population looking at just women, or people over the age of 50 or people who had poor glycaemic control at the start of the study. Subgroup analyses can be interesting, but if it wasn’t part of the original research design or study aims , you may be seeing researchers ‘data mining’, that is – looking for spurious significant results that may have just arisen by chance. Large clinical trials aren’t always powered enough to make strong conclusions about subgroups. It is a red flag for me when a study relies on emphasising the results of their secondary analysis and subgroups when their primary outcome didn’t show much of interest.
Discussion and conclusion
Finally, there is the discussion and conclusion. Here you will see the results discussed in relation to the hypothesis. A good discussion will put forward alternative interpretations to the research study findings, rather than just one dogmatic idea. Good discussions will do a lot of the critical analysis work for you and highlight in-depth all the limitations of the study. All research has limitations and I find that people who loudly call out the limitations of a research study do so because they don’t like the findings as they go against their world view of nutrition.
The conclusion is also an appropriate section to discuss potential future studies based on the new results. That being said, researchers also might hypothesise a potential mechanism of action, or point out ways future studies could improve on their study design. Research that ends with a statement along the lines of ‘more research is needed’ is likely a measure of lazy researchers who couldn’t think of anything else interesting to write. Okay, harsh I know, but really, of course more research is always needed – time to be a bit more original.
And finally, look to see who funded the research and are conflicts of interests declared? It is worth noting, but it never ever should be used to invalidate research simply because of COIs or the funding body. Researchers are NOT lining up to be paid off by industry so they can produce fraudulent research in the pursuit of upgrading to a bigger yacht. There are very, very few bad apples in the research community – most researchers are good people who want to do good work regardless of who funds it, but they also have the reality of needing research grants to develop their research program.
So, it could be that research that attracts industry funding is already helping to confirm prior positive work in the field, so the outcome is already pretty assured. It also could be that a study is designed to report a positive outcome by using a battery of outcomes where if any one lights up as positive, it gets the focus of the study conclusion. Or maybe, only positive research is allowed to be published and such industry funding contracts do exist. They are ones I would never, ever sign, but some researchers may.
It doesn’t stop at the influence of industry funding though. Researchers tied to a narrative with their research program (and maybe a large social media following and the odd best-selling book on Amazon or two) may find it difficult to do an about-face when conflicting evidence presents itself. As humans, we are all susceptible to this. The scientific method helps to reduce it, but it cannot eliminate it entirely.
So, that is a lot of things to consider when evaluating a study. Adding it all together allows you to form your own view on how solid the research findings are. And if you want to go a little deeper into it, the people over at examine.com have put together a very detailed and accessible guide on how to read a scientific study which I’ll link to in the show notes. https://examine.com/guides/how-to-read-a-study/
So that’s it for today’s show. You can find the show notes either in the app you’re listening to this podcast on if it supports it, or else head over to my webpage www.thinkingnutrition.com.au and click on the podcast section to find this episode to read the show notes.
If you find this podcast of value, then please consider sharing it with your friends and colleagues. Or maybe even leave a review. This all helps increase the ranking and reach of the podcast which means a big win for credible evidence-based nutrition messages while helping to dilute out the crazy and making the world a slightly less confusing place.
I’m Tim Crowe and you’ve been listening to Thinking Nutrition.