Why You Shouldn’t Trust Peer Review — From a Former Duke MD

They said the science was settled. It wasn’t.

In this episode, Dr. Andrew Kaufman shares the exact research method he learned at Duke University—and why it led him to reject psychiatry altogether.

You’ll learn how to spot bias in medical papers, understand the hidden levers of influence, and critically appraise studies like an insider.

Get the neurotoxin checklist that reveals what might be clouding your clarity: www.andrewkaufmanmd.com/brain-hijackers

Get the full show notes and transcript at the blog: andrewkaufmanmd.com/critical-appraisal-method

Transcript

Hello, everyone, and welcome. I'm Dr. Andrew Kaufman. And today I'm going to talk about how I do research for this podcast and to formulate my opinions on health and medical related topics. This technique that I use could be loosely called a critical appraisal.

Critical appraisal is something that I originally learned in my residency training in psychiatry at Duke University. This was a required technique that we learned and then had to utilize on a weekly basis at a mandatory journal club. In this journal club, we would be assigned clinical articles—usually clinical studies of antidepressant drugs or a type of psychotherapy. We would critically appraise these articles and then present them to the group.

This procedure was really for the purpose of rooting out what published studies are actually worth following in your own clinical practice. What are the takeaway messages? Can we actually learn something? Does this show that a treatment is effective?

Now, I want to give a personal example I witnessed at Duke University. This will help you understand how bias can enter into a scientific paper. It can be very difficult to read a paper and know that bias is present.

There are two main ways bias can enter that I'm going to talk about today. One has to do with the study coordinator, and the other with how study participants are selected and enrolled.

The study coordinator is essentially the person who checks in with study participants, reminds them about evaluations or treatments, and so on. If the coordinator is super friendly and hands-on, participants often have better outcomes. If they’re lax, the opposite happens. This kind of bias won’t show up in the paper—but it can shape the results.

The other issue is more severe: participant selection. Many times junior researchers are under pressure to meet enrollment quotas. But it's often hard to find people who meet the inclusion criteria. So what happens? People who don’t meet the criteria get recruited anyway.

I saw this happen in a study called the SAD Heart Study, which evaluated antidepressants in people who had experienced heart attacks. I evaluated a patient who had the cardiac history, but he didn’t meet criteria for depression. His issues were situational—his son and girlfriend had invaded his trailer, causing chaos. He wasn’t clinically depressed. He needed boundary support, not medication.

Despite this, he was enrolled in the study. The coordinator helped resolve his family issue, and his condition improved. But that would get reported as an antidepressant success—even though the drug had nothing to do with it. This is the kind of bias that makes it into the medical literature.

That’s why we have to maintain a high degree of skepticism when reading research. In fact, a famous epidemiologist from Stanford published a study in PLOS ONE showing that more than half of published research findings are false. So when you pick up any paper, assume it’s likely wrong unless proven otherwise.

One of the most important things to look at in a scientific paper is the methods section. It tells you what experiment was actually done. And yet, in many journals today, that section is hidden in the appendix or supplementary material—where casual readers won’t see it. This makes it even easier for studies to be misinterpreted.

Here’s another issue. In many virology studies, you’ll see statements like “we used virus X” with no explanation of where it came from or how it was identified. Without clarity on your input material, how can you trust the experiment?

It’s easy to get intimidated by scientific papers. So people rely on experts, news reports, or even AI. But I’ve seen AI fabricate case law and citations. You always need to check sources—no matter how advanced the tool seems.

Let’s talk briefly about the difference between science and engineering. Engineering has a built-in accountability—what you build has to work. Science doesn’t have that. So unless findings are reproduced, there's no built-in test of validity.

Reproducibility is rare. Most replication efforts fail. Sometimes it's sloppiness. Sometimes it’s fraud. And many prestigious institutions have had major scandals. There’s pressure to publish, and researchers are human.

Another layer is statistical manipulation. One example taught to me at Duke is the use of relative risk reduction instead of absolute risk reduction. This is how benefits get exaggerated in studies—especially for vaccines or psychiatric drugs.

Now let me walk you through how I do research:

I start broad. I look at questions and claims people are making online, in forums, in the media. Then I refine the language—because scientific papers often use different terminology. Once I find the right search terms, I move to PubMed or PubChem.

PubChem is great for looking up chemicals, drugs, or toxins. It has tons of links to related research. PubMed is my go-to for finding medical papers. Many are behind paywalls, but I use Sci-Hub (sci-hub) to find full-text versions using the paper’s DOI.

From there, I read carefully, especially the methods. I look for how the study was set up, what the actual intervention was, and how outcomes were measured.

Now let’s distinguish in vitro (test tube) studies from in vivo (animal or human) studies. Each has different strengths and weaknesses. Some information we can only ethically get from animal studies.

And finally, I want to give you a simple framework you can follow. I call this the 5-Step Critical Appraisal Method:

(Note: While I don’t enumerate them explicitly in this moment, the five steps are covered earlier in the episode. For clarity, here they are as described in my own words throughout the lecture:)

1. Identify the claim or question you’re investigating.
2. Seek multiple sources of information.
3. Examine the quality and potential biases of those sources.
4. Distinguish between correlation and causation.
5. Look for patterns across different perspectives and interests.
   
   

This is a powerful method that can help you in every area of life—not just health. It’s how I walked away from psychiatry. It’s how I protect my own family’s health.

So here’s your challenge:

Pick one small topic. Use my 5-step method. Leave a comment with what you learned.

You’re more capable than you realize. This is how you become your own health authority.

Thank you for joining me. I’ll see you in the next True Health Report.