Data Science x Public Health
This podcast discusses the concepts of data science and public health, and then delves into their intersection, exploring the connection between the two fields in greater detail.
Data Science x Public Health
In Theory, Statistical Significance Works. In Reality… It Doesn’t
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Statistical significance is one of the most familiar ideas in research. It is often treated as the dividing line between real evidence and random noise. But what if that binary framing is doing more harm than good?
In this episode, we break down why statistical significance often misleads, how threshold thinking distorts interpretation, and why effect size, uncertainty, and design quality matter far more than a simple significant versus non-significant label.
👉 Enjoyed the episode? Follow the show to get new episodes automatically.
If you found the content helpful, consider leaving a rating or review—it helps support the podcast.
For business and sponsorship inquiries, email us at:
📧 contact@bjanalytics.com
Youtube: https://www.youtube.com/@BJANALYTICS
Instagram: https://www.instagram.com/bjanalyticsconsulting/
Twitter/X: https://x.com/BJANALYTICS
Picture this, you're you're standing outside an exclusive club, right? And there's this ruthless VIP bouncer at the door.
SPEAKER_00Right, checking the list.
SPEAKER_01Exactly. Well, in the scientific community, that bouncer is a number. If a study's on the list, it gets past the velvet rope and gets published. If it's not, you know, it it's rejected. Today we're doing a deep dive into a text called The Tyranny of the Binary: Reclaiming Statistical Truth. Our mission is to figure out why statistical significance, the supposed gold standard for separating scientific signal from noise, is actually deeply flawed. Okay, let's unpack this. Why on earth do we accept this rigid binary system for deciding what is scientifically true?
SPEAKER_00Well, to really understand that bouncer, you kind of have to look at the sheer chaos of early science. I mean, researchers needed a practical way to judge if an observed effect, say a new drug-lowering blood pressure was real or just a complete fluke.
SPEAKER_01Like a baseline rule of some sort?
SPEAKER_00Yeah, exactly. That's what a p-value is actually meant to measure, is the probability that your result could have happened just by random noise. So the.05 threshold, meaning a 5% chance of a fluke, was introduced as a helpful rule of thumb to, you know, standardize how we deal with uncertainty.
SPEAKER_01Here's where it gets really interesting. If it started as just a loose rule of thumb, how did it become absolute law? Because right now, I mean, if a study gets a p-value of 0.049, pop the champagne, it's labeled the discovery. But if it hits 0.051, it's deemed a total failure and just forgotten. Are we really just drawing arbitrary lines in the sand?
SPEAKER_00We are, yeah. Humans love certainty. And over the decades, journals, reviewers, and grant panels just institutionalized that arbitrary line. It morphed from a tool into a shortcut for the reviewers. Right, because it's much easier for a journal editor to just glance at a number and see if it's under 0.05 than to deeply evaluate the actual methodology. What's fascinating here is how that artificial line actively distorts reality. A mathematically significant result can actually be utterly useless in the real world.
SPEAKER_01Wait, wait. How can it be significant but useless?
SPEAKER_00Because of how the math works. When your mathematical magnifying glass gets big enough, meaning you test a massive sample size of like hundreds of thousands of people, it will spot microscopic statistical anomalies. They are technically real, but practically meaningless for an actual patient.
SPEAKER_01Aaron Powell So what does this all mean for you, the person reading the headlines? Think of statistical significance like a metal detector on the beach, a really loud beep, a significant result. Might not be a gold coin. It could just be a rusty bottle cap that your massive machine flagged.
SPEAKER_00That is a great way to put it. And the reverse is where things get truly dangerous. Silence from the metal detector doesn't guarantee there's no buried pressure. This is what biostatisticians call low power.
SPEAKER_01Meaning this study just didn't have enough juice.
SPEAKER_00Exactly. Your study might simply lack enough participants to power the machine. You could be walking right over a chest of gold, like a real medical breakthrough. But because your detector is weak, it just doesn't beep. Treating a non-significant finding as absolute proof of no effect is a massive misinterpretation, especially, you know, when we use these imperfect studies to make high-stakes public health decisions.
SPEAKER_01Which is terrifying. If we can't trust the binary beep of the machine to find the truth, how do you and I train ourselves to look past those flashy significant headlines? Like what should we actually be looking for to spot good design?
SPEAKER_00Well, if we connect this to the bigger picture, it means changing what you look for when you read a study's abstract. Stop hunting for the p-value. Instead, look for the effect size. That tells you how much of a difference the intervention actually made in the real world, a tiny effect size. Even if the p-value says it's statistically significant, well, it often doesn't matter. You should also look for confidence intervals.
SPEAKER_01Which are essentially the study's margin of error.
SPEAKER_00Good biostatistics isn't about declaring winners and losers, it is about characterizing uncertainty, honestly.
SPEAKER_01So the goal isn't to burn significance testing to the ground entirely, but to, I guess, strip it of its final word status?
SPEAKER_00This raises an important question, though. What happens if we don't? Seeing the phrase statistically significant should be your cue to ask more questions about the study's design, rather than just accepting the headline as gospel. It is just one small signal in a much broader process.
SPEAKER_01Which leaves us with a pretty wild lingering thought. If these rigid.05 thresholds have dictated published science for decades, how many life changing medical breakthroughs or crucial policy interventions are sitting abandoned in a filing cabinet right now simply because they scored a 0.051? Next time you see a definitive scientific verdict, remember the bouncer at the club. Just because it didn't make it past the velvet rope doesn't mean it didn't have something important to say.