Poison in the Cache

Every time you type a web address, you're trusting a directory. A vast, invisible system that translates the names you know into the numbers that actually move data across the internet. You trust it the way a town trusts its well.

In 2008, a security researcher named Dan Kaminsky discovered that the well had no lid.

In this episode

• DNS - what the Domain Name System is, and why it matters
• Dan Kaminsky - security researcher and internet advocate 
• Cache poisoning - the class of attack Dan Kaminsky found hiding in the internet's foundation
• The patch - a secret meeting, fierce competitors, a deadline, and a synchronized global fix
• Trudy Kaminsky - Dan's mother, and a legend in her own right

Episode Music

• James Opie / Nihilore, CC BY 4.0
  • "Closest Strangers"
  • "Single Lane Tunnel" 
  • "The Dweller on the Threshold"
  • "A Different World by Night" 

Additional Notes

You can watch Dan Kaminsky explain the DNS flaw he found here. An audio only version is available here. He gave this talk at Black Ops 2008 after his original Black Hat presentation.

--

Transcript

Speaker: 0:13

Imagine a town built around a single, deep well. Every morning, the people in this town walk to the well. They lower their buckets. They draw the water. They take it home. They drink it. They give it to their children. They don't test it. They don't question it. They just trust it because the water has always been clean. But what if one night someone crept to the well and poured poison into the dark water below? The next morning the water looks the same. It smells the same. The bucket feels just as heavy in your hand. But everyone who draws from the well is compromised. And no one knows until it's too late. Every website is just a string of numbers. Human beings can't remember strings of numbers. So decades ago, engineers built a directory. You type a name, gives you an address. That directory is called DNS. It stands for Domain Name System. It is the well that we all draw from. Until a 29-year-old named Dan Kaminsky realized that the well had no lid. I'm Daina Bouquin, and this is Lore in the Machine. Dan was the kind of person who broke things just to see how they worked. When he was 11 years old growing up in San Francisco, his mother got a phone call. It was a security administrator for the United States Military. Her son, the administrator, explained had just hacked into their network. And they were going to cut off the family's internet access. Now his mother didn't panic. She told the government that she would take out an ad in the San Francisco Chronicle to announce that an 11-year-old had defeated the military's computer security. So they negotiated a three-day internet timeout for the boy instead. Then Dan grew up. He became a professional security researcher. And in 2008, he was looking at the internet's directory. He noticed something that made his stomach drop. Not a simple bug, a structural weakness baked into the very foundation of the internet, a vulnerability that would allow an attacker to quietly slip false information into the central directory, to rewrite the map of the web, to redirect your email, bypass your passwords. So that when you typed in the name of your bank, the well would send you somewhere else entirely. We call this type of attack cache poisoning. It's a security vulnerability where attackers can insert fraudulent entries into a cache, causing the system to return incorrect and malicious data. What Dan Kaminsky found was a specific type of cache poisoning that we now call DNS poisoning or DNS spoofing. When Dan brought this to one of the original architects of the system, the response was chilling. Everything in the digital universe was going to have to get patched. The loneliness of knowing a secret that enormous. You are holding a match, standing in a room, drenched in gasoline, and you cannot tell anyone why you're afraid to move. Dan couldn't go public. If he did, the bad guys would figure it out before the good guys could fix it. Instead, he convened a secret meeting. Picture a windowless conference room at Microsoft headquarters, 16 people sitting around a table, executives and engineers from the biggest tech companies on earth, fierce competitors who had no reason to trust each other. And Dan Kaminsky, the kid who once got a three-day time out from the military, walks to the front of the room. He opens his laptop. He connects it to a projector. And in about 10 seconds, he breaks the internet. He compromises a server running the software that controls 80% of global web traffic. He looks at the people in the room and tells them he is giving a presentation at a hacker convention in August. They have until then to fix it. And the crazy thing is they did. Competitors worked in absolute secrecy to build a synchronized global patch. On July 8, 2008, the patch went out. The well was locked down. The world had no idea it was ever in danger. A few weeks later, Dan gave his presentation at Black Hat. He wore a sharp suit. And as someone who always wore sandals, he had promised his mother he'd wear closed-toed shoes. He wore roller skates. A reporter asked him later why he didn't use the flaw to steal millions of dollars. Dan said it would have been morally wrong. And besides, he said, he didn't want his mother to have to visit him in prison. His mom, Trudy, was a legend in her own right. She would show up to these hacker conventions carrying homemade cookies. Dan Kaminsky passed away in 2021 at the age of 42 from diabetic ketoacidosis. When the tributes poured in, people didn't just talk about the time he saved the digital universe. They talked about his staggering empathy. He bought plane tickets for heartbroken friends. He built apps for the color blind. He used to say that the internet was never designed for any of this, just to move pictures of cats. Nobody planned for humanity to drop trillions of dollars into it, but we did. And when people asked what we were supposed to do about that terrible reality, his answer was simple. Some of us got to go out and fix it. Dan didn't just prevent an attack. He forced the architects of the web to build a heavy cryptographic lid for the well. A lock, so vital, that's securing it would eventually require an elaborate physical ritual. But that's a story for another day. Sometimes soon, you will open your browser. You will drop your bucket into the deep, dark water of the internet. And it will be clean. I'm Daina Bouquin, and this is Lore in the Machine.