Technical Postmortem

Buzzsprout DDoS Technical Postmortem

Buzzsprout was assaulted by criminals using a DDoS attack on February 21/22, 2021. We have always been and always will be fully transparent with those that trust us with their podcasts. This technical postmortem will provide additional details regarding the type of attack we suffered and how we responded.

Things to know upfront:

  1. The criminals demanded money to stop the attack. Buzzsprout will never pay criminals. This only makes our customers and us a target for future attacks and funds more criminal activity. It will never pay to attack Buzzsprout.
  2. No data was compromised in this attack. The sole purpose of this attack was to take Buzzsprout offline until we paid their ransom.
  3. We contacted the FBI immediately, and they are working to bring these criminals to justice. We will fully support those efforts.
  4. The entire Buzzsprout community suffered from this attack, but there is good that will come from it. We have donated the ransom amount to two great organizations helping people in Texas suffering from the weather crisis. Those organizations are Feeding Texas and Front Steps.
  5. Since DDoS attacks require a lot of energy to run, we also made a contribution to Project Vesta to more than offset the carbon footprint of the attack.
  6. The result of this attack is that podcast episode deliveries were delayed through the Buzzsprout platform for 7 1/2 hours, but people in need will get food, water, and shelter, and the global environment will be healthier.

What happened, and how did we respond?

The first attack began on Sunday, February 21, at 11:50 AM EST. Our team quickly identified that we were under attack and began implementing our mitigation strategies, including opening up communication with our DDoS mitigation service provider.

The attack was a combination of a distributed HTTP flood attack with requests to our site spiking 10-15x normal levels and SYN flood attacks on our infrastructure's network. The HTTP flood attack was first, and as we mitigated this attack, another SYN flood attack started at 12:30 PM EST. This SYN flood began impacting our upstream providers and taking down other websites in the southeast USA.

By 2:00 PM EST, February 21, Buzzsprout was back online. Our combined efforts were enough to dissuade the attackers from continuing, but it was clear that they would resume the attack at some point.

Now that we knew exactly what we were facing, we were able to take steps to further mitigate the impact of this specific SYN flood and implement some additional strategies to mitigate this specific HTTP flood. One of those strategies required a massive change to our infrastructure, enabling us to isolate our traffic from our upstream providers. By 1:50 AM EST, the engineers were ready to implement the changes. Buzzsprout was placed in a read-only mode allowing them to implement the new infrastructure. Buzzsprout podcasts were served up without issue during this time, but podcasters were unable to make updates. By 2:56 AM EST, the fortified architecture was in place.

Our Ops Team worked through the night to provide the best defenses possible. While there is no way to prevent DDoS attacks completely, the fastest way to shut them down is by keeping your site up for as long as they continue. These attacks cost the criminals real money to run. If they see that they are just burning money, they eventually give up.

The next siege was launched at 8:54 AM EST, February 22, 2021. We were able to defend against their initial attack pretty well, but they ramped up quickly. By 10:00 AM, Buzzsprout was fully down. Around 11:14 AM, we were able to get the site operable for some but not all. By 1:00 PM, we were able to block almost all attempts to make Buzzsprout unavailable. The attacks stopped around 1:15 PM, and at 1:24 PM, we declared Buzzsprout stable.

We remain on high alert in case of another attack.