The Catalyst by Softchoice

The Incident Response Episode: What Really Happens When Ransomware Hits

• Softchoice • Season 7 • Episode 9

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 24:42

When David Koopmans' IT manager started sending strange messages to employees, David knew something was wrong. By then, threat actors had been inside his network for 30 days.

What followed was a ransomware nightmare that cost $14 million, put David in the hospital, and ended with him being let go—despite years of warning leadership they needed to invest in security.

In this episode, we follow David's story from chaos to recovery, with expert context from Fortinet's incident response team on what actually happens when the call comes in (spoiler: it's always Friday afternoon), the critical mistakes that make attacks worse, and why 30 minutes a week of preparation could be the difference between survival and catastrophe.

Key Takeaways:

  • Why "we're not a target" is the most dangerous assumption in securit
  • The common mistake that lets attackers hit you twice
  • How tabletop exercises helped one company respond to a near-identical real incident
  • The 30-minute weekly habit that separates prepared teams from overwhelmed ones


Featuring: 

David Koopmans (CIO, MMT Ambulance), Josh Brewer (Softchoice), John Simmons (FortiGuard IR Lead, Americas), John Hollenberger (FortiGuard Proactive Lead)

====


This episode is brought to you by Fortinet

When a cyber incident hits, the difference between chaos and recovery comes down to preparation. Learn how FortiGuard Incident Response Services can help your team respond faster and recover stronger at softchoice.com/fortinet


====


Resources

• FortiGuard Incident Response Services: softchoice.com/fortinet

• Book: "Cybersecurity Tabletop Exercises: From Planning to Execution" by John Hollenberger (No Starch Press, October 2024)


The Catalyst by Softchoice is the podcast dedicated to exploring the intersection of humans and technology. 

Heather:

This episode of The Catalyst is brought to you by Fortinet when a cyber incident hits. The difference between chaos and recovery comes down to preparation. Learn how forta guard incident response services can help your team respond faster and recover stronger at softchoice.com/fortinet.

David:

You know, it was a huge issue for me. Uh, lack of sleep, stress, dealing with irate customers, lawyers, all kinds of things that were going on. Um, at one point I was actually ended up in the hospital.

Heather:

David Koopmans is a CIO. 25 years in it, the kind of career where you've seen everything. But he'd never seen anything like this.

David:

One of our IT guys, uh, was sending messages out to everybody at the same time. Subsequently, we were seeing messages pop up on screens that we had been ransom

Heather:

ransomed, that word that turns any ordinary day into a crisis that will consume your life for weeks, maybe months.

Josh:

They got completely owned. Just about as bad as I've ever seen it, and this is early in my career. I didn't realize how bad at the time, but after I've been gone through a few more now. It was one of the worst I've been through to this date.

Heather:

That's Josh Brewer. He's an account executive at Softchoice. He'd been working with David for years before the attack, and he watched from the other side of the phone as everything fell apart. This is a story about what happens. When the worst case scenario stops being hypothetical, when all those security investments you couldn't get approved suddenly became the only thing anyone wants to talk about. When you're working around the clock, your team is exhausted and the people who ignored your warnings are looking for someone to blame. But it's also a story about what comes after about the people who respond to these crises every day. And about the gap between the incident you've prepared for and the one that actually shows up. From Softchoice, a worldwide technology company, this is the catalyst. I'm Heather Haskin. This season, we're doing things a bit differently. We're making audio documentaries, real stories from the front lines of it, exploring the challenges of small teams chasing big dreams. Today's episode, what Every IT leader should know before their phone rings. We're calling it the incident response episode, act one. We are not a target before we get to the crisis. We need to go back. Back to when David was CIO at a different company, an electronics warranty repair operation based in Houston.

David:

At my previous company, I was the CIO and we ran an operation in the electronics warranty space, about 2,800 employees, mostly across Houston, but we had some global presence as well.

Heather:

Big operation, mission critical systems, the kind of place where downtime costs real money, and David knew they had gaps.

David:

We realized from an IT perspective that we needed to invest in some additional security technology. Uh, we were in a position that our executives, uh, my executive peers were not willing to invest. They didn't think we were a target.

Heather:

They didn't think we were a target. If you've worked in it for any length of time, you've probably heard some version of this. Maybe you've said it yourself or maybe you've been on the other side trying to explain to leadership why the threat is real. Why the investment matters, why it won't happen to us is not a security strategy.

David:

Everyone's a target. Um, and I explain that to them. Attackers really don't care. If they can get in the front door, they will, uh, which is what happened. And once they get in, you know, they don't care if they get a dollar, they get millions of dollars. They just want to get in and get whatever they can.

Heather:

Josh Brewer had been working with David for years at this. Point, he'd seen the gaps, he'd help document them.

Josh:

We had tried to become a little bit more strategic around, uh, David's environment from a security perspective. They didn't have any immutable backups. They didn't have very, what we considered high quality security solutions in place, but their organization continually just denied the budget. It wasn't David or the IT team that was not a, they were asking for it. They just weren't getting approvals on it.

Heather:

Josh and his team at Softchoice had put together proposals, assessments, a roadmap for closing the gaps.

Josh:

We had a laundry list of security action items that were known gaps, big gaps in the environment, and we had that list and we'd gone back and forth, but it always got shot down.

Heather:

And here's the thing that makes this story feel less like a cautionary tale and more like a mirror. This isn't unusual. John Simmons leads incident response for Fortinet across the Americas. He's been doing this work for a decade, and when he shows up to help a company in crisis, he sees the same pattern over and over.

John S:

Those kinds of, of companies that are, you know, they're, they're dealing with a lot of different, um, priorities and, you know, when everything's priority, nothing's a priority. And so you kinda get into that, that kind of mindset. And if you don't have the resources, right, the, the, the people on the ground and the skill sets, you know, outsource it. Right,

Heather:

but David's company didn't outsource it. They didn't invest. They ran lean and they assumed that because they weren't a household name, they weren't interesting to attackers.

David:

We had not planned, um, or playbook this out 'cause the company wasn't willing to invest in any of those types of activity. And everything was being done as, as they say, kind of on the cheap.

Heather:

And then came a Friday afternoon, act two. Everything's on fire.

Josh:

One day, I think it was, uh, pretty sure it was a Friday afternoon, which is when this stuff always happens. Uh, David gave me a call on his cell phone and told me We've got an emergency situation

Heather:

Friday afternoon. John Simmons will tell you, that's when these calls always come.

John S:

Those calls come in late, you know, at night. A lot of times.

Um, it's never Monday at 9:

00 AM most of these, these things, they're coming in Friday.

At 5:

00 PM

Heather:

it's not that attackers have a sense of drama, it's strategic deploy ransomware when it staff is thin, when response times are slower. When the pressure to just get it fixed before Monday is highest at David's company. The first sign was strange.

David:

A language that was being used and typed didn't sound like him. He said he wasn't at work, he wasn't in front of his computer. He remoted in, looked at his computer and said, oh my God, unplug it. Somebody he's in front of me. My screen doing something, and it was the remote threat actors leveraging him.

Heather:

The attackers had compromised an IT manager's account, someone with elevated access, and were using it to move through the network. By the time David's team realized what was happening, ransomware notes were. Popping up on screens across the company.

John S:

Those first few hours are pretty chaotic. Everything's burning down. We see encryption, we see ransomware notes. People are calling the help desk. Uh, everyone's freaking out.

Heather:

David called the cyber insurance company. Their response,

David:

they gave us the name of a firm that they preferred us to use. They said they'd be there in several days, and we said we don't have time to wait several days. Uh, we're a mission critical business for some of our customers

Heather:

several days. When your systems are down, when customers are calling, when every hour costs money, David found another way. One of their customers happened to have a cybersecurity division and offered to help within hours. The response team arrived

David:

within several hours. We had close to a hundred people within our facility, uh, helping us plan and remediate and physically segregate things. And they were experts in this field. They'd done it, you know, hundreds of times before.

Heather:

This is what major incident response actually looks like. Not a phone call with a help desk, but boots on the ground around the clock. And David's team, they'd never been through anything like this.

David:

Nobody on my team had been through this event before. So just following their lead on how to safely be able to recover and get up and running again, um, without the threat of the, the threat actors either being still in or coming back in.

Heather:

That last part is critical without the threat of them coming back in because one of the most common mistakes companies make during ransomware recovery is also the most intuitive one. They try to restore too quickly. I.

John S:

There is a lot of talk about recovering pretty early on, and so a lot of times what we see is that customers will start recovering before we even get on the call. They have a separate team that's already working on recovery.

Heather:

The instinct makes sense, the business is down, leadership is screaming. Every minute costs money. So you grab yesterday's backup and you start restoring.

John S:

But unfortunately, what happens a lot of times is that they're recovering from a day ago and threat actors dwell, uh, in environments much longer than a day, most times, uh, even the ransomware groups. So they're just recovering just to the point right before they got encrypted. Which could just establish the back doors that the threat actors set during that time.

Heather:

You restore the backup, you also restore the attacker's access, and then

John S:

some of the threat actors will stay in there and they will look for their back doors and they will come back and they will hit you multiple times with multiple deployments of ransomware. So you can't risk that.

Heather:

John has seen what happens when companies don't listen to this advice. He told me about a global medical company, thousands of servers, tens of thousands of endpoints. His team discovered a threat actor had been in their network for eight months. They advised the company don't block anything yet. Let us figure out where they are first. The company didn't listen.

John S:

The customer said, no, just give us everything you got now and we're gonna, we're gonna go block it. We said, well, we'll give you everything we have, but we recommend, highly recommend. Do not block this stuff yet. And so we gave 'em everything and the very next day, black Cat ransomware was deployed. Throughout their global environment, they got hit really hard. I mean, it was around 90% of their infrastructure got encrypted,

Heather:

all because they blocked too fast, and the attacker knew they'd been spotted back at David's company, the recovery was nonstop. Weeks of work and all of it took a toll,

David:

lack of sleep, stress, dealing with irate customers, lawyers, all kinds of things that were going on. Um, at one point I was actually ended up in the hospital, um, getting checked out. Everything was okay, but they, they asked what was going on. I explained what was going on at work, and, and they said, you probably need to take a break for 24 48 hours. Because otherwise you're gonna end up in a worse position.

Heather:

The hospital told him to take a break, but David didn't. He couldn't. There was too much at stake. Meanwhile, Josh was on the other end of the phone. Sometimes at two or three in the morning helping to spin up new infrastructure.

Josh:

We immediately started getting to work on spinning up net new contracts so we could rebuild fresh greenfield environments that weren't compromised. And this is just myself and my Microsoft licensing specialists working over the weekend after hours to do whatever we could to get them online because you realize their business was completely down.

Heather:

And then things got worse.

Josh:

Then all of a sudden we get a, a notification, which now is the last, the last thing I ever wanna see is a, a text message saying, do not use our Office 365 or Teams chat. The attackers have impersonator or admin are sitting in there listening to our incident response plans. So we went off of Office 365 and teams and went straight into cell phone and Gmail, personal Gmail accounts.

Heather:

The attackers were in their collaboration tools, listening to their incident response plans. So the team went dark, personal phones, personal email, and kept working. Eventually, David's team got the company back online, six weeks of work, a hundred people on the ground, and then about three months later, something happened that Josh still talks about with barely contained anger.

Josh:

David just got the job done. He didn't just sit there and say, I told you guys, you know, I told you so. Right. And then bounce, he sat in there in the trenches for those months putting himself into the hospital to get that company back online. Right? So he gave it his all. He like, he practices what he preached. He took care of the company and then the company threw him in the trash.

Heather:

The company that had denied his security investments. The company that said they weren't a target, they made David the scapegoat. Heather here with a message from our sponsor. Here's something we heard over and over again while making this episode. When an incident hits, most organizations aren't ready, not because they don't have security tools. They do, but because they've never actually tested their response plan. They don't have clear roles. They don't know who to call at two in the morning, and when the pressure's on. When systems are going down and leadership is demanding answers, that's the worst possible time to figure it out. That's the gap Fortinets for to guard. Incident response and advisory services was built to close. They help organizations get ready before an incident happens through readiness assessments, tabletop exercises, and tested playbooks. And when something does happen. Their team provides hands-on expert support, investigation, containment, and recovery guidance from people who do this every day. This isn't theoretical for guard's. Incident response. Team handles real world breaches, ransomware, nation state activity, cloud compromises, supply chain attacks. They've seen what works and what doesn't. The bottom line, cyber incidents aren't hypothetical anymore. The question isn't if it's when, and organizations that invest in readiness recover faster with less damage and less chaos. Fortinet and Softchoice work together to help you build fat readiness, aligning people, process, and technology before, during, and after an incident. Learn more at softchoice.com/fortinet. That's softchoice.com/fortinet. Act three, the preparation gap. Here's what I keep thinking about. David did everything right. He asked for the investments. He warned leadership. When the attack came, he worked himself into the hospital, getting the company back online, and it still wasn't enough to save his job. But it was enough to change what came next.

Josh:

When David got let go, he moved. He went to his next company after taking a well-deserved break, and we have come in there and from day one, night and day difference, his executive leadership over there listened to him. When he explained to them, this is what just happened to me. I'm not going to come and be in charge of your organization's it if you don't back and invest in security. And that business is just like, we don't want that to happen to us. Tell us what we need to do.

Heather:

David is now the CIO at a healthcare company. And the difference.

David:

My current company not only is willing to invest in all those things, but we have a formalized instant response, disaster recovery plan for these types of events, and we do tabletop exercises. On top of all of that, we have a retainer with a company that can help us recover. So it's literally a phone call in within 30 seconds, they're engaged

Heather:

and there's something else that's different now. The IR company they have on retainer. They're not strangers.

David:

They also help us as an arm or a branch of our cybersecurity. So they're familiar with our environment. They're not cold stepping into it. You know, if you don't have somebody and you call somebody up, they'll come and help you. But they're learning along the way, and that was part of the challenge that we encountered. The company that we have on retainer right now already understands that they know exactly where the crown jewels are. They know exactly what to go recover first.

Heather:

Tabletop exercises, IR plans a retainer with experts who already know the environment. This is what John Hollenberg calls proactive incident response. He leads that work at Fortinet and he literally wrote the book on tabletop exercises.

John H:

John Hollenberg, uh, proactive Lead for Guard Advisory Services. So basically, day-to-day for me on the Proactive team is, is really helping customers before an incident happens.

Heather:

Before it happens. That's the key phrase, because John Hollen Berger has also been on the other side. Before he was a consultant. He was an IT director.

John H:

In the past I was an IT director for a small nonprofit. It was myself and one other IT person. So I was running the team and also doing all the hands-on and, and security.'cause I was really the only security person on the team. Uh, so I totally get the, the small budget, the small team, uh, how to go about that.

Heather:

He gets it. The resource constraints, the compelling priorities. The feeling that security is important, but never urgent until suddenly it's the only thing that matters. So what does preparation actually look like

John H:

for a ransomware playbook? What are the things that we need to have in place even before an incident occurs? So looking at people, processes, and technology. You know, how are our backups? Are those backups in place? Do we have offsite offline backups? Do we have the right people assigned to to assist with this type of incident?

Heather:

And then there are tabletop exercises. John describes them like this.

John H:

It's essentially a roleplaying game for the incident responders.

Heather:

A roleplaying game. You sit around a table or a video call and someone presents a scenario. An inject in the jargon. Something like a user calls the help desk and says they can't access their files, and there's a weird popup demanding Bitcoin, and then you walk through it. Who do you call? What do you do? Who makes what decision

John H:

they, one of the biggest gaps we see with organizations is one, they don't know who's doing what. And two communication. They don't know who's doing the internal communication, who's doing the external communication.

Heather:

John told me a story that stuck with me. A colleague of his was running a tabletop exercise, not even a ransomware scenario, just a lost laptop, simple stuff. Everyone around the table said, oh yeah, if someone finds a laptop and calls the front desk, the call will definitely get routed to the IR team. So they tested it.

John H:

The call never actually made it. To the team that was in that room while they were there in the room, it finally gotta them later in the afternoon after exercise was already done.

Heather:

The call never made it a simple test of their own process and it failed. This is why you practice, not because you're checking a compliance box, not because an auditor told you to, because when the real incident comes, you don't have time to figure out who does what.

John H:

Outside of my day job, I'm a volunteer firefighter and EMT, so I like to pull in analogies to that environment quite often. And um, the last thing you want if you have a medical emergency is someone that's quote, a paramedic showing up that's never had training, has never pushed an iv, has never done anything. You don't want that person showing up at your door. It's really the same for us as, as cybersecurity incident responders. We need to be prepared. We need to be ready. Uh, we need to practice often. If we don't practice often, we're going to forget what we need to do. We're not gonna do the right things.

Heather:

So how often is often John's answer surprised me.

John H:

I would say 30 minutes a week forever,

Heather:

not a two day offsite. Once a year, not a massive exercise that requires months of planning, just 30 minutes every week practicing how you'd respond.

John H:

It's a marathon, not a sprint. You're not gonna have everything in place right away. You're not gonna be 100% perfect right away. Uh, but we need to start.

Heather:

And when you do the work, it pays off. John told me about a bank that ran a tabletop exercise with his team. During the exercise, they discovered a major gap. They didn't have an EDR solution rolled out across the environment. A month later, John got an email

John H:

and they basically were like, Hey, thanks. You just helped us get budget to improve our security.

Heather:

The exercise gave them the evidence they needed to get the investment approved, the same kind of investment. David couldn't get approved until it was too late. I asked David what advice he'd give to IT leaders who were in the position he was in, fighting for budget, trying to convince leadership that security matters.

David:

Read the news, tell your board to read the news.'cause it's happening every day. The threat actors are out there and they're constantly infiltrating companies, even some with good or great security. Even if you, you think you're a target and you're not willing to invest, then you're eventually gonna. You that guy on the front page of the news.

Heather:

But here's the other thing David told our producers, and it gets at something we don't talk about enough in it. When the crisis hit, David wasn't alone. He had a partner.

David:

It was not uncommon for me, like I said, I was working twenty four seven to call him at three in the morning or send him a teams message and he would jump right out of bed and respond and say, I got it taken care of. So he was a huge part of the ability for us to recover as quickly as we had.

Heather:

That person answering those 3:00 AM messages. That's Josh Brewer at Softchoice, and David says the relationship was different than what he'd experienced before.

David:

Typical relationship between a company and Avar is. Yeah, I'll talk to you on Monday, or, Hey, tomorrow morning I'll get back with you. We didn't get that experience from Josh. It was, whatever you need, just call me. It doesn't matter what time of day it is, and that made us be able to move a lot more rapidly on the recovery. Had we not had that, it probably would've been much more lengthy and ugly recovery.

Heather:

And when David landed at his new company, the one that actually invests in security, he brought Josh with him.

David:

I vowed to never leave his side as long as I'm a CIO. He's just a great partner for us and and for me personally, you know, whatever we need, he is there to take care of it.

Heather:

If you are an IT leader, listening to this and wondering where to start, whether you've got the right plan in place, whether your team knows who does what, whether you'd survive a Friday afternoon phone call. Softchoice can help. We offer security technology reviews that look at your current posture and identify the gaps, not a sales pitch, an honest assessment of where you are and what you need. Because the best incident response is the one you never have to use. And sometimes the most important thing is having the right partner before the call comes. Learn more at softchoice.com/security. That's softchoice.com/security. The Catalyst was reported and produced by Tobin Dalrimple and the team at Pilgrim. Content Editing by Ryan Clark with support from Philippe DMAs, Joseph Byer, and the marketing team at Softchoice. Special thanks to David Koopmans, Josh Brewer, John Simmons and John Hollenberg for sharing their expertise and their stories. Thanks to Fortinet for sponsoring today's episode. If the stories you've heard today have you thinking about your own incident readiness, visit softchoice.com/fortinet to learn how Forte Guard Incident response services can help.