The Audit - Cybersecurity Podcast

Critical Infrastructure: Everything is Connected and Vulnerable

• IT Audit Labs • Season 1 • Episode 77

Share episode

When hackers target the systems controlling your water, power, and transportation, the consequences go far beyond data breaches—people can die. Leslie Carhartt, Technical Director of Incident Response at Dragos, pulls back the curtain on one of cybersecurity's most critical blind spots: industrial control systems that keep society running but remain dangerously exposed. 

What You'll Learn: 

  • Why industrial control systems can't be updated like your laptop—and what that means for security 
  • How threat actors are using AI to generate custom malware for power plants and water treatment facilities 
  • The real state of critical infrastructure security (spoiler: forget about air gaps) 
  • Why commodity ransomware has become an existential threat to industrial operations 
  • The five critical controls organizations should implement right now to defend OT environments 

Don't wait until your organization becomes the next headline. Like, share, and subscribe for more in-depth security intelligence that goes beyond the buzzwords. 

#industrialcybersecurity #criticalinfrastructure #OTsecurity #ICS #SCADA #dragos #incidentresponse #ransomware #AIthreats #cybersecurity #infosec 

SPEAKER_02:

You're listening to the audit presented by IT Audit Labs. I'm your co-host and producer, Joshua Schmidt. We have the usual suspects, Eric Brown, our managing director, and Nick Mellum coming to you from an undisclosed location today. I'm in a different place when I usually am at. And today our guest is Leslie Carhartt. She's a technical director of incident response at industrial cybersecurity company Dragos.

SPEAKER_01:

It's my absolutely pleasure, absolute pleasure to be here. Thank you for having me.

SPEAKER_02:

Yeah, maybe you could start off by telling us a little bit about yourself and Dragos and what you've been working on and can I go from there.

SPEAKER_01:

Sure. I have a very strange job. Let's start with that. Um I am one of maybe under 100 people on Earth who responds to hacking of computers that don't look like computers. So industrial stuff, things like trains and ships and power plants and manufacturing facilities, cranes, things like that. And I've been doing that for quite a long time, almost 20 years now. Um I've been doing incident response in that space and digital forensics. Um and specifically, you know, response to places where somebody might have died, somebody might have been injured because of this equipment that is now digitally connected. So we're talking about physical devices, real life things that are connected to computers. And those computers don't necessarily look like computers that you have on your desk.

SPEAKER_02:

Are we talking like Internet of Things or are we talking about No, this is free Internet of Things stuff.

SPEAKER_01:

The Internet of Things has a place in there. There is industrial Internet of Things stuff, absolutely. Um, but this is like PLCs, RTUs. Um sometimes you hear about the overarching control systems that manage those devices, things like SCADA. Um, and they are running everything in society, and they have been for a very, very long time, long before there was like smart things. Like um there, there's been ICS for a for a long time. And uh ICS was mechanical at first. It was things like gears and pulleys that handled timing for industrial processes. And then it became electronic in the 20th century. So like the systems and power plants and things were switched always in wires and then transistors, and then eventually they became computers.

SPEAKER_00:

So Leslie, we we do uh have some customers in the same space where they have critical infrastructure and they're they're working with devices that control transportation or wastewater, things like that. Um curious uh in your in your work in um um looking into maybe attack surfaces or where threat actors may have gotten into those environments, and we we talk about those pieces of critical infrastructure being on separate networks or completely uh offline and not connected to anything, but inevitably something is connected somewhere or there there is some way to bridge that gap. Just curious in your in your work, what have you seen as far as how the threat actors are bridging that gap?

SPEAKER_01:

The gap is mostly a lie uh today. Uh in the last 10, 15 years, there has been such a need for connectivity to those environments. So think about like just in time manufacturing and think about trains. Everything, everything has telemetry now. Everything is computer controlled at very fast speeds, sometimes across multiple facilities. There's trucks involved, there's shipping involved. And when you're talking about things like utilities, like power and water, a lot of the people who used to do that have now retired or been laid off to be replaced by computers. So that means that their jobs that involve dispatch and going to places have also gone away and they've been replaced by centralized staff. So that means everything has to be connected now. I see maybe two air-gapped, really air-gapped environments a year, and I do this full-time. And they're in like things like defense and nuclear. Everything's connected now, and it's increasingly connected. We're talking about environments that have not just one way through from the enterprise environment, but tons of like external VPN concentrators, modems, things like Team Gear installed by vendors. The vendors want to be able to remotely access things too. And that's really hard to get a handle on when we're talking about sometimes remote facilities, things without proper, you know, modern security architecture. There's no ability to install modern monitoring. Um, and then you're dealing with increasingly connected devices. And a lot of those devices don't look like modern Windows 11 computers. They are things like really archaic things. I see Windows NT, Windows 95 on a routine basis, um, as well as Linux from that era. And then there's those very, very abnormal PLCs and things that are running custom firmware.

SPEAKER_00:

Why do you think it it's common practice? I won't say acceptable, but I'll say common practice for some of those devices to run those legacy operating systems and and not be upgraded to a modern OS.

SPEAKER_01:

It's very clear reasons, actually, and it's very consistent across multiple industries. When you go out and you go to a vendor and say, I would like to buy a crane or a power plant, what you get is a ton of different devices, a system of systems that works together. So you get the crane, and then you get the crane's motor and control systems, and that includes things like TLCs, but then the crane also has to have computers that display its status and can control it. And some of those are remote, and then there's the network infrastructure that connects to those devices and maybe the servers that update those systems and do things like domain services for them. So you get a whole network with your crane. Now, if something goes wrong with those systems and you say boo your control systems or your safety monitoring, you have to shut down at a bare minimum. It's a really big deal. At a worst case scenario, you're getting people injured or dead. So the systems, the systems of systems, as I call them in that process, are tested for like a year by the vendor in safe environments before they're put anywhere. They're duplicates of one another, they are structured and engineered to a super low latency and function in a very, very specific safe way. So if you want to upgrade those systems, you don't just swap out one of the Windows computers. Um, that is something that is tested, vetted, warranted by the vendor. And when you want to bring down, let's say, the power plant to do that, that's like, you know, you have to switch down power generation to another facility for a period of time and you have to shut down your power plant and you do that maybe once a year. So you have that consideration of not wanting to make the system have greater latency or instability or function in an unexpected way. And you rarely have outages where you can even do that maintenance. So these systems are expected to function much longer than the computer on your desk will. They are expected to have like time near even more life cycles for the Windows computers included with them because they were tested for two or three years before they even got sold.

SPEAKER_03:

Well, it's like I'm really curious to hear about, you know, some of the uh memorable cases maybe that you've you've worked on.

SPEAKER_01:

Well, I'm an incident responder and I do consulting incident response. And so I'm under very, very strict MDAs. Um tell you generalities. I can't tell you about specific cases. Like in terms of generalities, I can kind of divide my cases into three categories. The first one being commodity stuff. Ransomware impacts these environments too, increasingly so, because people have realized that this is a target-rich environment. But when it goes down, people really, really notice. So they've been targeting industrial verticals more often. And the ransomware doesn't necessarily again impact like those low-level PLCs running custom firmware, but it definitely takes out all the systems that let you control the crane and or make sure your oil platform is safe to be on. And that's a really, really big deal operationally, too. It doesn't matter that it's not the PLCs.

SPEAKER_00:

The adversaries are getting better, the systems are getting more connected, but we're still in that legacy mindset of the old operating system, and you know, it's comp it's complicated to um update because of so much testing that has to go into it because of the life safety aspects, and it's likely not going to get any better from the adversary standpoint as things become more connected. Do you see a solution on the horizon that would maybe reduce the testing time or force the manufacturers to not be able to run on legacy software? Like what are your thoughts there? Because otherwise it's just gonna get out of hand uh if it's not already.

SPEAKER_01:

There's a lot of different elements to that question. So the first one is legislation. Legislation is going into place in these environments, especially things that we would consider critical to our society globally. So not just in the United States, in Canada, in Australia, in Europe, in the UK, things like that. There's been um a push towards saying, well, you do have to have an incident response plan, and you do have to have some basic cybersecurity controls. You do have to have some kind of detection appliances and some types of segmentation and some plan for updating systems that you can update. Um it's hard. It's a it's a balancing act. Uh and organizations, they watch the news too. They know that there are threats out there. They know that there are um increasing cyber attacks that have made a dent in these organizations, and they don't want that to happen from a risk perspective. They're concerned about the same things. They've always been concerned about the process going down, people dying, their their business not functioning like they expect, being able to produce things like they expect. So cybersecurity is just one element of that, and they they realize that, but they are balancing that out with other risk and also the practicality of being able to do things that impact their process, like update systems, install monitoring, things like that, and the expense of that. So um what you've got to do, you've got to shift your thinking here. Um, as cybersecurity people were just concerned about like hackery stuff, like is the domain controller compromised? Is there an infection? Is there an intruder on the network? In these cases, your priority is life safety and process. You can have a network that is infected on every computer with five pieces of malware, and if it's not impacting the process, it would be more, more impactful, worse for the the process for you to shut it down to fix things.

SPEAKER_02:

So as we're thinking about our infrastructure and vulnerabilities, I was wondering if maybe uh Leslie and you, Eric, could shed some light on this question I have about like how vulnerable are our public infrastructures or just infrastructure in general, really. Like most people like me don't think about it every day. Obviously, we just kind of count on um the power going on, the water working. Um how often are we seeing threat actors attacking these um attack surfaces? And what's your take on that?

SPEAKER_01:

It's increasing. All all those three types of cases that I I talked about, the the insiders and the commodity actors and the state actors have all realized that this is a really good target for their various purposes. Um, because the cases make the news. They're they're they're very noticeable when an industrial product uh process is impacted by some type of cyber thing. And that's a huge noteworthy thing. We've seen it used as um a tool of war in Ukraine multiple times, and we've seen it uh impact operations of major worldwide manufacturers. And so for all those purposes, whether it's money making, sabotage, espionage, or just you know, getting back at an employer pissed off at, that's they're they're all uh viable cases, and so people are doing it more. And um in in a lot of ways, for like the sabotage and espionage and even making money, like it's it's a more efficient way to do that. And there these are not usually like teenage bored teenager, like that to have the capability to conduct these types of attacks and the knowledge you need to do it not as a hard role. Usually you're a pretty well-resourced organization or team of people, and you have a objective and a mission. You're a business and you're trying to do things as quickly and efficiently as possible. And if you have an environment that's relatively exposed and it's running XP embedded and it's mostly a flat network like 2010s and uh doesn't have EDR on it, and it's also doing really important things, it becomes a very viable remote target for all the things that I mentioned.

SPEAKER_00:

I think one of the one of the things that you know in corporate America we're used to is the structure of corporate America, where you know we we have meetings and you know, we we have people that we report to and we have uh reviews to give and you know, all of those other sorts of water cooler conversations and all of those sorts of things. Um and the the amount of time that we're dedicated to either threat hunting or um looking in the environment for things that we might want to improve or you know, what have you, is kind of all part of our job in information security. When we have those pointed threat actors, while they do have the they're very disciplined and and if certainly if they're nation-state well funded, then they're not gonna have the same constraints that we have, where, oh, I want to reboot this system, you know, oh, I got to go to CAB, gotta get it approved, gotta wait two weeks, you know, all of the sort of nonsense that we deal with in corporate America. Just think about operating if you didn't have any of that. You didn't have to attend any meetings that wasted your time. You could just sit and focus on one objective, how efficient you would be. So then you take the the side where we work eight hours a day, nine hours a day, roughly, and we have all of that nonsense that we have to deal with. And then you have the adversary that doesn't have a lot of the nonsense and isn't constrained to eight hours a day, and you could just see it's grossly unmatched. And it's really hard for corporate America to kind of wrap the mind around well, you know, there's this adversary that we can't see, we don't know who they are, we don't know how they act. Um, they're going after us, and sure, we can see the IOCs come through on the tools, but we don't really know what that means. And there's very few people in the organization that actually know what it means, and those people are are trying to defend the organization as well as do all of the other corporate stuff, it gets really hard. And then you could compound that with if they really are going after that organization and through social media, they could identify a person that they directly want to target, either through cyber attack or a bag of money, getting an insider um to help either knowingly or unknowingly attack that organization is really difficult. So um I think to answer your question, Josh, it it's really difficult on the corporate side when you have a motivated adversary.

SPEAKER_03:

And then you add in the fact that these threat actor organizations have like HR departments now and all kind, you know, all these resources, unlimited funding, you know, of basically, you know, it we're we're way behind the eight ball on that.

SPEAKER_02:

So Leslie, is that is that kind of where you're the tip of the spear and and and is that what you do at Drago's? Is that kind of your role to to kind of head up those campaigns and be a little more proactive so you're not stuck in a boardroom meeting and you're you're like actively hunting these kind of threat actors down, or is it more of a response type uh of a stance?

SPEAKER_01:

I personally, not not Dracos as the organization, but I personally am much more on the reactive side of things. Catastrophic things have happened. If you see me, I joke to my to my SANS class students, like, um I I understand if you never want to see me again, because uh it's very nice to get to know people in the community, but if I'm there, it's been a very, very, very bad day. I do do some like tabletop exercises and incident response planning services, and I of course teach. Um, but my work is very reactive. We have people who do proactive services like OT or operational technology process specific pen testing, architecture assessments, planning, audits, things like that. And that's very specific to their nuances in these industrial environments. And yeah, I mean, these are challenging environments to secure for a lot of reasons. All the reasons I talked about, like the things you can't touch, the things you can't alter, you have to get much more creative in doing the proactive security in these environments. Again, you cannot come in and just like update everything. And uh just to give you a sample, and I'm not trying to pitch something, but it's a good way to understand these types of environments because if you're if you're coming into them as a cybersecurity person and you're responsible for one, there's a white paper that came out through SANS a number of years ago called The Five Critical Controls for Industrial Cybersecurity. And the to understand the level that most organizations are in here, it breaks down for like leaders in that space, like five things you should start doing in these environments. And they are defensible architecture, like not having a flat network and having some segmentation, and then having some kind of incident response plan for these environments, having some control over your remote access and knowing what it's doing and where it's going, um, having some network monitoring, like something, like at least like passive network span port-based monitoring, and finally having some knowledge of what you can and can't update from a vulnerability management perspective. So they produced a whole white paper, and it's like pivotal in the industry right now. And it's like, this is where people are at. It's not like, oh, we're gonna do AI, ML, next gen, like whatever. It's like, hey, maybe, maybe we should start segmenting these environments. That'd be real good because adversaries are very aware of them now. Um, but that's like we're meeting people where they're at. Like they don't even have network maps or acid inventories in a lot of these environments. So um, you're not gonna tell them to do whiz bang 2025 security measures. You're gonna tell them to do foundations and fundamentals. And yeah, that makes the job challenging for the proactive people. Um, you have to get real creative and you have to think about a lot of the devices and protocols in these environments staying inherently vulnerable. There's no way they're ever going to not be vulnerable. Industrial protocols aren't encrypted, they on design, because again, if you hit the big red button on the wall to stop the process if somebody's arm is getting chopped off, you don't need like extra protocol latency in that or like a potential point of encryption failure. Like the button needs to work. So um, yeah, you you don't add a lot of security controls to a lot of the lower level devices, so they have to be vulnerable. So that means you have to think about creative solutions to keep those bastions very monitored and very secure.

SPEAKER_02:

That's really cool. Um I we also you know talked a little bit when we were getting to know each other about the AI panic versus the reality and how this kind of dovetails into the new threat surfaces that are appearing. And um everyone's either terrified of AI or is or is thinking it's magic at the current current state of things, it seems like. Um do you actually use it in your work or or how are you seeing that manifest in your day-to-day job?

SPEAKER_01:

I'm I'm more on the AI skeptic side of things. I'll tell you why. I I worked on AI in the 90s. Um I've I've been working with AI for a very long time and I understand very, very well how it works. And it's of course frustrating to a person who spends a lot of time understanding how the world works and how things impact our society to run into a technology that people grossly misunderstand in very dangerous ways. Um, in terms of using AI for my work, uh we've been using machine learning and detection and forensic analysis for ages. For well over a decade, we've been using it to analyze logs. What things like LOMs are good at is looking at big data sets and pulling out means and averages, getting answers to the most consistent common answers to questions. And of course, that's something we do in cybersecurity, looking at massive forensic data sets and logs over years, things like that. And we've been using machine learning for those purposes for a long time and it's gotten better at what it does, and it eliminates some tasks that you'd used to have to be done manually by analysts. And that's great. Um, if you understand what it does and doesn't do, and what it's doing is taking big data sets and finding the most common thing. And um, from a misunderstanding perspective, a lot of people think it's conscious, a lot of people think it thinks, a lot of people think that um it's creative and it's none of those things. It takes a big set of data and it pulls out the most common answer. Um, it is a screwdriver. You use it for the purpose it's it's good for, and if you're using it for other things, you're probably gonna jam a screwdriver into a light socket. Um, but anyway, uh in terms of what else I see see it doing in my job, I see it doing a lot of nefarious, horrible things in a space that I'm in. Because adversaries know what it does. They know that it's good at taking a big set of data and finding the most common answer. And what are some common answers that somebody might want to look for when they're doing like sabotage to industrial facilities? Well, um, if you were a, I'm gonna use it, I'm gonna use the term take a drink, APT, if you were an APT like 10 years ago and you wanted to like say poison the water. Um, so you wanted to break into a power plant and into a water treatment plant and you wanted to increase chemical levels or decrease chemical levels, alter them so that you could kill somebody, um, you'd have to go out and find a chemical engineer. You'd have to blackmail one, extort one, something like that, or you'd have to do a great deal of research as a team, but usually it involved finding some way to get subject matter expertise and maybe a lab uh to test your theories. Now that's still the case. Like the really well-resourced states still have like chemical engineers, electrical engineers, people who are specialists on target industrial systems, who they gain because they're nationals of their country or they're extorting, et cetera, et cetera. But now you can use an LLM and you can say, hey, if I had this model of power plant or this water treatment plant, if I had this municipal facility and I wanted to get into the water treatment systems, and I wanted to change the level of, say, chlorine in the water to kill the most people possible, what would that level be? And it's gonna go across all that that snarfed up internet data, and it's gonna say, well, the most common answer to that is XYZ. It's going to extrapolate that. And it might be wrong, it might be hallucinated. Who's hallucinating? It might be right, but uh that might be right possibility is real bad there, and you didn't have to call a chemical engineer. And uh then you need some some custom logic for that device to do that once you get onto it. And that's like weird ladder logic stuff that doesn't look like normal programming. And uh, but you can go on to the LLM and you can say, hey, can you can you write me some logic to increase the chlorine levels on a Dys YZ model that I see on this target environment to increase the chlorine to this value? And yeah, it'll write you some ladder logic. It'll go through Stack Overflow and and years of posts about industrial stuff, and it'll do as fast. And again, it might be wrong, might be right. And so that's a alarming development on top of the normal like LMMs for phishing, LLMs for for polymorphic malware that we're seeing in the rest of IT. So it's mostly mostly a bad thing in my world. Um, it's either something that's being woefully misunderstood for defense, or it's being used in really gnarly ways by adversaries.

SPEAKER_02:

I've seen that recently pop up that there is like an online cult now that thinks that AI is like some kind of a god, and it's actually creating kind of a movement of people that are are now guagulating around this idea that they're spending time talking to these LLMs and it's Where are you on the internet, Josh, that you saw this? Well, I'm going deep, man.

SPEAKER_03:

Believe it though. I have not heard that before.

SPEAKER_02:

Yeah, there's actually like it's I can't remember what it's called, but it's it's an AI religion.

SPEAKER_01:

So people are falling in love with them. Like people are getting married to them, so it's not very hard to see.

SPEAKER_00:

Yeah. We're gonna need another tinfoil hat episode.

SPEAKER_03:

I'm getting worried we're going down that route, but Josh, you gotta write. Sounds like Leslie, that's not a surprise to you. No.

SPEAKER_01:

Oh, I I I worked on I worked on primitive chat bots in the like Eliza AIML days, the very, very early days. And I was a I was a vulnerable teenager back then. You know, I'm I'm old now, I'm very old. It updates me a little bit. But um, even then when I had to code everything myself and I understood the the the holistic engineering of how that bot was working, it was still tempting to try to make friends with it. It's like, oh, if I meet a person, like and I can talk to it. And uh yeah, emotionally, yeah, that's very, very tempting to get emotionally attached to as a human being. And uh I I cannot even imagine being a vulnerable teenager today and working with LLMs that can appear very human through emulation. The Turing test was such a bad idea because the the smart people who thought, you know, created the the concept, the the common understanding of the Turing test, um, never anticipated people being happy with emulation as opposed to consciousness. Um and uh the the Turing test is supposed to test for consciousness, but what it effectively also mistakes for consciousness is just very, very good emulation of being a human being, what you get from exactly what it's described, taking a large data set of what every human being has responded to every question with and giving the most common answer, the most typical answer.

SPEAKER_02:

Eric, have you seen this AI stuff pop up in your in your like defense posture? Have you seen like any kind of unique attacks come through? I mean, we just saw that one in St. Paul. I'm not sure what the update is on that. Um, but I'm curious to know like what your experience has been in the last couple years on that.

SPEAKER_00:

We saw a neat one, or we were talking about a neat one last week where it was a side channel attack through a vendor that was using AI to write prompts on the end user's device to gather information and then encrypt it and send it back. So that that one was was pretty interesting, but it the the compare and contrast to me is just really stark and drastic. Where earlier today I was in a meeting um on the local government side with a with a local entity, and the discussion for a half hour was around um how do we govern an a trial of AI. So there's such a reluctance to even leverage the basics that are already around us, just figuring out how do we get these tools in the environment so the different teams can understand how they might use them in coding or how they might leverage them to you know to even do things like the basics around um making a document, um, translating it into a different language, or um you know, making it maybe more accessible for readers of different um levels of um uh language ability, right? So just some of the really basic things of AI, not even able to have those conversations because they're hung up. About how do we bring in AI into the organization in a way where, you know, it's not gonna exfiltrate data or the the data's not going to train the model, or you know, all of those questions that were answered, you know, a while ago are still being discussed. And I, you know, it would be okay if this was the first meeting that I sat in, but this is probably the 20th, where the topic it's the same topic. So, you know, here we have the threat actors that, you know, are are well versed in LLMs and machine learning and you know, the the the people that are are are are really diving in on the forefront, you know, bleeding edge. And then on the side where we're being attacked, we can't even figure out how to leverage it in the environment at the very basic level because we're too worried about what it what what the possibility of what it could do. And every day that we don't bring it in, we're falling farther and further behind as far as attracting talent and letting people really learn and develop a skill set around the tool. So then they have to do it on their home machines because people, you know, they want to learn and they want to get better. So now they're doing it on their their home machines and likely taking data sets with them that they shouldn't be taking with them. So it's like, you know, it just it gets frustrating sometimes around where we are, where we want to go, and how we get there.

SPEAKER_02:

Well, thank you so much for taking the time. And I know it's early in the morning, so thanks for getting up out of bed and and joining us here on the audit. Um, we really appreciate having you on and and spending some time chatting about this stuff with us. So you've been listening to the audit presented by IT Audit Labs. My name is Joshua Schmidt, and you've been joined by Leslie Carhartt. And we also have our managing director, Eric Brown, and Nick Mellum here from IT Audit Labs. Please like, share, and subscribe and check us in the next one.

SPEAKER_00:

You have been listening to the audit presented by IT Audit Labs. We are experts at assessing risk and compliance while providing administrative and technical controls to improve our clients' data security. Our threat assessments find the soft spots before the bad guys do, identifying likelihood and impact, while all our security control assessments rank the level of maturity relative to the size of your organization. Thanks to our devoted listeners and followers, as well as our producer, Joshua J. Schmidt, and our audio video editor, Cameron Hill. You can stay up to date on the latest cybersecurity topics by giving us a like and a follow on our socials, and subscribing to this podcast on Apple, Spotify, or wherever you source your security content.