Manufacturing facilities have never been more vulnerable to security breaches. As cyber attacks continue to rise, find out how you can be one step ahead with Work Cell Security.
Interested in learning more? Read Dave's blog.
Manufacturing facilities have never been more vulnerable to security breaches. As cyber attacks continue to rise, find out how you can be one step ahead with Work Cell Security.
Interested in learning more? Read Dave's blog.
Mara Fowler (host): Welcome! Today, I'm going to be speaking with Dave Cronberger, who is a business solutions architect within the industry solutions group at Cisco. Dave has been with Cisco for over 20 years, and has experience working in discreet and process manufacturing within the automotive, chemical, consumer package goods, and other industries as well. He's a huge car enthusiast and Michigan State fan. Welcome, Dave Cronberger!
Dave Cronberger (guest): Go green, go white! Thanks. Very much a kind invitation and introduction. How are you today?
Mara: I am doing well doing well happy that it's Friday. How about you?
Mara: Awesome. So what we're going to be talking about today is cyber security, which is a topic that you are very familiar with. So just to kind of give a bit of overview about the state of cyber security within the US in the world and manufacturing. So, within the United States, there are thousands of data breaches that happen every single year and that number has been on the rise since the early 2000s. Over 100 million Americans have been hacked or had their data stolen in 2019 alone. 8 billion consumer records have been hacked according to CNBC. So, 48% of manufacturers, have, at some point, been subject to cyber security incident, and half of those organizations suffered financial loss or disruption to their business. So, obviously this is something that is not going to go away anytime soon. This is still a very big issue that is on everybody's mind. So, Dave, from a cyber security perspective, what do you think is the biggest threat that manufacturers face today?
Dave: Frankly, Mara, I think we have to start with the directed attacks. So, in other words, intentional attacks where the motive is motivation is either espionage or the most disruptive of course, is a malware attack for ransomware and that is really that latter one is really got the highest amount of risk because it's the one that they'll use to close down the plant, which then, in turn motivates the victim to then turn around and pay the ransom.
Mara: Right and so do you think that those kinds of ransomware attacks that you've mentioned, they've obviously been in the news we're all very aware of, like the Wannacry example. Do you think this is something that, has been increasing and will continue to affect not only, you know, the average person, but manufacturers as well.
Dave: Well, I think manufacturers are certainly a major target, and we've actually heard about some of the things that happened during the crisis. And I certainly won't repeat a lot of that because it was so prevalent in the news. When you saw various auto manufacturers and other types of manufacturers shipping companies that were victimized by that attack. And I think, we have to remember that viruses and malware have been around for a long time, but oftentimes they have not been the kind of software that actually debilitated or stopped production. They were just kind of a nuisance. They created a lot of traffic on the network, but they really didn't stop production. And so manufacturers, historically, while they knew that they had contaminated networks with a lot of viruses and various types of malware, didn't do anything about it because the plant could still run and they could still make their product. They could still ship their product. What we're seeing now, in these newer, directed attacks is the ability for this malware to actually shut down the system to take it offline and then, of course, make a splashing announcement, say, and pay me so much in Bitcoin or some other way. And I'll turn you back on again and they're, in your systems now. The problem with that is sometimes they don't re-enable it. They just take the money and run and you're left, you know, quote holding the bag so to say. And so that is a big change and so and it's because of the experience of Wannacry that we're actually seeing some customers that are really moving forward with looking at various solutions, like our work cell security solution to mitigate and prevent these attacks from actually causing severe damage and mitigate damage, right?
Mara: Yeah. So you mentioned that, that and that idea of Work Cell security, or securing the cell in a manufacturing facility. So, for those of the listeners who aren't familiar with that concept, can you go into that a bit more?
Dave: Well, sure, manufacturing is typically broken down into a series of processes. I'll use the automobile industry, because I'm most familiar with it, but this is this could be true for any industry. It doesn't whether you're making cookies or consumer products, tissue paper, it really doesn't matter that much. What you have is a set of processes that are segmented that take you from raw material to finished goods at the other end of the plant and because of this logical segmentation in the manufacturing process, and due to some new networking technology and techniques, we can segment these processes in different ways, right now, with what we do with Work Cell Security is we use tags to classify the products and I want to be clear: this doesn't have to do necessarily with addressing or other ways that people can segment a network. This is really about how do I tag information and identify that it should talk between these two things, but not talk between these other two things or other five things. And so it's a methodology with a set of tools that helps you classify the traffic, and then can strain the conversation to that, which is appropriate and not permit conversations that are inappropriate. So, for example, a Windows based HMI should not be talking to an administrator's laptop in the front part of the factory, or outside of the factory for that matter, right? There's just no need for that connection to have taken place. There's no good reason for it. And so it's important to be able to say, okay, these things should talk to each other we're going to let that happen any time they try to talk to anything else we're going to prevent it. Okay. We're going to block it from taking place. And then that way, if a technician, for example, walks into the building, and plugs into a piece of equipment to do some troubleshooting, maybe loads new software and inadvertently contaminates the equipment in that cell. As soon as that equipment that's been contaminated, or that person's laptop begins to omit traffic that's trying to talk to other things, and contaminate other things when the routers and switches that have been enabled with the secure group tag evaluation. They look at that and they go well, you're not supposed to go there. So I'm going to drop the packet. So, that way, you may not get off scot-free where you know, nothing is contaminated, but what's good is the contaminated. Equipment is very confined. It's in a localized area and it won't cause the remediation of hundreds or thousands of pieces of equipment it may be 10, it may be 15, it may be five may be one, but because it was confined in that way, it's faster to remediate. I'm able to get my production backup because the reality of it is, if I contaminate one cell, I could probably stop production. However, how long does it take to fix it? And if I'm fixing thousands of things, it could take forever, a really long time if I'm fixing one, two, or three of something. I can probably do that a lot faster.
Mara: Right, you bring up a really great point just about the different, aspects of how manufacturers are really vulnerable to attacks. So, can you do a two-part question here? Can you touch a bit more on where you think manufacturers are most vulnerable? And how Secure Cell this Work Cell Security concept really will protect them from those from those vulnerabilities?
Dave: I think one of the classic things about attacks and attack vectors has always been that the highest risk comes from within your own business, within your own company, the disgruntled employee, the employee that's careless with what they connect to, and, you know, did they go then? Well, I mean, we've had situations where people sprinkle USB sticks in parking lots in the hope that people will pick them up and one of them will stick it into a computer. And there you go the viruses in the system and so what you want to make sure of is, you know, how do I protect myself from the burglar that's already in my building? All right, because perimeter security has been the classic mode of defense. They're on the outside, I'm going to build a wall. I'm going to put a moat in front of that wall. I'm going to make it really hard for them to cross the moat, to get through the wall. The real trick is what happens when somebody managed to go through the gate look normal, going through the gate in other words, the front door of the building, for example, or they're an employee that's been there for a long time and they're the one either because of or they're a victim or they're a perpetrator who causes the badness to happen the virus gets in the malware gets in, the damage is done, right? And so, this is what Work Cell Security is about protecting you from within and localizing and segmenting down your area of exposure as much as possible.
Mara: Right, and you bring up a great point, just how we have this, evolving understanding of how to best protect ourselves from cyber security threats. Right? This idea of protecting the wall so to speak, and now kind of like, protecting the little gates inside within those walls. So, how do you think, going forward, the cyber security tactics are going to evolve and that Secure Cell or Work Cell Security can continue to protect and to prevent those attacks in the future?
Dave: Well, we're going to have to always be aware of the kinds of attacks and how they operate. And one of the things that you have to do with Work Cell Security is you have to understand your own traffic flows. In other words, you have to have an understanding of how to classify things so that you can say this kind of a connection, and this kind of communication is appropriate and this kind of communication is inappropriate and that's always going to be evolving because you're going to be changing processes. You're going to be moving equipment around and the good news is, is that Work Cell Security because of the tools that we use relative to the creation of dynamic tags, for example, and the type can be dynamic as well as static we're able to react and compensate and adjust for these changes. Now, some of that stuff will be have to be planned and some of it will be able to happen as a matter. Of course, based on how our software and systems actually work. But it's always changing, it's always evolving and therefore you always have to be alert to the impact of a change on the method and manner that you are using for your security. So that in some cases you might say, well, this conversation between these two devices is normally not allowed. But under this very specific set of circumstances, I need to permit it. And so therefore ,you refine the classification and the way that this thing operates in terms of permitting versus denying, and the other thing to remember is Work Cell Security is based on what we call a whitelist approach what does that mean? It means specific things are allowed everything else is denied most security postures often work on a blacklist approach. This is what we know to be mad, bad, mad and bad. You know, into that, that category and so we deny it, we prevent it. We block it. Everything else is allowed. So there's two completely different ways to approach how you permit or deny traffic, and we feel that because of the way that these attacks have evolved, you really have to go with a whitelist approach, which is what secure group tagging does and it says, okay, this is what I'm going - this is what's okay and this is everything else can't do it right now. There is a process of refinement that you'll go through, but the point is, it's really interesting because the other thing to keep in mind with Work Cell Security is, I don't necessarily have to think about just blocking bad traffic or malware based traffic sometimes, I can use it to suppress broadcast traffic that is just making a lot of noise on my network and it doesn't really need to go everywhere. And so I can contain some things that I consider normal, but can kind of be a nuisance under a bad conditions and help out in that regard.
Mara: With broadcast traffic, can you go into that a little bit more and explain what you mean by that?
Dave: Well, broadcast traffic is where I want to talk, I want to send a message to everyone. It's, kind of like, you know, omebody on a megaphone in front of a crowd saying go this way to the exit or go this way to the auditorium or whatever or sorry, Elvis left the building, right? That's a broadcast. Okay whereas Unicast traffic is it's a 1 to 1 communication. So, for example, this conversation that you and I are having in making this podcast is a Unicast conversation. I talked to you, you talked to me and so, the question then is in the Unicast traffic what's appropriate what's allowed in a broadcast that's the same thing. Do I really need to send that broadcast to everybody? Or can I contain it to a proper audience and block it from going beyond that? So it's a way of lessening the traffic that goes out on the network, especially when you have large flat networks. So, anyway, the point is, I can contain and improve the performance of my network, as well as secure it based on what I tag, identify, permit, or deny and where and when I do the permission.
Mara: Okay, so you mentioned a little bit earlier about this idea, this concept of whitelisting versus blacklisting. Okay. Can you go into that a little bit more?
Dave: Yeah, sure. The thing that we're starting to find out about these viruses and a lot of things about malware is they're very sophisticated and how they engineer and attack and they sometimes can probe to see where there might be an opening. So when you lose when you use a blacklisting approach, where it's a deny some things permit everything else and the virus finds that okay, well, this didn't work. This didn't work. Let me try this other thing and it works. Then they've been then they're going to be able to be successful. All right, and it also is a situation generally where the targets become unlimited by comparison. Whereas if you do whitelisting, which is a model in, which everything is defined that's permitted, and all else is denied. That way if they make a probe, and they're successful at getting at something that the whitelist allows, you know, allows, they'll get to that, however, when they start to probe for other things, using other P ports, other ports, different protocols and trying to find things and realizing that it can't get a response, right? So then it's basically stifled and it's an important distinction because the old way of access control lists and, you know, permit or deny a few things and allow everything else is not going to work anymore because of the sophistication of the software and the fact that these folks are engineering these viruses in order to get around that kind of a defense, which also means that we have to be on the ball in the context of what we whitelist or what we tag in terms of what we're permitting and what's, you know, and how that's working because in the case of manufacturing, I may have a connection between two devices that are inside the cell directly, and then I might have to make a connection now and again to a device that's in the plant but it's not in the cell, so I have to go across a router hop, or some kind of a network boundary to get to it and I need to make sure that that isn't just a wide-open connection. In other words, if I'm going to talk to that device, I'm going to use this protocol, I'm going to use this port number, I'm going to use this, I'm going to use this, etc. And then under that context, I can talk to him anything outside of that I don't allow the conversation to take place and I also alert so, the other thing that's important is, it isn't just that you permit or deny. It's that when you have an event, that's an irregularity. In other words, I've denied this, but something keeps trying to talk to it. I need an alarm and an alert and that's the rest of the solution is we have these other tools that we take and are very looking at very closely. That's one of the reasons that we use net flow. On our switches as a part of the solution because what net flow does is it gives us the characterizations of the conversations both the good and the bad. And then that's used to inform how we create the tags and how we implement the tags and so therefore we're feeding that information into tools like stealth watch or Cyber Vision and allow that understanding of those conversations so that we can define: this is a good conversation, this is a bad conversation. And then mitigate that as required over time. Does that make sense?
Mara: Yeah, yeah, no, it definitely makes sense. I mean, so based on what you're saying, it seems like this idea of Work Cell Security, the whitelist, blacklist changing, how we approach cyber security - it sounds like as a game changer for manufacturers.
Dave: I think it can be and the thing that manufacturers have to understand is what works on security just like any other security technique or technology doesn't help you after the fact in other words, if you get attacked and the viruses there and it's percolating through your network it's probably already done its damage. Now, could you implement and enable Work Cell Security in that environment and begin to stop the storm of bad traffic yes, you could all right? But understand that while Work Cell Security is essentially a preventative technology implemented to anticipate that you'll be attacked. What we know statistically is, the attacks are increasing now it's not that one. Yeah, now, if you already have malware in your network, and it's already cruising around and getting into things and you then implement Work Cell Security, sure, it will suppress that traffic. It will begin to make it diminish, right? But you still have the business of remediating your systems, all right?
Mara: So, let's dig into that scenario a little bit more. So, let's say, you know, manufacturing facility gets attacked. They have some kind of ransomware, demanding X amount of Bitcoin, right? They're able to clean it up as much as possible they bring in this idea of a solution for Work Cell Security implemented and let's say that this hacker tries to come back and do the same thing again. How is that going to change the response from any for this manufacturing to set up facility?
Dave: Well, the big difference is the last time it had access to everything. This time, it's going to be constrained at the point of entry, all right? Now, yeah, you know, it's possible for there to be multiple points of entry but the thing is, if I restrict it to a very small chunk of my network, generally, in the context of a work cell, because usually, when troubleshooting occurs and machines break, they break in one place. They don't break all over the place, it’s just one machine breaks in one place. And then, the technicians coming in remotely or physically and it's at that point of making a connection to troubleshoot it, that there's a possibility of contamination. The idea is if I can then bring that down to that local entity. I'm going to have a little bit of exposure, as I said before, but I'm not going to have massive exposure and the real value in that is that the, the cost and difficulty has to do with how do I remediate a lot of equipment versus how do I remediate a few pieces of equipment? And so, what we're trying to do is provide a scenario where, if there's going to be remediation, it's a very small segment of the network.
Mara: Okay so this is what you're, what you're discussing is a relatively new innovation within a cyber security space, especially from manufacturing. So how much do you think going forward, how many manufacturing facilities do you think are going to start using this approach? Is Work Cell Security approach to their cyber security strategies?
Dave: Well, I think obviously what we're going to see is those that are hit hardest and in the near term, just like, we've seen some of the Wannacry victims have started to look at this technology now. And we expect that, unfortunately, it will get largely driven by people that are victims of attacks.However, we also believe that as other companies begin to implement this, because this is a new way to do things. And so, as the reputation of security gets out, because of the fact that it actually works. And it actually prevents a problem will drive the implementation because it's because it will cost justify how many billions of dollars can I afford to lose in lost production versus how much does it cost to put in Work Cell Security. And frankly, security, in relative terms, is not very expensive, because in, in the case of Cisco, the feature functionality that you need to use, comes built into the product. The, by the switch with the capability to do it, then it's a matter of getting some of the additional software that you need Identity Services Engine Stealthwatch, Cyber Vision, etc. to actually enable it and run it and operate it properly. So, I expect, I think the adoption rate will begin to accelerate when people realize the power of this thing to simplify the plant floor. And most importantly, the remediation of the plant floor based on an attack.
Mara: You bring up a good point because that's always, I think the bottom line within manufacturing facilities is the idea of downtime and, losing money, lost production time, all of that. Do you think that you said, initially, do you think it's going to really be a case where people who have been attacked, have faced these kinds of ransomware and Wannacry attacks are going to be the early adopters but how much do you think, broadly speaking, how much time and money do you think this this could save manufacturers by implementing this kind of security and this technology?
Dave: It might take two hours, but it's a measurement of a of a step in the process of the manufacturing of the product. And then most manufacturers break it down to in a cost per minute mode, at least in the discreet space. So, for example, an appliance manufacturer, their tack time might be, you know, 30 seconds and their 30 second cost is $700, right? Then you could have attack time of 40 seconds with a luxury car maker and that tack time is 40,000. right? And so, it really is a function of how many minutes of downtime happen because of the attack. And so therefore, the ability to identify there's an attack block, the attack and remediate the attack is really, really important. Because it's certainly a lot easier and a lot faster to remediate. For example, two or three devices in a cell than it is a thousand devices across that plant. So, that's where the cost savings comes in, and the value proposition comes in and in light of the fact that, you know, there's some software that we ask you to implement and take advantage of the existing features of, of some of our industrial switches...It kind of makes its own case. I think it solves itself.
Mara: It does, it does. Yeah. Okay. So, as you're mentioning earlier, you know, cyber attacks on the rise. It’s become very people are very aware of it in the public sphere right now, just within the manufacturing industry. So, do you think that helps increase in awareness? But you think that idea that this has become a lot more common within the public sphere is influencing manufacturers and how they're reacting to cyber attacks and how they're prioritizing investing in them?
Dave: As far as cyber security I mean, I think the big change in all of this is that manufacturers have been attacked, they're recognizing that they're a prime target for attack. They're therefore recognizing they owe it to their shareholders to make sure that they just didn't ignore it and then leave themselves open. And so, what we want to offer them at Cisco or from Cisco, is the idea that there is a solution that they can apply, right? And so, as they refresh their network or because of other motivating factors proactively move to upgrade their networks to be more secure were there for them with all of the right tools. At the right time to really help them address these concerns. So, I think, you know, it's kind of - I used to fly airplanes once upon a time and there's an old adage about how there's the pilot that will land with his landing gear up and do a belly landing and then there, the rest of the pilots are those that will land with their landing gear. So, it's kind of the same thing with security and cyber attacks. It's like, you've been attacked, or you will be attacked, but there's no, “I'll never get attacked that's off the table,” right? And so, it's an important thing to recognize. And so, you owe it to yourself to protect your shareholders to protect your workers from the harm that these things can often bring.
Mara: You want to make sure you have the right landing gear, so to speak.
Dave: Exactly, that’s exactly right.
Mara: Okay, so Dave, I'm giving you a challenge here. You got 30 seconds to pitch Work Cell Security to manufacturers, what do you say? Go.
Dave: You have been attacked, or you're about to be attacked. What we're going to offer you is a way to keep the cost of that attack down and your factory production time up.
Mara: Wow, you did that in about 10 seconds. Efficiency, I love it. Great. Okay, well, Dave, I really want to thank you for your time. This has been really informative. Do you have any final thoughts before we sign off?
Dave: My final thought would be take advantage of the technology that we bring to you in our network products that get you most of the way there to Work Cell Security just take the time to understand this solution and look at what it would take to carry you the rest of the way because it could save your business.
Mara: Well, again, thank you for your time, Dave.
Dave: Been really fortunate, really appreciate it. You're welcome.