Jim Labonty on Data Center, Manufacturing Cybersecurity

Former Pfizer head of global automation engineering Jim LaBonty joins the Nexus Podcast to discuss an article he wrote for Nexus on the need to secure data centers during kinetic conflict. He also explains the interlock between data centers and manufacturing facilities, and why a cyberattack against a data center can be devastating to the uptime and reliability of factory floors. 

This interview was pulled from Episode 2 of Nexus Digest, a monthly recap of content published on Nexus. 

Subscribe and listen to the Nexus Podcast here. 

Transcript

SPEAKER_01 0:27

And it's my team Jim T. Jim is the former global head of automation engineering at Pfizer. And he's brought some great insight via his blogs uh into manufacturing cybersecurity. So that's uh what we're gonna catch up with uh Jim today. How are you doing, sir? Good to see you.

SPEAKER_00 0:46

Yeah, doing pretty well. Doing I'm semi-retired, but you know, still still cranking away.

SPEAKER_01 0:52

That doesn't sound like a bad thing.

SPEAKER_00 0:53

No, no, no, it's it's it's actually very good. Sorry, but anyway, highly recommend it to anyone who gets in you know certain age.

SPEAKER_01 1:01

Absolutely. If you can do it, do it, right?

SPEAKER_00 1:04

Absolutely. Don't don't don't hesitate.

SPEAKER_01 1:06

Yeah. Um so let's talk a little bit about let's start with your last blog contribution to Nexus, and we'll link it here in the show notes and in the video. But um, it focused on data centers and how they are inevitably targeted by cyber attackers, even physical attacks as we've seen during the conflict in the Middle East. Maybe you could start by explaining kind of that growing relationship and how manufacturing and manufacturing data and data centers are all kind of interlocked there.

SPEAKER_00 1:36

What's happening? Yeah, no, it's uh it it's all this this um, it's not really new technology. It it there was always a growing need over my 40 years that I've done worked in in the manufacturing and industrial space. There's always been a growing desire and need to really capture and and leverage all the information and data um to get obviously business value uh from it. And so that's only accelerating, right, in in recent years, and majorly, you know, and of course now we have the AI technology, which makes it even easier to harvest the information from all that data and make sense of it and then use it um, right, to to to uh and that goes, you know, from the military standpoint, right? We see in the Middle East that goes into manufacturing to you know optimize the production process, make it go faster, higher efficiency, less people touching things or transacting data from one system to another. That should be, should be, right? Uh, all done automatically between systems. Um, so there is this growing relationship, and and you use the keyword interlock, right? There is an interlock between the manufacturing is providing safe and secure systems and technology. So that that's their responsibility. And this is their responsibility, it's reasonably expected that they provide things that work well and and and cannot be you know adulterated easily, um, and uh and you know, and work for the function they're supposed to perform. That's why you buy them. Um, and of course, that technology produces data. So, like how much data is like, as an example, I you I worked at Pfizer, you know, one biotech suite, one biotech suite that makes a drug like COVID vaccine, for example. Um how many sensors, how many you know, processed pieces of data are coming out of a suite like that? Thousands, many thousands of sensors. That's a lot of information on a second by second basis. That data is coming out all the time. Um, and it's used, obviously, in the production process to make the product, um, you know, and keep it running well. It's you know, the analogy I'll use is a car. You know, here's a perfect example of a full self-driving car. You're in it, right? It's producing a lot of data, hundreds, right? There's hundreds of sensors in the self-driving car today. Sure. Where's that data going? Some of it's used to obviously keep the car on the road, you know, and not hit anything else or hit anybody else around. Um, but it's also going back up, right? It's going back up to a big A, you know, AI system, better improve uh the firmware that that actual car is running from. So, and that, you know, eventually, not while you're driving, obviously, you eventually download the new firmware and you get a better version of self-driving, smarter, better, misses potholes now, you know, important stuff. Um, so that same kind of efficiencies in manufacturing are happening at a at an accelerating rate. Um, and so you would go back to the the manufacturers. Manufacturers need to provide safe and reliable systems. Cybersecurity is a big part of that, right? The defenses around it. You have to have reconnaissance of what's happening, and you have to have protection around that. And protection um is being a little forward thinking um in how we put things together, and we'll probably get into that a little bit as to what our issues are, challenges are, you know, with security and these environments today. But um, you know, the relationship is is a very key one. Manufacturers are safe and secure, users install it properly, the technology properly, and then put fences and put other technologies around all that industrial automation that come from the manufacturers and you know to protect it well.

SPEAKER_01 5:46

Yeah, so there's a somewhat of a shared security responsibility there between the manufacturers, the data centers to protect all that data, and of course, worker safety ultimately down the line. But who's responsible for what in your experience?

SPEAKER_00 5:58

Yeah, so again, worker safety, number one. Number one manufacturing is safety, safety, safety, safety. And that has to always be the fore you know forefront. You know, you always expect people to go into work and be able to go home at the end of a day. Uh, and that's the responsibility of both of the manufacturers. But for obviously, it's it's the actual firm that's um using all the technology. They have to install it properly, they have to verify it's installed properly, they have to operate with it properly, they have to have safeguards um you know around that technology. Um but again, that's that technology, if it's being controlled, it's being controlled by a computer. There's a brain behind it all, and that brain can be can be you know can be hacked, and which is a problem, a major problem, right? And so, you know, around any of these processes, you as the end user of that technology have to do your best within reason. Do your best. You just can't ignore it. Uh and a lot of places, we're gonna get into that, I think, a little bit, is you know, how old is this technology? Right, you know, really quick rule of thumb. Over half, right? Over half of the technology that's used today is outdated, right? It's the end of manufacturer's warranty, it's totally outdated and still being used. And why is that? The life, the life of manufacturing technology, and and that's the legacy automation controls as well. Typical life is 20, 30 years, you know, IT technology five, seven years. You know, there's a big difference, right? Four times uh difference in the life and use of this technology. And some would say, well, you know, just change it out. Well, just change it out.

SPEAKER_01 7:53

Not that easy.

SPEAKER_00 7:55

Like, hey, let's just go to the IT model of uh let's just you know evergreen everything, just put new in every you know, four or five years. I'm like, good God, right? Our example, we'd our estimate was two to three billion dollars. I'm like, okay, where are you? Where's that extra change laying around, right? It's not, it's not, and you're not gonna be spending two or three billion dollars every you know five years. Like, good time.

SPEAKER_01 8:21

Plus, it all works for the most part.

SPEAKER_00 8:23

I mean, it doesn't you put it all in and you run it and you optimize it for three or four years, then you're gonna go do this again, right? Right? I mean, you're creating work, you're creating churn, and it's yeah. I mean, the the the goal is just like a car, you know. You I love cars. I have a beautiful car and I enjoy it, but you want to keep running it. If it's running really well and you're enjoying it, you keep it running. Sure. Doing its job. Same thing in automation in manufacturing or in healthcare. It's doing its job and it's doing it well. Why do you want to change it? You don't, and that's the point. You don't. You want to do it cost effectively. Okay, cost effectively isn't you throw it all out and you put something new in. That's not cost-effective. You need to put some you know boundaries around it. Uh, we all know you know this technology, you know, from the 80s already like 2010, you know, 30 years, they they didn't worry about cybersecurity. They did. Yeah. They did. You only start to see some of that concern in the SCADA systems, you know, at the what's called level two in the Purdue model, um, you know, that are running real-time, not real-time operating systems, they're running a general purpose operating system like Microsoft, uh, you know, Windows technology or a based, uh, you know, a Unix-based platform. Okay, yeah. And in those environments, the general purpose, and there's a lot of avenues and protocols and communication capabilities in those platforms.

SPEAKER_02 9:56

Yeah.

SPEAKER_00 9:57

Which opens up Vandora's box per se of opportunities to be you know impacted. So, you know, it's just the the whole the environment that you have to, you know, you step back a second, okay, okay. So this this technology is out in manufacturing given it's you know, it's it's old. Old doesn't mean it's bad, it's just it's older, so that's what it is. You know, get your head around that and then understand that. And then, okay, how do we from a budgeting perspective, you know, how much is the cost? And that's in manufacturing, your IT budget, if you have automation um rolled into the IT budget on a manufacturing site, it's typically 50-60 percent of the IT budget is going to be the automation component and piece. I'm like, oh my god, yeah, that's a huge cost. Yes, it is. It is. So, you know, how do you use that? You're spending money, how do you do it smart?

SPEAKER_01 10:57

I mean so that that's a a good lead into the next question. Is like, how prolific is this introduction of smart technology versus two, three, five years ago, whatever the right time frame is? And you know, what are some examples of kind of how it's been integrated into the automation process?

SPEAKER_00 11:13

I've been dealing with smart manufacturing, smart technology for 40 years. Yeah. 40 years. So, like you know, way back to the early days in the 80s, uh, was that Eastman Kodak Company? And we used some technology, uh, Gen Sim G2 was one of the first AI engines way, way back when. Um, and that's it, it was very slow, and there's lots of reasons why it was very slow. Um, lots and lots and lots. Yeah, you learn from it. But you know, the early adopters are you know, you learn from that, like, okay, that sort of worked. We put a heck of a lot of energy and time into it, and took a lot of engineering work to get some value from it. Um, wasn't really cost effective.

SPEAKER_02 11:58

Yeah.

SPEAKER_00 11:59

You know, rapid forward now. Today's today, you know, in last, like I'd say about last five years is when the inflection plate started to really change. And it's some of it is the technology, some of it is I call it the vertical integration between you know manufacturing IT down into the production floor, you know, the opening up, adding what they call IoT, right? Industrial sensors, newer technology on the edge, down in production. And that couples with that, all that legacy stuff that's down there. Um, and so now you're starting to it's the mismash, it's starting to get you know some some really smart new sensors down there, but you've got all this existing stuff that you want to be able to get information from. And I use the word information because just collecting those thousands, like the example I gave for that you know, Pfizer biotech suite, you know, thousands of sensors without context, right? Without context to that raw data. It's a it's a I call it Mount Everest of freaking data. It's like, okay, where is something of value in all of this? There is, yeah, but without context to make sense of it. Um, and that's the whole key with like an AI model, it needs context. You give all the data, that's kind of useless. You know, it takes a lot of work to build them from un you know uncategorized information. So you know, the the key is we got a key key ramp, right? We have a very key ramp. Right. But when I say key ramp, we're taking advantage of about 20% of the mountain. Why do I say 20%? 20% is usable. 20% is you can get to it, it's on the network, whatever. There's a good 80% of manufacturing data that is still today unusable.

SPEAKER_01 13:56

Just lost, yeah.

SPEAKER_00 13:57

Well, untapped. It's still there. It's like oil on the ground. It's right, it's there. You could get to it. It needs to add context. And adding context to an existing manufacturing can be done. Um, you can do it live. But adding that context does take time. You have to go back in, right? You have to go back in and you have to understand what is going on, what is doing, where is the information, what is the context I need? What do I need out to add on this to make sense of it? You know, simple one is like what product are we making? What product are we making on this production area today or next week or you know, six months from now? Okay, now I want those data sets I want to be able to compare. Right, yeah, it makes sense. Well, I don't have in real-time data. What products being made, that's irrelevant to some degree. Um, you know, obviously there's there's a setup of information that tells the manufacturing what to make, but it's just the point being is you have to add all a bunch of context, you know. That's just one parameter. You need like 10, maybe 20 sets of data that add contacts in the keys to all that information to unlock it to get value out. And then you can then you can send that data with context and raw data together up into an AI engine and crunch away and build a manufacturing model of what's happening, and then you can use the model down in manufacturing, you know, on a day-to-day basis in real time and get value from it. So it's um there's cybersecurity around all that. So smart technology is great, it's moving, it's it's coming together, all the pieces are sort of there, but there's some some heavy lifting that needs to be done down in the core, and it's being done, and it's being done, you know, where where the bigger I call the better rich oil is in manufacturing. So we're coming around to you know, to do that.

SPEAKER_01 15:53

So, what are the some of the big cybersecurity challenges that this is introducing? I mean, there have to be some new exposures or uh some new risk introduced here.

SPEAKER_00 16:04

Yeah, that's really the challenge is unfortunately the challenge is um way back when, when we didn't worry about cyber too much, um there was a plethora of technology that was put in that was using, I call it the spaghetti mess of IT networks, in that a flat network, everything's on that. Your mail system, you know, your email systems are on that, printers are on that, manufacturing's on that network. It's all shared, right? All those wires are going into a common switch. So when I say, oh my god, it's like how are you going to protect a spaghetti mess of data streams? It's a mess. So I you know, an effort, one of the big efforts we did at Pfizer, because Pfizer is a conglomeration of like 45 companies. So you're picking up companies, some of them come in with the spaghetti mess, and you've got to clean that up. Yeah. You need to separate, at least separate in three buckets. My the way I call it enterprise systems, you know, email people. Yeah, that's one bucket, manufacturing IT, supporting systems, supporting technology, labs, whatever, you know, analytical instruments, fine. You know, they're supporting, and then there's the crown jewels. What's running and making manufacturing, your real-time you know, systems that are you need second by second, right? Those are three buckets. And you want to obviously protect the crown jewels, right? You want to put a little more fences and a little more structure around that, make sure that the crown jewels don't get stolen, don't get, you know, don't get impacted.

SPEAKER_02 17:42

Yeah.

SPEAKER_00 17:42

I mean, that's a big part of it. That's a probably the biggest challenge. Um, is is you know, they call it, used to call it i defense in depth. Yeah. Well, just set just separating out the spaghetti piles into three is a bit it will help a lot, help out greatly. Then you got to deal with again legacy technology.

SPEAKER_01 18:04

Sure.

SPEAKER_00 18:05

Um, you got budget constraints. Yeah, we understand that. Um yeah, those are the challenges. And here's one, here's the other big one in automation technology that my friends in IT used to give me a hard time about. It's like, Jim, why can't you standardize? I'm like, what? Good luck. Yeah, wait a minute, you just said standardize. We we're 45 companies, and you're you're gonna say, oh, yeah, yeah, just standard, pick up a certain technology or a certain vendor and just put it everywhere. First off, there is no vendor out there that does it all for everything in the automated world. There isn't it, like there's no one, and there's no especially no one any to cover the globe. You know, they're all good in their regional areas, and one's better than another, and you know, from a support and structure and supply. There is no standard. You know, one company you can go to and say, hey, you know, take care of my problem. No, if you're a global company, international global company, like you're going to be dealing with a plethora of technologies, systems, components, you know, and it's you can say the decision making isn't always mine to make, right? From automation. Sometimes it's from an engineering firm, you know, sometimes it's the manufacturer of all manufacturing line comes with their proven technology, and it's coming, you're buying it. It's gonna come with you know some suppliers' specialized equipment. What happens is you got ITs all great, it's all standardized, and then you get down to the production floor with real-time systems and devices, and then it just blooms. It's a play, it just like, oh my god, right? You're dealing with thousands of varying components and technology. Hopefully, you can narrow it down so it's you know, not thousands, but you know, 10, 20. Yeah. But yeah, it's you're gonna be dealing with a lot of different players and different manufacturers. That poses obviously an additional problem. You know, so bigger problem than you want.

SPEAKER_01 20:18

Exactly. So I'll let's wrap up with just a final question in terms of maybe what's your best piece of advice to the tech manufacturers and even the industrial and healthcare um end users that are dealing with these security challenges. What do they prioritize? Where what's their approach?

SPEAKER_00 20:36

Yeah, um, obviously they need to be testing their current equipment and and and the prior equipment. Um, right? They need to be testing equipment from you know vulnerabilities that are communicated to them, their own internal labs, you know, checking their equipment, verifying um, you know, what those vulnerabilities are. And if there's a known exploit out there, they got to be working to you know remedy that as best they can, and then be able to advise you know the community, the large community that's like, okay, we found these you know six vulnerabilities and we can fix it, and we're gonna give you an update, or we can't fix it. Right. And here's the bad one, we can't fix it. And like, oh, you can, you can throw out what we gave you, you know, 20 years ago and put this in place, uh, and that will take care of the problem, but um, or you gotta you know put some fence around it. You got and let's go back to that spaghetti mess in networks. You well, you how do I put a fence around all of them? Yeah, you don't. So you gotta separate out and protect out those technologies, segment them out from the rest of your you know, network communication. And they might have to go on an island all by themselves. That might be the ultimate you know, solution is like, okay, we gotta, you know, oh god, you know, what do we do? So yeah, yeah.

SPEAKER_01 21:58

That's like the go to company.

SPEAKER_00 22:02

Correct. Correct. It's like, okay. But if you I mean if you put three or four layers and you put some you know cost effective technologies um to be able to understand what's going on and and protect the environments, you're you're in a pretty pretty good place. I mean perfect place. You just you're in a pretty good place. You can sleep you can sleep pretty well. There's no perfect. There is no perfect. Um unfortunately not. Right. Um there's some really good technologies out there today, I have to admit. You know, over five years ago, we did a very large global program advisor. Um five, seven years ago now. Um and it was a struggle. Right. At that time, clarity was just just coming, you know, coming into being. Um you know, clarity is pretty new even back then. And um yeah, you know, it's like putting that all together and hitting all the needs that we had to you know secure, it was a challenge. Yeah. Challenge. And we did well. I have to say, you know, looking back on it all, we did very well at Pfizer. Um but this you know doesn't end, right? You have to keep right.

SPEAKER_01 23:18

It's continuous, absolutely.

SPEAKER_00 23:21

With AI, so here's here's the point. AI is a great technology. AI is also a great technology for the cyber attackers as well, right? So they can automate things now with these tools to really be it's kind of scary. So you have to I have to say, get on the front of this, use AI now to your advantage because what's coming ain't gonna be pretty.

SPEAKER_01 23:46

Yeah, I mean they're finding bugs quicker than ever, they're gonna build exploits quicker than ever before you know it. Correct.

SPEAKER_00 23:53

The speed of leverage is going to increase.

SPEAKER_01 23:57

Absolutely. All right, Jim, thank you so much. Great stuff, and it was really good to see you. Appreciate it.

SPEAKER_00 24:03

Always good to see you, Mike. And uh enjoy life.

SPEAKER_01 24:07

Thank you.