Enterprise Pen Testing

In this episode, ETAS experts Zachariah (Zane) Pelletier and Michael Scharl discuss enterprise testing, how it differs from embedded testing, and how they are starting to come together with the development of software-defined vehicles and increased external interfaces with the car (e.g., remote start apps). The conversation takes listeners through the realities of today’s vehicle having a broader attack surface, why OEMs need to understand this, and the importance of working together to bring knowledge and tools from enterprise and embedded testing together to secure our vehicles.

During the discussion Zane and Michael reference an article by Sam Curry; you can find it here. 

Thanks for listening!

• Email us at: contact.us@etas.com
• Learn more about ETAS on our website 
• Follow us on LinkedIn: @ETAS

Transcript

00:00:02 Voiceover

Welcome to the Empowering Tomorrow's Automotive Software Podcast, brought to you by ETAS, a single source of cutting-edge software and hardware solutions that make automotive embedded systems safe, smart, secure, and sustainable. Each episode, we'll be joined by ETAS and industry experts to discuss how electrification, automation, and connectivity are impacting the automotive industry. Now, sit back and enjoy the discussion.

00:00:32 Zane Pelletier

Hello, everyone, and welcome to another episode of Empowering Tomorrow's Automotive Software Podcast. I'm your host, Zane Pelletier, from ETAS, and today I am joined by Michael Scharl, who also works at ETAS doing enterprise security testing. Michael, do you want to give a brief background on you and your work? I know we've worked together before, so maybe a little on that too.

00:00:58 Michael Scharl

Yeah, sure. Yeah. As you already said, my name is Michael. I work at ETAS now, but I started out at a different company. Or actually, my journey started in my education already. I did a bachelor's degree in IT security. After that, I joined the company for 3 1/2 years doing enterprise security. And then I moved to ETAS, where I'm also still doing enterprise security, but with automotive relations.

00:01:27 Zane Pelletier

Great, thank you. Yeah, so I think a big topic today that we're going to talk about a lot is enterprise, what that means, how it's different from traditional embedded testing that I think a lot of folks are familiar with in automotive. So for those of you who don't know, I have more of an embedded background, but recently I've been working with Michael and others to start onboard into more enterprise-based focused projects. So I guess, Michael, do you want to give a background on what

00:01:57 Zane Pelletier

enterprises and maybe how it might be different from traditional embedded projects in an automotive sense?

00:02:05 Michael Scharl

Good question. Enterprise is a really broad field. We are doing a lot of pentests for all different kind of enterprise topics. It can range from the standard web application where you have just some company presence or some product described to you have a web shop or you have some integrations with other tooling.

00:02:27 Michael Scharl

We have mobile applications where you install on your tablet or smartphone. We have client applications where customers use on their computer to connect to other systems. We have embedded systems which have connection towards the enterprise systems because you normally manage them from enterprise systems. So it's really a wide spectrum of different technologies that come into play. But I would say most of the enterprise topics

00:02:57 Michael Scharl

are normally either web applications or client applications running on some operating system.

00:03:04 Zane Pelletier

Definitely, yeah, thank you for that. I think a good way to think about it is there's testing of...

00:03:10 Zane Pelletier

applications that are on board the vehicle, so systems that are actually physically within the vehicle bounds, and then there are off-board systems that are in the back end. So traditionally, when I'm thinking about this, I'm thinking about over-the-air updates to modules in the vehicle. The module that handles that, the telematics, usually the telematics module in the vehicle, will be the on-board system that needs to be tested for its security, whereas the server that it's getting those updates from and the PKI that's used to

00:03:40 Zane Pelletier

transfer all of that data and to make sure everything's secure that's hosted on a server somewhere else that it's connected to via a module on the vehicle. And so I think that's a good way to say that there's a lot of things that are coming together, I think, between the two right now. There's a lot of vehicles that are much more connected than they ever were before. And there's a lot of systems that traditionally were more focused on the onboard side of things that are now integrating with off-board mobile web apps,

00:04:10 Zane Pelletier

applications, web apps, like you said, and other such systems that just aren't really diagnostic focused on the vehicle.

00:04:17 Michael Scharl

Yeah, thanks as well. We see it quite a lot. In the past years, we have quite a huge shift toward cloudification. Everybody's moving to the cloud and putting their back-end systems there. And especially we have in the automotive sector a lot of back-end systems that are cloud hosted where the vehicle connects to either to pull software updates or to send some

00:04:40 Michael Scharl

Telemetric data for some monitoring or fleet management. Yeah, it's definitely getting more and more interconnected, especially with those cloud components.

00:04:50 Zane Pelletier

Yeah, definitely. So, I do also want to bring up kind of an experience that we had recently. I know that Michael and myself actually both competed together on the same team at the Car Hacking Village, which is...

00:05:03 Zane Pelletier

a CTF competition that happens at DEF CON. For those of you who aren't aware, a quick aside, DEF CON is a hacker conference that takes place in Las Vegas every year.

00:05:13 Zane Pelletier

There's all kinds of folks that come out to that. There are a lot of hackers there, and it's pretty much any domain that you can think of. So they have these different groups of people that come together to form what's called a village, and each village has a focus. So for instance, there's one that's focused on automotive, that's called the Car Hacking Village. There's one for aerospace, the Aerospace Hacking Village, and one for social engineering, and the list goes on. So this competition was mainly focused on

00:05:45 Zane Pelletier

bringing together hackers in the industry to come together to solve extremely difficult challenges that were constructed by industry experts.

00:05:54 Zane Pelletier

to try to basically compete together to show off skills. And there was actually a prize at the end. They gave away a Tesla. So that was a pretty good incentive as well. But I know we competed there together. And one thing that I noticed in particular was that there were some challenges that were more focused on traditional, what we would consider to be enterprise systems, right? Like cryptographic infrastructure, things that have to do with transferring data from an off-board system onto a system that's traditionally in the vehicle. Whereas I

00:06:24 Zane Pelletier

I had been at that competition for a few years, and I hadn't seen anything like that before. So it was really good that we had you there, Michael, because you were kind of one of the folks, I think, that headed up working on those challenges that were more enterprise-based.

00:06:37 Michael Scharl

Yeah, so I was really glad that it was actually that way. It was my first time being in Vegas at the DEF CON, and I didn't know what was coming.

00:06:49 Michael Scharl

Up, so I thought I might can support you in some way, but not being a big deal there, because I'm only enterprise-specific, or before that I was only focused on enterprise, and I was really positively surprised that there were actually quite a few challenges that were either...

00:07:08 Michael Scharl

really enterprise challenges or it had its roots in enterprise technologies that I could transfer my knowledge to and actually use it and be a useful part of the team there. So yeah, I think the challenge that was most enterprise-based was the package, the software that we had to compromise to simulate a malicious update and then through that get the flag. And I really enjoyed the challenge. Yeah, great experience for me there.

00:07:37 Zane Pelletier

Yeah, definitely. I think everybody was a little bit surprised by that. But I also think that that's indicative of kind of where we're heading, right? I think that this is not going to go away anytime soon in terms of what we're seeing. I think that if anything, we're going to become more and more integrated. These 2 traditionally more separate systems, the onboard and the offboard systems, are going to kind of come together and they're going to just become more integrated. There's going to be more interaction happening there.

00:08:07 Michael Scharl

Definitely.

00:08:08 Zane Pelletier

I think that's important.

00:08:09 Michael Scharl

I see it in my daily life. Like a few years ago, I was still surprised when my friend told me that he had a mobile app where he could like preheat his car. So when he went to the gym, he could just like said, oh, I'm heading for the shower. It's winter, so I just kind of preheat my car so that when I come out, it's already ice free and already preheated so I didn't have to freeze in my car. And I was like, wait, that's possible? Like on a mobile app?

00:08:37 Michael Scharl

that you can just remotely control something in your car. And I instantly was interested in how it is secured and if there's some potential to mess with. But yeah, I see it in more and more kind of domains where you get more and more external interfaces with the car, either via keyless vehicle, via

00:09:01 Michael Scharl

Cloud updates or some maintenance backend that is able to remotely dial into the car and pull some diagnostic data, it's getting more and more interconnected.

00:09:11 Zane Pelletier

Yeah, definitely. That's a good example. Yeah, I think that there definitely is something to be said there for the fact that not only are more and more...

00:09:22 Zane Pelletier

auto manufacturers integrating these systems into their vehicles, I think that there is really a sales that are driving that, right? People are interested in those features. Those are things that people really want in a car. They want to be able to control it via their phone, via mobile application. They want to be able to control things in their vehicle remotely and also get information, right, about systems that are running in the vehicle while they're running them. So

00:09:48 Zane Pelletier

I think that this is, again, only going to increase in the interconnectedness here. And I think that this also kind of relates to the shift that's being made towards the software-defined vehicle as well, because all of these features that we're seeing here are largely software-defined. They're running on the traditional hardware systems, and there's integration with backends. But at the end of the day, this is a mobile application, or this is a web application that you're using to interface with.

00:10:18 Zane Pelletier

with the vehicle. And I think that kind of brings traditionally 2 separated domains together. And I think this is a pretty good transition into saying there's a article that I know that I had read and you had read too by Sam Curry. So for those of you who don't know Sam Curry, he's traditionally like a more enterprise-based hacker. He, you know, does a lot of web app stuff, does back-end systems for traditional enterprise systems. But back in 2023, there was an article that he wrote

00:10:48 Zane Pelletier

and did a lot of research on that is titled Web Hackers versus the Auto Industry, Critical Vulnerabilities in Ferrari, BMW, Rolls-Royce, Porsche, and more. This was released in January of last year. So it's actually getting a little bit old now. I remember when this first came out.

00:11:06 Zane Pelletier

But this one is interesting because you have someone who's traditionally in a more enterprise-based domain, and he's coming in and he's starting to get all of these exploits on systems that are more vehicle-based, right? I mean, if you look at the list, I think at the beginning of the article, he has a huge list of all of the different car manufacturers he was able to hack. I mean, you know, it's so long. Yeah, I

00:11:27 Michael Scharl

really love that article because every time we have to argue why enterprise security is relevant for the SDV business, I like to

00:11:36 Michael Scharl

pull up that article because it perfectly showcases how enterprise applications are more and more critical for the automotive industry because you don't compromise one car anymore, you compromise the entire fleet with it. And you said the article's getting old. Yeah, it's already, what is it, two years ago, but it's getting more and more relevant each day, I feel like. More and more car manufacturers move to those kind of integrations and

00:12:06 Michael Scharl

all these findings that Sam Curry showcased, that's what I see in a daily business. When we test those applications, we see those web vulnerabilities where you have an API endpoint and when you compromise it, suddenly you're able to like reissue a key for a car or you can change the ownership. Sam Curry actually did a second article where he just did it for Kia again, where he was able just by the number plate or the VIN to re-enroll the car to

00:12:36 Michael Scharl

the new owner and thus taking control over the car. And so it shows it's still more relevant than ever. And yeah, you already teasered it. It's all the big manufacturers' names you can find in there. So it's not just one that they found that did a bad job. It's not me saying they're doing a bad job, but you can find those findings throughout the bench. Every manufacturer has the same topics to tackle.

00:13:05 Zane Pelletier

definitely, yeah. I mean, there's big names in here. There's Ferrari, BMW, Rolls-Royce. I mean, they obviously have quite a bit of resources, financial resources to spend on things like this. And like you said, we're saying the exact same issues across the board. So I think it's less of a resource issue and more of an awareness issue where you might have people in these organizations that traditionally have worked on more embedded systems.

00:13:32 Zane Pelletier

And now they're making the leap. They're like, okay, I'm familiar with automotive controllers. I've been doing that for the last 20 years. And suddenly you ask the same person to come in and make a web app that makes it so you can remote lock, unlock your car, start the engine, right? Get precise location of it, all these types of features. And they are not familiar with some of the common security pitfalls that are involved in systems like that. So yeah, that creates issues.

00:14:02 Michael Scharl

Yeah, exactly. It broadens the attack surface by a huge, like I remember back in the day when I read about automotive security, I always thought about CAN bus systems or did I have to physically plug into the car. And I know there was some kind of exploit where you could like pull out the headlight, connect to the CAN bus, and because it wasn't separated, you could somewhat unlock the car, but it always would require that initial step that you somewhat get physically

00:14:32 Michael Scharl

in the car or near the car to at least do some radio frequency kind of stuff. But now you shift the entire tech surface also to the internet. You're getting interconnected and it doesn't need to be anymore that you are physically present to attack a car.

00:14:48 Zane Pelletier

Definitely, yeah. And I think that this definitely is...

00:14:52 Zane Pelletier

kind of being shown in full force here. And I don't really think it's even fully constrained to traditional like automotive sectors. I know that one of the things in the original Sam Curry article from 2023, he also hacked a couple electric scooters. So, you know, there's electric scooters that are now connected to mobile applications that you can use to manage, you know, that type of vehicle.

00:15:15 Zane Pelletier

And I think the big takeaway for me from that was a lot of these systems he compromised either through a single user account or he was able to get into some of the back-end cloud infrastructure, I think, in a few instances. And that was really huge because in an enterprise environment, if you're able to control the cloud infrastructure, I think, from any aspect, that compromises everything, essentially, because that's the infrastructure you're using to host all of these services. And

00:15:45 Zane Pelletier

I think that with the most recent one where he was able to get the vehicle identification number and get the license plate information and basically arbitrarily register information from a vehicle just by knowing those two basic pieces of information.

00:15:59 Zane Pelletier

It's not something that just affects one person's vehicle. This is something where if anyone has a vehicle that was made in a certain number of years from a certain manufacturer, it's going to cause an issue for them too. So it's so much more widespread than just like taking control of 1 car. You're attacking an entire brand. And now the enterprise security is becoming, I think, more crucial to that brand image of the OEM and being able to say, yes, we have a secure product.

00:16:29 Zane Pelletier

because nobody's going to be able to steal your car, take it over, do anything nefarious with it, right?

00:16:36 Michael Scharl

Yeah, I wouldn't say that the focus shifts. I think it's still important to secure the car locally, but it definitely broadens the spectrum by a lot. And as you said, it's not just that I attack one car in specific, I attack the manufacturer, and then I take control over an entire fleet or a product line. I remember that it is only for some specific model

00:16:59 Michael Scharl

versions that they were vulnerable, but it's still thousands of cars instead of 1, and that's suddenly a way bigger impact. And it doesn't need to be stealing the car physically that the car is gone. There's so much more damage potential towards it. can be just getting the location data for stalking or tracking or knowing when somebody's home.

00:17:23 Michael Scharl

We mentioned earlier, there are some really big companies in there, and some of them, like Ferrari or Rolls-Royce, are known to have customers with a little bit deeper pockets, and knowing when they're home or when they're out on a drive is already something that is in potential damage. Then there's, of course, breaking into the car or breaking into the house. There's also denial of use. If you

00:17:49 Michael Scharl

somewhat bricked the system remotely for thousands of users and they can't use their car anymore. It might also be a huge damage. I remember that picture on Reddit where somebody couldn't drive his Tesla because some update went wrong and it said, oh, you have to call your manufacturer or the car needs to be towed. It can't start anymore. That's a huge damage. Suddenly, thousands of cars have to be towed or manually fixed. So much more damage potential out there now.

00:18:19 Zane Pelletier

Right, I think that definitely, I think that's illustrated here for sure. And the fact that really no one is safe. I mean, I'm looking at the list of manufacturers and it's pretty much everyone's car. It's not just one niche group of people, to be honest.

00:18:36 Zane Pelletier

so a question I get a lot is, what can be done here? Because I know traditionally, you do penetration testing on the embedded side. You have to, I think for a lot of manufacturers, there are requirements to do penetration testing on the modules and the controllers themselves, and then also on the back-end systems, the enterprise systems that are being used.

00:18:56 Zane Pelletier

But a lot of times I get the question, what can we do to kind of simulate something like this? What can we do to simulate a similar scenario to what Sam Curry presents here, where we kind of are able to combine these two systems together and have a full end-to-end attack scenario? And the first thing that comes to my mind when I think of that is doing more red teaming exercises and using those as kind of a way to determine, hey, not only how secure is my system, but

00:19:26 Zane Pelletier

What could happen if something goes down? What could the manufacturers do to prevent something like this? And do they have a playbook in place for if there's a catastrophic event where a lot of their vehicles are getting taken over in this way?

00:19:39 Michael Scharl

I think it's a multi-step process. The first one is to realize that you have to look domain overarching. It's not like being in your own domain and saying, I'm only responsible for the car. The cloud backend is managed by somebody else. I don't care about that. is already the biggest mistake that we see doing a lot, that they only focusing on what their responsibilities are. But you need to have somebody that has the overarching view on the entire infrastructure that you're not

00:20:09 Michael Scharl

Just have to look at the car, but also on the cloud back end, and of course then you have to do pen tests for both to find vulnerabilities and fix them as soon as you can, and then also the inter...

00:20:22 Michael Scharl

operability is important that you don't just test one without the other, that you only look at the enterprise back end and then you only look at the car. You have to do that together. You have to see what's happening. If something in a car is happening, do you see in a cloud? Do you have some monitoring? Do you even realize if somebody is breaking the car or sending some malicious packages? And also the other way around, how do you

00:20:46 Michael Scharl

Ensure that, in the car, you notice when a malicious update is being pushed, or if there has been some malicious changes, so first you have to realize that...

00:20:58 Michael Scharl

those people that have those responsibilities work together and not only look at their own. And then you already mentioned into red, blue, purple teaming kind of scenarios that you bring the red and the blue team together, that the blue team that tries to detect something like that if something malicious is going on, and also the attackers that they play well together, and you identify those potential attack scenarios by doing real-world simulations.

00:21:27 Zane Pelletier

Yeah, definitely. I think red, blue, purple teaming comes into play here a lot. I know that that's traditionally what we would recommend. if you're in a situation where you want to see, hey, how do the two sides of the security of this product come together to figure something out, right? I think a really good example to kind of illustrate this is exactly what you mentioned as was the challenge, right, that we worked on at DEF CON, or rather you worked on at DEF CON, that system of a malicious update getting

00:21:57 Zane Pelletier

pushed through the backend system to a vehicle, that is a perfect red teaming exercise that could also be detected by the blue team on the side of looking at detection, looking at incident response for something like that, how to fix the issue, what to do if it goes wrong. That is an excellent exercise to go through because not only is it something that could realistically happen, in my mind, that's like worst case scenario, right? Because essentially the attacker has control over the firmware on the

00:22:27 Zane Pelletier

On a physical device, then, and that could cause some catastrophic issues on the vehicle itself.

00:22:33 Michael Scharl

I think that basically would be the worst case if you're able to push an update in a legitimate way because you compromised the back end.

00:22:42 Michael Scharl

you have full compromise of the vehicle in most cases. So that's definitely something that, yeah, you should work together with the blue and the red team, that you monitor what the red team, what a potential attacker can do, and then adjust your systems, your blue team, to be able to detect them in the future and also write playbooks what you're going to do in that situation when something like that is happening.

00:23:08 Zane Pelletier

Definitely.

00:23:09 Zane Pelletier

so maybe we can shift here now to talk about what can be done to kind of break down traditional silos here. So I know that a lot of, for me, I come from an embedded background. I kind of transitioned to doing more enterprise now, which is, I would say, kind of non-traditional. Usually people go in the other direction. Sometimes they go to more, you know, something that's more broad like enterprise to something more specific like embedded. So yeah, I guess, Michael, what do you think we can do to have a better joint effort between subject matter experts for

00:23:39 Zane Pelletier

Testing in both of these areas?

00:23:41 Michael Scharl

That's a really good question. That's also a challenge that we are facing now, because that's something that we have to tackle in the next few years, because that's where...

00:23:53 Michael Scharl

where we have to improve. Because right now we see a lot of teams think in silos. And the first thing is realizing that you're in a silo, that you are only looking at your own stuff and reaching out to the others like we did now. I joined the enterprise team. I joined from the enterprise team, the automotive team, to see how they're working, what their tooling is. And I also introduced the enterprise testing towards the embedded testing. And that's starting that knowledge sharing, knowing what the

00:24:23 Michael Scharl

domains are about, and also realizing that there is potential for synergies. A lot of the technologies that I see now being used and embedded is stuff that was developed for enterprise, but now being reused and embedded, and bringing that knowledge from the enterprise field into the embedded field is really important.

00:24:46 Zane Pelletier

Yeah, certainly. Yeah, I agree. I think just bringing people together and discussing commonalities and the

00:24:53 Zane Pelletier

issues you're having, that can be a very, very powerful tool that we can use to our advantage. And I know that this is not a singular event that we are witnessing here at ETAS. I think that this is happening industry-wide. There are a lot of people thinking about this issue. And I think it would benefit us all to kind of come together a little bit more to try to just bring

00:25:15 Zane Pelletier

The not just different types of testers together, but also the red team and the blue team that traditionally have been on opposite sides to say, Hey, how can we create a more holistic solution here? And I know we've also talked about this a little bit, but one...

00:25:34 Zane Pelletier

I think penetration testing in general and doing advanced hacking is a very specialized skill set. You cannot be a subject matter expert in every type of red team attack, right? It's not really feasible for someone to do that. So at a certain level, we kind of, we do have to specialize. It's not going to be possible to hire the unicorn that can do everything, right? I know that's a myth that happens a lot in the industry, but I just don't think that that's possible for someone

00:26:05 Zane Pelletier

Yeah,

00:26:08 Michael Scharl

I think you hit the nail on the head. German saying, I don't know if that translates well, but I think you made a really good point. When I started in pen testing, I always felt like I have to learn that, I have to learn that, and I went into so many different topics, and you have that kind of learn curve where you at first think you know nothing, and after a few years, you believe you know everything, and after a few years, you realize how low you are.

00:26:34 Michael Scharl

little you know yet. So I'm at that point where I realize I have like a broad scope now, but to go deep into every single domain, it's impossible. You can't do that. You can't be proficient in web, in mobile applications, in client applications, in reverse engineering, in attack simulations, in embedded devices. It's so many different technologies that are coming together. You have to specialize at some point. And that's why it's

00:27:04 Michael Scharl

It's not possible to create the perfect pen tester for every single project. And that's why it's important to bring the people together from these different domains where they specialized in and bring those people into those projects and let them work together and find a way how they can efficiently work together to improve the security.

00:27:22 Zane Pelletier

Awesome. Yeah, I think we're on the same page there, definitely. And I think that this is a good takeaway, I think, for anyone listening who is encountering some of the same issues, I think, that we've seen.

00:27:34 Zane Pelletier

I think the best tool to remedy a lot of issues is just good communication and finding the people that can help you with the specific problems that you're having. And I think that, again, no one person is going to be able to do that for you. It needs to really be a number of people coming together to discuss this in an open setting, certainly. All right, thank you, Michael. Really appreciate you joining the conversation with me and talking this through.

00:28:02 Zane Pelletier

Are there any last thoughts you want to share with the audience before we end to just kind of talk about how you feel about the entire situation and the coming together of the enterprise and the embedded sides of security here and what that means for automotive?

00:28:17 Michael Scharl

Yeah.

00:28:17 Michael Scharl

From my personal perspective, I think it's really interesting where the trend is going. We are having more interconnected vehicles, more attack surface. For me as a pen tester, it's always interesting to see new vulnerabilities, new attack scenarios. So I'm really eager to see where it's going and how we tackle those challenges. It's important.

00:28:40 Michael Scharl

It's important that we work together, that we learn in those fields together. It's an ever-changing field. So staying on a ball, always getting involved in the newest trends, technologies, and as we already pointed out multiple times, bringing the people from those fields together will be a really important step in the next few years.

00:29:02 Zane Pelletier

Definitely. Thank you for that, Michael.

00:29:05 Zane Pelletier

this concludes our episode with enterprise security expert Michael Scharl. Check back soon for a new episode. We'll have one for you shortly. Thank you, everyone.

00:29:16 Voiceover

Thank you for joining this episode of the Empowering Tomorrow's Automotive Software podcast. Please leave a comment or review with your feedback or what you'd like to hear in future episodes. To learn more about automotive embedded systems and ETAS's capabilities, visit our website at ETAS. That's E-T-A-S.com.