#008 - Are Embedded Manufacturers Ready for New IoT Security Compliance Demands with Francois Baldassari

In this episode, Jacob Beningo interviews François Baldassari, Memfault CEO, about IoT security compliance demands. They discuss embedded manufacturers' readiness for new security regulations, the challenges they face, and potential solutions.

They also explore the differences between the EU's Cyber Resilience Act and the US's Cyber Trust Mark. François emphasizes the importance of OTA updates, using open-source software, and building security teams within hardware companies. He also highlights the need for collecting the right data and observability to improve security posture.

Takeaways

• Embedded manufacturers are not fully ready for new IoT security compliance demands.
• Regulatory frameworks like the EU's Cyber Resilience Act and the US's Cyber Trust Mark are coming into effect and will require certification of cybersecurity guidelines.
• Challenges include the uncertainty of the regulations, the additional costs and effort required, and the lack of established infrastructure and best practices.
• Recommendations for compliance include implementing OTA updates, using open-source software, adopting SBOM scanning, and ensuring observability of devices.
• AI is not currently a solution for compliance, but it may play a role in the future as more data is collected and analyzed.
• Joining the conversation around open-source products and following security best practices can help improve device security.

Transcript

Jacob Beningo (00:00.472)
Hello and welcome to the Embedded Frontier, the podcast where we dive deep into the ever evolving world of embedded systems. Whether you're an experienced engineer looking to stay ahead of the curve, a newcomer eager to learn, or a director looking to understand the latest industry trends, this podcast is your gateway to understanding the intricate world of embedded systems. I'm Jacob Benigno, CEO and founder of Benigno Embedded Group. We're an embedded software consulting and education company.

dedicated to helping teams and developers modernize their processes and enhance their skills. Our goal to empower you to succeed in your career and projects by developing affordable, reliable and secure products efficiently and on time. Now in today's episode, we are going to be talking with Francois Valdessari from Memfault. He is the CEO and founder, and we are going to be talking about security and particularly trying to answer the question.

Are embedded manufacturers ready for new IoT security compliance demands? All right, well, welcome Francois to the Embedded Frontier podcast. How are you doing today? I'm doing great. I'm so glad to be here. As I was just telling you earlier, I cannot believe it's taken us five years to chat. So excited to finally make it happen. Yep, me as well. So I've certainly seen.

You guys at conferences and had conversations and we've done some stuff, but it's definitely good to sit down with you and have an opportunity to talk with you about embedded software development and where things are going in the industry. So maybe just to kind of kick things off here, do you want to maybe give a 60 second introduction about yourself and kind of your background and maybe how you've gotten started in embedded systems and what you're up to today.

Sure thing. So my name is François Baldassari and I'm the founder and CEO at MemFault. At MemFault, very simply stated, we help hardware companies understand how well their product is working. And when it's not working, we get them the telemetry data, the diagnostics information they need over the air so that they can figure out why and they can fix the problem. I think more pertinent to this conversation, I'm an embedded software engineer. This is how I came to

Jacob Beningo (02:18.47)
to building this company. I've been building connected devices my entire career. I actually have an electrical engineering degree from college. I think many of your listeners have probably followed a similar road as mine, right? You do some electrical engineering in college, you do some simple PCBs and some analog circuits, and then you run into software and you realize that instead of having to wait six weeks for your PCB spin, you can just...

edit, recompile and run. And the fast iteration cycle is very addictive. And that's how I fell into software. I ended up getting a job at Sun Microsystems, where I built firmware for their big servers for what I think ended up being called the M7 systems. They were big kind of fridge sized servers. And then went on to work in consumer electronics, including at a company called

Pebble that many of you may remember, one of the first smartwatches. We could have a whole show about Pebble, to be honest, because it was a STM32 -based device that ran a multitasking operating system where you could run third -party applications. had a GUI. It synchronized data to the cloud. It really, really cool embedded software project, really. And most recently, I worked at Oculus, where I

built the firmware and led the team that built the firmware for embedded virtual reality headsets. mostly consumer electronics, a little bit more serious stuff at Sun. And over the years, realizing that the tools that we use are really tough. And so building a company to try to build better tools. But that's not what we're here to talk about today, but I have to kind of make my little spiel anyways.

it's definitely good to know, you know, your background and where you're going and that sort of thing. And certainly I'll, I'll, we'll certainly encourage people to check out MemFault in more detail for sure. We'll at the end there, we'll leave some info for everybody. And I know today what we're going to try to do and talk with you about is we want to talk about, our embedded manufacturers ready for the new IOT security compliance demands, that are coming out. So, from that perspective, can you give us a little bit of, detail or some ideas of what types of.

Jacob Beningo (04:42.424)
What types of things are coming out that developers need to be aware of? Yeah, first I'll say that, you know, the law of headlines is if the headline is a question, the answer is always no. And so of course we're asking the question because in, you know, at least in my conversations with folks, think as an industry, we're not ready for the regulators to turn the crank crank on us on cybersecurity. I can tell you that some of the products that worked on were woefully insecure, right? The Pebble watch.

We didn't do signed firmware updates. We had customers who would binary edit the firmware and load their own firmware on the watch because we didn't do any verification. They did some cool things with it, but from a security perspective, it was a nightmare. so, you know, but the regulators, they are coming for us. Not because they are mean or bad, but because there's increasingly critical infrastructure that's connected to the internet that if it gets compromised en masse, can have a real impact on the...

on national security and on the life of people. So what are they doing specifically? Well, two things that I've been looking at very carefully, you may have others you'll be excited to share, is in the EU and the US, we've got two big regulatory frameworks coming together. In the US, it's the cyber resilience, in the EU, it's the cyber resilience act, which is a law that is expected to come into effect over the next few years, what will require

some amount of cybersecurity testing and better posture and hardening of devices before they can enter the markets, you will need to certify that you meet the cybersecurity guidelines to get your device in the market. You won't be able to sell it otherwise. Whereas in the US, we've got the cyber trust mark, which is actually in effect. It's a regulation that the FCC has put in place.

It's a very American approach, is it's actually a voluntary scheme. So you can elect to comply with it. And then you get the cyber trust mark in return as opposed to the EU approach, which is very mandatory. And they're surprisingly similar. You can tell these regulators are talking to each other and thinking about this together. I'm sure we'll talk more about what each of them entails. But the last thing I'll say on this topic is that this is actually

Jacob Beningo (07:06.158)
the results of a long process that's been taking place over the last several years. The NIST, one of the big scientific and standards body in the United States, published a report in September 2022 giving an overview of IoT security and some recommendations that they make for regulators. And lo and behold, both the CRA in the EU and the cyber trust market in the US.

quotes the NIST reports and take a lot of their recommendations. So a lot going on. think most of the companies that I talked to, and maybe the same for you, aren't actually following this yet. They're not really fully aware, perhaps even, but it's happening. over the next few years, which is not long when you're building a device, over the next few years, it will be part of our life as connected device makers. absolutely. And my experience has been that a lot of embedded companies, you know, they're

looking at security, but as far as implementation or being ready for it, there's not a lot of companies that I think are in that, that are in that ready to go kind of space. You know, even going back to what you were talking about with, you know, signed firmware updates, I've seen, you know, a sad number of companies that don't even have that. So, you know, and this is well past the dates of Pebble, right? So, you know, so, so,

kind of thinking about these types of, you know, some of these laws are coming out. And as you mentioned, you know, in the US here, obviously, we were like, we care about security, but we don't make it mandatory. Which is kind of funny in and of itself. But what are some of the challenges that you see for these manufacturers as these regulations roll out that they're going to have to overcome in order to actually meet the compliance with these laws and recommendations? I think so.

many challenges. think the first one is that the regulation is still ill understood. So for the first products we're all going to build that we're going to try to comply with this regulation with, think we're going to make some guesses as to what complying means. There's not yet all the consultants that you need to help you come. If you want to comply with FDA regulation, there are people who can

Jacob Beningo (09:29.622)
walk you through that, we've done it for a dozen products. No one's really done it for the CRA yet. And so we don't really know what complying with looks like and we look like, and we don't really know what enforcement will look like. So there's a lot of uncertainty. That's number one. Number two is that building IoT products is already really costly, really effort intensive. And most programs, most IoT programs are already behind schedule.

And so how are you going to add to this an entire new concern around cybersecurity when you don't have the knowledge in -house and you're already on a super tight schedule, especially in consumer, when you're trying to hit the holiday season, make or break, right? So how, like where is ultimately this means there's additional costs to these programs and where is this going to come from? How are we going to make the, and number three is I think there is very little infrastructure and

best practices that are established that we can... You're talking about, we were just talking about, you know, firmware updates and especially signing, firmware update signing. You know, today there isn't an off the shelf firmware update signing solution that everyone uses. Most of your customers and mine, Jacob, I'm sure have built something custom in -house and it may be good, it may not actually.

And what is certain is that it had to be built from scratch. And so we can talk about MCU boot and some of the open source projects that are coming together that makes this a little bit easier. But between schedule, regulatory uncertainty, and the lack of expertise and prior arts that we can build upon, there's a really, really tall task complying. Absolutely.

And there's certainly solutions that some of the silicon vendors have started to provide. again, lot of those are examples, right? So they're not production intent. They're just designed to kind of show off, here's how you might go about doing it. Exactly. And they say themselves, like not for production, or they say, this code. think most of the semiconductor vendors give you code under a BSD license. Love that. It's great that they give us code. But what does the BSD license say? It says this code is provided as is, you know, no warranty.

Jacob Beningo (11:53.442)
You do it as you will, we're not responsible. So, so, we're thankful they do it, but it's not going to be a hundred percent of the answer. Exactly. So what are, what are some of the things that you, you, you see, you know, we've, we've just talked about some of the challenges. Do you see any primary solutions that people can start to look at to solve some of these problems? Yeah, I think first, it's about finding expertise somewhere that is going to help us understand what it will take to comply. So.

There, I think I'm a very big fan of the PSA certified initiative that ARM does. I think if you are PSA certified, you're following their recommendations, you probably are. We don't know for sure, but you probably will be compliant with the CRA and you probably will be eligible for the cyber trust market.

So that's a really good place to start and ARM, as always, does a phenomenal job of building up the ecosystem. There's also some consultancies and especially some audit firms like Tove Sud, one of the big audit firms that people use for safety audits and others. I assume Underwriters Laboratory and others will also get in on this game. My guess is that they're going to start building some best practices that we can build upon.

If I, but abstracting from that, I think the first recommendation that I personally would make is, you know, to, start thinking about OTA upgrades as the very first step. Why? Because number one, I personally believe that OTA upgrades, if your device is going to be connected, the most secure device is the non -connected device. But if your device is going to be connected, then the, the last line of defense, the best chance you have to, to.

to give yourself a cybersecurity incident is having a way to update the devices securely so that if they're vulnerable, you can patch those vulnerabilities. And in fact, don't take my word for it. You'll see that all those regulations as well as the NIST reports all mandate OTA updates as part of the compliance requirements. Why? Because they know.

Jacob Beningo (14:14.83)
vulnerable device that cannot be updated is vulnerable forever. And these IoT device, we know they're in the field for 18 years on average, right? So in the industrial space. you know, 18 years of a compromised device on an assembly line or in a power plant is a very, very long time. So OTA, think still today, not a majority, but a large proportion of devices don't have OTA, even if they are connected.

So I would tell listeners and device manufacturers that they should today make a plan for how they're going to start delivering OTA for their devices. They might build something themselves. They might use AWS IoT. They might use MemFault. We sell an OTA solution and there are others. And I would say that that's actionable today because we know how to do it and we know it's required. That's an ambiguous. Absolutely. And I think getting

the audience engineers recognize the need for that OTA update is absolutely critical. You probably don't know this 15 years ago when I started Beningo Embedded Group, our first product was essentially an OTA. I looked at more of a bootloader solution for folks and it was very interesting to see how hard it was to actually push that. Not push it, but just get people interested in being able to update their firmware remotely. Now, of course, this is 15 years later and it's absolutely, I think, required. But getting the...

getting the signing correct and getting a really good robust solution. It's not trivial. Absolutely. And it's not something that you want to do yourself. yeah, you shouldn't be doing it yourself at all. you know, coming from someone who's written probably 30 bootloaders in their career, you don't want to be doing that yourself, but it is fun. But for production, you don't want to be doing that. you know, one thing that I, this is a little bit of an aside, but because you were mentioning some of the regulations and PSA,

It might be worth noting, I don't know if you've seen this, but ARM just released their PSA certified 2024 security report just like a couple days ago. So, don't know if you've seen it and certainly anyone who's listening, I figure I'll just point it out and I'll try to drop a link to where you can give that, but, or where you can download that. But I personally haven't gotten through it, but they do have some interesting key findings. You know, that just over like, for example, over the last 12 months,

Jacob Beningo (16:39.022)
people have said that security, 73 % of people have said security has increased as a priority in the last 12 months. So that certainly is good that folks are starting to look at security a little bit more seriously, Hopefully getting away from the security by obscurity concept, right? Absolutely. Have you seen a lot of folks starting to think about moving away from that? Or what's your take then? think still in conversations that I have,

I see companies who point at their security by obscurity practices as the proof that they're doing something, and that's better than what they were doing before. And so for those of you who don't know the term security by obscurity, by the way, means obfuscating in a way or not documented interface or making it hard to find out how the device communicates with the outside world, but without explicitly securing it.

So you have an undocumented API. It gives complete access to your device, but because it's hidden, you think it's secure. But of course, keeping things hidden doesn't keep them secure for long. I think from our European customers, we definitely are hearing, are seeing a culture shift. They're more aware of regulation. I also think they're more attuned to privacy and security concerns, maybe even culturally.

And so when we talk to our European customers, they ask us very precise questions about security, how we contribute to the security of their products or not, and how we can help them or what advice we might have for them to make their device more secure. European, the American companies, I have to tell you, I think it's still a long way. It depends on the industry, right? Defense, critical infrastructure, of course, they're thinking about it.

but most IoT device makers that I talk to are not critical infrastructure. And so they don't think about it much. And they may not even be aware that there's regulation coming, which is why I'm talking about it more. Absolutely. And some stats from within the US, something along the lines of 65 % of embedded projects are delivered late. And that's not taking into account security regulations and requirements and things like that. So we're already behind schedule and over budget.

Jacob Beningo (19:05.774)
which then makes it that much more challenging, which then might bring up an interesting question here of, do you have any recommendations for folks that maybe are smaller companies that are looking at, who find that, yes, we should be complying with these. What are some steps that they can also take? You've already kind of mentioned, you know, adding, you know, signed OTA updates. Do you have a couple more thoughts of things that they could do? Yeah. So,

Perhaps maybe it's worth spending a minute first, if you don't mind talking about, the best of our understanding, what are the few headlines on the regulation? What might they need to do to comply with it? And then we can talk about recommendations on how they can actually do that. But as best I understand it, and I'll preface it by saying I am not a lawyer and I'm certainly not your lawyer, but to the best of my understanding, they require OT updates.

they require that communication to and from the device be encrypted. In effect, they require that the device maintains some sort of, that the manufacturer maintains some sort of a built software and hardware bill of material, like know what's inside the software and the hardware. And then that vulnerabilities be tracked and inventoried. So if you find out about the vulnerability that you have a system to track them and, you know, a ticketing system, maybe even just Jira.

and that the device be aware of its own security posture. So the device do some amount of monitoring of itself, hopefully might be able to detect that it is being compromised. So those are kind of some of the five main things that come inside of those regulations, at least the ones that I took away. And so how might you do these things? Well, first, think open source software. I've been banging that drum for a long time, but we just said like, don't...

build your own secure bootloader that you will mess it up most likely. And if you don't mess it up, the effort is just too much. So I'm a big fan of the Zephyr projects, but there are certainly other projects that are great. But if you're building on Zephyr, what do you get? You get a secure bootloader that's built by many and used by many. You get a secure operating system that comes with

Jacob Beningo (21:34.144)
its own vulnerability tracking, some amount of compromised detection, all of these things built in. So for example, the Linux foundation, which maintains the Zephyr project. If you go on the GitHub for Zephyr, you'll see that they maintain actually a list of security issues and they issue CVEs whenever there's a problem with Zephyr. This is not the case for many other RTOSes.

using good, well -adopted open source software enables you to benefit from the security investments of other groups. And I think as far as small company, that's definitely one of the recommendations I would make. Number two, use some sort of SBOM scanning. Nowadays, especially if you're building an embedded Linux device, for example,

There's all kinds of tools that can scan your Yocto build or scan your Debian build and spit out a file that has all of your dependencies. And you can stash that somewhere and make sure you know what your dependencies are. And once a year, check them against a CVE database and see whether you need to update one of them to avoid being compromised by your own dependencies. Of course, put together puts.

adopt some OTA solution. I also recommend not building it yourself, but I'm biased here. you know, do with that as you will. I hope you'll purchase mine, but certainly purchase one of them and start using it. Make sure you know how to use it ahead of shipping your products. And last but not least, you know, some amount of observability. I always tell companies like just by measuring how many bytes of network traffic are coming out of your devices.

you can start catching big problems, right? If all of a sudden your temperature sensor goes from sending two kilobytes of data per day to a gigabyte of data per day, you know that something weird is going on. Like just that can already help you detect and improve your security posture significantly. I could go on, but I think those are pretty good. What am I missing, Jacob? No, I think those are some great ones to...

Jacob Beningo (23:57.57)
to have folks start to look at. mean, those are probably top of my list as well. maybe the one thing I'll mention, because you've mentioned the OTA and kind of your, know, your firmware update process a couple of times there for folks that are listening, that process is actually really critical because in most systems, you end up starting that process with a hardware based route of trust. And that ends up being your anchor point for your secure updates, for your secure solution.

Oftentimes you partition your software into secure and what we call non -secure areas of memory. that whole process, it's not simple and trivial to get correct. And that's why a solution that's already been kind of worked through can be extraordinarily useful and helpful to get you started because you start already out with that anchor point, which then the rest of the trust and security of your system is built upon that from the moment that the system boots up.

I think that's maybe the two cents that I'd throw in there about that. And if I can share one more thing that was coming to mind for me is work with vendors who have good security practices. So nowadays, for example, all of the big semiconductors have a security team and they publish security updates. Subscribe to those. Subscribe to the security updates. Have them go into some mailbox that you look at every once in a while so that if you're

Your semiconductor vendor pushes an update to their SDK because it turns out, as happened with one of the big Bluetooth vendors about four five years ago, turns out their implementation of key exchange in Bluetooth was broken and completely compromised. You ought to know about that and update the SDK, the chip SDK in your project and push an update to your devices so that the Bluetooth that you believe is secure remains secure for all of your customers.

I think some of these things are not rocket science, but they require a change in mindset. Yep, absolutely. And I think that's absolutely key, especially here in the US when we're very focused on getting something out the door, developing as fast as possible. We often tend to overlook security and security itself is one of those things you can't, just like quality, can't bolt it on at the end of the development cycle. You have to, from the very beginning, build security into your system.

Jacob Beningo (26:21.198)
Yeah, and you might think that your consumer device is who cares, right? But there are some stories that are horrifying and also some stories that you wouldn't believe can happen. I have two things that I think of. The first example, I don't know if you've heard of it, is the Mirai botnet, which a number of IP cameras got compromised by hackers and they were able to use those to

install a custom firmware on them, and then send distributed denial of service attacks against websites. So all of a sudden, your security camera in your house would start flooding some random website on the internet with traffic. Another one that I think is a little bit more problematic is also a camera, a baby monitor company who was compromised and hackers were actually watching the camera feeds of random children on the internet. I mean, that seems

pretty bad, right? So even consumer devices really need to think about security. It's not just an industrial concern. absolutely. It certainly goes far beyond just industrial. I always like the, when I teach my security courses, I always joke about an internet connected coffee pot, right? And you'd say, well, who would need security on that? But for that exact reason with the botnets, you have a super cheap thing that is connected, but that has zero security. You can use that for other purposes, right?

And if it has a heating element, I mean, it's even more dangerous, right? Like a hacker can set your coffee pot on fire. That's not unthinkable. Yep. exactly. And even in extreme cases, if you had enough of those in one area, right, you could try to simultaneously turn them all on at once and cause a big spike on the grid, right? see if you could turn on transformers or do other crazy things with the infrastructure, right? So this is absolutely why we need to make sure that we comply.

with some of these new security requirements. kind of speaking about those, as these roll out and as we've been talking about how we should be complying with these, what do you think some of the new compliance demands, how do you think this will actually influence how folks design and develop future IoT devices? Well, I think it's actually going to lead to a consolidation of the

Jacob Beningo (28:46.84)
types of software we built on these devices. Today, there's how many RTOS's out there? I can't even count them, like a dozen probably, right? There's a dozen popular ones, but there's actually over a hundred out there. Exactly. So many RTOS's. And I think that as we have to consider security more carefully, we're going to have to choose these vendors more carefully and there'll be fewer of them that pass the bar.

That's one thing that's most definitely going to change. think counter -intuitively, some companies might choose to remove connectivity from their products because they might decide that the security burden is actually not worth it. For example, I don't know, perhaps a connected dishwasher doesn't create that much better a user experience.

that connectivity is warranted. And so perhaps some dishwasher makers might say, actually, I don't want to comply with this regulation. I'll just make a dishwasher that's not connected, washes the dishes just fine. And last but not least, I think we're going to start seeing the emergence of security teams inside of hardware companies. Like a whole new job function. We will not build security device without security engineers. We can't just train ourselves to become security experts.

And so if you look at some of the big electronics manufacturers, they've already taken that step. mean, certainly Apple has probably one of the best security team in the business across software and hardware industries. even smaller companies will have security engineers, right? Certainly, Pebble, we never had a full -time security person.

Today, a 200 person company making millions of devices should really think about hiring their first security engineer and building up that function, building up that muscle. Absolutely. I couldn't agree more. Perfect. So let's see here. We talked a little bit about the two standards. I kind of want to hop back there a little bit. Are there any key differences that come to mind that are different between the NIST and the C? Yeah. So by far, the biggest difference

Jacob Beningo (31:08.198)
is that the American standards, the FCC, the cyber trust mark, the American standard is voluntary, which means that you choose to comply, you get a sticker you can put on your device, whereas the European is mandatory. You have to comply or else you can't sell your device. The second thing is we know the Europeans with their data protection. We've all heard of the GDPR. And so the CRA,

has one requirement which the Cyber Trust Mark doesn't have, which is simply stated, you must only collect the data you need and no more for your device, right? Which is a way of saying like the best way to avoid data being leaked is to not collect it in the first place. Don't collect stuff you don't need. And by the way, you should document why you need it, right? It's not just.

a determination you make, but something you write down that the regulator might be able to ask you about. I think those are the biggest differences. I'll note that there's other security standards, like the UK has published an IoT security law, but it's much more limited. It basically says, don't reuse the same passwords for all your devices. Like don't have a default password. Don't make it admin admin. know, every device needs to be provisioned with a unique password, default password.

Does the California passed a... Go ahead. I was gonna joke. Passwords aren't allowed to be password. Exactly. Passwords aren't allowed to be password or admin. California has passed a cybersecurity regulation in 2020, actually, but it's also very limited. It's around, you know, same thing, passwords, encryption, and then Singapore just passed one. that's, you know, that's going to make our life more difficult, actually, because now if you're selling a device worldwide,

You're not just going to have to comply with a CRA. You're not just going to have to get the cyber trust mark if you care, but you're going to have to make sure that you comply with the UK and with California and with Singapore. you know, next thing you know, you've got a really complicated regulatory compliance job on your hands. And so what I think will happen there is we'll see the emergence of some standards and, you know, certifications that kind of

Jacob Beningo (33:27.778)
I'll allow you to say if you're certified with this, you're compliant with all the regulations. PSA certified might be it, but I'll give you an analog in the software world. If you're a cloud company, Memfold, we make cloud software, that's our business. And so we have a certification called SOC2. And SOC2 is a privately administered certification that specifies some amount of data and security best practices.

And then you work with an auditor and you get audited. And then you can say, I'm SOC 2 compliance. And the advantage of that is that instead of having to say, comply with this long list of regulations in all these different countries, you can say, I'm SOC 2 compliance. And it implies that you meet or exceeds a large number of regulations because it's a standard that we've coalesced around. And so I think the same thing will happen with IoT.

Maybe it will be PSA certified. Maybe someone else will come up with a different standard that gets widely adopted. ISO oftentimes has done a good job of creating these standards. And then you'll work with an audit firm. You'll get audited. And you won't really think about the CRA because you'll already be certified at a higher level. Excellent. Yeah, think that's a great point. So it's something for us all to certainly look forward to, right? Yeah, maybe.

Who knows, maybe Benningo and Beniggroup can be creating one of those standards, right? Like there's, yes. I mean, there is a market opportunity here for sure. there absolutely is. So I think that'll be, yeah. Interesting to see how things just kind of pan out for sure. So, yeah. Now I've got one other question for you, which kind of looking at where this, where these things will go and how manufacturers can comply. Everybody loves to talk about AI, right?

So the question of course is, do you see AI playing a role in helping to comply with any of these, or even helping manufacturers to comply with some of the security standards that are coming down the pipeline? This is going to be a very disappointing answer, but near term, I don't think I see AI being a solution because here's why. Long term, think we're going to be collecting more signal from our devices.

Jacob Beningo (35:49.814)
And I can absolutely see that we'll be using AIs to look at anomalies or to look at potential compromise and automatically detect using AI that devices might get compromised. But we're so far from that, right? Most companies, most manufacturers don't collect nearly the right data to even start feeding an AI this security. So here, you know, as I was saying earlier, I talked about just collect the amount of

network in and out traffic that can already give you some data, but 99 % of manufacturers don't do that. So before we talk AI, let's talk, you know, collecting the right signal in the first place. Another interesting signal that I've been thinking about is with some of our customers is if you track the IP addresses that your device connects to, because actually most IoT devices

The that connect to many phones is a little different, but most IoT devices connect to one server, the server of the manufacturer. And so they should really only be talking to one IP address. So if you collect the set of IP addresses the device talks to and the database somewhere, you do a count of those. And if there's more than one or two, probably there's a problem. These are traditional analytical systems we can build.

to improve the security posture without AI. And as we become better and better at that, we'll have more data and then AI might make sense. The last thing I'll say on this topic is that AI on device is likely a security, complicates a security story rather than simplifies it. Because we don't know how these AI systems really behave. They're black boxes.

We don't know how they change the security posture of our device. And so in all likelihood, if we're using AI at the edge, we're probably more at risk. Doesn't mean we shouldn't do it, but it's not solving our problem. Absolutely. think by many people listening, you've probably seen it as well, where a single pixel is manipulated as part of the algorithm and it causes the system to behave totally differently than what they expect it to do. And so that becomes a huge vulnerability.

Jacob Beningo (38:09.432)
when you're looking at these Edge devices. How do you audit that? I'm not sure you can. Yeah. Yeah, it's certainly a... I agree with you. It's an interesting problem that I'm not sure it has a really good solution. So at least certainly not in the meantime, so... or short term. yeah, so with... I guess with that, what I want to say, I really appreciate you taking the time to talk with us today. It's been extremely fascinating.

Do you have a couple of thoughts to leave with the audience? Maybe even an actionable item for them to go back to the office and check something out or something that can help move them down the path to starting to secure their embedded devices or at least keep it on their radar, right? Yeah. So you know what? I'll talk about AI, which is here's a very simple way to educate yourself about this regulation.

You know, we both have technical difficulties today. Very easy way to educate ourselves about security, the security regulation and security best practices, which is grab the text of those security regulations. You can download it on the European Union website. You can download it on the FCC website. You can then use some of those AI chat bots to feed them the PDF and start asking it questions. This is actually what I did when I started to educate myself.

I use Claude, you know, it's one of the AI chat bots out there. hope, but it, accepts large files. So that's why I used it. I uploaded the whole FCC cyber trust market. And I said, I'm building an IOT device. How might I comply with this? And of course, you know, it's not going to give you a perfect answer, but it, you know, it's taking a 300 page legal document and giving you reasonable human understandable answers. So I think that's actually a really good hack.

to start working with these really long specs and legal compliance documents. So that's one thing that I'll say. I think think about observability. This is one of the things that we do at Memphold, but there's certainly other ways to do it. The best way to detect vulnerabilities and compromise is to have real data about the behavior of your devices in the field. So that requires starting to collect that data, building up that muscle.

Jacob Beningo (40:27.526)
So I recommend observability. then last but not least, join the conversation around some of these open source products. Look at the security page on the Zephyr Wiki. Look at the security page on ARM embeds, the embed TLS or some of these libraries that you depend on. They've actually been thinking about the security of their device, of their libraries. And they might give you some really actionable...

advice on how to secure your device. Perfect. Thank you very much. We really appreciate it. It's been a pleasure talking with you and thank you for that advice for everybody to go back to the office and start to follow. then certainly if people want to continue to follow you and kind of check out the type of work you're doing, do you have any recommendations on where they can go to learn a little bit more about what you've been working on? Yeah, you can follow me on Twitter. My handle is my last name, Baldassari with

the first two letters of my first name, FR, so Bardasari FR. I also write on the Interop blog. So we love the Interop blog. We constantly write firmware best practices there. And there's a community developing that I find really exciting and the conversation is always very stimulating. And then last but not least, my email address is françois at memfold .com. I love receiving emails from embedded software engineers. They always have interesting things to say. So...

Don't hesitate to reach out to me if you want to talk about security, observability, or other things. All right, perfect. Thanks again. We really appreciate it, and we'll look forward talking with you again here in the near future. The pleasure was mine, Jacob. Thank you for having me, and I hope we'll do this again soon. Absolutely. Thanks. Bye.