The Embedded Frontier

#0003 - The Fight for True Open-Source RTOSes

Jacob Beningo

Carsten Gregerson shares his background in the embedded systems industry and how he got into it. He discusses his work at Nabto, a company that provides remote access to small devices using peer-to-peer technology. Carsten then delves into the topic of real-time operating systems (RTOS) and the battle for open-source RTOS. He explains the difference between a real-time operating system and a high-end operating system and how the introduction of the internet into embedded devices has increased the need for RTOS. He also discusses the consolidation happening in the industry with the acquisition of RTOS by big tech companies. Jacob shares his thoughts on Zephyr, an open-source RTOS, and its place in the market. He also explores the challenges of adopting Zephyr for embedded developers and the potential future of embedded development with the integration of interpreters and AI.

Jacob Beningo (00:00.046)
Can you tell me a little bit about your background and how you got into the embedded systems industry? Yeah, I actually, I wouldn't say I'm quite old, but I definitely grew with the whole tech industry. Started out when I was around 13. Bought my first home computer, which at that time was something called the VIC -20 with around 5 kilobytes of RAM. Started, yeah, bought...

my next computer which was a Commodore 64 which probably is more well known. But you know, very small 8 -bit computers. And I actually started to write the small programs in first and basic and then in assembler language actually. I then moved up to Amiga and PC and so on. And actually when I finished high school, my idea was to be some kind of engineer. I mean, building engineer, mechanical engineer, something like that.

I thought that you can say computers would just be something I did in my spare time because I mean we are talking.

middle of the 80s or something like that, 90s. So nobody knew yet, I mean, these computers, what's going to happen? I mean, how big is it going to be? And you can say basic engineering stuff, you know that's going to last, I wouldn't say forever, but at least the next 100 years or so. But I kind of became an engineer, an expert on...

Unix based system. I was running Linux, I mean, before nearly everybody else. I mean, some of the first versions downloaded via the internet. And so, yeah, that's kind of my way into the computer industry. I also, when I was finished, I actually worked in, you can say the high -end internet industry for some years. Before I came up with this, why not use...

Jacob Beningo (02:10.382)
Skype for IoT, which is basically what we do at the company. I mean, remote access to small devices. So that kind of made my way back to microcontrollers because I had a long period where it was more high -end server stuff. So, you know, kind of talking about embedded systems and everything, and you kind of alluded to the fact that you're running a company. Could you...

Tell us a little bit about the name of the company, a little bit about what you do there and how long it's been around and some of the stuff that you guys are doing there. Basically, the vision is to use the same technology that is beneath Skype, probably also what we're using here for this recording, something called peer -to -peer technology, a way where you can traverse firewalls and get a direct connection.

on two peers on the Internet. And the vision was basically why not use this for IoT? If you have a client device, that could be a smartphone. So we started out before there was something called a smartphone. So when we started out, it was actually how do you connect a PC to an embedded device? But today the use case is mostly how do I connect my smartphone directly to my device via video surveillance camera?

and be in real -time communication with this video surveillance camera. Or it could be an alarm system. And the benefit of this is, of course, very low latency. Excellent. Very cool. So now I know one of the things that we had talked about wanting to discuss here, and I think a lot of people will be interested in, is talking about real -time operating systems, and specifically starting to talk about the battle for...

open source real -time operating systems. So do you want to maybe explain what your idea of a real -time operating system is and why it's important to people who are working on connected devices today? Yeah, gladly. I will start saying what... Instead of the, you can say, the direct definition, I would probably rather say...

Jacob Beningo (04:32.046)
do some examples. Because I have moved, I've been on all the levels, you can say. And a real -time operating system compared to, you can say, a high -end real operating system is a complete operating system. It's that a real operating system like Linux or Windows or Mac OS is something that totally abstracts away the hardware. I mean, programs cannot access the hardware.

directly, at least not without getting admin root access or something like that. So the idea is to completely abstract the hardware away so that programs cannot interfere with other programs and so on and so forth. And then you of course have in the so that's the that's one of the that's a high that's a high end. And if you go all the way down, as I say, bare metal, which were where I started.

was it's more like where, well, you do everything yourself. And I see a lot of, 10 years ago, a lot of devices would actually just run on bare metal without any real -time operating system because I mean, the type of application you would do at that point was normally very simple. I mean, a thermostat,

A simple thermostat, I would say, would just wait until the temperature is at some level and then it will start out or shut down the heating. And you don't need much operating system for that. And that's what I have seen in a, if we look at it at a historic aspect. I mean, all this IoT actually has pushed the embedded industry from cook.

run on a bare metal system, if you didn't have any real time, you of course have applications where real time and all this, you have to stop and start and add exact time slices and so on and so forth. Then of course you would have a real time operating system, but a lot of the devices out there, I mean, they didn't need a real time operating system. But with the introduction of internet into these devices, things have just gotten so complex that you had to move up.

Jacob Beningo (06:51.342)
And that's what I've seen. I mean, the last five, maybe 10 years, I mean, the industry has really, really matured up to at least real -time operating system level. And sometimes even all the way up to higher operating system. Normally Linux, Microsoft tried with embedded windows, but I don't think they do that much more. I think it's still out there, but I don't see much traction. So normally it would be Linux. But I...

I also believe that the last step for this type of application for embedded use cases is not too much because you probably want to do something directly in hardware and also you don't have the need for multiple, I mean the separation of multiple programs running on top of each other because you have full control, you know exactly what is on the embedded device. So.

So, and that's not the case, I mean, with real operating system, you never know what the user is going to install. So, there's a big need for this total separation of the hardware. But whereas in an embedded system, this extra complexity moving up to a real operating or high -end operating system is maybe just too much. And then of course you have a lot of hardware guys out there and they are not that keen on...

on being moved all the way up in user space and not being able to access the hardware directly. So I think this is what we're gonna see embedded devices at least for the next foreseeable future will probably run, most of them will run on a real -time operating system. The bare metal system will be caught off and that's because of the internet. And again,

Because on the internet, you get some communication from somewhere and then an interrupt happens and then this has to be handled and you have an IP stack. And again, it's also nice to have memory allocation and being able to stop and start and synchronize threads and so on and so forth. So I think that's what is happening and what's gonna happen is we're gonna see much more traction on the real -time operating system.

Jacob Beningo (09:12.846)
which is just validated by Microsoft buying third X and Amazon buying three R tops. I mean, so it's a, I think it's even today it's old news because it is already there. Yep. Well, most definitely. And like you mentioned the, you know, purchasing of a lot of these, you know, real time operating systems over the last couple of years has really been, I guess, staggering to some degree. It's even affected how people are going to develop their systems. You know, you mentioned,

ThreadX being purchased, you had FreeRTOS, you actually had MicroCOS 2 and 3, which Silicon Labs bought. So it's a little bit different, where Microsoft and Amazon were big cloud players in Silicon Labs is more a microcontroller vendor. But there's a consolidation that's kind of occurring in the industry. And it was a smart move. I mean, yeah, and I could say I saw it coming.

I cannot reference anything, but it was a smart move for them because again, a lot of the decisions out there, they're starting in, if you look at how you develop products, start in product management, they're going to find out a new idea, they write a lot of specification on what this product has to do with this and this and this. And at some point they go to the engineers, they'll convert those, you can say high level.

requirements to something more specific and find out, okay, how many input outputs do we need? How they're trying to estimate how much compute power do we need and so on and so forth and what preferable interfaces do we need and so on and so forth. And then they're going to look into what kind of software is supporting this device that they now found, or this microcontroller that they now found.

And you better be there with something that can integrate to your cloud system. Otherwise, they're going to choose whatever software they can get support from, from the microcontroller or in any other ways. And if it works, that's probably going to chip in on to which cloud to use. So.

Jacob Beningo (11:39.438)
But that's also what I see right now. I see this silofication. So if your choice is FreeRTOS, probably you're going to build your complete system up on top on Amazon services. Yeah, and so I think that's interesting. So if you look at kind of talking about Amazon and FreeRTOS, you know, FreeRTOS, you know,

was an open source real -time operating system for years. Azure RTOS, which was formerly known as ThreadX, was always a commercial one. As we start to look at this idea of open source real -time operating systems, do you have a definition in your mind of what a real open source real -time operating system is and what that looks like today? We should look at how is it controlled? I mean, who...

who decides which features to put in there and not kind of, I think I'm not sure I can do a definition. I mean, like in math, definition or something like that. But I do think there's some elements in there that has to be there for it to be real open source or true open source for me. And that is especially the decision process of the operating system. I mean, how...

who decides what to put in and so on. It's not so much how rigid it is because if we go to Linux, there are some stories about some of the maintainers being very rigid on what can go in and what can not go in, but it's an open process. People, they can say, I would like to do this instead and we all vote about it and so on and so forth. But...

As you say, Free Artis was what I would say was a real, true operating Free Artis, real -time operating system. And I really like Richard Barry. He has actually been visiting my home and me and my family had dinner with him and everything. And I really greatly admire his work. I mean, he has done so much. And I fully understand why when Amazon came to him.

Jacob Beningo (14:02.19)
He did what he did. I mean, I would do the exact same thing. But to me today, I mean, it's not a true operating system anymore because it's so much in there, even though you can say it's still rich at imagining and it's handled by a very large organization that, I mean, their whole purpose is making profits. And whereas the Linux Foundation is, yeah.

It's a nonprofit organization and well known and have established stand out on how to do things. So I'm much more, I mean, CIFR is open source, FreeRTAS not so much anymore. Yeah, I think you're absolutely right. The idea that open source, it's really the whole community is contributing to the direction of it. There's, like you mentioned, there's discussions and...

people to the community deciding what goes in and what goes out. Whereas now, when you look at some of the other ones that are being purchased by a large tech company, not that necessarily they're going to do things that would still help the community, but they are driven by a different beat of the drum, shareholders and things like that. So, and maybe even the integration paths in the future for other products that they're developing. And again, I mean, it's not the...

That's how the system works. I mean, that's how companies work. So it's not, I mean, I also run a company and of course, if a company can kind of manufacture, you cannot do a monopoly on IoT, but you can at least try to get people locked in and then you can kind of have your own monopoly inside your own silo. And definitely that's what they're trying. They're trying to lock people inside their silo so they cannot get out.

And then of course they earn money. I mean, that's, I think that's a pattern that is all the way through all kinds of tech companies out there, Microsoft, Apple, and so on, Google, and so on and so forth. It's all about making these silos. Yeah, absolutely. And there's a lot of benefits to, you know, IoT developers and embedded developers to be able to leverage a silo like that, right? With all those integrated stacks, you can...

Jacob Beningo (16:26.83)
Instead of having to integrate all that yourself, the internet's just there. I can focus on my application code for my product. If you can live with the silo, then there's a lot of benefits that can come with that. I have read a lot of documentation in both silos, I would say. And both of them, as I say, I only see those two major silos right now, Microsoft and Amazon AWS silo. And both of them, they state that, of course, you can move this.

application code that is on top of the real -time operating system to something else in theory. Because as you say, the amount of work you have to put in to moving this and implementing it, that are not going to happen. If you are a developer out there and you're going to tell your boss, okay, I would actually like to use, let's say, Cepher. I just have to move all this application code over to Cepher and...

it's only going to take me something like 100 hours. No way you're going to get that decision through the board or management. Yeah. Yeah. And so I think it's interesting because we have been talking about these kind of commercial solutions and the IoT stacks that Amazon and Microsoft provide. There are developers out there. There's always been a very strong movement for using open source software within the software community. We love our open source tools. I mean, you look at

GitHub and all the other different repositories out there that are open and have open source software. I mean, there's a lot of people contributing to the community. And so why don't we talk a little bit about Zephyr and some of, you know, you defined it as kind of a true open source real time operating system. Could you elaborate on that a little bit and see where, let us know where you see its place amongst, you know,

when you have these two big commercial providers, where does Zephyr fit in there for people? Right now, definitely on none, you can say internet, none IoT systems, because then the complete stack on top is not important. I do also see it from... I'm just guessing.

Jacob Beningo (18:49.582)
But I do see that it seems like some of the microcontroller vendors, they're like, okay, we have to support both Azure RTOS and Free RTOS, Amazon's RTOS. Let's just go with Zephyr and then we're on the sideline. And then if people want the other two, they can just support it themselves. I've seen some of that, especially in some of this new low power threat, if you have worked with Threat Matter.

some of these radio modules and stuff. It looks like it actually has traction on Cepher. I don't know why. It could also just be historic reasons for some of these microcontroller vendors. But it could be, you could say that some of the microcontroller vendors doesn't want to choose and they definitely do not want to support both silos. So they just go on the sideline and say, well, let's just do Cepher and then we have...

We haven't chosen anything yet. That could be. But otherwise, I actually see Sephora having, it's going to be a hard time moving forward for open source. Because, I mean, these two, I mean, Microsoft and AWS, I mean, they're big players. I mean, they have a lot of resources. So they can make a lot of things easy for you if you choose their environment or their platform, a silo, you can say.

So it's gonna require a lot of open source developers out there to be able to match that.

Okay, gotcha. And I have seen, so I don't know if you've seen this as well, so I'd be kind of curious to hear what you've been seeing. So with Zephyr, I have started to hear or see a lot of traction. Like they have, for example, I think it's in Prague coming up in June, they have like a developers conference. I've started to see some of the Silicon vendors pushing Zephyr a little bit more, like you kind of alluded to. Yep.

Jacob Beningo (20:55.95)
So you're really seeing this as more of kind of a disconnected solution or do you see, are you starting to see traction with Zephyr in such a way that we may see, I don't know, third party stacks or things that help you connect to AWS or Microsoft? I can tell you what we do. We want to support all of it, but we also have a very agnostic mindset, you can say. So...

But again, we also have to choose some platforms and again say some platforms will help you or will do it together, I mean, for our customers. But also, so today we do our software is agnostic. So we have an abstract layer and then if you implement that, you can run on the one or the other. And Cepher is one of the operating systems we say we have to support that.

because we do see traction. But also to tell people, well, we have, of course, FedEx and free authors, but we also have a, you can say, open source alternative if that's what we want. But I do also see, I wouldn't say the market is mature, but maybe it's not that mature yet because my guess is, or my experience is that some of the projects out there or some of the companies out there,

It's maybe only the first evolution of their real -time operating system product. So, I mean, it's also about them getting up to speed. And I mean, if it's the first project you do, or maybe the second, you probably not want to take too much risk. So you're going to go with Free Art House or FredX or...

or something that is fairly well known and have everything implemented for you, or at least a lot of it. And then I think when you start playing around with the... When you start playing around with the real -time operating system, you maybe go and say, okay, maybe you should not get too much locked in into one platform. And then... So it's maybe also just a question of time.

Jacob Beningo (23:13.102)
Again, I don't know, but right now, the exact moment here, I see Zephyr having a hard time, but I also think it's maybe just we should wait and see what's happening. Yeah, I think it's I find it interesting because when I originally heard that the Linux Foundation was going to create the real -time operating system Zephyr. Being a microcontroller person, my

my response to that wasn't, well, who cool that there's going to be this open source RTOS. It was, dear God, here we go, another RTOS. Another standard. Another one, right? But since then, I've kind of seen, like you've been talking about, FreeRTOS is kind of a standard open source one. We had ThreadX, Azure RTOS. But Zephyr really has, in those years, kind of shown to be, at least within the last couple of years,

a kind of a third major popular one that, you know, like you said, I think has some issues or some struggles going forward, but is also maybe starting to gain traction enough to maybe be able to overcome some of those. I'm not 100 % sure yet. I guess time will tell, but. I could try to pitch another crazy idea. It could be actually that Free Art House and the Amazon silo could be in trouble. And I...

If you follow my argument now, I'm not saying it's going to happen, but this could also be what's happening in the future. I mean, so you have Microsoft because what I see right now is a lot of companies, IoT is moving from something that they just want to show on a trade show. I mean, something where it was not real. I mean, we have customers where the only reason we had a project from them was that,

Okay, they had to show something on a trade show relating to IOT. And then we were there and we implemented the project. And suddenly, some time went on and suddenly they were like, okay, all our products are now nearly customers actually asking for IOT. So now we go from some marketing gimmick to real production. When you get to real production, suddenly the IT department is involved. I mean, the ones running servers, running...

Jacob Beningo (25:38.382)
your mic, your laptops, your PC stations and all that and probably also your ERP systems and your website and everything. And these guys, I wouldn't say are all Microsoft, but I would say mostly Microsoft minded. I mean, because if you go to the industry, I mean, most, of course you would still have Apple. But I see that mostly by developers.

If we go out in the industry and you see a person from sales or so on, a lot of time it's PC, Windows based. So what I see right now is this movement from, yeah, you can say marketing gimmick to production actually involves getting the IT department more and more involved in the process of the product. And these guys, without any specification or anything else documented,

what they will say is we have to shoot Microsoft because that's what they know. They already run probably Office 365 and so on and so forth. So right now I see, even though Microsoft was a little late or later than AWS, it definitely to me looks like they're getting more and more traction in the IoT industry and they want to be in there.

getting people to decide that they have to go to Microsoft. So it could be that in the future you have Microsoft being the winner of the embedded IoT development. And then you have an open source operating system. And then you have a minor player being Amazon. I know it's a crazy idea because right now they have lots of traction too. But so it's just a crazy idea. But it could be in the future actually.

Cephyr is the open source initiative and then you have the good old Microsoft being closed source and all the good and bad things about that. And then you have something where Amazon kind of falls to the second place. I don't think it will ever be completely out of the market, but maybe not as dominant as they are today. I know, crazy idea.

Jacob Beningo (28:00.782)
But you know, I thought that's what you know, that's what business leaders and stuff like that industry do, right? We look at the landscape and we we have a crazy thought or idea, but it's a prediction for what the future might bring. Right. And so I think it's you know, it's a it's a valid prediction. So so for sure. So I find that to be very interesting. And I think that there is a Zephyr Zephyr. You know, you mentioned some challenges with connectivity stacks.

I've kind of seen Zephyr maybe as having a, I'll say training type of issue because Zephyr borrows a lot from like the Linux style of doing things. It has a device tree. It uses tools that are very similar to Linux based development. What kind of challenges do you see between like a micro, you know, someone who traditionally works in embedded microcontrollers?

adopting Zephyr versus maybe someone who is using Linux already and looking maybe to do some embedded work or some IoT work. Could you maybe talk a little bit about that? Yeah, I actually want to maybe change the direction a little because what I actually have seen a lot of discussion about is it's actually we need more embedded developers basically because right now everything is...

everything is moving on to Internet and I see a lot of discussion where are these people are going to come from. Of course, the education system is pushing out a lot of people, but are we going to catch some of the Linux guys and get them down to doing embedded stuff? And one of the...

things if we have talked a lot about the future because one of the things I see is some kind of combination of real -time operating system and interpreters. So maybe not the usual suspects like Java or other type of but more specific languages that are made for.

Jacob Beningo (30:21.646)
for embedded programming. And then you could maybe get some of the more high -end programmers to actually develop on the embedded stuff. And you can reuse those guys, the competencies that you already have in -house could actually be used on that. And that could also be kind of one of the rescues for Cepho.

because then suddenly it doesn't matter that much which real -time operating system or micro control or that matter you run on. If you can just implement the interpreter, yeah, you're up and running and then you can choose from any silo or any environment. So that could be a way that the Cepher could actually...

win the complete market. Well, not the complete, but yeah, not be left out, I would rather say. But again, I think I also talked about in the store, I do see some pushback on this because especially from the old embedded engineers who are used to compiling their own C code and moving bits around and talking directly to hardware.

they are not that keen on using these high -end languages. To them, they're getting way, way, way too far away from the hardware. So they don't like it. But who knows? Because right now we have had this push from you can say bare metal up to real -time operating system. Next step could be up to some higher type of abstraction. And I don't think it's...

Many people say, well, it's going to be Linux or some other kind of high -end operating system. But I think that's too much complexity. We have, there has to be something else in between, or at least because running Linux on a small device, I mean, too much complexity for what you actually trying to solve. So it could be these other environments, interpretive environments where it's much more easier to reuse code and much more easier to, yeah.

Jacob Beningo (32:42.894)
Yeah, push out new code and so on and so forth. I probably went too far away from the path you set out, but it was just an idea that I can remember. I thought about here the last few days. Yeah, no, I think that's great. I think that's a real important point actually to keep in mind for people. So I know several years ago, I wrote an article about...

the soon to be extinct embedded software developer. The reason I wrote that was because I saw this trend of we need so many bad developers, but we can't produce enough. And there's so many high level app developers that at some point you just and the computing technology for microcontrollers has evolved so much from, you know, the basic eight bit microcontrollers we use, even though they're very popular, surprisingly in the market. But, you know, there's this elevation that's essentially required away from.

the low level stuff, just like we saw originally in the computer industry, so many decades ago where it started low level and then it moved to Windows and Linux and high level operating systems. The big question is probably, did you write that article a little too early? I don't know. Probably. I wrote this probably like five years ago or something like that. Every now and then I get, maybe it was pre -COVID for sure. And every now and then I will get a person emailing me saying,

I read your article and should I go into embedded systems? And they're all worried because they're a third year college student and they've already got started down the path. And it's, no, no, no, there's high demand for embedded developers. But if you can do a little bit of both, I guess you could say, or just be aware that the industry evolves quite rapidly. So.

We're all going to be replaced by AI anyway. Yeah, exactly. Not to start a totally new breath here. Yeah. You mentioned that literally right before our call here for our discussion. I had another call with a company that's doing essentially using machine learning to read data sheets and allow a developer to create a model of the chip.

Jacob Beningo (35:01.198)
or the part and that allows them to very quickly develop hardware abstraction layers and the code for those chips. And so, you know, again, kind of same. We are impressed too here on how much it can actually help you. I wouldn't, I would never, I mean, let it unguided or what's it called, unmanaged, but actually to help people with.

So we have this abstraction layer on the operating system. And our customers, if they come with a complete new type of system, some kind of internal real -time operating system they have done, they have to implement this layer themselves. We are actually experimenting with the AI on if it could actually supply us with...

And it was quite good. I mean, it could read our documentation and it would actually tell in the real -time operating system documentation. And then it could actually get quite far at trying to make these adapters. We were very impressed, I would say. Yeah, it's really amazing. And just the fact that it's, in my mind, still...

quite early, right? I mean, it's, you look at where the industry's gone and just, or where it seems what's evolving over the last six months and just the exponential curve and capabilities that are coming out, it's quite amazing. So. Yeah. And going back to where you said, well, you have people asking you if they have the right path doing embedded stuff. I do think that my advice for people when they ask these kinds of questions is that,

you shouldn't be too much focused on just one part of the industry. Because, I mean, it's not servers and not that much different from microcontrollers. It's just, I think, they're just bigger, run faster. And of course, they have an MMU and stuff like what you don't see on the small. But again, I mean, today, I think it's hard for people to stay focused and...

Jacob Beningo (37:22.862)
and just do one part, a little corner of the industry. I mean, because today it's getting, I mean, if you go 10 years back, you could have your small device and you can just stay in there. But today it's internet connected. It has to be connected to apps. It has to connect to servers. Probably installers have a PC program they have to connect. So you have to be, at least you have to have some knowledge of the different systems. And...

It's just so much easier if you do. And also your choices are so much bigger. So normally when people say, I want to be an embedded developer, I'll tell them, well, maybe you should just try to be a developer. And then with that focus on embedded, that I think is actually the best advice I can give people like that. Because yeah, you never know. And again, to...

To get back to the point, we're going to be replaced by AI anyway. Yeah. Well, of course not. But I think we will. I don't think like you're like yourself, I don't think we will be replaced, but I think we will be very much augmented in the development cycle by AI. All right. So.

Once again, I want to thank you for your time and sharing your thoughts today. Where could listeners potentially go to learn a little bit more about you and the type of work that you're doing? Well, they can go on our website. That is naptow .com. N -A -B -T -O dot com. And they can actually, you can test out our products for free. Once you sign up on our console.

You can actually create three or two free devices for free. So it's, and again, we have, we try to support all kinds of operating system and even bare metal. So it's, we are trying to support it all. So it's, yeah, of course you can also go on Twitter. We have a Twitter feed and also a Facebook feed. So, and feel free to contact us in our chat.

Jacob Beningo (39:40.206)
because we always have embedded engineers there to help you. Perfect. Thank you.


People on this episode