#0004 - Embedded DevOps to the Rescue

In this episode, Jacob Benningo discusses the importance of DevOps in the embedded software development industry. He emphasizes the need for DevOps to improve project delivery, efficiency, and product quality. Jacob provides insights into the principles of DevOps, the implementation of CI/CD pipelines, and the impact of DevOps on software development processes. He also shares actionable steps for implementing DevOps in embedded software development.

Takeaways

• Embedded DevOps is crucial for improving project delivery, efficiency, and product quality in embedded software development.
• The principles of DevOps include incremental value delivery, improved collaboration, automation, and continuous improvement.
• Implementing a CI/CD pipeline is essential for automating software development processes and improving code quality.
• DevOps transforms how embedded systems are developed, leading to cleaner and more efficient software development processes.
• Actionable steps for implementing DevOps include defining an ideal pipeline, creating a roadmap, and setting up a DevOps framework for remote software building.

Transcript

Jacob (00:00.698)
Hello and welcome to the embedded frontier podcast. I'm Jacob Benningo and I will be your host for this episode. just like always in the embedded frontier podcast, we are going to be talking about topics that help will help you, become better embedded software developers, better teams and help you deliver on time and on budget. And of course, as part of that, we look at what's coming up, what the frontier looks like for us embedded software folks. Now for us today.

That topic that we are going to be focusing on is DevOps. Now you might think DevOps, that's not a very exciting topic. Is that even an embedded topic? And the answer is yes, it is. The larger software industry has been working with DevOps for years and many embedded software teams are just starting to realize the power that it can offer your development cycle. And so today we're going to talk about DevOps and I'm going to provide you with what DevOps is, the values it entails, what it makes it.

up what makes it up and what it looks like for us embedded software developers. I'm going to try to give you a couple of action items so that you can start to leverage DevOps to improve your own development life cycles. Okay. So with that in mind, let's dive in and talk about DevOps. Now, before I actually tell you what DevOps is, or we look at any real definition, we kind of need to frame this through an important lens that has a lot of embedded software teams struggling today.

If you look at any of the latest surveys, you're going to discover something that's a little bit dismal. If you look at how often we are able to deliver an embedded project on time, guess what you're going to find? Only 35 % of teams are able to deliver successfully. Now, if we were back in school, what are you going to find? That that's a failing grade, unless you're taking quantum physics or something like that, right? 65 % of projects are actually delivered late.

And I'm going to be honest over budget. Right. And that's a big problem. And it's a problem that we really can't live with as developers, as managers, and to be completely honest, as companies. Right. We can't make our customers angry because we're delivering a product late to them. Right. I, one of the first jobs I had, I actually worked as an engineer at a startup where we had a company management that was continuously promising that the product.

Jacob (02:27.738)
was just three months away. I worked at that company for two and a half years before it went out of business. Okay. And that product never fully came to fruition, which to be completely honest was a little bit disappointing, but it's an aside and not probably important for what we're going to be talking about today. DevOps for us is solving an important challenge. It's helping us to be more efficient. It's helping us to speed up our deliveries and it is helping us to improve the quality of the products that we are actually delivering. All right.

That is what our ultimate goal is. We're trying to still get our projects delivered on time and on budget. All right. So if that is the case, what is DevOps and how can that actually help you? Well, as it turns out, there's actually four principles in DevOps that we need to take into account. All right. DevOps is actually about focusing on providing incremental improvement, incremental value to the users and the customers of our products.

And we should be doing frequent and incremental improvements. All right. That's kind of the first principle that guides all of DevOps. The second principle is that we're trying to improve collaboration between development and operation teams. Right. If you're going to successfully deliver a product to the market, you need developers, you need operations, and we need quality assurance. Right. All three of those silos have to be able to work together in order to be successful.

And so what we're doing is with DevOps, we're breaking down those silos and where all three of those intersect. That is what we're considering to be DevOps. All right. And you'll sometimes hear DevOps called DevSecOps. We'll actually talk about that next week with my guest, but DevOps, you know, DevOps itself is all about development and operations. We had DevSecOps in there. The sec is the security, right? But we'll save that for another time. Cause as a bed software developers, let's be honest, security really

Isn't that exciting to us, at least for most of us. All right. So we've got improving communication protocol, communication between these different silos of the business, breaking down our barriers, improving communication. We're also trying to automate as much of the software development cycle as possible. As you can imagine, how many of us today in embed software do a lot of manual repetitive work? How often do you verify that your build is still working by compiling your code? How many times are you running a linter?

Jacob (04:49.05)
How many times are you running static code analysis? How often are you beautifying your code? How often are you running metrics analysis on your software, running your test cases? I hope the answer to that is often and frequently, but the problem for most of us is that we're doing that manually. Okay. You want to be able to automate this process. If we automate it, that means we can push new code, let it go automatically in the background while I'm then working on something new. Right. And I can get a report that points out, you know, where some issues are, then I can go and fix those issues.

And so on and so forth. All right. And of course, the last principle that we're thinking about here is continuously improving the software product. Right. So this should allow us to think about this. One of the big problems that we have is we deliver late. I see so many companies that try to throw in, you know, the kitchen sink and everything with it into their product, and they don't want to launch until it's all in there. Well, if we follow these principles, one of the things we should be doing is actually, first of all, identifying our core features in our product, what the minimum viable product is. And from there.

Once I have those products, those features implemented in my system, then I can start adding in the extra stuff, right? But if I don't have the, you know, if I don't have my core features, I'm never going to be able to get my product out to market. In fact, one of the problems with that, that startup that I mentioned that I worked at when I was a young entry level engineer that, you know, didn't make it. One of their biggest problems and reasons they didn't make it was that they never focused on the minimum viable product. They were constantly chasing the latest and greatest feature.

that had to be in the product. But the truth of the matter is it's 15 years later, maybe 20. And some of those features we had to have before we launched that particular product still doesn't have those types of features in it. And so we were, you know, the, the management of that team was chasing a dream and they weren't focusing on what the minimum viable product was. Once you have that and you get that out to market, yes, continual incremental improvements of those features and adding new features is absolutely.

you know, something that customers are going to want. But until then, you know, you want to make sure that you're hitting the minimum viable product. Now that's going to help you. Right. So those are kind of our core four principles that we're looking at here today. Now, as you stop and think about those four principles, I hope they sound kind of familiar to you. Where have you heard those before? Incremental value to customers, improving collaboration, communication, automating development, life cycle, continuous improvement that I've heard that somewhere before.

Jacob (07:16.442)
Well, that improvement that you've heard before probably sounds a lot like the agile values that were written down over 20 years ago. Right. And I often joke that DevOps today is nothing more than re, you know, restating the agile values in a more modern way. Right. In the old days and 20 years ago really isn't the old days, right. Maybe I'm dating myself, but the, the agile values talked about.

in, you know, individuals and interactions over processes and tools. You know, that was core, you know, individuals and interactions. Well, what are we doing? We're worried about communication here, right? Okay. So communication is important. We're worried about working software over comprehensive documentation. Well, you know, with DevOps, we're really thinking about incremental, small incremental improvements that are working, right? And so that piece is still kind of there. Customer collaboration over contract negotiation. This again is still just all about communication and then responding to change over following a plan.

Right. Continuous improvement, small incremental improvements again. And so we're really looking at these core values, really find their way into the dev psych, that's, DevOps type of, you know, principles that they're following. So they're just kind of restated in a slightly different manner than they ever were before. All right. And when we stop and think about DevOps and how we can leverage this, the idea here is that we can get.

our customers, our company, you know, our team to essentially create a value feedback loop. Right. If I get a minimum viable product out to my customers, I can create a delivery pipeline through a continuous integration and continuous delivery pipeline. That means I can build my software. I can test it and then I can deploy it to them. Right. That's a good delivery pipeline. But the problem with that is that's one directional, right? What we really also need is feedback. We need a feedback pipeline that allows us to monitor our embedded products.

and generate reports on how users are using them. Any bugs that are popping up, performance issues and those types of things. Right. If we can get that, if we can get that observability into our systems, guess what we can do? We can identify new features. We can identify pain points for our customers, which that means we can create new features, fix those challenges that they're having. We can then build, test, deploy those through the delivery feedback. The customer gets happier over time, new features, their lives are easier. Guess what?

Jacob (09:42.618)
We notice that through our feedback and observability pipeline. And guess what? The process repeats. And this creates what I call a value feedback loop. All right? Now, you might be saying, OK, this is great, Jacob. But this is great for maybe people who are in management, who are running teams. What about the low level implementations? How do we implement a DevOps type of solution? Now, for embedded software developers, you know,

Well, first of all, this should be a cultural change within your organization. All right. But today, given the time that we have in a podcast, I'm just going to talk about what we do with embedded software developers. OK. And what that looks like, you know, for implementing a CI CD server and pipeline. Right. As a developer, one of the first things you do is you think tactically. All right. You're not thinking about strategies for DevOps and long term pictures of interactions with customers. That's really what what folks in management do. Right.

And I wear multiple hats and being a consultant and a business owner. So sometimes I am helping, you know, business owners and CTOs and directors of engineering come up with the strategy that needs to be put in place for developing a reliable embedded DevOps solutions. And sometimes I'm working with the developers and we're technically saying, how are we going to implement this? How are we going to use Jenkins? How are we going to use GitLab to do this successfully? And what I'm going to do with you here today is I'm going to talk through some of the basic parts of a CI CD pipeline.

what the minimum viable pipeline you need as embedded software developers. I'm going to talk a little bit about the different types of builds that you actually should have in your embedded system. So if I were to stop and think right away, the way that a CI -CD pipeline works is I have continuous integration. And then once I continuously integrate and get to a certain point where I'm happy with the feature set that's there, then I can continuously deploy or deliver that particular product. And right there, you might be saying, well, Jacob.

I'm working on an embedded product. We don't just continuously deliver to our customers. We release on a quarterly or yearly basis because our system is special and we can't just be pushing updates to our customers. And in some cases, why would you ever want to do for an embedded system continuous delivery? As a consumer, do I want my microwave to be continuously updating its firmware? Probably not. I just want the thing to be able to cook my food.

Jacob (12:02.714)
I don't need daily updates or to be annoyed by it when I go to use my microwave that it can't work because it's doing a firmware update, right? So you do have to decide what delivery means to you. In many cases, delivery can just be, you know, to your own team locally to be able to use and kind of flesh out and test and validate things, right? Maybe someone in quality assurance or something like that. But I do digress there a little bit. What I want to start to focus on here first is the CI piece for you. OK, and that's probably really what we're going to focus on here for the rest of.

today's podcast. And if I am thinking about continuous integration, the first thing that you need to do, of course, is first pick some type of DevOps framework. Use Jenkins if you like open source. I personally like GitLab. It's the one that me and a lot of my customers use the most. You can certainly use GitHub or any of the other tools that are out there. Pick whatever works best for you. But you have to get some type of Git solution with a DevOps framework associated with it.

Once you have that, you know, you can actually start to implement tactical pieces there. But before we start to implement the, you know, the, the day to day stuff, you had to stop first and say, Hey, what is it that I'm trying to do? What is my strategy? What is my end goal? And our idea here is that we want to be able to automate as much as possible to free up developers, to focus on the developing new features, right? We want to remove the manual labor. We want to remove anything that's going to be a roadblock for us to be efficient and deliver faster.

more and higher reliability software. And so when you do that, one of the first things I recommend that you do as a team is you design your ideal CI CD pipeline. All right. This is, you know, open up PowerPoint, open up Visio, open up Lucid charts, open up whatever your favorite tool is. Go ahead and get that up and running and then just think through, you know, what does a pipeline look like to me? And a pipeline is just, it's nothing more than a collection of jobs that run in a sequence or even in parallel.

that perform specific tasks that you would normally do manually. And you might be saying, well, Jacob, what are some of those things? Right? Well, if you look at kind of a very basic standard CICD pipeline, you might have a build job. You might have an analysis job. You might have testing jobs, you report jobs, and then you may get to a point where you have a merge job and then a deployment job that's going to deploy your software. Now, to be honest, I also have like release jobs and other things built in there. But again,

Jacob (14:27.93)
We're going to keep things relatively simple for today's podcast so that we don't confuse anybody, but hopefully still give you some good value added ideas of things that you can do. All right. Now, thinking about those things today, what we'll think about is we'll need a build analysis, testing and reporting pipeline inside of our CICD server. Okay. So what does that look like? Well, I don't know about you, but there's lots of times where I've gone to build software, but that may be a colleague developed or something. And I just found that I couldn't build it successfully. Right.

And so the first thing you got to do is make sure that you can actually build your software. and don't get me wrong. I've probably given software to colleagues as well that, that didn't build successfully. Right. And so, it's not just, colleagues, not me. It's no, we all make that mistake at some point. so the first thing we want to do is make sure that we can actually deploy our, our first thing we want to do is build our software in multiple environments. All right. So that's the first job that we have to do. Now, when you're building your software, there's actually five different areas that I always say people need to be able to build their software for.

We often always just think about one way. You know, we often just think a debug build, right? But a modern software development cycle actually has to build in five different ways. First, you need a debug on target so I can deploy to an embedded system and I can debug it. Second, I need a release build for that embedded target. Okay. Those are the obvious ones. The non -obvious ones become the third type of build, which is testing. I need to be able to run unit tests, integration tests, regression testing.

system level testing, all of that kind of stuff. So that's the third type of build I need to be able to do in my build system or in my CI CD pipeline. I also might want to be able to build for simulation to be able to simulate. And then I also want to build for analytics. Okay. Now some of those types of builds you'll find, if you think back to what I was just talking about, they actually find their way into the CI CD pipeline. Right. I said, I want to be able to build, analyze, test and report. Okay.

So those almost correlate to all five of those, right? I had a testing build, a reporting build for it, an analysis build, and then a debug and release, okay? So I want to make sure that I can do those things. And oftentimes I'll use something like a Docker container or some other virtual environment that you can actually deploy into your CI -CD server that will be able to go through and validate that, yes, I can actually build my software for my target platforms, okay? So you can verify all those builds and that's great, all right?

Jacob (16:52.794)
That's what you're low hanging fruit. I'm going to be honest. You can probably get that up and running in half a day or less. All right. Depending on how familiar you are with tools, you know, oftentimes the first time you set it up, it takes a lot longer. You know, it might take a couple of days. You know, if I go and set up a new CI CD server with just build capabilities, it's, you know, literally, you know, I've done it so many times, it's like an hour or less. Right. But part of that is I can actually leverage the tools and scripts and things that I've already developed over the course of several years of doing this kind of thing. Right. Okay.

So we've got the build that we have to do. We then also want to be able to analyze our software. This is the area where I think a lot of teams can actually benefit. And this really helps to improve the quality of your software. Instead of leaving up code quality to code reviews, making sure a developer runs a linter or a code beautifier, or looks through it and make sure that you're following certain standards or miseracy, or just checking the cyclomatic complexity of your code.

I can't tell you how many teams I come across that don't have a way to check the cyclomatic complexity of their code. It's kind of like a embedded software, even software engineering, one -on -one type of thing to do, right? And there's lots of free tools that allow you to do that. But, you know, I find teams don't do that type of thing and they end up with very complex code that's very hard to change, maintain, and just gets them in a lot of trouble. And so having a build, or not a build, but a job in your pipeline that analyzes your code for

Cyclomatic complexity verifies that you're meeting your coding standard and your coding styles. Again, this is something that once you do that, you can build artifacts that will stay on your CI CD server that can then be tied to releases that can be tied to daily reports. There's nothing like walking in in the morning and getting a report that says, Hey, these things are good. Hey, here's the cyclomatic complexity reports. There's these five functions that were developed yesterday that aren't meeting our standard, right? They're too complex. Nothing like getting that thing. First thing in the morning, you fix them.

And then, you know, you move on to the next features, right? Verifying that the quality of the code you wrote yesterday is actually meeting your standards. And then if it isn't fixing it right away, before you totally forget about it, I can't tell you how many times, you know, in the past where I would like to deliver code to a customer. And then you got feedback like three months later, by the time they gave you feedback, you're like, yeah, I don't remember what I did on that. Or you start, you work on a project, you deliver it. And then they come back to you three, four years later and say, Hey, we need this change.

Jacob (19:18.266)
And then what are you doing? You're going, yeah, I don't have the processes in place. I don't remember this code. It's going to take a longer time to get up and running. Right. Well, with, with DevOps and running these pipelines, if there's something not right, you catch it right away. And so that removes all that extra overhead to try to figure out again where you were, what you were doing, what is this code doing, all that stuff. And even if you document your code, good, there's always gaps that you have to worry about. All right. And so this is kind of that second pipeline we build and we analyze. Okay.

The third, which is absolutely important, which I see embedded developers getting better at every year is testing. The testing that ICS starting to do the most is unit testing, which is great. It forms the foundation of our code, right? But I start to find that we struggle a little bit more with functional integration, system level testing, performance testing, even power testing. All right. just the other day, I wrote an article on embedded related where I mentioned how, you know, the drool scope.

220, the JS220, it can be used in CI -CD pipelines to monitor hardware in loop testing and kind of track how much power you're using. If you're building an IoT device that's battery operated, that's huge. Instead of getting to the end of your project and discovering that your battery is not big enough or you're, you know, then try to say, OK, well, what features that that's using all this power, you can track in a CI -CD server throughout your entire development cycle what the energy consumption is. Then that, you know, you can profile that system.

without you having to sit there and watch it and deal with it. And that right there, man, that's so powerful. All right. And so testing our unit tests, regression tests, being able to identify problems right away. Again, yes, we have to maintain these things. Yes, we have to write good tests. If you write bad tests, it doesn't mean anything, right? If you can get a hundred percent code coverage, that doesn't mean you're, you're bug free. That just might mean that you have gaps in your tests and you know, yeah, you're covering all the code, you know, the basic minimum branches, but it doesn't mean that everything's working the way that's supposed to. So we still need.

You know, other pieces of accountability requirements, traceability and things like that in our development processes. But, you know, thinking besides that, just focusing in on DevOps, you know, again, being able to test your software like that. It's been a game changer. I know for me as, you know, when I work in the development environments, being able to write tests, unit tests with my code, it opens up a whole new ball game to be able to detect defects early decreases how much debugging time I spend.

Jacob (21:41.882)
I used to spend like 80 % of my time debugging code when I was a younger engineer. You know, when I adopted test -driven development and DevOps, maybe like, I'll say over the last five to seven years, actually it's probably been five years, completely has changed the way that I develop embedded systems. It's also changed how I architect systems. My code is much more decoupled now. It allows me to implement more simulation and testing on target, get product in front of customers sooner.

on hardware that isn't maybe the final version, but that still represents how the system will behave. And so that type of feedback in that value feedback early is absolutely huge and important. And I'll be honest for you developers who are listening, that may not seem like a big deal, but those of you who might be managers, directors of engineering, or running your own small businesses, getting that feedback, being able to simulate having to not wait for hardware to arrive before you can start testing the stuff with your customers. That right there can be a night and day change in the way that you run your company.

and deliver product. You have to remember, you know, if you do simulation techniques like that, that that's not the final product. So while the customer might seem to be like, all right, I'm ready, let's ship it. You know, you still got to wait for your embedded hardware to show up, but at least allows you to to kind of get the application, the business logic pieces of your code done early. And, you know, of course, continuously tested as part of your CIC setup within your DevOps. Right. And then the last part here that we'll talk about before we close out for the day is reporting.

Again, looking at, being able to get results for code coverage, checking the metrics of your code. it's just so important. I really do come across a lot of teams that don't track those types of things. And you know, whatever gets tracked gets managed. All right. So if you don't have metrics that are defining what your software should be doing, you know, what levels of complexity, what the cohesion looks like, what your power profile should be and all of that kind of stuff.

Then I'm going to be honest, you're going to end up getting tripped up later in the development cycle is going to cost you more time. It's going to cost you more money. It may even cause customer headaches. It's going to cause your team to go into reactive engineering. I personally prefer proactive engineering where instead of fighting big fires and whatever's coming up in the biggest priority of the day, that's reactive, right? And that's not a good way to run a business or an engineering team. You want to make sure that you're proactive, that you're able to.

Jacob (24:06.458)
see those fires, put them out before they happen. Yes, there's always going to be stuff that comes up, but when stuff like that does come up, just use the Eisenhower matrix, right? If any of you think back to it, you know, it's urgent versus important versus not urgent versus not important. Very, very well known. I actually I'll tell you, it's so important to leverage that type of thing to be proactive versus reactive. I actually over here on the right hand side of my desk here have posted up on the wall. OK, so.

that allows me to remember when something comes up, Hey, should I actually be looking at this? Is this a distraction for me? Is it a waste of time? Is it actually a necessity? Is it a productivity related activity? And that helps me to analyze on the fly to make sure I'm not reactive. But instead I'm being proactive with the way that I am, you know, working with customers, running development teams, writing my own code and, you know, training people and working on things like that. Right. Again, I wear a lot of hats oftentimes, so I have the different perspectives.

based on what it is I'm doing for that particular day. But, you know, with that in mind, we've talked about a lot of things here today, and I want to make sure that despite all these things that we've talked about, that you have some action items that you can go back and take away from this and use them to improve your own development cycle. All right. We don't necessarily want to be using, you know, buzzwords like DevOps and doing things that maybe waste our time or cause bigger struggles for us. We want to do things that...

are initially low hanging fruit that we can go out and do. And that will save us a lot of time and effort and free up time then to be more proactive with our development cycles. And so I've got just a couple of really basic action items here for you today. First of all, I'd like you to stop and think about what your ideal pipeline might be. All right. I've given you an example, something as simple as build, analyze, test, report, merge, release, deploy. All right.

That is right, right there. A very basic pipeline, but that's, you know, maybe some of that I would put in my ideal pipeline. I don't know what you need in yours. And so I'd recommend you sit down, just kind of map out what an ideal pipeline might make might look like for you. All right. So that's the first thing. Secondly, based on that ideal pipeline, start to put together a roadmap to get it done. All right. Strategies, ideas are fantastic. If you don't take action, they're completely worthless. Okay.

Jacob (26:32.41)
So first of all, put together your strategy of what that ideal pipeline looks like next. Let's get actionable. All right. Put together a roadmap of when you're going to implement different pieces of that pipeline. All right. The roadmap might be wrong. You might fall behind. You might be ahead of schedule, at least put some dates on the calendar, build it into your calendar. All right. So that you start to work on it. You know, one of those things I absolutely love is day stacking. All right. From a skills and development standpoint, if I make just 30 minute progress or an hour of progress every single day,

or I improve by 1%, it doesn't take long before within two, three months, I've made huge strides in whatever it is that I'm working on, right? And so while you might not have the time to sit down and knock out a DevOps solution today, schedule two hours a week to start working on it. Maybe you work on it for two hours at the end of every Friday. Maybe you work on it 30 minutes a day or an hour a day or whatever it happens to be. Maybe it's just during your lunchtime, I don't know. Whatever works for you, okay? But get it on your calendar.

And then start working towards that those solutions. All right. You're going to find the first thing to do once you, once you have those two things in place, you're going to have to get a, a DevOps framework of some sort. Like I mentioned, GitLab, GitHub, you know, whatever Jenkins pick your favorite, whatever works the best for you. And then get an environment set up where you can actually build your software remotely. Okay. That's probably going to involve you looking at a solution like Docker or some type of VM. And just going through that process.

and getting your code to build remotely, you're going to find that that actually helps improve the way that you distribute software within your team, on how you set up your own tools and all of that kind of stuff. Okay. And so I'm telling you right now, once you start going through this process, it will revolutionize the way that you develop your embedded software. It's going to change the processes you use. It's going to make set up times for your software. When you onboard people much less, it's going to help improve your insights into how your software is being developed, how it's operating.

and so on and so forth. Okay. Sorry, if I'm getting too excited, there's just a lot that I've seen that has allowed me to transform the way that I develop software. And maybe even more importantly, the way that it has helped me to transform the way that my customers and the people that I coach and help train, help them to improve the way that they do things. And so at the end of the day, if you just take a couple of little action items like this, work on them, work on them weekly, you're going to find that you will transform the way that you develop embedded systems.

Jacob (28:59.482)
long before you realize it, you'll be like, wow, this is insane. So kind of day stack like that. And from there, I hope that this will help you to develop embedded systems a little bit cleaner. And then next week or next time, in the next episode, what we'll do is we'll go and dig a little bit deeper and talk about DevSecOps. And we'll look at some of the strategic points there on how this is actually done in industry. And I'll have a colleague that will be joining me for that discussion then.

So with that in mind, I do want to wish you happy coding and I look forward to talking with you again in the near future. So good luck with your DevOps endeavors and certainly feel free to leave a comment or reach out if I can be of assistance in any way. I've set up so many of these and trained so many people that I'm happy to share my insights there. So thanks again and look forward to talking with you soon.