EDGE AI POD
Discover the cutting-edge world of energy-efficient machine learning, edge AI, hardware accelerators, software algorithms, and real-world use cases with this podcast feed from all things in the world's largest EDGE AI community.
These are shows like EDGE AI TALKS, EDGE AI BLUEPRINTS as well as EDGE AI FOUNDATION event talks on a range of research, product and business topics.
Join us to stay informed and inspired!
EDGE AI POD
From Ideas to Reality with Edge AI Expertise from embedUR and ModelNova
How can we revolutionize everyday objects into smarter, more responsive entities? Join us in this exciting episode as we explore the cutting-edge world of Edge AI, fresh from the vibrant showcases at CES 2025. We promise an in-depth look at the future of technology deployment with our special guests Eric and John from EmbedUR. With their extensive expertise in embedded products, they bring a fascinating perspective on the importance of blueprints for tech commercialization in today's rapidly evolving landscape.
Our conversation with Eric Smiley and John Marconi introduces the innovative ModelNova, a transformative tool designed to speed up the journey from ideation to proof of concept for tiny edge AI devices. Discover how ModelNova's curated model zoo and datasets can empower developers, ensuring efficient performance even under resource constraints. We dive into practical insights like community contributions, adapting large AI models for small devices, and how tools like Model Nova democratize access across various chip platforms, turning ambitious ideas into reality.
This episode doesn't just stop at the technical nuances; it goes further into the realm of edge AI product development. We share a compelling story of transforming a bicycle helmet camera with object detection to enhance rider safety, illustrating the complexities of selecting the right hardware and the critical role of blueprints in this journey. From MLOps integration to cloud connectivity for continuous updates, our discussion emphasizes collaboration within the tech ecosystem to tackle the challenges of AI deployment. Tune in to learn how these advancements are not only reshaping technology but also enriching everyday life by making objects around us smarter and more responsive.
Learn more about the EDGE AI FOUNDATION - edgeaifoundation.org
Thank you. Okay, good morning everyone. Good morning Morning Davis. Welcome to 2025, pete, I know this is our first one in 2025. Yeah, we got with CES and everything else. It took us a while to kind of get back on track, but it's good, good to be here. Did you have a good, good break?
Speaker 2:Yeah, ready to rumble back to it. I mean glad you survived CES. Personally, I wasn't there, but I'm still hearing and seeing lots of things that set the tone the tone.
Speaker 1:It's like 150,000 people there or something. It was a good crowd, shoulder to shoulder most of the time. Nxp had a really cool. For those who have not been to CES, they always have this cool pavilion quasi-tent thing out in the parking lot in front of the main hall.
Speaker 2:Central Plaza, I think it's called.
Speaker 1:Yeah, it was really cool. You guys had some really good demos in there.
Speaker 3:It was really nicely put together.
Speaker 1:I have to say so, whoever did your booth layout.
Speaker 2:It's a team effort, though there's a lot of the Invite only, though. Invite only, that's what I was going to say. There's the invite only, and then there's the secret. Demos are where it's really at.
Speaker 1:So I mean, I don come Speaking of demos and cool things, so we're going to have Eric and John from EmbedUR today. So this is the Blueprint show, and today's is a bit of a meta episode, right, because we're going to talk about Blueprints, about how to do Blueprints.
Speaker 2:Yeah maybe we'll come back to the ground a little bit. I think so. Since we first formed the group EmbedUR team Eric and John and company have been really involved and very supportive of the initiative here. So I think this is the culmination of a lot of discussion and a lot of good material and, like you said, speaking of demos, this is kind of a recipe for creating not just demos but compelling use cases. And what Blueprints is all about are those real production deployments, which I think we can everyone on the stream can relate is harder and easier said than done, so hopefully after this live stream we'll have a few, you know, interesting tips and pathways to make stuff happen, which is what we're all about.
Speaker 1:Well, that is the hot topic. That was also the hot topic at CES was all around deployment and commercialization. So you know, for years we've had a lot of IOT and Edge and Edge AI sort of development and proof of concepts, and now people are like how do you deploy and support this stuff as solutions? So it's a good, good timing to get this stuff going Cool, Should we? Let's bring on Eric and John. How about we do that? Perfect? Let's see here Good morning John. Hey guys.
Speaker 2:There you go.
Speaker 1:We'll give you guys the fine spots there. Welcome, Eric and John from EmbedUR. Where are you guys calling in from?
Speaker 4:I'm in beautiful Chapel Hill, North Carolina, except it's really cold outside today, which is unusual for us.
Speaker 1:It's kind of an East Coast thing, yeah.
Speaker 3:Yeah, I'm in Chicago and it's negative nine degrees Fahrenheit here.
Speaker 2:Good Lord, that's nothing. I'm north, I'm north of the border, so yeah, that's relative.
Speaker 1:Oh, that's true, yeah, and you're in canada, davis, so I'm I'm out here near seattle, so uh good, good.
Speaker 4:It's funny that I call myself in a cold spot and it's probably the warmest of.
Speaker 1:uh, you know one's going to compete with John and Davis on cold, that's for sure.
Speaker 3:Not today.
Speaker 1:Cool. So what are we talking about today? Let's talk about blueprints. Let's talk about blueprints for blueprints and what embed. You are, and you guys are in the solution business, so it's kind of really appropriate, I think, to have you guys on.
Speaker 4:Absolutely. Thank you, pete. Thank you Davis. Yeah, we've been in the business now for over 20 years helping silicon companies and OEMs build embedded products anything from wireless access points to smart intelligent doorbells, to health products, medical products and partnering with the Edge AI Foundation over the past couple of years now has really been exciting for us to venture into the world of edge AI and see how we can build upon the expertise we've built over 20 years building connectivity and intelligent and cloud systems to the edge about really how to take products from the ideation stage to building a proof of concept to productization.
Speaker 4:And that's even more important today with Edge AI, it's a wild west out there and it's a little bit confusing as to how you get from point A to point B.
Speaker 1:Right. Yeah, they say the Edge is a team sport, right? So no single company can walk in there and say I alone will fix this. You know it has to be a I feel like we're at a really interesting inf.
Speaker 4:So no single company can walk in there and say I alone will fix this. I feel like we're at a really interesting inflection point in changing how the industry thinks about, comes up with ideas and then designs products. It used to be for us over most of our history that a vendor will come to us knowing what they want to build and then ask us to help them build it. And oftentimes they would come to us with ideas that the technology wasn't even ready to do yet, and so we'd have to work with our silicon partners for the next generation of devices. But we're in the mode now where the technology has not just caught up but almost surpassed the ideas. Interesting yeah.
Speaker 4:What you can do with silicon and software now is is beyond the use cases, and so now these blueprints are even more important, trying to highlight to the developers of products what can you do with this amazing technology that you know silicon and software companies have been launching over the past few years great it's, it's a moving target.
Speaker 2:I think is one of the analogies that we've increasingly tried to understand and wrap our heads around. But I think you captured it quite nicely that there's a different shift where you're not coming with specifications, trying to meet those specifications, both the specifications, the solution and the technology landscape, especially if you talk about AI models and the landscape with LLMs and Gen-A moving so quickly. Fantastic use cases, but how do you match them up with the production equipment?
Speaker 1:So hopefully in the next hour, we'll figure out a little bit more about how to do that. We actually got a lot of, I would say, fortune 500-type companies coming in to our events. By the way, austin, february 25th and 27th, I need to do a little PSA, be there for three days' worth of edge AI stuff and meet all these celebrities, but we're going to be talking about solutions. A lot of folks come in and say what can I do with this stuff? Like they have a set of problems you know canonical problems but they're not quite sure exactly what can this stuff actually do? Like what happens when I pair up a generative AI you know model with a sensor, set of sensor models, or you models, or what can sensor?
Speaker 1:models or AI vision do, and exactly how much precision is there and power and cost and all sorts of things. So they usually come in with a good, well-defined set of problems, but they're not exactly sure what can this stuff do for me to help solve it? So it's great that you guys are kind of in the breach, as they say. Clarifying that.
Speaker 4:So that's good, absolutely Right. And you know there's plenty of really smart people out there building AI models for many different use cases and there's plenty of models used out there where a developer or a researcher can go find a model and start experimenting with it on their server or their computer or online on a, you know, like an AWS system on a GPU. But when you really start talking about how do you take that model beyond just tinkering with it and turn it into a product that's running on a tiny device with an MCU and a sensor and you know every sensor processes data differently. Every MCU has a different set of use cases and different set of applications that it can run versus another one and trying to get that model working with high performance and reliably on the edge device, that's where a lot of the complexity lies.
Speaker 3:Yeah, right, and I think a lot of times you don't even have a concept. That's even possible, so you don't even know where to start. Or if you know, can I do this on an mcu? I mean, you need somebody to kind of show you the way, um sure before that idea pops up, that okay, I can yeah yeah my battery powered device cool, let's get into it.
Speaker 1:By the way, I want to give uh, there's a uh john here, and uh, I just want to do a shout out to chapel hill.
Speaker 2:So just uh, let you know he's, he's watching absolutely and yeah, that's a good way to show that for, for those watching, you know we're going to hear from you, know, eric on the bd side, john on the technology side. We also want to hear from the audience questions. You guys have what you want to find out, so this is a super efficient way of bringing up a burning question clarification. You need. Pete and I are here to elevate that to the speakers throughout the conversation and we'll definitely have time at the end to have a form of discussion. But water in mind, everyone watching that this is a great chance to talk to the experts.
Speaker 1:Definitely yeah. Whether you're on LinkedIn or YouTube or Twitch or StreamYard or whatever, type it in Chapel Hill, canada, we'll get it answered. Yeah, exactly, cool, all right, well, why don't we give you guys the floor? And do you want to bring up some slides now, eric, or how do you want to do this?
Speaker 4:Yes, please do and thank you, pete and Davis, for that great introduction and, as we have been talking about for the past couple minutes, what John and I want to bring to you today is a little background on how EmbedUR got into the edge AI space and what we've built with Model Nova to help you and the community to go from that ideation to proof of concept to productization, which we'll talk a little bit about. Why there's so many challenges with that Real quickly. Some background on EmbedUR we are not a new company. We've been around now for 20 years, completely organically growing over the past 20 years to being a very successful, trusted entity in the embedded software space, ranging from expertise in the connectivity realm, where we started with wireless Wi-Fi, bluetooth and DOCSIS and all types of connectivity solutions, to now the intelligent edge and bringing intelligence to edge devices and products in general. So we bring a unique expertise not just on the AI and the model side but down to how those models interact with the platform itself, the MCU, the sensors and so on, and we're going to get into why that's been so important and why we've really found a good opportunity for us to work together with the Edge AI Foundation and its members and the community to help accelerate innovation in the edge AI space. So we have, for the past 20 years of our existence, partnered with silicon companies to help bring software solutions on many different types of embedded devices.
Speaker 4:And as we were working with some of our silicon partners who were releasing ai accelerators and chipsets that were focused on edge ai devices, what we learned is that there's a plethora of models out there in the um, in the market, um ai models that is, but those models do not easily just run on edge devices. Every mcu has, um you know, a different set of functions that it can execute, and it can be different from the next one, and so it's a challenge to make that model work on the edge device, let alone optimize it to work on sometimes a resource constraint device, excuse me, where you're targeting low power, maybe even battery power in some cases. And so, beyond not having those models ready to go on tiny devices, there's also just a lack of real world examples. What can I do with these devices, like I said a few moments ago, and so that led us to build what we call Model Nova. Model Nova is a model zoo that's catered specifically to tiny devices towards edge AI devices, where we take a repository of publicly available AI models in the vision space, the audio space, to go target many different use cases and applications, and our embed you, our engineering team, ports those models to edge AI platforms, optimizes them and then tests them to get the performance metrics and we publish those metrics on model Nova. So the whole idea is to shorten that time from ideation to proof of concept. Then what we decided to do, in partnership with with the edge AI foundation, is to launch a bespoke version of model Nova that we call edge AI labs. So we definitely invite you to check out not just model Nova but the labs page off of the edge AI foundation website, which is an instance of model Nova, which is a very specific, curated set of models and data sets, also with community interaction. So our idea here is that we want you as a community to contribute to Model Nova, specifically to the Edge AI Labs site, so that you can provide to us your own ideas for models and data sets and blueprints to further help the community to accelerate innovation in the edge AI space.
Speaker 4:As I mentioned, the three main tenants of Model Nova are models, data sets and blueprints, and so we take vision and audio models and port them to tiny platforms such as Raspberry Pis and platforms from some of our silicon vendor partners, optimize them on those platforms to work on the minimum resource constraints that are available on these small platforms and then create performance benchmarks so that you can understand how a particular model is gonna operate on a platform.
Speaker 4:Once you have a model, then you need a set of data in order to train that model to meet your specific needs, and there are some data sets out there, but some of the data sets are higher quality than others, and so what we've done is we've curated a few data sets.
Speaker 4:Now we are going to expand this set of data sets in the future on Model Nova to target very specific applications that make sense for Edge AI products, and so one thing I'd like to ask you is please feel free to submit your suggestions to Model Nova and Edge AI Labs of specific applications you're trying to build solutions for, and we could help to build data sets for that.
Speaker 4:And finally, blueprints that's the purpose of this whole show. The idea of a blueprint, which we now post on Model Nova as well, is to give you an idea of how you go from step A to step B in order to build a particular use case, such as pedestrian tracking, like we've seen in previous instances of this show. And now what we're talking about today is a blueprint for a blueprint. Soon John is gonna walk you through how you can use the curated content on Model Nova in order to get to a proof of concept of a particular use case. So, john, if you don't mind, I'd like to hand it over to you to give a few examples of how Model Nova accelerates the development of edge AI.
Speaker 3:Yeah, sure, so you know, in the world without Model Nova. So, if I think about it, there's a lot of AI development going on in the world. There's a lot of research papers, a lot of really cool stuff coming out in mainly PyTorch, tensorflow people doing a lot of cool stuff with transformers, and these things tend to run on NVIDIA or GPUs. Right, they're huge models, they're designed to solve everybody's problems and they operate on extremely power hungry and expensive hardware. But you know, all of those models in some sense can be shrunk down and can fit onto small devices, onto your Raspberry Pis, onto your NPU, accelerated MPUs, onto MCUs. But how do you actually get started doing that? That's one challenge that we found that a lot of chip vendors actually are trying to take on themselves. So, essentially, like they've designed NPUs and they want to bring the idea that you can run these models on their NPUs. So what they do is they go and they have a team and shrink down these models and get them optimized for their NPU, and what you'll see is that pretty much every chip vendor is doing the same thing. They're all developing the same type of YOLOv5, yolov8 models, mobilenet, basically just to demonstrate that it can be done, not necessarily getting to production. So what we find is like if you want to do this edge development, you really have to go through chip vendors or, you know, find some places on the web you know that have some you know demonstrations of how you do this.
Speaker 3:And so what we're trying to do with Model Nova is kind of make that I'll use the Chad GPD word democratize this across all the different chips that are out there, essentially make it easy for anybody who wants to develop edge AI solutions to find models that can work on their devices right out of the box.
Speaker 3:And so what we have is we've ported a bunch of these open models to TF light, to Onyx files. We've done the quantization, we've shrunk them down to a state where now you can actually just take these files, integrate them with your chipset vendors, tools that will basically convert them onto the you know language that their npu knows and get them up and running quickly. So that's kind of what our deal is is that we want to basically take users to 80% of the way of getting their POC running, give them the recipe for finishing that on their device and then letting them get up and running so that they can basically see, is this solution even going to work for me, like, is this model going to be fast, is it going to be accurate, is it going to work well with my sensors? And from there, that basis of a proof of concept of running Edge AI on their device, you know, is going to lead to a lot of market opportunities to create products based on this.
Speaker 4:So, john, are you saying that if we go to a model zoo today and see a model that makes sense for an application that I'm trying to run, that it's not just easy for me to take that model and port it to a Raspberry Pi or port it to an MCU or a platform of my choice that I'm experimenting with, that it's not just easy to do that?
Speaker 3:Yeah, I mean. So there's a variety of different steps you have to go through depending on your platform. So if you're running a Raspberry Pi, it's a little bit easier, right, because you're running Linux, you've got Python, you can run PyTorch and TensorFlow. You don't actually necessarily need the smaller models, but if you want to run the models quickly on a Raspberry Pi, you actually do need to do some of this quantization and shrinking down of the input weights and things like that. So that is important even for MPUs that have a little bit more horsepower.
Speaker 3:Going to MCUs is a whole different challenge, right, because now you don't have Python. You basically have C and C++ and some other tools. You're getting more direct access to the NPUs and so, like all of these like abstraction layers that you get from going from a GPU down to Python that just runs on your CPU are gone and you have to basically do all the pre-processing and post-processing and model integration yourself. So that's one key area that we really wanted to focus on is how do you take these AI models and actually get them running on an MCU where you don't have Python?
Speaker 2:Hey, John, can I jump in with a question? Yeah, so how do you guys? You mentioned some of the challenges that typically block companies. I think you have a good vantage point to speak to that. What is? What is some advice or insights you've seen on how to actually curate the model zoos that you guys work with? Do you look at the latest, most trending repos on github? Do you look at literature? Is it word of mouth? What? What guidelines do you use to curate your own model zo?
Speaker 3:Yeah, right, so I do think it's important to kind of keep up with the latest trends, so keeping up with some of the newest models, and one of the reasons is because, even if some of those models don't necessarily perform very well on your MCUs and MPUs today, as you said, davis, the world is moving really fast and what wasn't possible on these devices a year ago is now totally possible. So we're seeing a lot of like LLMs coming up on these devices that we you couldn't do last year.
Speaker 2:Don't get discouraged kind of sounds like some don't get discouraged, right.
Speaker 3:But? But there's still. There's still a lot of you know use cases for older you cases for older YOLO models and older models that basically are less power hungry. They still can serve a purpose, they can still be very performant and you can still solve your problem with those. So we want to basically have a good mix of things that are smaller, but some of the newer cutting edge stuff as well, Cool, Cool.
Speaker 2:Thanks.
Speaker 3:So kind of we wanted to talk about, like the, the journey of an edge AI product, you know. So you know, basically, like when you start off, you have this idea that I want to put some kind of like you know, object detection into the camera that goes into the back of my you know bicycle helmet or something like that, right, so maybe maybe wants to detect when traffic is coming too close and kind of alert the bike rider as an example. So you know where do you start with that? You obviously can't put an NVIDIA GPU into your helmet. You know you need something that's going to be running off a battery and you need something that's not going to overheat, right, I mean. So there's all these different things that you have to consider. But basically you have a you know a variety of MCUs and MPUs out there that could serve this purpose, but how do you even know what's out there? It's a little bit of a challenge. So you basically have relationships with chip vendors and you have to word of mouth and whatever online literature is out there, but also you need to know what kind of model can you even run on these things, because you might need to pair an MCU with a certain model to do your use case and you have to find that right combination. So that's a bit of a challenge. Even to start is knowing how to choose a proper CPU as well as AI model. And then, once you've even got those pieces together, how do you actually put that code together and fit it onto a chip? So it's easy to take, you know, a Yolo v 10, and see it running on your on your PC and your GPU and get that model working. But then, when you want to put it on your MCU, how do you do that? It's it's a little bit of a mystery and and there's bits and pieces of information out there and, um, you know, there's definitely tools that will help you, but we're trying to make that part of it a little bit easier. Um, so what I'll do is I'll go ahead and show you, um, the demo of um Model Nova does so.
Speaker 3:First let me walk you through a little bit. So Model Nova, like Eric said, has models, data sets and blueprints and will soon be launching contests. And if you check EJI Labs site, there's actually a contest already running for the Wake Vision data set. So if you haven't checked that out, you definitely want to take a look there because it's intended to help grow the models and datasets that we have today. But I'll start with the blueprints. This is the blueprint show about blueprints. In here, what we have is we actually have some blueprints for creating systems that you can basically use as a recipe, essentially for how do I create a solution. So I'll take one case refrigerator ingredient detection. So we've seen some appliances that basically can take a picture of you know what is in the fridge or what is going into the oven, and what does that do? It actually allows the you know appliance to detect what kind of food is there and then make a recommendation for how should I cook this food, what recipes can I put together.
Speaker 3:So in this example blueprint and you can go onto the Model Nova site, scroll down here. Yeah, it has essentially an example of what kind of code would you need to run? What kind of pre and post processing is needed? How would you build a Python system on a Raspberry Pi to do this? What are the different hardware components you need? What are the different models that you need? What kind of data set do you train this on?
Speaker 3:So again, this is kind of a recipe to kind of get you started. For if you wanted to build something like this, here's some ideas of what it would take to actually put a camera in your fridge and get pictures of what you're seeing and run it. And so this is an example demo that we actually run on an MCU with an NPU and we're able to kind of simulate this running with one of the models that we can download from the site. And so there's a variety of different blueprints here. If you take a look just again to see the idea of how do I create a product and as so as embed your like, as we're working through a variety of different products and services we're, we're basically enhancing this blueprints to kind of give our ideas of how do you go and create these products um.
Speaker 2:The other piece is now go to the models page before we get there actually I think, pete, there was a question from the comments about, uh, model nova and whether it was for poc or production. If you want to share that one with the audience if possible. But it'd be great if the there's a question from the comments about Model Nova and whether it was for POC or production.
Speaker 4:I don't know if you want to share that one with the audience, if possible, but it'd be great if John or Eric, you guys can address that one for the audience. Absolutely, I can go into that at a high level. So, as John and I have been talking about, one of the biggest challenges of just getting past that idea stage is figuring out how to test your idea on an actual edge device versus just on a PC or a server. So the main purpose of Model Nova is to get you past that hurdle. So, instead of taking weeks or getting frustrated and giving up to get from a model that you found online to an actual platform, model Nova's main purpose is to shortcut that process down to minutes instead of potentially, you know, days or even weeks, and so that you can get to a proof of concept, see how your idea is going to run even if it's not with the optimal data set yet and the optimal, you know peripherals on a evaluation platform and showcase that to your team and get approval for budget to move forward in developing the project. And then, as far as moving forward to production, you can absolutely start with the models that are pre-ported and pre-trained on Model Nova.
Speaker 4:Most likely, however, you're going to need to further optimize that model, likely retrain it for your specific focused use case, make it interact as you want it to with the peripherals on your end target platform, probably create a user interface or maybe a cloud agent if this is going to communicate to the cloud from a user perspective, and either your team can take that on or that's where EmbedUR can come in. We build custom engineering teams to help our customers and partners and users of Model Nova to take a product to production. So, to answer your question, yes, this is meant mostly for proof of concept and then take what you've built from that proof of concept through to production, together with embedur's help, if you would like. Thank you for that question.
Speaker 3:That's a great question yep, and we'll actually go through a little bit about some of those steps that are needed after, after you get your poc, like after you get something up and running, like what are those kind of steps that you have to take to get to a product? Because it's not it? What you'll see is it's not just about the model, there's a lot of other stuff that has to happen to basically create the whole solution. So what I've done here is I've navigated to the models page on Model Nova and if you see, here there's we have about 55 models that we're currently, that we're currently putting onto the site, and I'll show a demo of one of these. Can I show you how this works?
Speaker 4:I'm actually running this web browser on my Raspberry Pi, so it's a little bit slower than John, you're going to show us how to get a model onto the Raspberry Pi from Model Nova.
Speaker 3:Yeah, and I'll show you. Yes, that's cool.
Speaker 2:I mean, as he's pulling this up here, yeah, it's one thing to say, hey, you can take it down from weeks or days to minutes. I mean, you guys are actually about to do it for us live. I know it's tricky to pull up stuff in real time here, but I think this would be a good example of what you guys have alluded to, and I'm personally quite interested to get into more of the how-to, as John just said.
Speaker 3:So I mean we're all familiar with.
Speaker 2:Raspberry Pi platforms too, as John just said. So I mean we're all familiar with Raspberry Pi platforms.
Speaker 3:I know they've made a lot of updates over the years and AI is increasingly a relevant workload nowadays on Raspberry Pis Right, and we basically chose Raspberry Pi because they're just easy to get into, they're easy to find, the tools are very easy to get started with, but not necessarily that this is your end product.
Speaker 3:But right you have how to run these things. So what we have here is a blaze face face detection model and we look here. So each of our models has a variety of data sets that could be trained on, and you'll definitely see this on the Edge AI lab site, where some models are trained with something like the Cocoa data set and some models are actually trained with the Edge AI lab's approved Wake Vision data set. So one thing is that you know the question before was about production grade models Definitely that data set is a key area that needs to be focused on right, because the better that data set is, the more accurately it was annotated and how much data is in there is definitely going to improve your model. And then we have a platform that can be selected, so in this case, raspberry Pi, and then you can select the different types of file formats. For this one we have TensorFlow Lite, but for most of the models we have Onyx, pytorch TensorFlow as well as TensorFlow Lite, and I'll go ahead and download the model. So one thing that we do here is, when you select Raspberry Pi and model type, we give some estimates for the inference time and the energy usage and the model type we give some estimates for the inference time and the energy usage and the model size that you're actually dealing with with this model, as well as the input size. So if you see here this is a 128 by 128 RGB image that is going into this model and these metrics are actually key for understanding whether it's going to work on your platform. That inference time can vary quite a bit based on the capabilities of your system and the model size. Again, like you need to fit this, this model, into ram at some point or into flash. So those are, those are key attributes when you're selecting something to know whether it's going to work.
Speaker 3:And one thing that we have here is under the metrics is basically the script for how you would run this model and when I'm using the, how you would run this model and when I'm using the Raspberry Pi example, we give the example in Python, but if we would select other platforms like ESP32 or another MCU, there would be like C++ code. So you can basically go and download the script for the model, which is, in this case, will be a Python script, but you can also look at the different components that go into making this. So, basically, what are the different packages I need. How do I set up the environment? What do I need to install from Python perspective?
Speaker 3:And then it kind of goes into a little bit of detail on preprocessing and detection and post-processing, like what's being done there, and then steps on how do you hook up the camera and how do you actually run this thing? So I've downloaded those files. So I've downloaded those files. So what I'll do is so I'm essentially going to run the, the Python script that I downloaded, the blaze face face detection Python, as well as the tflight and again, this TF Lite file is most, I would say, a large percentage of NPUs out there have tools that can convert TF Lite or ONNX files into the systems that the NPU needs to understand. And one thing is that not all the NPUs can support all the operations of TF Lite or ONNX. So what you'll see is some things get emulated in software and some things are done on the NPU. So, depending on your platform, there will be definitely some inference differences. But let me go ahead and run this.
Speaker 2:I think we should be focused on the command terminal at the top left right. Is that the idea, john?
Speaker 3:Yes, yeah, I just typed in the command. I'm just waiting for it to come up. Oh, no worries.
Speaker 2:I know we were joking virtually backstage that this is like Saturday live or Tuesday morning live, so I know sometimes with live shows but I is like Saturday live or Tuesday morning live, so I know sometimes with live shows. But I appreciate you guys taking the risk and I think there we go, there it is.
Speaker 3:Right. So basically it loaded up the TF Lite file, put it into the TensorFlow Lite interpreter and is doing the pre and post processing. So just like that, you're able to basically take a model that you could put onto your NPU and run on a Raspberry Pi and see how it actually operates.
Speaker 4:And that really took nothing more than just entering a couple commands. Right, versus having to get a model, port it into that Linux environment and execute it. It just into that Linux environment and executed it. Just all that's built into Model Nova.
Speaker 3:That's correct, Yep. So typically, like the processes, you know you go and download a, get a get repo. You know you go in there. You probably have to. You probably don't have a pre-trained model. You probably have a model that needs to be trained. You do the training process, you do the validation process and then you finally get like a, a pie torch or a TensorFlow model that you can actually run. Um, and what we actually find is like, depending on, some of these models are fairly old. Some of them have um, you know, use like kind of outdated um, outdated versions of TensorFlow, and just even getting through that process can be sometimes challenging to get the model that you need even started. So what we're trying to do is take all of that hassle away, get it up to these tflite and ONNX files that are more standard and then just run those directly in the interpreter to get people up and running fast.
Speaker 4:And then, john, I know you and I were both recently in Las Vegas at CES and we had the honor of demonstrating some of our solutions with a couple of our silicon partners, and I know we spent quite a bit of time in getting those models optimized and running on those edge reference platforms themselves and running at high performance. What kind of steps are involved then in going from here, once you have a model that's running, to a model that's running at a production worthy performance and quality?
Speaker 3:Yeah, okay, yeah. So, eric, there's a, there's a slide. If you want to bring that up. It's after the demo, right? So? So, basically, if you look here on the left side, your edge AI POC is alive, right? So, basically, if you look here on the left side, your edge AI POC is alive, right, you've got this thing running on your MCU. But what do you do after that, right?
Speaker 3:So a typical IoT product write-up and you have to integrate all your other sensors, you have to integrate cloud communication, you have to go through all the QA, you have to make sure that that thing is rock solid, you have to make sure there's an update path. So all those things that you typically have to do for any kind of like, any kind of edge product, ai or not. You have to go through still. But also, even just the AI model right is still fairly large and it takes up a lot of power. So you have to still tune the performance of these models. You have to squeeze them down more potentially for your NPU, you have to also get a lot of these NPUs also run in parallel to the CPU, so you can be basically running stuff on your MCU doing post-processing for one inference while you're doing inference on another set of data in the background on the NPU. And so all of those steps need to be combined to actually make these MCUs perform very well. These MCUs perform very well, reduce power and get good performance.
Speaker 3:And then the biggest piece, I think, is sensor variations and custom data sets, right? So it's one thing to take a Cocoa data set or whatever kind of data set, but if your sensors don't produce the you know the images in that same format, like if they're, if they're darker or they, you know they're they have less resolution than what you typically train it on Some of these models you have to go back and retrain with, you know, the data from your sensor, and the whole Qa aspect of that is another part of it, right? So combining like, okay, now you get into this loop of does it work with the sensor and does it work at this angle? Um, you know, and does it work? Um, under this condition and this condition and this lighting? So all of those things are are parts of the whole journey of creating a product with AI. It's not quite as simple as just having a model that's thrown on your platform. It's ready to go, because you basically need to make sure it's integrated with the system very well and totally tested in every scenario.
Speaker 2:Hey John, that actually, as we get into kind of the latter part of this, and I definitely encourage those watching to jump in with their thoughts. I know there's definitely people have some questions here. What are the? You guys just hit a button on the head or nail the head. Uh, you know from from my experience. Take automotive, for example. So automotive has has, you know, adopted aiml in cabin outside the car as well. You know what. What are the role? What is the role that model nova or blueprints can play in this scenario where, for example, maybe you've trained on, you know, standard RGB images and now you have to use a fisheye camera and that's what you know. You kind of have this loop where you go back and retrain. What is the role, going forward, that something like Model Nova and Blueprints itself can play in addressing these kinds of go-to-market challenges where you do have a hard fork in the preparation and people want to alleviate the time cost of that without sacrificing quality, of course?
Speaker 4:Well, in my view, david, and then John, if you want to get a little bit more technical with it but in my view, the main idea here with Model Nova is to help get past that discouragement point that you were talking about a few moments ago, that you have this great idea, you might even have a hardware platform.
Speaker 4:Maybe you already have a camera or a sensor in your vehicle, so you have a source of data already, and so you have this idea of what you can do, this use case that you can drive through your platform to add more value to the consumer, to the customer.
Speaker 4:But you can't even get to the point of a POC to prove that it's a viable concept.
Speaker 4:And so you get discouraged early on when you're trying to showcase to your team that this is something that will work. Model Nova helps get you over that hurdle and then, like John was just saying, once you get over that hurdle now you still have 80% of the work left to turn this into a product, but if you can't get past that first 20 percent, then you're never gonna get anywhere. Um, so model nova gets you to that first step and then you can engage your own ai development team and your embedded software development team or seek the engineering services support of outside help like from embed ur to then help you train it on your data or help curate data that makes sense for your particular use case, as well as for your particular hardware targeted towards your cameras or your ultra wideband sensor, whatever it is that you're using as the source of data, then our, our team can help you to to calibrate that and um, create the training data set, retrain the model and and further optimize it.
Speaker 3:But before you even get there, you have to have that initial proof point, and that's our goal that we're trying to accomplish with model nova right and we give we do give some recipes on the site for how the data was trained, like what was the source of the data set. So so if you wanted to take that step, you can actually go and retrain it with your own data set, or, you know, as Eric said, there's a variety of tools out there in the market to help do this as well.
Speaker 1:I had a question like so we're talking about sort of the you know development going from POC to something that's ready to deploy. Talk more about like, how does this integrate with typically would integrate with, like you know, the dev cloud, devops or MLOps platforms that are out there, whether that's you know, azure Arc or from AWS or you know Edge Impulse has a big platform for model development and deployment. You know a lot of MLOps, kind of middleware out there. So what you know, how does this get from here to there?
Speaker 3:Right. So yeah, these tools are basically great for doing some of this part that we're talking about on the slide here, where you know, for instance, like on a Jim pulse, you can use the actual sensor of your device to collect data. So you know, the idea is that you know you don't need to come up with some synthetic interpretation of your device, you don't have to, of your sensor, you don't have to basically like have it gather data offline and then annotate it yourself. You can actually do it online in some of these tools. You can actually do it online in some of these tools and the MLOps and basically, like controlling the data sets, you know what the quality is doing the annotation, all of that stuff is super important for you know, especially if you have a custom data set, getting it into the right format to be production ready.
Speaker 1:Sure, so that's getting stuff ready, but then, when it gets deployed, there's, you know, orchestration of like updated AI, you know models and you know things that have been retrained and, like the, the real commercialization happens at the, at the, you know at the provisioning, basically of the devices in the field. Yes, so what's the, what's the bridge to get into, you know getting this to connect and play nice with like hyperscaler system, backend systems and things like that.
Speaker 3:Yeah, I mean. So I think, I think a lot of these. So I would say, from edge AI devices with AI, probably most of them are going to be cloud connected. You know they're going to have IoT, wi-fi or Bluetooth or Matter be able to connect back to a central system through.
Speaker 3:QTT to some cloud and that's where, on the cloud side, you can definitely do a lot with what's running on these devices, basically capturing data when people are reporting problems, you can. You can potentially like actually capture what, what are they seeing? You know why is the model not working? But then also like having the whole life cycle of the model, being able to update it on the fly. You know all of those things are super important when you, when you have a. So, basically, if you just throw out this AI model and it doesn't work for somebody, what do they do If you don't have that, that cloud system?
Speaker 1:there's stuff.
Speaker 3:Well, and also there's going to be.
Speaker 1:You know, drift over time there's going to be all kinds of you know this is. This is not the old days where you sort of flash and forget it and come back in 10 years.
Speaker 2:You know so, especially with Edge.
Speaker 1:AI. There's going to be drift, there's going to be trainings, there's going to be tunings, there's going to be things where, like you know, let's say, you have a camera reading a meter you know an old brownfield solution like that, right, you deploy a camera to read the meter and then the meter gets dusty and dirty, you know so. You have obfuscations like real world things change. You know, uh, as they say in the big lebowski, new shit has come to light. You know so. It's like things change in the real world once these things are deployed, and so you need the ability to tune these and change these models.
Speaker 1:I mean, there's also the programmability of like, oh, I want to add new capabilities, so, or I want to really refine the ability to read that meter, but there's also the drift. So the ml ops here is really important. Um, and I feel like it's kind of like the, the secret ingredient to a lot of this stuff, because I think, as I mentioned at the top of the broadcast, there's a lot of, uh, development of pocs and things like that. But when you get to get to provisioning and deployment and then ongoing maintenance down the road, you know that's kind of what makes the difference between kind of a one off and something that really scales.
Speaker 4:And I think that's exactly what what the term journey means.
Speaker 4:You know this is not just you know step A, step B, step C, and then you're done and you have a product out in the field that's serving its purpose, from starting from idea to proving that concept on a general purpose system like a server or a cloud GPU, and then taking that step with Model Nova to get it running on an edge device, showing that that is going to work on a power-constrained, resource-constrained device.
Speaker 4:Then moving on to all these other steps that John has been talking about, about how do I go beyond just the model but now focus on the model, the data set, the data pre-processing, the data post-processing, the connectivity with the cloud so that you can look at the data and have that available to solve the challenges that you just mentioned, pete, when things change in the field. And so it's a very cross very cross discipline approach to building a product. I don't think I've seen product development life cycles quite like this before, where there's so many different components going into making a product and supporting a product. You know, from the model to the embedded, to the support, to the data, and it's it's kind of exciting to see where we're going to go next with what we can do with this.
Speaker 1:There's actually a question here from Hardiker, kind of reading my mind, I guess. So you know, supporting the retraining or fine tuning of edge AI models directly on edge devices to adapt to new environments or changing data patterns. So how does Model Nova? Is Model nova more on the pre-commercialization side and then this is more of the kind of deployed ml ops side or where? What's the? What's the connection here?
Speaker 3:yeah, that's, that's right. So it's basically to kind of get like an idea of what these models will look like on your platform. Um, the, the retraining and fine-tuning is not something that we're doing from model nova today. It's just kind of like something that you would have to do as a part of that productization stuff.
Speaker 2:Yeah, yeah that makes sense. I like this question. I think it it kind of begets the need for blueprints, right? I mean this discussion, I'm just kind of listening to you guys go back and forth a bit. I think this is why, like when you want to build a house, you want to figure it afterwards, you should use some different shingling. You want to know before. You know that you can observe and adapt, that. I know you guys are getting to the end of your material here. I think this is a great line of discussion and there's a few kind of percolating thoughts myself as well. But yeah, that was a good one Cool.
Speaker 1:Good.
Speaker 4:So, as more questions are coming in and please bring them our way over the next few minutes I just wanted to leave you with some QR codes here, which, of course, you can look at on the recording afterwards, but please feel free to visit Model Nova and sign up for a free account in order to access the models and the blueprints and the data sets that are up there right now, as well as the Edge AI labs version. As I mentioned earlier, this is an instance of Model Nova hosted by Edge AI Foundation that EmbedUr has sponsored, and our goal here is to really bring community driven models and resources, and so please, you know, come and contribute your own ideas for solutions and new blueprints.
Speaker 1:Cool, good, and I'll reiterate my PSAs. So for those that wanna talk about these things in person, come to Austin, texas, february 25th to 27th. We are. The registrations are filling up.
Speaker 1:There is a capacity limit there, so I would say, if you are in the area or wanna make the trek register, I would suggest registering today rather than tomorrow, um, and there are discounts for students and uh and for commercial customers as well, so I would encourage that. Another PSA on the 23rd of January, just in a couple of days, we're going to have our first career edge live stream. So that's with 5V uh who are a um, a staffing partner. So that's all about how to build your career in Edge AI, whether you are a student or early in career or a tech veteran. It's going to be a great panel of hiring managers and career coaches on that live stream as well. So those are my two PSAs. Any closing thoughts here? Davis, it feels like you have a question or something in your head.
Speaker 2:It's not visible. I think I have a PSA myself and actually I think I bet you guys will. So at the Austin Show there's also the Blueprint Awards and I'm going to couple of questions, but this is a place where I mean, if you like being recognized in front of your peers, if you like holding trophies, if you like knowing that you've won, I definitely encourage people to submit. There's different industries, there's different categories which will be judged around. We have a committee in place that will help review these. So, please, I definitely encourage you. You can find a link to submit directly from the site for the summit at JI Foundation, and I would pose the question back to eric and john. I'd love to hear from each of you guys uh, one favorite use case commercial blueprint, like where this all culminates, is in real stuff, right? So I'd love to kind of pick your brain a bit on what you think is one of the most exciting your hall of fame challenging.
Speaker 2:Yeah, yeah, yeah what's the blueprint award you're going to submit?
Speaker 4:there were a lot of cool ideas that people were talking to us about at at ces, some of them simple, some of them complex and um, but all of them with with important value for um, for us as a society, and and some of them were um, you know everything, ranging from um how do I have automatic controls of my power tools so that my power tool doesn't injure me when I'm using it? You know there's a lot of you know safety systems today that are out there in these tools that are only working in very specific situations. But now, with AI, the AI can collect data from the tool itself, from how much torque the tool is impacting on whatever the object is that I'm using the tool in the tool, can even have a camera to see how close my body parts are to the tool, and can use AI to very rapidly make changes to the configuration and setting of the tool in order to protect life and protect limb. That's exciting.
Speaker 4:And John was mentioning bicycles. It's amazing how much technology is going into bicycles today, of course, with e-bikes, but now, with the ability to put cameras and radar sensors on your helmets and on the bikes, you can indicate to the rider and to the drivers around you that they're getting too close or they're going too fast. So I think there's a lot of cool things there. Another thing yeah, go ahead, davis.
Speaker 4:One other thing that I saw a lot of was just personalization, you know, making my life a little bit easier on a day-to-day basis If I walk up to my coffee pot and the coffee pot can recognize me because it has a small camera locally. But from a privacy perspective we want all of that data to stay local. We don't want my face to go to the cloud.
Speaker 4:Yeah, but it would be pretty slick if my coffee pot knew who I was in a secure manner and made exactly the type of coffee that I like versus that my wife likes, or when John is visiting, that John likes. So there's a lot of cool ideas out there right now that are possible with technology, and we really want to help our partners and people building products to get there more quickly. John, you have any more?
Speaker 3:thoughts there. No, I mean, I definitely like the personalization story because I think all of these things in your house that are kind of just inanimate today that you have to go up to and toggle or curse because the TV remote doesn't work, I think a lot of this is going to become much smarter and just react to who you are I'm me versus my spouse and you know like it's just going to make things a lot more seamless.
Speaker 1:You know, interacting with everyday devices, Cool, nice Well, final comment here from Sarah, just a thank you, which you're welcome Appreciate that.
Speaker 2:Thank you, Sarah.
Speaker 1:Thanks for tuning in and, yeah, sounds great. Really appreciate you guys coming on and educating us about kind of this key part of the whole ecosystem.
Speaker 4:Absolutely. I appreciate the time. Pete and Davis, thanks for the invite and I'm excited to see where our team and the community, through these blueprints and Edge AI Foundation, take Model Nova and Edge AI Labs. I think the future is wide open as far as new models and new data sets that we can bring to the resource to help develop products of tomorrow.
Speaker 1:Sounds good, absolutely All right, take care everybody. Thanks everyone, thanks everyone. See you later, take care.
Speaker 4:Bye-bye.