PART 1: Navigating the Realms of Tiny Edge AI with Industry Leaders Evgeni Gousev of Qualcomm and Gopal Raghavan of Renesas

Prepare for an exhilarating ride as we navigate through the intricate realms of Edge Compute with industry leaders Gopal Raghavan from Renesas and Evgeni Gousev from Qualcomm. Get ready to unravel the technologies that empower the Tiny Edge, the people fervently pushing their development, and the revolutionary transformations it can trigger. 

Our guests share their profound insights into the potential of microcontrollers (MCUs), the challenges of executing workloads on ultra-low powered devices, and the remarkable advancements in connectivity, algorithms, and developmental software tools. We shed light on the pivotal role of the three V's of data - Volume, Velocity, and Variety - in making edge computing a success story, particularly in the automotive industry. Gopal and Evgeny also highlight the importance of a balanced mix of low power silicon, smarter workloads, and efficient connectivity in reducing the energy footprint of AI.

As we bring our enlightening conversation to a close, we anticipate the future of technology and connectivity, the inevitability of smart devices, and the evolution of the software stack. We emphasize the need for robust deployment tools and stringent security measures and explore the potential repercussions of poor coding on energy dividends. Join us as we dissect Edge Compute to foster learning, growth, and acceptance, and propel ourselves towards an exciting future. Don't miss out on this opportunity to gain insightful perspectives from industry leaders and expand your knowledge horizons in Edge Compute!

Want to scale your edge compute business and learn more? Subscribe here and visit us at https://edgecelsior.com.

Want to scale your edge compute business and learn more? Subscribe here and visit us at https://edgecelsior.com.

Transcript

Speaker 1: 0:13

When you ask people what Edge Compute is, you get a range of answers Cloud Compute in DevOps, with devices and sensors, the semiconductors outside the data center, including connectivity, ai and a security strategy. It's a stew of technologies that's powering our vehicles, our buildings, our factories and more. It's also filled with fascinating people that are passionate about their tech, their story and their world. I'm your host, pete Bernard, and the Edge Celsius show makes sense of what Edge Compute is, who's doing it and how it can transform your business and you. So let's get started.

Speaker 1: 1:00

I get things rolling here and I can even see sort of the jitter and packet loss of our session. It's pretty good, so I'm pretty excited. Cool, yeah, so well, this is going to be kind of a very special episode of the Edge Celsius show because we are going to have a three person conversation with Benny and Gopal, who I'll introduce in a minute. Oh, so one other thing, just as a housekeeping thing, is try to silence your alerts or any of your loops and loops, because they can't really edit this out. No, don't want to do that. Yeah, I mean, it's not Sergeant Pepper or anything here, but we're trying to keep it professional yeah.

Speaker 1: 1:44

Try to keep it professional. Cool, and this is an audio only recording, so don't worry about how you look.

Speaker 2: 1:54

Okay, I found it oh good.

Speaker 1: 1:56

Yeah, I had someone I was recording and they had it all set up in the hair and it looked really good and I said you know, I appreciate you getting dressed up, but this is an audio only thing, so whatever makes you feel good.

Speaker 2: 2:09

So what is your typical audience?

Speaker 1: 2:11

Well, you know, so this goes out. I mean, it's through iTunes and Spotify and all that and I actually are getting I'm getting a lot more views on or listens on YouTube now Probably probably 10 X on YouTube that I'm getting on, like iTunes and Spotify Just in the past, like month or so month or month and a half.

Speaker 2: 2:34

Is it like like a general public or some engineering? Like what are we? What kind of Well?

Speaker 1: 2:38

I don't know exactly who the who is the audience. I mean I'm assuming they're self selecting to be interested in edge computing. So I would find it hard to believe it. If you were not into it, then you'd actually listened to the whole thing.

Speaker 2: 2:53

So people are somewhat familiar with the area.

Speaker 1: 2:56

Yeah, I think so and I, you know, I think it's. We've. We've had Intel on the show, we've had different analysts on the show, leonard Lee has been on the show, we've had Dave McCarthy from IDC on the show, we've had Microsoft on the show. So lots of different players in the kind of the edge computing space. So, yeah, it's, it's all around the the edge stack Cool, and you know so. Today I thought we could talk about I'll introduce you guys in a second here but the talk about the tiny edge, the edge of the edge, and which is something that a lot of people don't focus on as much. But before we get into it and all the different topics, let me kind of go through. Maybe you guys can introduce yourselves. We have Gopal Raghavan from Renaissance calling from Southern Cal. So, gopal, do you want to give a two second? I'm in two seconds. Give yourself 15, 20 seconds A quick background and who you are.

Speaker 3: 3:59

Thanks, pete. You know I've been working on ML on edge devices for about the last 10 years, first in a company that I started, and then I joined Microsoft, where I was working for Pete actually doing the same thing, and now I'm at Renesas, where I'm coordinating the AI strategy across a range of edge devices, from MCUs, which are tiny devices, to MPUs and some BFIR AI accelerators.

Speaker 1: 4:33

Great, great Appreciate that. And, evgeny, do you want to give everyone your info?

Speaker 2: 4:40

Absolutely, and it is a pleasure to be here, pete, thank you for inviting and to be able to share our thoughts about this exciting area.

Speaker 2: 4:51

So I've been with Qualcomm since 2005 for almost 20 years, a lot of different projects and more recently, in the past about 10 years, we've been working on embedded compute, low-power compute, machine learning, edge, ai applications, hardware, software basically the full stack. And I also serve as the chairman of the Board of TinyML Foundation, which is a non-profit organization of many global companies doing business together in the area of edge compute or the edge of the edge compute. Like what you say, tinyml software, hardware applications very diverse, very interesting ecosystem, so it's actually fun to be part of it.

Speaker 1: 5:36

Cool. Yeah, no, that's great. And yeah, we've done some work together in the past as well, some good projects. So, yeah, and I wanted to get both of you together because I think we've all occupied sort of a similar space over the years.

Speaker 1: 5:50

You know, a lot of times when people talk about edge computing they tend to talk about maybe some of the heavier edge stuff you know in servers and gateways and kind of big things like that. But in fact you know, as many people may know, in terms of high volume, you know, billions of devices out there are running much lighter weight, lower power, compute capabilities, a lot of those in the past, like MCUs. Everyone you know probably hopefully everyone who's listening to this podcast knows what an MCU is. But microcontroller, they're everywhere, they're in everything these days, everything from your bathroom scale to your toothbrush to your, you know, whatever toilet seat beyond the bathroom, even in the fridge and everywhere healthcare. But what's been happening interestingly is that MCUs over the past several years have become much more capable. So they're not just kind of doing very simple kind of functions, but they're actually able to do compute and they're able to communicate and they're actually driving workloads.

Speaker 1: 6:56

And so because of that and we can talk a little bit about some of that architectural advancement. These platforms are becoming really interesting, useful platforms, including doing, you know, ai and other kind of edge compute. So that's kind of a frontier, and the challenge, as I'm sure we'll talk about too, is how do you do that in an architecture that's really designed to be ultra low power, you know, many tons, battery operated and also with very low cost. So these are typically single dollar or so type of chips in very low cost things. So that's kind of the frontier and I'd love to kind of get both of your take on. You know what has been changing in sort of this MCU space and this tiny edge space. It's now making it a lot more usable and feasible to do things with. So either you guys can chime in.

Speaker 2: 7:50

Yeah, I think there are a couple of fundamental things happen in the past, I would say five years or so and it's all given by road map technology, road maps, but also innovations in the systems area and also talent development, specifically and also to specifically silicon is becoming more and more capable. So, like one, one point of comparison, what we can run on a small microcontroller today which is just like five millimeters square, this small microcontroller has as much horsepower as a Kenji computer. Was what 100, not 150 years ago? So you can imagine big desktop. You can run so much workload on this tiny couple millimeters square piece of city. That's one Second.

Speaker 2: 8:43

Algorithms and models are becoming more sophisticated and they're becoming both more capable and lightweight in terms of model size. And there are other techniques like how do you make it even smaller using quantization, pruning and so on, and people build this small models. And the third driving force tools, software tools. There are many companies who offer the software tools. As a result, there is a big developer community around this, so it's easier for people to use all the way. To like no code programming, you can kind of develop your own code without even knowing programming languages. So I think all of these forces Silicon Innovations, algorithms and tools. They drive this massive development in this area. That's on the technology side, and we are seeing more pool on the end user and application side, which I'm sure we're going to address later in this.

Speaker 1: 9:37

Yeah, yeah, go. Paul, what's your take on what's changed with good old fashioned MCUs to make them new and exciting?

Speaker 3: 9:46

Before we get into specifically MCUs, one of the points you made, pete, which was a very good point, is there have been advancements in connectivity as well as compute True, true, and what this does is transforms the AI problem into a continuum. So, given a problem, you have a choice of where you want to solve it and why. Would you want to move towards smaller devices or towards the cloud? That depends on the three V's of the data you have. So like, depending on the data volume, the data velocity and the variety of data, it makes sense to solve or do AI either on the tiny device or, let's say, a more capable MPU, or even an on-prem data center or a server All the way up. Next up would be the server provider edge and then finally the cloud.

Speaker 3: 10:51

So there is this whole thing, and so what happens is, as more capabilities are available in MCUs, to me it only moves your problem up or down. If your MCU is not very capable, then you move to the next level. Up, right, you keep moving. Having said that, mcu's and Silicon for machine learning, all the NP use there's a whole host of companies doing that and software that Ivingly mentioned also. All these are pushing the compute access, but without the real big changes in connectivity we have seen recently, I don't think the edge market would be as interesting.

Speaker 1: 11:39

Right, well, I mean, it's a good point. The edge solutions are defined by a continuum of devices, everything from the hyperscaler to the potentially the tiny edge sensor and maybe a few hops in between. That actually distinguishes it from maybe traditional IoT solutions, which may be sensors sending data one way up to a cloud or something like that. You're right. I think the ability to now think of these solutions as a continuum of compute I would throw in there's management and orchestration of those workloads is also pretty important. But getting these things connected and we could talk a lot about connectivity for the tiny edge, whether that's LPWA or lower-win things or NB IoT but also, like you were saying, is just thinking about the ability to plop the workload on the right piece of the edge at the right time to get the right result. I think that's the thing that's now bringing a lot of the tiny edge capabilities, making them much more valuable in completing these types of solutions. Right, exactly.

Speaker 2: 12:48

I think that's an excellent point. We're not talking about tiny versus cloud. It's a distributed compute depending on the workloads. The winners in this games are those people in company who know how to partition your system in a such smart way you can get most of your compute in terms of energy efficiency and cost, because that's basically the rule of the game today.

Speaker 1: 13:15

Yes, exactly In my experience working with customers, they want to solve the problem for as little cost as possible, that's, fast as possible. Anything that solves those problems is good for them, because they don't really want to spend any extra on it, right? No, it's interesting, I think, also for our listeners. Openai just had their DevCon, their developer conference, and so there's been so much oxygen in the room has been taken up talking about generative AI, which is fantastic. It's wonderful. I use it too. But the types of AI that are happening on the lighter and tinier edges is not generative AI, it's other types of AI, it's anomaly detection, it's vision AI, it's object detection. I mean, can you guys talk about what are some of the scenarios and use cases that really snap in well with the lighter edge of compute?

Speaker 2: 14:09

I think there are also probably two driving forces here, pete. On one hand, we have all these cloud-based AI techniques and technology has been developed, and I think what you see is that these types of approaches and techniques are being adopted also to the edge type of devices by making models smaller, smarter, more application-specific. That's kind of one driving force, right. And on the other hand, we have a bottom-up type of evolution, like sensor type of companies ST microelectronics, Borsch, all these companies. They try to make their sensor smarter by adding ML capabilities there.

Speaker 2: 14:43

So those are kind of the two things that are connecting the big AI and the tiny AI world.

Speaker 2: 14:49

This is kind of a general comment, but application-specific.

Speaker 2: 14:54

I think there are many areas, basically, where you need to bring more intelligence to the edge and do it in a way that does not violate especially privacy and kind of is beneficial in terms of latency because you don't need to connect to the cloud or sometimes you cannot connect to the cloud right, like, for example, some applications in the industrial IoT.

Speaker 2: 15:20

Yes, you connect it through Wi-Fi, but Wi-Fi in a big industrial building may not be reliable, right, it may be on and off, type of things. So I think the latency and the library are also important. That's kind of where this HAI tiny ML comes to shine, because you don't have this type of constraints in terms of energy, in terms of dependency on networks and so on. So those are kind of general things and if you look at by verticals, I think we are the very beginning of this. I think it's kind of hard to make kind of big, big predictions yet, but we definitely see some very interesting trends, like, for example, in the industrial IoT. There are many applications like, if you mentioned, predictive maintenance, for example, is one of them.

Speaker 3: 16:06

How do you?

Speaker 2: 16:07

make your machines smart, more reliable, more predictable in a way, and anomaly detection is kind of related to this. How do you detect things that are about to happen before they happen? Right, so they can. You replace a motor based on some vital signs from from this engine? Right, so that's one big vertical. Another one is consumer electronics. There are many, many applications of these type of devices in consumer electronics. Just like v8, call-call, you just release the product in a laptop business, for example, using this type of technologies or wearables as a big big application augment the reality and the mixed reality, xr, because all these devices they have constraints in terms of battery size or energy. I think that's kind of where these technologies come to to play a big role there. That's that's a second vertical. In the third one, I would say it's in the healthcare and medical type of applications. I think that's kind of another big, big opportunity, but there are more modern ways.

Speaker 1: 17:11

But those are kind of three yeah where we see quite a bit of traction you mentioned about some of the kind of privacy, security things.

Speaker 1: 17:20

I mean, as we know, and some of these, especially industrial solutions, they're airgapped for a reason, you know, and and also in sort of military and government things, so there is no cloud access or there's occasional cloud access, and so you know the types of kind of compute and AI that needs to happen, happens, needs to happen at the edge, you know, without relying on a cloud.

Speaker 1: 17:43

Then, as you mentioned before also, there's just the latency and, frankly, cost and ingress of all the you know where back in the old days, maybe years ago, like healthcare stuff maybe would collect some data, dump it up to the cloud, get some analysis and get the data back, maybe at some point. But now a lot of that stuff, especially in the healthcare, medical fields, that can happen like on device, instantaneously for doctors and patients to see without having to go off-prem and do calculations. So, yeah, no, it's interesting. I think probably we'll eventually see, you know, the now a lot of these mcu's and kind of very kind of light edge, rather than AI being sort of an interesting added feature, it'll just be sort of a standard way of running compute. But go, paul, any any thoughts on that? What's your?

Speaker 3: 18:33

take on the hot areas might take on the hot areas, I think the ones which are closer in part of the problem has been a lot of these AI edge applications die after the Pock level and, depending on who you believe, it's somewhere between, let's say, 70 and 90 percent. Yeah, that sounds about right. I think we're both familiar with that too. Pete, that's right. Yeah, well, so the question is, when we try to force AI into applications where there is no how would I say it, but there's no concrete or provable benefit, economic benefit it usually tends to die out.

Speaker 3: 19:17

So, while we've seen a lot of applications, I think the two big verticals I think, as Ewing mentioned was one of them is, of course, industrial, and the other one, which you know may not MCUs do play a role, not as big is the automotive, because the automotive is absolutely the edge. You know and again to me them the way I identified things that the edge is based on the three V's of data Volume, velocity and variety. So if you have a huge volume of data coming, it obviously does not make sense to send it all to the cloud. Similarly, the velocity is very high. You're not sending it to the cloud. The only other thing is when you close the loop. You need latency if you need to act on this data, and so those are the two that I feel sort of warmish about. I'm not really hot about any of these Because I'm not seeing the money, but I feel warm. Automotive certainly is a big one.

Speaker 1: 20:23

Yeah, in this.

Speaker 3: 20:25

Yeah, go ahead.

Speaker 1: 20:26

Well, I was gonna say automotive, I mean. Yeah, I mean that that whole segment and I mean Qualcomm and Renaissance are both doing big business, and automotive these days the you know that the rise of the software defined vehicle and kind of consolidating all these Legacy ECUs into these kind of central vehicle computers and stuff, it's a huge engineering project For for everybody and there's a lot of real-time. You know, safety critical stuff needs to happen there. So that's actually a really good example because it's it has to be deterministically real-time and it has to be and you know, there's.

Speaker 1: 21:02

I mean, we've all driven cars Maybe not everybody, but you know they go pretty fast, they have brakes and things, and so you need to sort of make sure that all that stuff is it's instantaneously actionable. Uh, so yeah, I know that's a really good example of one of the most complex and interesting Edge platforms that are out there, right?

Speaker 2: 21:20

and actually some of the other more features may be actually mandated because they will be a safety or mission critical, right, right, and they can't be dependent on any kind of cloud connectivity to work. Like for example yeah.

Speaker 3: 21:35

Actually, the funny thing about this is one of the reasons we tout edge is to work well with intermittent connectivity. But I think, if you saw the news over the last, I don't know, in six months, when I forget which of these automated cabs in San Francisco they were, I forget which one they lost connectivity and all of them went to this one street in San Francisco.

Speaker 1: 21:58

Oh really.

Speaker 3: 21:58

Yeah, and they all jammed it. So I think behavior with intermittent connectivity needs to be worked on some more. Yeah, yeah.

Speaker 1: 22:20

Thanks for joining us today on the Edge Celsius Show. Please subscribe and stay tuned for more and check us out online about how you can scale your Edge compute business music, music, music. You.