Hello, everybody, welcome to the mind plex podcast. We're very excited today to have Ryan sternlicht. Hi, Ryan. He is an expert in many things. And we've worked together on some virtual reality projects, artificial intelligence projects, neuro tech, all sorts of exciting things. Over the years, it's been exciting for me to learn from him. And so when it was time to start covering VR, and xr in these shows, I like Oh, I know the person to call for sure. So he came on before telling us about the latest in headgear for VR. And we looked at the Apple vision Pro and and all the latest headgear. And you can find that on the show on our channel. And today, we're going to talk about the AI 3d revolution. When he got back from GDC, and was telling me about all the stuff he saw, basically, there is a revolution that has happened, it is clear in scanning and rendering AI has made it so you can not only do very high quality scans with your phone now. But you can even rerender your own scans into better models with this technology. And so that's all that's new and exciting. And I thought that's it, we've got to do a show on that that counts. And I will let Ryan, take it away. And until you tell us a little bit more about yourself, Ryan, and then jump into this amazing shit that you saw at GDC this year.
Ryan Sternlicht:Yep. Thank you so much, Lisa. So I'm Orion sternlicht. I was on here a few weeks ago or now?
Lisa Rein:Yeah, time flies. Yeah.
Ryan Sternlicht:And I do a lot of different research for different people, as well as I helped advise people on different upcoming technologies quite a bit. So these days that mostly is focused around AI, VR, different types of neuro tech and brain computer interface stuff, as well as some 3d manufacturing, and different forms of like, how this stuff integrates with the consumer side, but also the manufacturing or business side a different stuff. And recently, I went to GDC, the Czech I've gone to GDC. This one my ninth year. Yeah. And so
Lisa Rein:I really want to know, you know, I want to jump right in and talk about these one by one because there's a lot to talk about. There's basically three or four or five, I think, specific technologies. Yeah, I really wanted to make sure that we covered. Yeah, so let's Yeah, so let's start off. Well, you can you can let us know where to start off. Yeah. What's the scanning? The rendering? Probably the scanning?
Ryan Sternlicht:Yeah, are okay, let's do the scanning one. First. Okay. So, 3d is scanning. So I think a lot of people here have heard of 3d scanning in some form, whether it's, oh, we 3d scanned a building, or we are turning a object in real life into a 3d model for putting in a video game or for historical preservation is actually one of the most critical uses of 3d scanning. And they're over the past like 20 years, it's changed a lot. I first got started 3d scanning back in 2013. When I first learned about the basics of what is kind of the back end of all this which is photogrammetry which is creating 3d models from photos by using triangulation between photos to find the distance to different points, um, I and then I talked with a number of companies at GDC and There's been a lot of different changes in the past year or so, like the technology I was using in 2013, it would take me a week to get a 3d scan of my head with about three unread photos, I did take a video going around my head with my phone, I can I convert that video to a set of photos that I have used thing called structure from motion SFM to convert that into a point cloud, that would then needed to be converted into a 3d model mesh. So about a week.
Lisa Rein:So let's let's wait and to compare the three different kinds of scans like we're going to, because I really want you to jump in and tell us about the new stuff. Okay,
Ryan Sternlicht:yeah. Okay, so. So you got query engine here. Yeah. So that's what one of the companies at GDC. And they are one of the bigger players in a phone based 3d scanning tool set. They have been around for of, I think, a few years now. And they, their stuff is really, really cool. In that it is very, very high quality for phone based stuff, because instead of doing any processing really on your phone, it uploads it to their cloud where it can be processed much more efficiently. And somewhat the big is things that for years were difficult to do with photogrammetry is things like PBR is Physically Based Rendering and textures, which that is things like color reflection, the normal maps, pretty much all that stuff that makes something look like a real object and not just a blob in that's of a certain shape. They have implemented multiple different forms of 3d scanning pipelines, including stuff that utilizes the LIDAR scanners, on iPhones, you can use video to photo set conversion, you can 3d Scan small objects, you can scan rooms, it is very, very cool. And
Lisa Rein:is that if your phone isn't set up, it utilizes everything that your phone has. But if your phone doesn't have the processing power, it makes up for it on the cloud. Is that what I'm hearing? Okay,
Ryan Sternlicht:pretty to a degree. So most phones cannot process a 3d scan, they don't, they can render one once it's processed, but they can't process it directly. So it has to be uploaded to the cloud, which is a lot that that stuff. No one would ever do something like this about five years ago to do free exports of 3d scans, because the amount of processing required was so substantial that you you just would be losing money hand over foot because of how inefficient that was. But now there
Lisa Rein:was a company doing it. Remember at the beginning of the pandemic, that company that I was doing all those scans? Yeah. And they just went away. Do you happen to remember their name? Do you I can't remember their name,
Ryan Sternlicht:but I will try to find it. i It's really hard.
Lisa Rein:Yeah, I couldn't find it. But that was years ago, you know, that was like 2021 and they got bought and then I haven't seen the technology pop up, you know, anywhere else. So it's, it's but the thing that they did was that depending on how you did your scan, it might not upload it was a huge undertaking to do the scan. You know, with your phone, you had to walk around the image and over and over again and, and do all that stuff. And basically, it's not such an arduous process now during the scan, right, that's completely different when you use your phone to do a scans. So explain that difference too, because that's a big difference. Yeah,
Ryan Sternlicht:so this is like, that's actually kinda explains some of that stuff they're doing. So there is a paid model, it's not very expensive for the quality of stuff it's doing. But things like with the free one, you do get LIDAR rooms scanning, you're able to use different things like low poly conversion, which is very valuable. If you want to use the model. With the paid version, you get read topology with quad mesh, which is a very valuable thing and PBR materials. And then this AI powered featureless Object mode. So this is, the big problem for a long time with 3d scanning stuff was reflective, transparent, and translucent materials. Because the way a lot of this stuff was originally processed, it couldn't handle that sort of stuff very well. So they, over the years, different places have gotten much better at 3d scanning, and understanding how to do different things, mitigate those issues. So there's four, 3d printing you This is an example of kind of the workflow, you go through to do some of these things where you would take video of a person's head converted into a set of photos, then you edit it in 3d modeling software, then you could 3d print it. And you could, this used to be weeks long process to do all of this with some of the tools you have. Now, you can do this in a matter of hours. Like here's a brick, and you can put it in a game you could 3d These, Dan, objects you really love, but you're worried about breaking 3d, scan them, 3d, print them, and then paint them. And you can do that in an afternoon. Now, that is was not really possible before. So doing what a lot,
Lisa Rein:what are they doing to the it's with the AI? It's a number of different
Unknown:techniques are using Yeah,
Ryan Sternlicht:into that, can you talk a bit about some of the stuff they're doing. So the main things that are happening right now, in the past year and a half, some new AI driven photogrammetry models that are using MLMs, and a lot of different new AI tools because we everyone's seen the proliferation of the new AI eyes in the past couple of years. And of course, that stuff would be applied to different things such as photogrammetry. So the one of the main companies that's been pushing this is Nvidia and Microsoft have both been doing stuff. So this is actual that is not a video that is a 3d model made from scans of images. So this is using a thing called nerfs, which are neural radiance fields, it uses a neural models do pretty much more advanced comparison of the information between the two images, to get better 3d depth data, color data, and other types of stuff. So it as all of these 3d scanning technologies have a very very similar first two or three steps of you take photos, figure out where they are in space, then you process them. But this processing is what's completely changed in the past year and a half. But that, because those first two steps aren't the same as they were 1015 years ago, you can still reprocess old photos and videos for 3d scans, but because that process has not changed in 20 years, which is really cool. So a lot of people like me, have been starting to try and find our old 3d scans to reprocess.
Lisa Rein:So they're backwards compatible.
Ryan Sternlicht:Yes. Which it kind of insane. That
Lisa Rein:is, I'm surprised. Yeah.
Ryan Sternlicht:And the different types of so let me find a
Lisa Rein:question that was asked earlier in the week when I was researching this. Is this, anything like upscaling when it's adding information? Is it? Is it getting more information? My question is, is it getting more information out of the file? That's there more point information? Or is it doing AI stuff to kind of upscale it just to make it look better?
Ryan Sternlicht:If I were both? Kind of in between? Like, I wouldn't say it's upscaling like it, you can actually do that you could apply a up to a resolution upscaler at the beginning. Right, right.
Lisa Rein:I know you could do that. But that's not what they're doing. Right? No,
Ryan Sternlicht:no, this is generally the issue with nerfs. for wildlife. This was first kind of design nerfs work first, done a few years ago, but they were still pretty slow. They were a lot faster than old stuff like this stuff that took me a week to do what nerve would take about eight hours now.
Lisa Rein:Yeah, the nerves require GPUs of GPUs, right? Or
Ryan Sternlicht:photogrammetry always has required GPUs. So
Lisa Rein:three of the systems required GPUs. Yeah, all odo
Ryan Sternlicht:gravimetry. Pretty much all 3d scanning is incredibly GPU intensive. Both for processing, but often, it's also pretty hard for the rendering. Like we, one of the places I helped out in SF Noisebridge, hackerspace, we got to really high end 3d scan of our space about seven years ago. And even on a really, really top of the line $10,000 computer, it would crash that computer because the file was so big, and there was so much information that it would trying to render all at once that it would just crash the computer outright, every single time.
Lisa Rein:So this makes sense. So it's actually like the way that things are processed, it's being done more efficiently, in a way, that's not only going to make a better model, but it's going to render better, it's going to be less susceptible to things like crashes and yeah, and stuff like that. Yeah.
Ryan Sternlicht:So this, this stuff is very cool, then, like there's, people still get kinda confused about nerfs versus photogrammetry. Like nerfs is pretty much a AI or or a more advanced AI driven photogrammetry methodology.
Lisa Rein:Right that titles a weird title, right? Because it's not really versus it's, it's yeah, it's not first one's a subset of the other. Yeah, yeah. So it's all photogrammetry and then it's just whether it's literally someone walking around with a camera taking pictures or Yeah,
Ryan Sternlicht:so like, this is actually great image of the, in essence, usually second step, or third step of photogrammetry and of nerves and of what we'll talk about in just a minute. This is where you're figuring out of view perspectives of each of the images that you are then going to process. So you take you usually photogrammetry is done On, on with an outside in methodology where the object you're scanning is inside of your perspective of the camera. So you walk around the object, but you can flip it the opposite way and do like room scans where you're doing inside to out. Which often gets confusing, but they use the same terminology in both VR and motion capture for different things, but they generally mean the same thing of what the camera perspective is. Um, so that's,
Lisa Rein:that's part of the fun in in VR a lot of times is you're inside a large object. You know, the book was burning man exhibits and stuff, we were literally flying around inside the exhibits. So if you would, that would be pretty useful to scan from the inside, you know? Or if you're making a building, you do an outside scattered inside skin. Yeah.
Ryan Sternlicht:Yeah. So this is a kind of actually interesting way of showing what a basic point cloud kinda looks like. So this has nerfs, and photogrammetry. So the this it, point, clouds are what you generally get out of this information, which point clouds are not that useful in a lot of situations currently, because they don't have any geometry. Point Clouds are. But the thing is, if your point cloud is dense enough, it will look indistinguishable bowl from geometry. But generally, it looks kinda like this. It looks very blobby, and it has no color to it. It has no reflection. It's just a map blob. Yeah. So that's, you're losing a lot information that those images had, when you just have the base, like photogrammetry, and point cloud. But AI has really gone along different distance in allowing you to get all that information back. And also infer information that you didn't fully have. So the next technology that kind of came out recently, is Gaussian. splatting for so are actually wait, that one's in just a sec, first test. So this is what came out a little while ago, back in August of last year, that was really, really interesting. This was taking a lot that that technology that nerfs had, which was nerves were still pretty slow at the time. And figuring out what can we do instead of nerves that's more efficient. And that was 3d Gaussian. splatting. Which I'm trying to find. So how
Lisa Rein:is this implemented? Like, where's somebody going to see this in action?
Ryan Sternlicht:Okay, so now our first 3d, here we go. So they both use different types of AI models. So the 3d godson is a, it's pretty hard to explain in simple terms of you, it uses a lot more parameters related to the information to process it. So,
Lisa Rein:yeah, so it's using machine learning to process. Yeah, yeah, that's fine. We don't have to go into the math. The point is, it uses math to machine learning to to process it. Now if you're in a virtual world, and the objects are around you, all the different objects, they could have all been made different ways, right? Well, yeah, I'd have been a scan when my be golfing splatting when might be a nerf. And the point is once it's done and rendered, it's just a rendered object in the in the world at that point, right? Yes,
Ryan Sternlicht:but Okay, looking better. But the other thing with a lot of these is, there is a issue a bit with a lot of these, which he Yeah, so as this actually is a great demo with this steps to it. So you have your ground truth 3d model image set, which there are standard image sets, this is one of them this bike, you do structure from motion, which basically what I used to use, this is what I usually got, after a week,
Lisa Rein:when you said standard image set, you mean like a bike is a bike as a bike so that all the AIS will know what a bike is, or what do you mean,
Ryan Sternlicht:literally a image set that everyone can use as a comparison or benchmark for their 3d modeling. So
Lisa Rein:okay, so suddenly, I think it's an 3d modeling thing.
Ryan Sternlicht:Yeah. Or this is just in general, every industry has a benchmark set.
Lisa Rein:Right? I'm talking about the one that you're talking about right now. Yeah. That
Ryan Sternlicht:one specifically for 3d scanning. Okay. And for photogrammetry. So this is what a lot of people use. And yeah, so here's an example of multiple different systems, all rendering the same bike scene using different tool sets. So that and PSNR. That is the I can't remember but what the P is, but it's like signal noise ratio, which the height in this case, the higher the number, the better, the more it's realistic. And then if you look at these things, these have to do with their render time. And training time and stuff. So like, MIPS nerve, which used to be a peak signal,
Lisa Rein:signal. Thank you Reiter.
Ryan Sternlicht:It's MIP nerve used to be one of the best variants of this a few months ago, and it took 48 hours to train, which is not bad, but it's not great. Then you get into some of these things. Like
Lisa Rein:what? Well, I'm sorry, what are you talking about to train it to do what a bike is?
Ryan Sternlicht:Or no, it's not gonna know yet. Or were talking about the 3d scan? Pretty much it took 48 hours to process, this 3d Scan.
Lisa Rein:We have a bike of what yeah. So second, about a scan in general on the system, that it took two days to process the scan. Yes.
Ryan Sternlicht:Okay. Um, and give you an output with a good piece are using AI, they are considered training. And then you can see like this, like, this is the same scene, but it only took 51 minutes, and it has a better PSNR than the one that took 48 hours.
Lisa Rein:So better models faster. Yes, yeah. Yes.
Ryan Sternlicht:Which, and, like the difference in when these two technologies released, it's about eight months. So that's about a, like, order, like 48 times faster in six months, which, that's pretty crazy.
Lisa Rein:And so again, which two things are we comparing now the guys in splatting and the nerfing or the
Ryan Sternlicht:soap, nerve, this
Lisa Rein:cladding. Okay, so the nerf is the fast one, right? Just checking out.
Ryan Sternlicht:I've seen splatting in this case. Oh,
Lisa Rein:in this case, is the faster one. Yes, I thought we were saying before, so it matters what you're processing as far as which one is going to be better for that kind of model. Ah, okay.
Ryan Sternlicht:Um, and a lot of these are all very easy to like, run yourself. Like, you can get GitHub models have a lot of these. But the thing is, is in the past, like, month or two, a lot of this is like, stuff is changing really, really dang fast. So let me find the recent one. So like this the one from two days ago, yeah, April 8 2024. Like this stuff is, like improving every week, basically. So one of the biggest issues with Gaussian splatting, compared to nerfs, has been, there's been a lot of blurring issues in certain parts of a scene. And there's, in just last week, there's was a new proposed methodology to fix or like, solve some of this problem, which is doing this, which is, in essence, improving processing, on certain parts of the scene, and slightly modifying the model for what's
Lisa Rein:gonna stay, there's a little bit that smells like upscaling to me in the sense that you're, you're making up for information you don't have in those areas. So I know are a blur, the blurry areas are where they don't have enough pictures. To put it together. Now what I was, I was literally taking pictures with our friend Bernice, with a camera in the beginning, okay. And when you got your model back, you know, and you you ended up doing things two or three times because you weren't sure if you got you were like, oh, did I get that angle, and then you try to come up with methods of making sure you got every angle, and it was just, it was really hard, you would get the model back and you would see what you missed, because there would be these blurry areas that you missed. So are you saying that like this? is a way of using math to to fix that missing information? Or is it actually deriving that information from the information that it has, instead of sort of trying to patch it up and make it look better? It's actually getting the information from somewhere else in the file? Yeah, so
Ryan Sternlicht:try, it's pretty much running a lake better. So generally, you run an entire scene on a single model, per se. But the thing is, is models are very flexible. So in this case, it's looking at the model figuring out like to kind of like simply looking
Lisa Rein:at what is that right now? Figuring
Ryan Sternlicht:out which areas are having issues being rendered. And then
Lisa Rein:it's rather Bryce if we're talking about Yeah, okay, those images.
Ryan Sternlicht:So, um, yeah, the Yeah. Yeah. So it sounds
Lisa Rein:like both to be honest. Right? It sounds like a little bit of using the information you have in different ways with math, and then using AI to smooth it. To make it look right, what you think is right. And, and it's, yeah, comparing what it thinks is, right.
Ryan Sternlicht:Yeah. So, um, but I want to talk about one that is really cool on top of this, which is Smurf, which is a modification of nerfs, specifically for real time large scale visualizations, because nerfs can be kind of hard to render at times. So they're called streamable memory efficient radiance fields. And this technology
Lisa Rein:nurse with a buffer screaming nurse, okay.
Ryan Sternlicht:Yeah. So let me find the right one. Ah, here we go. Here's the actual thing. So the, this is a running in real, this can be running in real time on a phone, which you use to not be able to run a 3d scan this nice on a phone like this was done using a similar photos set.
Lisa Rein:You'd have to make a video out of it. Yeah. To look at it on your phone. Yeah, or
Ryan Sternlicht:Yeah, Are No not a video? No, you can walk around this space. This is a full 3d Scan. You know, I'm
Lisa Rein:saying in the in the olden days, if you wanted to run on your phone, we had to generate MP fours the model and just look at it and be like, yep, that's my model. You weren't doing shit with it. But but you can look at it. Yeah. So this is something you can walk around. This is something you can go in with a headset or what are you saying? Yes,
Ryan Sternlicht:yeah. Okay. There's so that like, this is pretty key. So here are some of the different well known scenes. So let's use laptop with strong GPU. Oh, yeah, this is going to take just a sec to process.
Lisa Rein:topics. So pretty fast, though. All right,
Ryan Sternlicht:is running in real time on my computer right now at 240 frames per second. And I can move, I can zoom in. Can I move around in this one? This
Lisa Rein:is running in the browser. What does this actually running in? Yeah, yeah,
Ryan Sternlicht:that's running in the browser. So there's a plugin.
Lisa Rein:There's a Smurf plugin or something for running? Yes, the browser? Yes.
Ryan Sternlicht:But you can also view this in a apple vision Pro, any number of different tools? And the thing is,
Lisa Rein:what's the actual file format? That you're looking at?
Ryan Sternlicht:That one for this? I am not sure. I have to look at the back end. But yeah,
Lisa Rein:just look at the end of the URL and see the name of the file format? See, I can't see it because it cuts off. Yeah,
Ryan Sternlicht:I am not sure this, you can't just go to the running. And so
Lisa Rein:sometimes if you get Okay, so it's not a so it could be it could be different formats, is what you're saying. It's not it's not like a native format. It's one of those formats that you can generate all the different formats that you that go in and out of unity and stuff like that.
Ryan Sternlicht:But yeah, let's see with the laptop one that will take minutes. So that's a
Lisa Rein:real time scanning. That's so that sort of everything all at once in that model we just saw, right. Yeah, the error correction, oh, stuff like that. Okay,
Ryan Sternlicht:this is, like, best one has some issues right now running. But the other thing that is really cool with nerves and 3d, Godsey and splatting is how it handles transparency, and reflections. Because view based reflections are something photogrammetry never use deep in dream of handling. So you see, you can see the reflection of that pillow in that TV. And it right now, it's it's a bit wonky. But when it's when things are processed correctly, you can have view dependent reflections of other things in the scene. So
Lisa Rein:and this gets into the real time physics that Yeah, yeah, are built in now to some of these scans. Right. That's new. Yeah, that's an AI thing. That wasn't there before.
Ryan Sternlicht:Yeah. Tell us about that. Yeah, so the amount of base is able to understand a lot more of, in essence, photonics, like light interactions like physics, which, like we aren't, it's built
Lisa Rein:in to the point where, if you are the objects identified or something so they know how to reflect. I mean, how do you how do you enable that, depending on what system you're using? Yeah,
Ryan Sternlicht:like here. It's a really cool one. Look at this wineglass a old 3d Scan would never be able to handle doing this at all. This wineglass would just be an opaque object.
Lisa Rein:Or saying literally couldn't scan transparent objects or it was a process
Ryan Sternlicht:that it was transparent. Whatever
Lisa Rein:didn't come out, right. And he and he couldn't do it. You would be like, where's my glass? Okay. Yeah, I remember that because they had, you would have to make your glass you know, blue were, you know, give it some, some transparent, like color. So you knew it was a glass. But I didn't realize that couldn't be transparency. I didn't realize I thought that was just an artistic choice at the time. I didn't realize it didn't You didn't think about it, nothing's transparent. Um, yeah, so this real time error correction for these models, this is a good time to talk about how it goes into 3d printing. Because there's the connection, I think, when the revolution light went on in my head, it was when it was the combination of the real time physics and things like that for the 3d models, and then the ability to get all that information back out. And into meatspace. World. If you wanted versions of your objects that you could hold in your hand, and then they're printed out faster and better than ever. So tell us Oh, yeah, yeah.
Ryan Sternlicht:So when you have tools this powerful, you can use them for so much, because a lot of people have wanted to be able to create things a lot easier. 3d printing has come a long way compared to what it used to be. But it generally had a lot of issues with speed, accuracy and cost. But now it is getting to the point where all of those are not really an issue anymore. With speed during the pandemic, actually, a bunch of 3d printing people were like, We have good quality now. But it's slow. Why don't we all try to do things faster? And they did. So the standard 3d printing benchmark object called the benci, a normal 3d printer in 2020, took about an hour 30 To print a decent one. Then, people were like, Let's print this as fast as possible. So they created a thing called the speedboat race, to see who can print this fastest. And right now, the world record is about two minutes. Do this 3d print, which is insane. The problem is the quality is kind of lost. But the algorithms they've used to get this fast, actually mean that the quality version instead of being an hour and a half now, it's about 30 minutes. On every day home 3d printers not on high end machines, right?
Lisa Rein:The error correction, yeah.
Ryan Sternlicht:A lot that this error correction for the bending of the 3d printer is, in that sense, machine learning and like, literally running a algorithm on the data of the 3d model to understand the literal physics of how the machine is going to be moving. And correcting for momentum is the big thing. So we're moving the 3d printer is moving so fast now that it it can overshoot
Lisa Rein:thing intim interesting. So actually, while the while the nozzle is going from one.to The next dot, that's a thing that, that has properties and, and might make a boo boo like, well, it's moving over something depending on the shape of the objects, right? Like when you when you watch one of these things get made, you know, and you're seeing it literally, it's like a look a little liquid laser doing it, you know, well depends on what kind of printer it is. But what the point is they can actually predict depending on the object where those errors are going to be when printed fast and then adjust for it in the processing. But I'm still it's interesting. So they have to do all that they have to fix all that stuff before it gets to the printer because by the time it's being printed, it's got a No it's just gonna come out like it is. So did it was It must have taken was it machine learning to do that error correction? How did they event So the
Ryan Sternlicht:sounds used for that. It depends a lot. So it's different stuff. Okay? Yeah. So like, here's a example of a two minute Banshee like the problem is less that we will you won't be able to actually see this thing printing at actual speed, it's really going because it's faster than the camera can capture for the most part. So this is right now it's just calibrating itself. Now it's printing you, the camera can't actually capture how fast
Lisa Rein:like they used to be. Yeah.
Ryan Sternlicht:But the amount of forces on this machine are kind of crazy. It's, yeah, so you can see the actual real world counter and the machine. It's, and this is only about a three or $400 machine with probably $400 In mods on it. Um, but yeah, so AI processing has a number of companies now have little out AI algorithms kind of built into the 3d printing process pipeline. And, yeah, help do this stuff faster. It is so
Lisa Rein:strong. I'm surprised it doesn't break. I'm surprised it can hold on. Oh, yeah. As it's. So that was the first two minute benci here. And when that happened that happened two months ago. Two
Ryan Sternlicht:months ago. Yeah. But yeah, so I'm sad that this is not running. Correct? Yeah. Yeah, that's just, yeah, I.
Lisa Rein:So let's move on. So we're talking about the about 3d printed objects. And, um, basically, it's interesting to me that this converts into wearables and, and virtual clothing, that or, you know, actual, actual meatspace clothing. And that whole connection, and the error correction was a big part of that, too. You were telling me because, well, of course, if it's you're trying to be fashionable, if it's clothing, you know, it's, it's got to look good, it's got to look nice. And then what's neat is you can take your real clothing and make it 3d scan it, bring it in to your virtual world. Or you can take a virtual world object that somebody's wearing, and be like, Oh, I think I want an actual jacket of that. And you can take it out and send it somewhere and have the jacket made. So that's here. That's a thing. Yeah, no, no. Tell us about stuff. You saw GDC, you know?
Ryan Sternlicht:Yeah, that's so one of the there's a few companies working on clothing, there's like two big ones, there is Marvelous Designer and style 3d, both of which are just now really starting to try to do a lot more AI stuff with being able to take drawn images of clothes, and turn it into 3d models, they, at GDC, they didn't have any demos, but they did say they were trying to work on that stuff. Because that for video game characters, if you want it to be like, Oh, I drew these really cool clothes, it would be really cool if I could put them on my character. But for a long time, you had to go through a bunch of steps to make clothes for 3d characters. Or to make them such that you can actually cut and sew them in real life. And there's, but the other side of other thing clothes, trying to get better avatars has been a big deal. And that is actually one of the really cool things I saw at GDC pretty much on the very first day we're just here is some really cool stuff related to getting better at Jr's into video games using AI. So there's a few companies working on this, but the most notable is called the most tech. I've been around for quite a while. And are you say right now? Oh, we didn't? Yeah, I, here we go. Okay. Um, so they had to talk at GDC. On the first day that I went to that was absolutely fantastic. Talking about how they got their chat avatar system, which I'll show you in a minute, into a game. So this game Earth revival, which is a PC and mobile game, they put their avatar creator in this game, to both generate NPCs as well as improve the character creators for its users. So this is all using
Lisa Rein:Well, LLM in the character creators. Yes.
Ryan Sternlicht:Okay. Or it's, it's using actually three different sets of AI pipelines. So there's the dream face, which is the one where they're actually dream face is that one of the back end parts of the chat avatar system, that they're using some system, some search methodology, like aI search and modification tools. And then they also have a image to 3d models set of tools that they're working on. And like, oh, show, I, I sadly, don't have this game currently, but I can show like, some of Deimos is stuff. So the reason the most can make such high end avatars is because they have a background in the VFX industry of 3d scanning faces. They have 1000s of faces that they've 3d scanned over the years to, and then they trained a model on all their faces, they've scanned, they've actually trained many different models. So doing face replacement, they did facial reconstruction stuff. And they have a very high end lighting kit. So they, they are really cool with this stuff. And they have been able to do different types of yeah, let's see, oh, I think one of the viewing Oh, there we go. Um, being able to, let's see.
Lisa Rein:So you can use their tools to to, and give them a picture and it will create this model and then you export it back out and importantly, to whatever you want it for, right?
Ryan Sternlicht:Yes, but they've made it much easier using this tool called hyper human with or hyper human is a project series. And chat avatar is one of them, where you can either drag an image into it. So you can so like, here's a original image someone uploaded, which that looks like it was generated using a scan system, then you upload that image, it generates the base, head shape, and then it generates the skin mask for it. And so literally, you can like use a text generation tool that generate an image, upload that or You can actually just directly upload like facial images of real faces, do it. Or let me find one that that text based ones.
Lisa Rein:They let you give some prompts to I was messing around with a little bit. So you give it your picture, and then you give it the prompts. And I couldn't get it. I got weird models out of it, but I was giving it weird pictures. Yeah, that I was it was an angle and the lighting was weird. So you really do need a photograph that shows you know from the front that shows you straight on to get to get a good result. But then then there is yes. Or you
Ryan Sternlicht:can also just say use, you can also just use text. So like if I generate from text, so let's see.
Unknown:Oh, man
Lisa Rein:let's see, this is great. This is actually creating a 3d model just from text. Right?
Ryan Sternlicht:Yeah. Let's see. Oh, wait, I forgot it requires sign and I can't do it at the moment. But I'll. But the other project they have right now, which this is only their early beta i Right now, it's only a multimodal 3d search engine, it works pretty well, you can upload an image and it will search all the different web 3d date databases. Find what type of model you want. But the thing is, is this is their point one beta at GDC. They've showed their 1.0, which is a 3d model generative AI, which this is something that they've done, people have wanted for a very long time. So Deimos says show you different things. So they literally you upload an image. And then it will generate a 3d model of that object that you can then immediately export. And, like open, so that was in less than a minute, taken from a image to a real 3d model. And they are able to do this really, really easily. Right now. They have a closed beta. But they're hoping, hoping to open this up in the near future. And yeah, so I also wanted show like, what those those chat avatars, things you saw earlier, those were just the skin meshes that when they are outside of something being used, so Oh, wait, it wasn't the whole
Lisa Rein:avatar, it was just yeah, that would just skin.
Ryan Sternlicht:Yeah, you can attach it in unity, unreal, Blender to actual 3d models. And then you can have them fully rigged. So facial rigging, which then after you do that you can use AR kit or any number other real time facial capture tools to convert it and allow you to use it. So this, this pipeline used to be incredibly complex. But now there's automated rigging tools for rigging faces. There's things like AR kit and different types of tools for facial motion capture, which facial motion capture has gone through a lot in the past about three years. A big part of this actually is due u two V tubing and different people wanting to use facial motion capture in real time. It all actually started about 10 years ago with a thing called Face rake, which was a like Steam like app you could use that allowed you to do webcam based facial tracking and apply it to an avatar. Then when Apple released the iPhone 10, and the facial unlock system, that tool set actually is very, very good for facial motion capture. So many people started playing around with Hey, can we use an iPhone to do facial motion capture? And yes, and they actually use it in Hollywood at this point, literally just a iPhone on a head rake. That
Lisa Rein:yeah, helping that out? Right? I mean, yes.
Ryan Sternlicht:Based off different, like aI help identify the features at the face, and then focus on tracking those features in real time. It and on that subject, there was actually another really cool thing at GDC, which was real time markerless motion
Lisa Rein:capture. Great. Yeah, that was the last thing that was the thing. I wanted to make sure that we covered. Yep,
Ryan Sternlicht:yeah. So I got to do a demo of a company called ar 50. Ones, marker lead, capture, going, like they're doing, you know, later on and sharing that with all of you. But it was incredible. So a little bit of background is full body motion capture. So like where you're moving around. That is very hard to do. Because generally don't have a good sense of depth. You need a lot of information from cameras, which, for a long time, these easiest way to do this processing wise was to use markers on your body that these cameras could easily track. So if you've ever heard of bicon or Optitrack, those are the two big companies that do this. So
Lisa Rein:it's when you're talking about markers. Are you talking about like a body suit? Gloves? or Yes, yeah. So like the app so that the camera can see it and know where the different parts of your body are like how the VR headsets work?
Ryan Sternlicht:Yeah. So yeah, they updated track is probably the most well known one. And they are expensive. Like, very expensive as
Lisa Rein:the expensive two. Oh, yeah. Yeah, they're all the pieces are expensive.
Ryan Sternlicht:Yeah. I don't even remember if they? Yeah, so like the suit without anything. Is
Lisa Rein:out the sensors? Yes. Basically.
Ryan Sternlicht:fabric you are putting on and then putting sensors on it. It's already three unraid. Yeah, dollars, then gloves, then you have to actually get the markers. Which the markers are. These little balls. See?
Lisa Rein:Yeah, and the more the merrier. Right. The more points you have, the better. It will see you. Yeah,
Ryan Sternlicht:and you need these marker bases on this suit. So it's, it adds up
Lisa Rein:on a suit. Where does that go? That thing that things like this sticks
Ryan Sternlicht:on the suit using usually Velcro. Then you put one of these balls on that little post, every single so
Lisa Rein:every ball I post
Ryan Sternlicht:every Monday until lot,
Lisa Rein:so really, it should just be 3d print and all that stuff. Right? You 3d print stuff. Yeah.
Ryan Sternlicht:Or the balls? You can't these are retro reflective things. No, yeah,
Lisa Rein:he was Velcro it on why would you have to put it on that stand? It? The whole thing's ridiculous. We don't need it anymore. Right? So we got markerless. Yeah. Like,
Ryan Sternlicht:we're talking good setup with this is anywhere from like 50 It's 1000s to like,
Lisa Rein:like one person or to people or something if you've actually trying to do it. And Raiders letting us know for the best tracking, you still need it. Okay. Yeah, but yeah, but anyway, tell us about the markerless thing because it's pretty cool. You're showing me Yeah, darn good.
Ryan Sternlicht:Like, or, actually, the funny thing is, is proper steam trackers, like on your headset actually have higher tracking accuracy than these do. If you do everything correctly Lake, and they tiny want higher than, like, two millimeters of precision precision, you're going to have a bad time, no matter what tech you're using. But one of the companies at the NGDC ar 51 was demoing markerless motion capture, which they were doing just using similar ish camera like high frame rate cameras. There. But the big thing is they're using AI to help track where you are, so that you they can then render it in different ways. So like, and the thing is, is you can apply this really easily and it's very cost efficient. It's more expensive than the IMU based motion capture. With so a IMU based motion capture suit is generally about two to $5,000. But that's pursuit, which you need one soup per person. And they have a inherent flaw of drift
Lisa Rein:that's a basketball game.
Ryan Sternlicht:Drift. Yeah, so let me find there. Yeah, so it's
Lisa Rein:really funny we do it's like it's like somebody's just avatar just kind of floats. Yeah,
Ryan Sternlicht:yeah, no thanks. People build IMU based system before and they're like, oh, drift won't be a problem. I they get their algorithm running. And within three seconds, their hand just floats off. And it's like gone. And it's like, yeah, it's a problem. So that
Lisa Rein:would be the system not understanding in the 3d space exactly where their avatar hand should be. Right? It
Ryan Sternlicht:has to do with the earth having this thing called a magnetic field. And most objects in modern everyday society have these things called electronics in them, which I am use use a magnetometer do the full six degree of freedom tracking, so they use a accelerometer to do roll, pitch and yaw, then a magnetometer to do X, Y, Z translation. And the thing is, that magnetometer gets affected by any magnetic field.
Lisa Rein:So after all that the Earth's gravitational force is just going to toss it all out the window. Anyway,
Ryan Sternlicht:I mean, generally you can correct for the earth, but you can be
Lisa Rein:a big magnet. And we have a magnet on one side house. Yes.
Ryan Sternlicht:Or if your house has metal in it, that attracts magnet because magnetometer uses a tiny little, incredibly sensitive Mac. So
Lisa Rein:is there any way to Fix it. Don't get it how you're supposed to fix it, though. We're not We're not read as an Earth's magnetic field. That's,
Ryan Sternlicht:you can saying a thing called a Faraday cage. But the problem is,
Lisa Rein:you can't do your VR. What are you going to do go into a very big cage everytime you maybe are. Yeah, good luck with that. That's yeah, no collusion. But yeah.
Ryan Sternlicht:But you could theoretically use multi precision magnetometers where they are referencing each other's local fields. But you're you get into all these extra
Lisa Rein:things like you just want to take it out of the picture. You just don't Yeah, pray to be able to pick up on it. Yeah,
Ryan Sternlicht:you want to take it out? Did the picture you don't want to wear a suit with all these trackers. So that's where marker lists is such a big deal. So like,
Lisa Rein:so tell us about talk about your you actually, were in the booth at GDC. And you were talking about how they were basically tracking you from the moment you walked into your booth? Tell us a little bit. You have a video to show. Did you bring your video?
Ryan Sternlicht:I I needed. I
Lisa Rein:will link to it in the description later. Okay. But yeah, let us know. Tell us more about that. That was sounded
Ryan Sternlicht:really Yeah. So it, it was quite amazing. Just they had a booth set up with like, 14 cameras, I think it was, each of these cameras is only about 800 to $1,000. All those were plugged into a computer on the ground in the corner. Wait,
Lisa Rein:how many do you need to do this? You
Ryan Sternlicht:can scale up or down. Depending on how well you want stuff tracked. You can
Lisa Rein:pop, the average person is maybe going to be able to buy maybe three of them at the most right, three or four?
Ryan Sternlicht:At the cameras currently. Air Base. So
Lisa Rein:the price will go down hopefully.
Ryan Sternlicht:Oh, yes. Very, very fast.
Lisa Rein:Probably you need at least three or what are you thinking? I mean, the the question is three, when it when it had the sensors, it wanted three of them.
Ryan Sternlicht:So these AI algorithms are getting good enough that you can use a single webcam and a mirror. So then
Lisa Rein:you don't need an$800 camera or you do need an $8 camera. Okay,
Ryan Sternlicht:that is just because if you want a really professional setup that's tracking at high frame rate and has like sub millimeter, like has millimeter accuracy, so the
Lisa Rein:AR 15 one system has that, yes. Okay. But
Ryan Sternlicht:and it's scale, but like you can use like four cameras for a like, for what a normal person might have at home, which might be a 10 by 10. area.
Lisa Rein:Probably four cameras. So it's still right now that would still be, you know, a few
Ryan Sternlicht:dollars. Yeah. But the thing is,
Lisa Rein:it's the webcam mirror thing you're talking about that. So right
Ryan Sternlicht:now, part of the reason using AI for this stuff is so great is these algorithms are changing just as fast as all the others. So everyone in VR has wanted easy, cheap, full body motion tracking for years. And a bout a year ago, someone was like, let me just point a webcam at the mirror, and then run a AI algorithm to figure out what the person is doing. And it worked. And then there some people have played around with having a few webcams or a few mirrors to get better three, the positional accuracy. But every one all what software
Lisa Rein:is it using at that point with the mirror the webcam, what software is using if I was gonna go set it up right now in my, in my house.
Ryan Sternlicht:It's like usually like a get you go on GitHub. It's usually I think Python or it's just a little app running that is then sending data to whatever motion capture lakes. It's sending it to either VR chat, Unity or Unreal, which all of those can pick up make
Lisa Rein:to make an object out of it, too. Yeah, yeah. Okay,
Ryan Sternlicht:it's, um, but the thing I really loved about AR 51 thing is it, it just is immediately tracking you. And it can track as many objects as it can really process which it they talked about that on the first day, they add a little event there, it would tracking 20 people in their booth.
Lisa Rein:Did it show you the model that was being generated from that? Yes,
Ryan Sternlicht:yes, it was. That's the other cool thing they were showing you in real time, what it was seeing? The thing is, is, as more people go in processing increases by dates,
Lisa Rein:oh, the quality doesn't go down.
Ryan Sternlicht:The frame rate goes down. So if you have like 14 cameras and a decent computer setup up, like we're still talking at, like, five days$1,000 computer, but it's, or more if you really want to go, Pam. But generally, it was they said that their setup currently with their 14 cameras was running at nine milliseconds latency for tracking the entire area, which this was a 10 foot by 20 foot area. And they said it doesn't increase. Like 12 or 13 milliseconds tell you if like six or seven people, which for VR, the threshold for good movement is 11 milliseconds, which is 90 frames per second. nine milliseconds is right around 120 frames per second. So they were and if you're ever doing something like live streams on YouTube, you only need do 60 frames per second. Which means you can have like, six, I can't remember if it's like 16 or something milliseconds latency. It's
Lisa Rein:but you're saying the frame rate can be affected. What I meant by quality was just not looking as good. The frame rate affects how it looks. Yeah. More. Yeah, that's a live scan with 20 people. But that was what the 14 cameras, right. Yes, yeah. So this is new, though. It hasn't had a price hasn't had a chance to go down yet.
Ryan Sternlicht:Yeah. And we'll go well, so both the price and the processing requirements, because as these algorithms improve, you can do it on lower and lower end machines. And as cameras become cheaper and better, you can do it on lower and lower end cameras. And those cameras are going to be better cameras than what they use staff. So right now, like a lot of people I tell them, they still have webcams, which like this. I'm using a Logitech BRIO 4k webcam, which is okay. But this camera came out 10 years ago, I think at this point or nine years ago, and up until very recently, it was one of the only 4k webcams and I'm not even running it at 4k. I'm running it at 1080 p 60. With HDR. All
Lisa Rein:right, but doesn't really do 4k. It doesn't really stream 4k. Yeah, if
Ryan Sternlicht:but it's like yeah, in the past couple of months, a company called OSP bot released a thing called the tail err, which is a 4k PTZ webcam, which PTZ means it's robotic. It can follow you around. It's pan tilt zoom. And that thing is amazing. It's it's about $500 but it's literally better than PTZ cameras that conference centers pay five to $6,000 for and it's tiny, and it's about$500 and that that also has AI tracking built into it. They'll follow you around and it it also has gesture based control so you can turn it on or off Have a track, have it switch scenes do different things. Like I plan to get one probably because it's so cool. And using AI for all these camera things is very useful for both, like making your conference or like podcast setup better. But this stuff also is applying to the very high end cameras that they are currently using in motion capture. And it's quite amazing the level. So let me show a little bit of AR 50, one's markerless motion capture. So this is so to give you an idea of what's going on is these two people are in VR, you can on the bottom right screen see the perspective of one of the people. And on the bottom right see kind of a world perspective. And these are paper scissors. And it's tracking them in real time. Without them wearing anything. It's so the finger tracking is being done by the not division pros, they're wearing the quests that but their movement, their real world positioning was being done using the cameras around so ar 51 has led me find that there cool video of it running in. So these two people so they're in a capture studio with this capture studio looks kinda crazy or VR space, but down on this TV or laptop, actually, oh, same. I know that laptop there. That's what is going on in the scene. And you will see in a sec, they're going to get up. And it's still tracking them as they move around without them wearing anything. And it is for it to be been this good was basically impossible of few years ago, probably even a year ago, that would have been like substantially more expensive than it is now. All right, and see people running on in the booth, and they're all being tracked. So that is pretty amazing to just like walk on and off and be tracked. With a rate like it seems like a absurdly high end drag of this technology, it would only be about 20 $40,000 which is a lot less than marker motion capture. And the thing is, that also means if you're in like the game dev industry or any industry that just needs to rent this stuff or rent a studio, the studio rental prices will be much lower because very few people need these setups at all times. So most people rent them and a lower cost. Setup means they probably will have a lower rental cost as well as you don't have the suits to worry about and the soup cost. It's literally once you have no more costs at all. And that actually might like as algorithms get better you You don't really even need to upgrade in the for a very long time. Intel's like better
Lisa Rein:and better with software. Yeah.
Ryan Sternlicht:Yeah, that's, that's like the crazy thing at this point, a lot that this hardware doesn't really need to improve. It's just let the software improve.
Unknown:Yeah. Yeah.
Lisa Rein:Well, Ryan degi, thank you so much. I'm looking forward. What is the next thing coming? Do you think down the pike that you're excited about? Before we Hmm.
Ryan Sternlicht:So the next conference I'm going to go to is display week in mid May in San Francisco, which is the big display technology conference. That is where a lot of the big display manufacturers show a lot of their research. So these are things like TVs, and phone screens and stuff of things that aren't coming out for five years.
Lisa Rein:So what are you looking forward to seeing these specific, I'm trying to find a specific new thing that's coming that maybe you may you might see there.
Ryan Sternlicht:I'm hoping to see a lot more I'm like, I, I view independent 3d displays a lake, we're getting close to a point where real time Hello graphy. And like, certain types of three dimensional viewable content is going to be much easier to display and will actually look good, it won't be like wearing the blue and red glasses that the cinemas, it will be you independent, which means you don't have to be in a specific spot for the effect to work. And a lot of that has to do with processing and light field. Things as well as a lot of upcoming VR displays. Last year, when I went, I found a bunch of different technologies that I would like I want to smash these all together, because they get
Lisa Rein:your wish. Yeah, yeah, that's maybe what's what's happening is a lot of things are kind of combining Are there ways of doing that. So
Ryan Sternlicht:having better displays is something very important for VR. I mean, if you if any of you tried the apple vision Pro, it is nice. But it also still fails in a number of visual tests, mainly brightness. And that's been a big issue with VR and actually just displays in general, if anyone, even with the really nice phone has ever brought their phone out in like direct sunlight and trying to view anything. They're like, Oh, God, I can't see this. And that's because our displays can't get bright enough. Even though our phone screens are actually better than most. That's why I put
Lisa Rein:your jacket around your phone. Yeah. Okay, great. Well, thank you so much, Ryan, for coming on. It's been a great show, everybody, remember to subscribe. And we will. We'll see you soon. Oh, yeah. And thanks so much for coming on the show, Ryan. Really appreciate it.
Ryan Sternlicht:I'll be sharing all the links to all of this stuff short. Yes.
Lisa Rein:Ryan has spreadsheets basically that go shows just to make sure to have direct links to everything that was covered. And we'll have that up in the next probably 24 hours. All right. Thank you so much, everybody. Sweet dreams.
Ryan Sternlicht:Thank you.