Patrick Wallis and Marc Goldman about Esri, AI and Gaussian Splats

We explore how GIS connects BIM, CAD, and reality capture into usable context for design, construction, and operations. Gaussian splatting takes center stage as Patrick Wallace explains how it preserves fine detail and enables point clouds, meshes, and object detection at scale.

• Indoors product ingesting DWG to power floor-aware campus maps
• Difference between authoring tools and GIS as the system of context
• Drones, LIDAR, photogrammetry workflows for operational models
• Weekly drone flights for 4D construction review and issue forensics
• Gaussian splatting fundamentals, benefits, and capture best practice
• Filling gaps with open imagery and managing artifacts
• Building point clouds and meshes from images and video
• Cloud-first processing, streaming data, and single source of truth
• AI deep learning models for solar panel and asset detection
• Living Atlas curation, partner data, and scalable access

Transcript

Welcome And Guest Intros

Roopinder 0:19

Hello and welcome to FoDES, the Future of Design and Engineering Software Podcast. My name is Roopinder Tara, and on the show, we will have guests that will discuss tools and technology that engineers will find interesting and useful. Today's guests are Marc Goldman and Patrick Wallis. I was introduced to Marc at a recent event. Marc is the director of architecture, engineering, and construction industry at Esri. I asked Marc what at Esri would be the most innovative and fitting for our future of design and engineering software podcast. Since Esri is a GIS company and has been around since 1969, I was expecting a short answer at best. Instead, I was to find that Esri was involved with all of the groundbreaking technology we engineers are interested in AI, LIDAR, photogrammetry, as well as technology we are most curious about. For example, Gaussian Splats, which have been able to produce stunning and very accurate 3D images. Patrick is head innovator at Esri's Innovation Lab. He holds a master's degree in architecture from Tulane University. Who better to explain how exactly Gaussian splats work?

Marc 1:32

Let me give a little bit of info to Pat and take it from there. I joined Esri six years ago now, just before COVID. I work remote, headquarters at Esri is in Redland, California. And as soon as I got my arms around what I was supposed to do at Three, I wanted to figure out who else in the company I could understand the status of things. And Pat's name came up, and he's been someone for six years that I stayed connected to. We try to have a weekly or bi-weekly call to understand what the other is seeing in the industry. And I think what the work that Pat's doing at Esri is some of the most interesting and creative. He knows what a Gauss House flat is.

Roopinder 2:05

Yes. Okay. All right. So that would be very interesting to talk with Pat.

Patrick 2:08

So I think when you came on, Mark, indoors was a new product, and that was one of the products I helped launch. In fact, I was hired by Esri to start an indoor mapping and facilities practice. So that all that grew out of that work.

Roopinder 2:22

In my background.

Patrick 2:29

talking about full-off facilities mapping from room scale up to floor to the building to a campus, to multiple campuses, and so on.

Marc 2:38

But it doesn't require scanning. We'll start with a CAD view.

Patrick 2:41

We don't really care where you the provenance of your data was as long as it can be transformed into the vector format that's needed to power the indoor product. You know, the indoors product is floor-aware, it understands how buildings are organized and you know how they're operationally organized within campuses and sites and stuff like that. So it's just it's already optimized to do those things.

Roopinder 3:06

You can handle a DWG file.

Why GIS Matters To AEC

Patrick 3:08

Yeah, so there's you can actually just upload your DWG and you say, Oh, these layers make up my rooms, and this annotation layer are the room numbers, and it'll just process it and turn it into a map. We launched the Indoors product in March of 2019. We had worked on it for two and a half years prior to that. That's the part that I led, and then I kind of handed it off to a full-time product manager. It's focused on buildings and what happens in buildings and campuses and the stuff in between the buildings. Because if you're running an organization and you're a facility owner, you care about, you know, you care about the buildings and all the stuff in between them, including the underground infrastructure. So so uh we have a lot of customers that use that to do the types of things people used to do in likes or Trirega and or even Maximo, for example.

Roopinder 4:00

Yeah. I just came back from a Bentley conference, and as you mentioned, underground, and they're big into that if I underground mapping. Yeah, is that a pretty close competition there, Pray?

Patrick 4:11

People do a bazillion things with GIS. People use indoors to do security operations, they can use it to do facility operations, space management. So we don't really see any one of these as real direct competition. In fact, we would say if you're at a certain scale, we would expect you to be using these other technologies. And at some point, you're gonna want to use the power of where and location to bring everything together.

Marc 4:37

Yeah, we're not an authoring tool the way an AutoCAD or Revit Civil 3D or MicroStation or the other Bentley products would be. We're not someone's typical the blank screen and line laying down lines that eventually become an engineering drawing or a model. You can do some of that, but you're more likely to generate that type of content in one of those authoring tools. Right. We consume it.

Roopinder 4:58

I see make sense out of it, make it orderly, data. We put it into context.

Patrick 5:04

We're seeing more and more where reality, the state of the art and reality capture, we've been dealing with reality capture forever, like airplanes and photogrammetric processing, stuff like that. Uh IDAR. And now we have additional sensor types, we have consumer devices that are lightweight or commercial technology that's a lot cheaper with all the different types of drones you can deploy, whether terrestrial or airborne.

Marc 5:32

We just got the car actually packed on with uh maybe what's called a potential partner who's got a robotic dog and service at the earliest type of sensor. Oh, yeah, yeah. Like a BLK devices.

Indoors: From CAD To Campus Scale

Patrick 5:46

Take that content and we'll process it into usable data, like mesh data that could also produce like a vector four-plan type of layout that's very clean, very accurate because it's based on actual conditions. And the users they're using it for operational purposes, they don't care about the level of detail. I'm gonna say geometric detail, but they don't care about everything that's inside the wall, for example. They just need the architectural background model, and they just want to know that the geometry is accurate, and they don't care about all the properties that you would find on an IFC element or a Revit element, for example, because they just care about what they're doing. What are they doing? They're doing like project management or campus security or field technician workflows and for a variety of use cases. What's inside the wall and what's well, I'd say for a majority of use cases, that could be true, but then we have customers who care about what's inside the walls. They'll take advantage of, you know, we have a direct connection to Autodesk, for example. So if you're an Esri user and you're in our technology, you can also enter in your Autodesk ID and authenticate. And our contents panel, you'll see all of your Autodesk stuff, and you can just drag in your Revit models directly into ArcGIS, and you'll see the full level of detail. And you know, we have users that leverage our GeoBIM product capabilities to manage construction projects.

Marc 7:16

We've also got another example. We all heard the term eating your own dog food, and I like a better term. We drink our own champagne. We flew drones around our newest building on campus, start construction right before COVID. And I think every Friday we'd fly a drone around as construction was occurring and uh a dashboard that allows us to look at that model over time. So we're able to kind of peel back the layers of construction. And let's say that the northeast corner of the building starts to get a leak. We can go back in time and see that the flashing wasn't installed correctly at some particular window. And we can combine it with BIM file to see what was really planned and whether there's some mechanical, electrical, some plumbing behind that wall that we need to be aware of. Okay. 4D model is actually 4D model. Okay, okay. That uh, you know, you've been in AEC, but not really GIS. And uh, you know, that kind of you know hit me in the side a little bit. That I was in the same camp. I've barely not expelled GIS when I joined Esri. Yeah. The projects I've seen where you, if it weren't for GIS, the decision making, the design decisions, the context just wouldn't have been there. You know, I know I'm a cheerleader sometimes for this stuff, but it's really amazing how separate the camps were for so long to now see solutions where BIM and GIS and schedule data and imagery and underground and above ground are all combined, almost you know, Star Trek in nature compared to 10 years ago, uh becoming I don't think commonplace, but certainly a best practice.

Patrick 8:40

I just dropped a screenshot in there of some stuff we just processed. This is our beta. It's in a product that's gonna be released in the end of the year, and so it's in its third beta internally, and this is uh aerial data that we collected like four years ago or something, and we reprocessed it. And this isn't a mesh, this is a Gaussian splat, and it's super clean and pristine.

Reality Capture Beyond Authoring Tools

Roopinder 9:06

Yeah, let me find it. It popped up on my screen there for a while now on the chat. Okay, got it. I see. Let me take a look at that because yeah, splat. I really want to discuss because every time I have to look it up, what the hell does Gaussian splat mean, right? And then I figure it out and I promptly forget it. But for one, my understanding of it is it can really make clear what would otherwise be a kind of a messy image on photogrammetry.

Patrick 9:29

Like all the BIM structures, like fences, light posts, you know.

Roopinder 9:34

I don't know, things that are like one one pixel one.

Patrick 9:36

Yeah, like if you put this in detail on the roof of the building I just I put in there, you'll see yeah, you'll see all these very fine details.

Roopinder 9:46

Right. So now Pat You could probably tell me how that's done because I still don't get it.

Patrick 9:51

Basically, I'll look at all the rays that that could be traced from these different images and how they're bundled together, right? Like you'll you'll have like a you'll want to close the loop between all the camera poses that comprise the data, and then you're gonna look at gaps, you know, between the image sets, and then you're you're trying to predict, you know, what what are in those gaps. So you use a little bit of some generative AI to predict what should be missing, the missing bits. And then that's how you can create novel views, you can synthesize novel views based on this technology. So I could put the camera anywhere I want to finish processing it.

Roopinder 10:31

So that would be AI then that determines that that line of pixel case.

Patrick 10:35

Well, thin line of pixels are already there, but there could be some missing element. The other thing that is unique is that is just the function itself, right? So these are like ellipses, bigger small that carry all this information. And just because of the nature of this, you're gonna be able to pull out these fine structures, whereas a mesh would just eat it up. That's why you have like lollipop-shaped trees and farm balloon trees and every mesh model of the earth that you've ever seen. Whereas the trees here are pretty realistic. A LIDAR is a type of splat, it's just a it's a point. It's uh, but so this is a Gaussian splat where the where you actually do have a point, but each point also has, you know, it's like the origin for the ellipse that's connected to it. Yeah, ellipse carries, you know, the texture information and all other stuff.

Roopinder 11:27

You said it's not AI that changes it and from line line of pixels to a cable. Well, where does the AI come in again? Tell me.

Patrick 11:35

It's just gonna find the missing. I mean, what's your your backgrounds in engineering? Yeah. So I I went to architecture school and you know, I studied how to create perspective drawings. I I studied how to create them from you know, even from photographs, right? So how you could kind of decompose the photograph into the picture plane, horizon line, and all these different elements.

Roopinder 11:56

Oh, yeah, yeah. Vanishing points and all that stuff.

Patrick 11:59

So photogrammetric processes you use all of that knowledge. So you're at this instance, you're trying to correspond, you know, the vanishing line, vanishing points and and lines with all the different images to one another based on correspondence between the same thing and different um. So if you do that, you do what you're calling you're calling 3D reconstruction. So you can you can reconstruct 3D reconstruct space just based on images of the space. Um, but often you'll have uh just some missing frames. Like you'll like if you were doing it perfect, there'd be no missing frames, right? Given the nature of the beast and field collection, you're gonna miss some things. In this instance, the radiance field and the way it leverages AI can predict you know what's in some of these missing regions. But the more you're gonna see like some really weird artifacts where you're missing a lot of data, it's just gonna make up a bunch of stuff that's unusable. So they'll call them floaters. Like if you can look it up. What is a floater with a Gaussian? It just looks like garbage.

Roopinder 13:05

So like what they might call artifacts in it.

Patrick 13:07

Yeah, these are artifacts that you want to erase just because there wasn't enough data. If there is enough data, then you can make a prediction about the gaps between any any gaps in the data. But if there's not enough data, the prediction's gonna be garbage.

Drones, Dogs, And Operational Models

Roopinder 13:21

Since that imagery is georeferenced, it knows where it is on the planet Earth, it knows the address, as it were, how close are we to say to that's actually a this building owned by this company, and it and using that that volume, that uh graphic imagery from street Google Street Maps, for example, or whatever's available, even pictures that are taken in the area, using that to fill the leg gaps instead of trying to conjure up detail from the image that exists. How close are we to a world where we add in images from other sources to complete the data?

Marc 13:57

Didn't Microsoft try to do that some years ago? Aggregating public imagery and use it to generate 3D models of of landmarks and commonly photographs.

Roopinder 14:08

Maybe, maybe. There was a there was a project here. Oh, you worked at somewhere you worked at Autodesk. I don't know if you were there when they did that whole reality capture adventure, I'll call it. But you know, they gave up on it. I think they didn't it didn't make up enough money for them, but they were uh yeah, Recap.

Marc 14:23

Well, Recap had gotten, I think, uh breath of life blown back into it, it seems.

Roopinder 14:27

Oh, is that right?

Marc 14:28

Yeah, they probably didn't it was yeah, it was a lot of interest and excitement for a while. I think for whatever reason it died down, but I'm seeing something come come back to life in some way.

Roopinder 14:37

But what I was going to going getting to with that was that they had projects in which they took uh they used the statues that were destroyed from the Taliban uh the Middle East, and they would take the publicly available like Flicker images, yeah, and they would take them all and use them to from Yeah, you can yeah, this is viable and possible and being done today, right?

Patrick 15:00

You could imagine. I mean, this is stuff we were looking at like a decade plus ago, where we were being asked by unnamed organizations to take historical data and create reconstruct three environments of these like like historical aerial data. It was a pretty fascinating project. But but it's only better now because we just had a project about a year ago, year and a half ago in Hawaii, where you know, we were working with SkyDio, uh partner who does drones, and we were flying around the state capitol in Hawaii, and because of flight restrictions, we couldn't fly over the top of the capital. So we just we scraped open source images from the web that didn't have any licensing restrictions and found some that of the top of the capital, and we just added that to the Gaussian processing, and it worked just fine.

Roopinder 15:49

Wow, that's amazing. So it is being done in real life. The other example I thought of, which also was Autodesk was involved in for some reason I got a lot of stuff from Autodesk because I'm right down the street practically from them. When they were redoing the new when they're helping out the Notre Dame reconstruction after the fire, I think they used a lot of photogrammetry because they luckily they had scanned it beforehand. Yeah. I'm pretty sure, I'm not clear on this, that they used also publicly available data because Notre Dame Cathedral, it's been photographed millions of times certain details, and why not use that instead of trying to where things might be missing or coarse, they could get the fine imagery from photographs that are available. That's that should be I think that's what they did to some extent. So the gargoyles and things that were reconstructed, I think they used photos.

Patrick 16:40

Yeah, I would imagine I would imagine they they would have leveraged content like that. But you know, I mean, I could imagine, I mean, just based on the high level of fidelity and the results that I'm seeing from our own Gaussian splatting technology. Like I'm I'm super impressed. I'm kind of a tough customer to please.

Roopinder 16:59

What kind of detail can you get out of Gaussian spec? Can you get down to like a quarter centimeter?

Patrick 17:03

It depends on the type of images you started with. Oh, okay. Yeah, right. Yeah. All depends on the type of images you started with.

Marc 17:09

So what quantity, like you said earlier, how many gaps you might have or what distance you were capturing imagery from.

Time-Lapse Construction And 4D Context

Patrick 17:15

I mean, there's definitely new there's definitely new rules for data capture if you create beautiful splats, right? So flying a site cross-hatched with a drone with a camera facing straight down or nadir, as we would say in the mapping, is not going to create a beautiful three-dimensional uh product. You're gonna need to have the camera at an angle, right? Fly the site, right? So you can pick up, you know, all sides of the elements across that site. You know, so there's just some like different different rules of the road now that we have this new technology. I think it's pretty exciting. We've been we were recently looking at some work where we were trying to use all this reality captured content to create usable data for digital twins. So not sourced from BIM or CAD, but just sourced from the actual reality capture content. And uh, you know, we came to the conclusion like, well, ultimately we need to get to a point cloud so we can create a nice mesh from it. So we've worked a lot on developing really nice meshing techniques for infrastructure, and it it was looking great. And we realized that with the splats, you can create pretty robust point clouds, you know, from those data too. Like there's a lot of open source projects popping up where people are doing this. And the beauty of that is that you just have to have cameras. You could even just have video cameras and capture and recreate these 3D environments and then create a really robust point cloud, and then you can create these mesh products that you would want to use to turn into more traditional formats, right? Like the vector lines and geometry you would you would more traditionally see.

Roopinder 19:01

You mentioned taking high uh high resolution images, photographs. That's like megabytes of data right there, right? Right? And then if you add LIDAR scans, you're using like millions and millions, maybe even a billion points of data, right? That's a lot of data. When you compare that to vector format or some kind of more uh shorthand format representation of image uh objects, right, it just seems so inefficient.

Patrick 19:28

Well, it depends on how you're looking at the problem, right? You would say what's more efficient that I start with a dense point cloud, then I can create these derivative products that are really lightweight. Yeah. Or is it more efficient to start with a bunch of personnel hours, you know, creating BIM files and going through that whole process to come up with something on the other side that probably has too much level of detail. Now I could just create an initial data capture and then there we could chew on it. Spit out some more lightweight products that people use day to day. Or you can even just start with, like I said, with images. Just start with a video where you don't even have to be that well trained. You're just running around with like a 360 video camera and collecting data. And then you're synthesizing that back into a 3D environment. Yeah. And create then you can create point clouds from that.

Roopinder 20:21

I read an inspection being done with drones, and they captured the very detailed model down to where you could see cracks in the bridge. And it saved bridge inspectors from having to hang off cables and you know drive to the site and saved all kinds of time and all kinds of effort because they had that vast amount of data that they could later process and do anything they wanted with. I suppose it's the same with large fields of data, big square miles, although maybe not down to that crack level, but certainly down to the building level.

Patrick 21:07

You know, repair planning without ever, like you said, uh you know, one person goes out, does the survey for a day, drones, and gets back to the office and has vast amounts of information to then plan a really efficient project.

Roopinder 21:21

I guess I'm more or less stuck in the past with this idea of too much data because I'm thinking of how much can my how much can my computer store? But I guess that's not an issue anymore. You got the cloud, right? So the biggest issue is more or less like sending, transmitting your information over to the cloud, getting it back.

Marc 21:38

Yeah, and you said something earlier, Window, it it and it reminded me that you and I, and and Pat maybe to a couple of the years, less degree, come from a file-based world where the size of your file was a limitation and you had to plan around and you uploaded your files to FTP sites, or you still might even be worrying about shipping a hard drive literally across the country. Yeah, I I think I'm still in that mindset.

Gaussian Splatting Explained

Patrick 22:01

Yeah. I mean, I don't I don't think that's all foregone. Like even in the work we were doing in Hawaii, we uh, you know, the tyranny of distance is still a problem, right? For compute. I mean, your compute wants to be next to the data regardless of whether that's on site or in the cloud. So even if you're processing in the cloud, you need to upload potentially terabytes of data, which just takes a lot of time. You're not gonna get rid of that problem. Just got to figure out where you want to deal with the problem. At least if you're if you have a cloud-first approach, you could put the data once somewhere, you create products, derivative products that could be accessed by everybody. That way you don't have to have like dozens of people having the same data on, you know, the same source data on all their machines, right? You could just you could just log in and you could say, Oh, I want to I want to download that product. Like in fact, I could we have a product called Site Scan, which is like what our all our drone data goes into on the web, and it does all the processing, right? You just upload the images and and then based on what you want, you just say, Well, like I want the point cloud version of it, and you can just download the point cloud of it. And that way, only the people who wanted the point cloud, or only the people who wanted the true Ortho product from that get it when they need it. Not everybody has to have all the source images.

Marc 23:23

And we're also, I think, in an era where down the streaming our workflows, we're not shipping the entire project set to someone who simply needs to view and inspect a particular area. Like you said, Pat, they process the data one time, and then others who need access to it zoom into the area that they're interested in, and the system, let's say, streams to their desktop through their browser that allows them to interact with it. And never outside of that original generation of a massive file that needed to be processed, right? We worrying about files. Instead, we you know, or we might still be concerned about our bandwidth and our input. But we're we're consuming data that is specific to our needs versus receiving giant files that everyone then needs to process uh synchronously.

Roopinder 24:07

That massive file could set in the cloud somewhere, and every user can just access the bits that they want from it. And it's all it's all great because it's a single source of truth. It's not like you have to have the same model everywhere that replace and risk uh duplication and confusion. Really that's really been a big benefit of the cloud. But yeah, there the problem still exists that there's you said Pat, they meant a turn of distance. We're limited, we're limited by the speed of light or transmission, right? It still takes time if you go great distances. We are putting up terabytes of data. You gotta wait till it gets there or till it gets down from there. Pat, you said we were you were I don't think you said invented reality capture, but among the first companies that Esri was one of the first companies, or how did you say it? But was we been in the reality capture world yeah. Would you say that Esri was the first?

Patrick 24:58

Or no, I mean we we uh look to me the whole revolution in reality capture is like a it's like Marty McFly and Back to the Future for us. Like we've been we've been dealing with this for eons. It's just the sensors have gotten way better.

Roopinder 25:16

You have the flux capacitor?

Patrick 25:17

Yeah, so you ain't been working with flux capacitors since 1970. So it's it's like uh it's kind of exciting because it's a little bit of a blast from the past, but it's new tech. And you know, I think for the world that I was trying to solve for, like dealing with buildings and what's inside of buildings and at the campus scale, it's just super exciting because honestly, no matter how easy you try to make processing BIM or CAD into GIS, it's still always gonna have some level of tedium and painfulness. And and it'll carry a level of detail that our customers honestly don't really care about 90% of the time. So if I can use reality capture processes and techniques to create superior information products in beauty and fidelity to fidelity to the truth and how geometry is in real life, then that's if I were to try to do this, get this fine type of image that you sent to her.

Roopinder 26:16

Is it like like photo in Photoshop or is it Esri product or do I have Esri product?

Patrick 26:22

I'll uh here I'm trying to load this up right now so I can just share with you how this the image you sent, that's our GS Pro, right? Let me share my screen. Let's see if I can do this. Um I I can share, so let me try this. And then share. Can you see my screen? This is a building at Esri that was constructed about five years ago.

Roopinder 26:45

Okay. As as I I'm looking at it, I'm thinking the splatting has been done just on the building itself, but not the trees are left unsplatted.

Patrick 26:53

Well, the the capture was focused on the building, right? Okay. The trees were incidental. So some trees came in like this. So do you see that?

Marc 27:02

Yeah, what is it like? One inch strut, maybe a little.

Filling Gaps With Open Imagery

Patrick 27:05

Yeah, it's like a strut for the ducting. Oh, yeah, yeah. That's a crazy. I can probably see the micro inverters on the on the solar array. I don't know. Yeah. Um, I can probably let's see in the atrium here. Yeah, I can see into the atrium.

Roopinder 27:20

Uh yeah, no, this is a this is a exquisite amount of detail.

Patrick 27:26

Well, the beautiful thing about something like this is I could, I bet I could do here. Let's try this. Like the level of detail is so high on something like this. I could probably give you a quick demo on how we integrate AI.

Roopinder 27:41

Is it just a button that press that says go go splat?

Patrick 27:44

To do splat, yeah. We actually have a ribbon here. It's grayed out right now. I can't do much with it, but um, on this other panel, like I have all the drone data, for example, for a flight over the University of Redlands. If I go to the reality mapping tab, I can create other products. Right now, I don't have enough data on this particular data set. I could create these other products. And so it is just pressing a button, but but you're gonna have to come back a couple hours after you press the button.

Roopinder 28:12

Processing is done on the cloud?

Patrick 28:14

Uh, not for this, but I would imagine, like for our site scan product, that you will have cloud processing for splats. Oh, okay. But I just don't I don't think that that's not on the roadmap for this next release. Let's take a look at some of these trees.

Roopinder 28:30

At some point, some AI is coming in to say, okay, this should be a straight line or this should be a certain object. Well, let's just we could just do a what what level are you using AI for recognition of objects?

Patrick 28:43

Well, we're not using any eye to recognize objects in this, like for the splat itself. So that's that's what I was gonna show you. So because the level of detail is so high, I could probably go to our image analyst, I could go to our deep learning, I could click classify objects using deep learning.

Marc 29:03

A lot of questions this year, you know, what is Esri doing in the last couple of years really around AI? And that's we also have been involved with for many years longer than most folks have been talking about AI. We have a source of content called Living Out of the World that includes deep learning models for some hundred, maybe by now, different object detection algorithms. So let's say you're you're a talent, you're a city planner, and you want to see who in your community has a swimming pool. We have the ability through our deep learning toolkit to upload your image that you received from Maxar, a satellite image company, and you're learning model for pool detection. And almost like clicking a button, all of those pool are automatically detected, and you now have a new feature layer of all the swimming pools in town. Probably gives you all the addresses or those. And you can run that against your records to see who actually applied for a permit to build this. Um, you know, uh, so you get an image of your community, your town, your city, your state, and you have these deep learning models against the image, and it generates an inventory of the different types of trees because the uh the pattern recognition, the computer vision, AI, if you will, that's how to recognize trees or bird migration, wildebeests versus zebros clean, you know, with uh the desert in Africa.

Roopinder 30:33

Um exactly, exactly. Yeah, then that the AI state of the art for that is is pretty well developed right now, I would say. Because I can even take okay, so on my smartphone, my iPhone, I can uh I can take a picture of a shrub or a tree or something or flower, and it'll tell me that's it'll match it up with objects from public. So I imagine that's just a matter of time before that gets into professional.

Marc 30:56

Where there is well, we have a tool called Survey13, where you put up together a survey type application like Monkey or Survey Gizmo. Ours is called Survey 123, and I've used it for a whole bunch of survey purposes, but in there is the ability to generate, to take a picture and have our AI tell you it's a fire hydrant, it's a manhole cover, it's an edge. So we do have those segment classification tools running as well combined into various workflows.

Capture Rules For Better Splats

Roopinder 31:22

If I want to just take a picture of a building, it should kind of know what that building looks like from the street maps. It knows where my camera is and knows what direction I'm looking at, it can see the image, right? Now can it line that up with that model? And and then you know, all that information is available, right? AI should be should be able to use it somehow. It should be able to get a lot of information just by pointing to it and asking what that is, just by recognition. Google to their generosity has made all of these so much video image be available off of the built world.

Marc 31:56

And a whole lot of other partners. I mean, we are uh capable of Google images, but we have dozens of other partners who are going around collecting images off of garbage trucks driving to the streets. Or Max mentioned satellite imagery, I mean fixed wing, helicopter, drone imagery we talked about. So yeah, these are just inputs in the IS model now.

Roopinder 32:15

Where is all this being stored? Is that is Esri have like giant data centers?

Marc 32:20

Like, we data curation. We aren't necessarily storing all the data that near mounts are in generated, but you can access it through our tool almost regardless of where it's stored. I see.

Roopinder 32:31

So as Esri does not act as a as a data collector of this of this uh Earth imagery.

Patrick 32:37

Well, we do curate certain data sets, but usually it's in partnership. Like if it's imagery, we're part we have partners that provide some of the imagery. You know, we don't maintain a fleet of airplanes, we have partners that that do that type of stuff.

Marc 32:51

But if the sense is done every 20 years, collecting demographic data, get that law of data, you can massage it yourself and you can make it available and ready to be in a GIS. But we all share that and organize it and make it ready to be a feature layer in your map if you want to understand transportation corridors relative to particular demographics. Yes. So there's half a dozen ways, if not more, that a user or tools would access public, private, open source, secure data sources.

Roopinder 33:20

It's a fast-moving subject, and more and more things are being uh recognized as intelligent objects, and you can make use of them. One thing I wanted to ask though, you would show you the image which has uh all those reflectors on it. Solar reflectors. I was thinking, wouldn't it be nice? Put this in your wouldn't it be nice camera? If I could like tell it on the screen, okay, this is a solar collector. How many solar collectors like that are there? Like a conditioning unit. Conditioning unit. How many of them are there on the roof? Is it you see anything like that happening? AI to find objects that are similar, even though they may be of a different show, a different perspective, right? We can maybe infer that hey, this is a solar collector over here. This is the same in color if not in shape, but it's close enough by similarity or by proximity. It's probably a solar collector. So I'm gonna count you know, all that reasoning. That's sort of my impatience on expectation. Let me share my screen for a second here, Pat, while you marry her on.

Patrick 34:19

Uh I'm I can share now. I mean, I think that I move this up. So let me let me share.

Marc 34:24

This is go ahead first. This is our living networks of the world that I mentioned. And I split solar, and the third or fourth tool here is the solar panel detection, the learning packet. So I have this package to detect solar panels against my images are Redlands, California, St. Louis, Missouri, San Francisco, yeah.

Roopinder 34:43

I take a business for spies in the sky kind of thing. Like I count them for uh Yeah, that could be for all kinds of purposes.

Marc 34:49

It could be wanting to understand where your untapped solar potential is. Ah, okay. More panels exist and where should new ones. This is this library, thousands of layers of information, and these are just the ones that are tagged. Yeah, most of these worked off of images. I could try to use the text SAM model, which is segment in. I could enter a text prompt and say solar panel. And nice. Now these are all it would create polygon features for every solar panel.

Patrick 35:17

Okay. So these are all algorithms or what would you call modules that are available to everyone? These are deep learning packages or modules that we have available.

Roopinder 35:25

Uh craft detection. Freely available? If I just have an Esri subscription, I can come to the code.

Patrick 35:30

Yeah, if you have an Esri account, then you can download these. They're all in our living atlas. Oh, nice, nice. Do maintain quite a bit of data.

From Images To Point Clouds And Mesh

Marc 35:39

Curie, yeah, we curate. These aren't necessarily big data sources. These are algorithms that run against images. And the images could be ones that you license from one of our partners or that you have generated yourself as a as an engineering firm, perhaps. I see.

Roopinder 35:52

Oh, so so they there is a a possible ability then for a customer of Esri to uh say, okay, I developed this on my own. It might be useful for other people.

Marc 36:02

I can volunteer it to we've got like 3,000, maybe 4,000 partners who um range from engineering firms.

Patrick 36:08

We have a whole team dedicated to Living Atlas. So they're the ones who who create, curate, consolidate, all that and maintain it.

Roopinder 36:16

Thanks for bringing a mechanical engineer up to speed on the GIS. It's uh quite an education for me, and I think very useful. I try to cut across disciplines because uh all engineers at our specialists in one little niche, curious about other other areas at that level. I think this is going to be very interesting for other disciplines. It's probably old hat for you guys. It's like, ah, I'm surprised you don't know this. You should get out more, right? But but for people that don't dig too deep, I think it's fascinating. So thanks, guys. Thanks, Mark. Thanks for volunteering for this. And and Pat, Pat, thanks for uh coming in. Probably at short notice.

Patrick 36:55

Yeah, no, no worries. Uh hopefully it was useful. And then uh there's a lot of resources out here out there too on splatting that can better explain, you know, how it's making these inferences in terms of like stitching together the missing missing gaps. But it's super interesting technology for sure.

Roopinder 37:14

I think with your explanation, I'm gonna retain that definition of splatting now for some longer. Thank you for listening to FoDES, the Future of Design and Engineering Software Show, brought to you by ENGtechnica. I hope you have learned of a new application or technology that will help you with your job. If you have an application you think would be of interest to other engineers, please let me know by emailing me at roopinder at engtechnica.com or message me on LinkedIn.