What happens when you merge the world of public works with drone technology? Our guest for today's episode, Brian Simmons, a Principal Engineer with Bolton & Menk, and tech enthusiast, shares his insights. A seasoned expert in the field, Brian provides us with an engaging primer on drones and UAVs, distinguishing between the two, and underscores the growing popularity of these devices in public works and the emerging applications.
Gartner Hype Cycle
Bolton & Menk UAV/Drone Services
University of Minnesota LTAP Technology Exchange Article: City of Fridly, MN, Tests Drones for Public Works
Welcome to the Public Works Nerds podcast with Mark and Mike. Welcome to the Public Works Nerds podcast. I'm Culver and I'm Mike Spack. We're your co-hosts. Today we're talking about using drones in public works with Brian Simmons. Brian's a principal engineer with Bolton Menk here in Minnesota, out of Bolton Mank's Chaska office, to be precise. Not only is Brian a principal engineer, he is a tech nerd like me, and Brian and I have geeked out numerous times already about asset management, sensors, meters for sanitary and water and just a little bit about UAV. So I'm really eager to get into this conversation and learn some more about how we can use UAVs in the Public Works realm. There's a lot here. So welcome, brian. Thank you, thanks for having me, thanks for joining us. I saw you guys start the podcast and then, with the word nerds in it, I was like, oh, I got to get in on this. Yeah, you're a nerd, they're my ilk, yeah. So, yeah, talk to us. I mean, give us a little bit of a primer on your background, first of all, sure, and then, you know, bleed into. Well, first of all, like, what's the difference between drones and UAVs? But first talk about yourself and how you got to. So, first off, I shouldn't be here. I'm a municipal engineer, yeah, so that's my training, that's my experience. But I've been doing this for 20 years and just sort of fill in the gaps and keep me from getting bored. I've had some very supportive leadership that saw that I needed to do some other things to keep me busy. And so, yeah, starting at the beginning of my career, there's a handful of things that were like, hey, that's got technology associated with and alongside civil engineering. But Brian would like to look into that. And now, you know, 20, 15 years later, a handful of those things have turned into complete. I'll just call them product lines on their own, you know. So I've dabbled in a number of things now that we as a company do. You know it's sort of a primary function, and UAVs, reality capture, is one of them. So, yeah, talk about drones, uavs all the jargon acronym. Yeah, what does this mean? So everybody knows what a drone is. Right, it's a little quadcopter, a little toy. You can go to Best Buy and buy that. Or when we first started getting into this, the term drones was sort of associated with the global hawks, the reapers, you know. So we sort of set aside drones as the PPPU, the bad drones and that's more entered the public vernacular. Now Everybody understands that and so that's sort of just become accepted. But a UAV is an unmanned aerial vehicle and the industry is shifting toward SUAS, which is small unmanned aerial systems, and you're going to see that unmanned be switched for uncrewed, because it's 2023 and the world is inclusive. So how long you've been playing with? I'm just going to call them drones, yeah. No, for the purpose of this discussion, we can just say drones. We can just say drones. Yeah. I started dabbling actually in the data side of things in photogrammetry in 2013-2014. I used to attend Autodesk University on the regular and that particular conference, you know, a lot of education, a lot of showing off software, and I realized that I could use my iPhone to build 3D models, and so I got way into the data side of things before I'd ever actually touched a drone. And then you know, really, the Phantom 4, the Phantom 3s and the Phantom 4s, which were the most publicly accessible, relatively inexpensive and fairly technologically advanced. Those were out in 14, 15. You know, I went out and I bought one, and so I was flying a drone on my own, knowing that once this became commercially possible. We wanted to be into this, or at least we wanted to know how we were going to enter this. So for a while, I was kind of operating on the fringe. The real trigger for us as a company was August of 2016. Prior to that, you could operate with a pilot's license Right, yeah, you're just going to ask about that. Yeah, yeah, you could get all these acronyms but there was a 333 exemption, where someone who had an actual pilot's license could operate a drone commercially. And so then, in August of 2016, they dropped the Part 107, which is the commercial drone operator's license. In fact, there's two of us Part 107 licensed pilots in the room right now, and that was really the trigger for this industry. They came out with a separate drone pilot license, a drone certification, and from there things really took off. So what's involved in getting the drone certification? It's a written test. Okay, you know, it's really not about flying a drone. It's about how do you operate the drone in the vicinity of other aircraft. Manned aircraft is the sense you get from the FAA, and this is true, but the drone always gets out of the way. The manned aircraft always trumps, you know. So the safety is the drone operator's responsibility and there are some interesting things to try and separate those two. Drones can only operate for the most part in that zero to 400 feet. Oh, and general aviation should be 500 feet above. But there are lots of fringe cases where the drones in the manned aircraft will interact, for example, aircraft taking off and landing. Obviously they're below 500. Right. So there are challenges working in, for example, a community right off the end of a commercial runway in MSP, or you're in a rural community where somebody has permission to fly below 500 feet like crop spraying. So then you have to learn how to interact and that's mostly what the Part 107 is about. So if a public works director wanted to start using got a drone, wanted staff to start using it, that operator would need the 107 certification. So technically, public agencies could operate under a different set of rules that they refer to as a COA, a certificate of authentication. But for that COA to cover up an agency, they would need to come up with their own program. So the easiest way is for you know a city, county, whatever to say we're just going to have our pilots get Part 107. Rather than come up with their own process and then they're covered, and this is a case for any application where you're doing it for work, like any commercial type of application, right? Yeah, I mean, yeah, what kind of drone would you not need a license for? It's less about what kind of drone and more about how and why am I using it? If I go out on the weekend and I'm not licensed and I take a bunch of pictures you know fireworks and then I turn around and I sell them to the local newspaper if I am not licensed, that is against the rules, got it? But if you don't sell the pictures to the newspaper, then you're still on your Instagram. Yeah, you just post them on Instagram. Yep, no, I did this as a hobbyist and there are a set of hobbyist rules. There is a short written quiz for hobbyists, but that's very easy to obtain. Actually, I think both are, but the Part 107 is a more stringent licensing process, for sure, right? So is there a course for the 107 or there's lots of study materials out there? There's lots of study materials. The FAA has a lot of resources themselves, especially since, like I said, a lot of it is about understanding where the manned aircraft are going to be and knowing how to read airspace. Do you have to do an annual renewal Every two years is a re-up and thankfully, during COVID that went from a shortened version of the test in person to an online quiz. It looks like it's going to stay that way. So awesome, we'll call that a win. Yeah, how much does this two-year certification cost? That's free. Oh, the initial test, I believe, is about $150. Okay, yeah, it's not financially a huge burden. So a UAV, that would be a drone that would be of quality, sufficient quality and capability for actual public works use, whether you're just shooting aerial video or whatever else you're doing. How much would that cost? There's a pretty big range. Yeah, there's a lot going on right now in the industry surrounding what we'll call blue UAS, blue or green UAS. These are two lists that are being curated. Blue UAS is the Department of Defense and green UAS is a parallel list that the industry is policing themselves. But these are both concepts of we're trying to look at and identify drones that are have mostly American-made parts or are assembled in the United States, but what's really behind it is they don't send data outside the country. They're not sharing any data. So if you're working on a project that has federal funding and they say they follow Department of Defense procurement rules, that takes a whole bunch of drones off the list. And also, is there actually a fear of like just buying some commercially available drone and that data is unbeknownst to you being sent? Yes, to another country? Yes, and I think some of that has actually been substantiated. Wow, and here's the bummer part about it is, is that DJI, who makes the most user-friendly drones on the market, the most accessible drones on the market, is very much a Chinese company and there is some Chinese state involvement in sponsoring DJI. So is this data? Do they think this? They've substantiated it, so they theoretically know how this is happening. Is this happening through a satellite connection? Is it happening through some sort of internet internet connection? So you have to connect drones to the internet for a number of different things, got it? One of them would be if I want permission to fly off the end of the runway in MSP, I have to connect the drone to the internet to submit an unlock, and so now I have shared with any number of parties outside of my control where I am operating and when, and I have shared with them a bunch of my contact information, and I've done that to unlock the drone, so, and that's a sort of a DJI specific thing but they have their own version of the FAA airspace map and they lock it down because they don't want me to go to Best Buy and spend $1,000 and then use that drone in some sort of terrorist activity, and so then they lock it from being able to fly immediately off the runway. And the ability to unlock it is how we go there and do work, but I've shared that data and it could go somewhere else. That's something that I had never even considered, as we talk about using these drones for commercial purposes or public works purposes, that we could actually be sending data to China. What's up? Others, russia or something. Yeah especially our public works infrastructure data, right, yeah, well, and those of us that are in public works, we sort of think about that nonchalantly, right, right, oh, you know what's so important about the locations of my catch basins. You know how is someone going to harm the public with that? But I think we maybe take that for granted a little bit. Yeah, also, I will say that, like as an organization, we sort of keep our own data somewhat secure. You know, we try not to send a lot of it out and actually the data piece of this is kind of where I want to go, because you know it's possible to go to Best Buy, spend $1,000, come out the door with a drone. In our case it's probably more likely we're spending between two and five thousand One of the biggest triggers we get way into. We won't go there today, but you can get way into the weeds on the camera types and there's a difference between a mechanical shutter, which you know a full iris that opens and closes, versus an electronic shutter, which they call a rolling shutter. A rolling shutter would go from open from top to bottom, but if the drone is in motion, that can produce a little bit of distortion to the photo. And the reason that's important for us is because we don't take just one picture. We automate the drone into these lawnmower paths a lot of times so that we can cover, you know, a whole project, a whole block, a whole community, and then we stitch them all together. And when we stitch them all together, we want those photos to have the same characteristics, you know, taken with the same optical settings, the same white balance, exposure, all that stuff. So the photos look the same when we stitch them. So they look like one big picture instead of a hundred individual pictures. Yeah, if you're going to apply some type of machine vision to them, then it's a lot more accurate, consistent, exactly, exactly. I mean we run into that already with, you know, like clouds coming through and you can see the factors change. We've done a couple of projects where it would be like a five mile linear trail, for example, and if you decide you're going to fly that down to one end, turn around and come back. So you've got some overlap, but the lighting conditions may have changed from when you left. That origin point went down, turned back, came back. You know, when you return to the beginning of that five miles, those two photos might not stitch? Yeah, because the lighting has changed. You know, the sun position has moved enough. If it's taken you a long time, those photos look different, yeah, and so they're difficult to stitch. Are there best practices, like time of day? I think of looking over at Tony, our producer, who's done a bunch of photography like I mean, don Dusk are like the golden hours for, like, wedding photos. Yeah, that kind of photography, but it's almost the opposite when you're trying to get consistent, well-lit or surveying purposes, yeah, yeah. So, don Dusk, great for drama For photogrammetry and mapping. You want no drama, you don't want drama, so I don't like drama. Direct sunlight not necessarily the greatest because it creates shadows. So an ideal capture day for mapping with a drone is, you know, 10 am to 2 pm, overcast but still well-lit, okay. So overcast, you know, diffuses the clouds, so then the or diffuses the light in the clouds, so then lighting is pretty consistent. One of your questions on here what are our most common applications? We are flying, mapping flights and taking photos, probably most commonly. A lot of times they are to stitch together and a lot of times they are just one-off photos. We do a fair amount of video capture, but a lot of that is decision-making tools or communication tools. And now we're going to make a before and after out of this project or we're going to render in a proposed improvement, and so we take it a video flying down this roadway and then our visual communications department, you know, will take that. They will map the position of the drone and then bring in a model from Civil 3D or something, impose that in there and then we'll spit that out. Someone could take that to a decision-making body you know, city council, whatever and say this is what it's going to look like and they say you know, they bang the gavel and say yes, we'll spend our money on that. So it's a lot of decision-making tools externally. And then internally we use it to augment survey and I will jokingly call it faking in survey. You know, my ideal operation is you know a surveyor who's out there doing a topo survey and then launches the drone and does a mapping flight to augment his own shots, and also they'll fill in around. Or maybe there's some obscure areas. You know there's a dog barking behind a fence behind the house and we don't want to crawl back there. But then later on during the design of a project we might need to Dig that person's water service close to that fence, and it's really helpful for us to be able to see from the aerial perspective that there is a paver patio right there in an In-ground pool. Okay, we need to be careful what we're doing, excavating there. So how accurate can you? Are you physically drawing in off of the the aerial into the survey or how does that look like? Did somebody pay you to ask that question? No, so accuracy is something that I have spent a lot of mental energy. Okay, I there's a lot of people in our organization that have spent a lot of mental energy on talking about. You. Guys asked me how I got started in this. In 15, right before the licensing dropped, we formed a UAV committee and a handful people within Boltman Bank, myself included, and some surveyors, because we wanted their input right and want to be able to talk about accuracy of data, and so we kind of landed on two separate worlds. One of them is the photogrammetry, the stitching of photos, and We've got a handful of tools we use, like ground control points and Augmenting that with actual surveyed shots to bring those, the accuracy, to a higher realm. We have an accuracy statement where I would tell you from just straight-up photos. I can place those points in a one-foot box, so I won't guarantee those within a foot Relatively, which means in relation to themselves, we are getting very high accuracies. In fact, if we use ground control on a hard surface like pavement, I we can get you know almost survey grade, so within centimeters. Okay, if we employ all those tools, and it's a great day and conditions are ideal. But that isn't consistent enough for us to be able to advertise all photogrammetry Centimeter act. Does that make sense? It does. Is anyone advertising Absolutely, absolutely. I actually would prefer, I would prefer to come on here and say that our accuracy statement is whatever those other guys said they could do. We're doing that too, you know, because it like there are, there are some in the industry that are out here saying, you know, we're getting within millimeters with photogrammetry and I say, cool, come on up. No, okay, and so then we you know what we'll demo it and kind of pick it apart a little bit. Yeah, it's sort of unfairly. We have a site that we have referred to as our UAV lab that has been captured with probably every scan and photogrammetry and survey product known to man. Okay, so if anybody comes says I'd like to sell you a $50,000 drone. That's millimeter accuracy. I invite them up to scan that particular site and then, okay, when they send me the data, I can send it back with a no thank you. So, or hey, this is great, I'd like to buy it. What about outside of just video or or Photos, like using other sensors? Yep, so that's a great question. We, at the same time that we embarked on that, we said a one-foot box isn't great. What does it add? A true survey grade version of this look like. And so we went down a path that led us to Purchase a half million dollar lidar sensor. Well, which is essentially an aerial laser scanner. You know, lidar is light detection and ranging, and so that's a spinning laser that captures its origin point and then captures the reflections of the laser and with that we can build a point cloud. And so that you know, we have various versions of that, both Terrestrial, so like set it up on a tripod, or mobile based, where you mount it to a vehicle, and then this is just another iteration of that, where we hang it below a drone. This is not casual. We've got a half a million dollar sensor below a $60,000 drone way to put it. Yeah, no, it's not casual. We're like. You know, brian just like sits in his office and chews on his nails and waits for a phone call that the drone didn't crash, but it's well insured. We've moved past that. Yeah, so that was our, you know, solution for the idea of survey grade and and the lidar is Bronkers, because we can, we can collect so much data. This is, you guys hear, a theme for me. Yeah, the data is where everything heads. Yeah, so are you moving towards keeping this lidar out Every day? I mean in replacing survey crews? I mean, whoa, whoa, we don't, we don't displace any survey. But but there are some very good examples where we, you know places like Very, very active County roads. It's very difficult to close that arterial route. Oh, it's very difficult to ask a surveyor to walk down that center line with active traffic if you're not gonna close it. So there are lots of examples where it's much easier to fly that and then it would be to send a surveyor out. There is always some survey involved. We're gonna set ground control. We're gonna, we're gonna do ground Verification shots because we're checking in on ourselves with the data. Not not everybody does this, but this, this is our best practice, and so We'll combine those two into the the most accurate point club we could get. Okay, and I would also think, when you're dealing with rivers, wetlands and want more accurate data in those situations, and the surveyors would prefer not to put on the waiters or even the scuba gear. So so water is death for lasers. Okay, what water eats laser? Essentially Okay, so we don't get any returns on water or below water, but what that does mean is I get laser returns right down to the edge of them, to the edge right, yeah, and and there's a, there's a thing coming Right now. This exists in two realms. There is a thing called green laser LiDAR, which uses a different color laser, and then we have these. You know, aerial Lightweight will bowl LiDAR applications, where they've taken the laser in the mirrors and they put it in a compact package. They are just starting to integrate the green laser LiDAR with those packages. Green laser LiDAR is is the right wavelength to, so it's like essentially polarizing. You know, it's a difference between putting on your polarized fishing sunglasses and being able to see through the glare on top of the water. Green laser LiDAR gives us the ability to do bathymetry from above so well, you'll get hits on the water and then you'll get another set of hits, you know, whatever you can see below the water. Wow, how about will that penetrate? That's a good question. Yeah, I'm not selling green laser, lidar, I don't know. Yeah, but something to test, I you know I haven't. I it's not to the point where we buy one yet, right, one of the five hundred thousand dollars. Yeah. Yeah, I go to thought casual, but for the fishermen a fisherman out there you might get more accurate. Well, but bathymetry is a very real industry, mm-hmm. You know we partner with a couple of firms that do a lot of. You know, I don't know what we'd call that area, I don't know if it's really shoals or not, but the sort of the shallows of the south end of Lake Superior, and I mean that's truly coastal engineering type of things, right, putting up peers and things to support the shipping industry, and all of that they have to handle, you know, everything but major tides, and so we've done some laser captures to support them, lidar captures, but you know they had to add the bathymetry themselves. Okay, cool. So one, one interesting thing that Mark and I talked about previously that I kind of want to add to this is that being involved in this industry now for a couple of years. You know we're almost 10 years now, yeah, and drones have gone from sort of the toys spectrum, mm-hmm, you know Mark and I were joking about the hype curve prior to this and and I won't, I won't spend a lot of time talking about that, but on this particular graphic, you know we were reaching the, the, the slope of enlightenment, right, and we're headed towards the plateau of productivity. That that the marketing company that came up with that graphic is gonna sue me someday if I keep talking about this. But you know, we've gotten to the point where they're not just novelties. We're actually employing, you know, drones and UAVs on a daily basis to replace Situations where people can't go or humans can't go safely, you know, or captures that would have taken us a long time. You know one of the things that, yeah, there you go. Yeah, so we're through the trough of this illusion. Well, we'll show this in the, in the, in our episode notes, but it's, it's a, it's a cool little. I love the, the terminology, you know, like, like he was saying it, you know, and it's just basically kind of mapping out acceptance of new technology and and Applications and things like that. And you know it starts with this technology trigger, which is your zero and your XY Scale, and then you go up to really high Ascension, up to the peak of inflated expectations. Yeah, when, what do you mean? My new iPhone can't do everything for me, right, right. And then it dips down pretty quickly again almost the same slope down to the trough of disillusionment, which Is a. I love that. Why did we spend half a million years the trough? Yeah? and I know we're wallowing in the trough I've definitely wallowed in the trough and then it goes about halfway up to where the peak of inflated expectations was that it plateaus and and that's the slope of enlightenment is that it Ascension back up to the, the plateau of productivity. So once you've actually, you know, really honed in the technology and kind of mastered it, now you're at this Plateau and I think that's just a great way to talk about new technology and applications in general, you know, and I think it really Manages our expectations, for it helps us really. It makes us laugh about it a little bit, you know, puts it in a little bit of a humorous Tone, but it does help us really manage our expectations of how is this Adaptation going to occur? Yeah, for anything new. So so where are we in the in that in the UAV market? Where we are we clearly in that plateau of productivity? Are we still in that slope of enlightenment? We're in like the first, just the very the first inches of that plateau of productivity when we know what we can use it for, we have some idea of the limitations and also the the trough of disillusionment is behind us. You know we've been disappointed at some things that we thought we could capture and we did it and that didn't work. And so now it's like here, here are the actual applications, here are the things that we're we could do all day long, and so let's start. Let's start doing them. Okay. So I mean, one of the first applications I heard of probably 15 years ago was getting out there on bridge inspections and just you didn't said any bucket trucks and all the hardasses and stuff, so kind of that inspection level. We have some surveying applications, we have some kind of graphic simulations. What are other areas that can be applied to a public works for kind of using these? One Thought I've had is we had Mark Ray on the podcast. I'm such a fanboy, mark, I can't miss an opportunity to say that. And in talking about just kind of the disaster response and kind of my brain has gone to, I mean, in a military application we're flying these UAVs across the world. Yeah, I'm a command center. Is there kind of an application for like, sending the UAV out the top of the public works building but having those video feeds into a command center? Or could my team just sit around and, like at our staff meeting, say, oh, we want to check out that thing over on X property and quick, send the drone out and like, get the live shot as we're sitting in our staff meeting talking about it, like to answer the question real time. Are there things like that? So we should be thinking about it. So there is one yes to almost all of that. Yes, I'll tell you that. The aerial perspective, you know, like the advent of Google Maps and the accessibility to satellite imagery, you know, I think that sort of changed all of us internet users. I'll just say right, I know, all of a sudden, I could access an aerial photo of my own home. Right, I could look in my neighbor's backyards. Drone imagery and the stuff that we do with mapping and stitching takes that to the next level. Okay, because we're flying at a much lower altitude, almost regardless of how crappy or nice the camera is right, or whether it's. 12 megapixels is sort of the bottom of the range, you know, and we dabble up into the 80 with, like, a Sony a7R4. That's almost too much data, honestly. Well, yeah, your file size is too, yeah, hard to pick. Yeah, oh, trust me, yeah, word, word, it doesn't like us for this. We have our own. So we have our own storage devices and our own server for this stuff. But in drone imagery, you know, we can identify things like gate files and manholes and pavement. We can see joints and concrete, you know. It's just that next level, beyond the satellite imagery and on a consistent basis. So I mean, are we to the point where we can do and this gets? I think we're going to talk more about data and what to do with the data here but I mean, are we to the point where, instead of driving our streets and doing like pavement surveys and the set and the other or maybe even sign inventories and surveys, can we use our drones for this? Yes, yep, we can. Okay, we can, and that sort of takes it into like the asset management role. Right, you could do updated imagery. You know we can fly it tomorrow and you could do extraction from that stitched photo of your assets, you know. So if you don't have time to send somebody out with a handheld GPS, but you want to have some idea of where your, you know we want to start mapping our manholes, our underground structures. Yeah our manholes, our signs are all of the that infrastructure. Who cares if you're down to the millimeter, right, I mean, if you don't need that level of accuracy for your inventory system, well, and there's very much like some data is better than no data, right, even though there might be higher levels of accuracy. If this is all you got, and especially in our case, if we've already got things, I'll just say, in the can, right, we've already taken and stitched a whole mapping flight, a whole series of photos into one big photo. What else can we do with it? You know, one of the very first things we did in 16 was for a community in the Metro. Here we augmented their entire pavement management plan and their process was to send somebody out, you know, and it was the same gentleman who, I'll just say, was probably getting close to retirement. Everybody was panicked about it, because this same guy has been doing our pavement ratings every year, right? What happens when he retires? How do we bring some subjectivity to this right and take it out of the like? We trust his judgment, but how do we move on from this? And that's not to speak ill of him, but to say like, let's apply some science here, and so then we involved that same gentleman, but we flew all of their pavements, we took pictures and then he used the photos to identify where it looked like the pavement was. You know, alligator, cracking and blocking and the different types. You know where are the distresses the most serious. And then he went out and he drove those with a drone. You can't rate ride, you know, but we're getting there. Yeah, you know we're getting there. We can put sensors on dump trucks or snow braids and we can measure that now. So we've got all sorts of options. The drone I'll just say the drone isn't always the best tool, but oftentimes it is a very accessible tool. Hey, everyone, I just want to take a quick moment to thank our sponsor, bolton Mink, who is producing and editing our podcast.Bolton & Menk:
At Bolton Mink, we believe all people should live in a safe, sustainable and beautiful community. We promise every client two things We'll work hard for you and we'll do a good job. We take a personal interest in the work being done around us and, at the end of the day, we're real people offering real solutions.Hosts and Guest:
It's so. Do we have kind of that machine vision, ai level of just plug and chug, like I'm thinking? When I was at Maple Grove as a young engineer, I would be driving around and say, hey, that tree needs to be trimmed because it's halfway blocking the speed limit sign. Do we have automated tools to be able to say, pluck out all of the signs, in that we flew on these roads and say which ones are blocked by trees and then turn those into work orders quickly and over to public works to get the tree trimming? Those systems all exist. I don't know that they talk to each other yet. Okay, I want to answer that with two things. One of them has been really interesting to be involved in this industry and sort of spectate what has happened as we came through the trough of disillusionment. You see everybody and their brother essentially go out and buy a drone because you can go to Best Buy and buy a drone, and now we're sort of at a point where some of those unfortunate individuals are being weeded out a little bit. You have a lot of people that are competent flying the drones but they don't know what to do with the data or they don't have the ability to do things with the data. They can just hand somebody some photos. That's great for real estate. They're not asking for any AI to be applied. In our industry, we are more focused on what is happening with those things after they are captured, and so it didn't take me very long to figure out that we could have a whole crew of competent pilots, but if we didn't know what to do with the data, there was no point. We needed to be able to deliver the data and own the data and be able to certify. This is what we're gonna use this for and this is what we're not gonna use this for. You know what I mean. So that's sort of one consideration. You asked about AI and machine learning. Right after we did that particular community, we had all these pictures of pavement in various states of distress, right, and so we approached an AI company or a fledgling AI company I'll say this was 17, you know. So there was no chat, gpt, yeah right. But we said we have this whole data set. Could you train something with this? And that was a really interesting interaction for me, because what they came back with ended up essentially being like a Photoshop filter, where they recognize what I asked and then looked at our photos and said well, everywhere where the contrast is different, there's this dark line in the pavement. That's what he told us was a distress. So they met just the very front of what I asked them to do and they delivered this product and I was like, okay, well, this isn't really an AI. I'm going to smack the next person that call this an AI, because it's not. It's a filter that turns up the contrast and then says this is how many changes in contrast there are in the pavement on this photo. And they were like done, and I was like, okay, that's not intelligent at all, right, interesting though that I asked it this way and they were able to sort of get there, but not in the way that I wanted, and so one inefficiency we found in that was that if you do crack ceiling, that shows up as a huge change in your contrast. But we would consider that that's protected from the elements. That is a maintenance activity that we want to do on that. That actually improves the PI, not the Right, right, but from their oversimplified perspective, they saw that as a change in contrast. They counted that as cracks, and so now a street that had recently been crack sealed, they were telling us was in terrible condition. Okay, if public works went out and they seal coded a road without filling in the cracks, that was also going to skew what we could see, because now it looks smooth and black. Right, you know you can't see the cracks as well, but they exist and they're going to cause freestyle in a place for moisture to penetrate. So that's a problem. Yeah, so they just applied a simple algorithm, but it wasn't true artificial intelligence. And the way AI was explained to me is think of AI as a toddler and you walk your toddler out to the street and you show them 10 buses and those buses have variability to them, but eventually there is a point the first bus they don't know what a bus is, but at some point, 10, 15, 20, 30 buses later, they understand what a bus is, but they can't articulate the variability and that's kind of the same as AI of. You show the AI system enough of what something looks like and you give it that name. Eventually it will pick out the nuances and the details, but we don't actually know the brain of the AI and how it's doing it, and it may actually do it different than a human brain. But so what that company didn't do was take all of what bad looks like and train it on bad, and then also go what good looks like and train it on good, to make sure it was excluding. They just said, hey, this kind of to us looks like the lighting is different, so we're just going to say it's that and that's just a simple static algorithm. That's AI. That's why I jokingly called it a Photoshop filter Turn up the contrast, count the cracks done Right, and even internally, since 2017, one of the things we will do is we'll fly. You were asking me before about drones in static situations. You can power a drone from a tether. You'll see this a lot in emergency operations, where they can hover a drone over a disaster site or whatever it is, over the emergency response trailer and they can get that aerial perspective from 400 feet, but they can power it indefinitely. That's one of the drones in traffic data collection. We get out and do 48 hour Haiti counts, or we go out and we do at least two hour turning movement counts. Well, these drones don't lie. The batteries don't last for hours, let alone days. Getting that at the non military grade Right, right, right. But who's going to go out there and tether and then do you have to stand there the whole time it's tethered and like just the economics blow up where no, we're going to use other ways of getting this data. Yeah, so where is the battery technology? Where are things currently at with run times? So I, up until now, sort of our industry standard, like the standard tool would have been a DJI Phantom 4, which you could buy for $1500 at Best Buy they're not making those anymore but that had a 20 megapixel mechanical shutter camera and that would run for 37 to 42 minutes per battery. So if you were asked to do a continuous capture, we would have to either send two pilots with two drones and a whole rack of batteries and then you needed to have overlap between the two video captures or you needed to tether. Where we're at now, sort of our standard drone has shifted to a DJI Mavic 3, which is a little bit spicier meatball. That's like a $6500 drone but a more enterprise grade platform. That is the replacement for the Phantom 4 in a mapping capacity and then that'll run almost 50 minutes for battery. So one area where I used we hired somebody in Goddus Video was work. I had a little practice where I was working on school operations kind of the pickup, drop offs and what are the walking, and it is absolutely amazing to have that drone footage for a half hour to an hour during either the pickup or the drop off, just hovering above the campus, because you really can see at most elementary high schools you can see the whole campus on that bird's eye view and it is just so different See from that versus sitting on the street corner trying to see across the parking lot to what it. So that's just one personal application where it really changed how we operate. So one thing that is very accessible right now that we actually do sometimes internally is we'll go out and we'll capture that kind of footage right, we'll try to hover the drone in a static position. It's kind of fun to speed those videos up and watch them. Well, then the patterns become apparent right To your naked eye. When you're just watching somebody, a walking pace or traffic is stopped. You're like yeah, yeah, okay. But then when you speed the video up, you start to see the jerkyness of where it's smooth and where it's jerky, where real time you don't necessarily pick up on. Look at how long that TU got. We didn't see that many cars back up until whatever 937. And then in a product like TensorFlow, which is one of Google's publicly available AI engines, we can process that video with stuff that's available for free right now. You could go in there and you could use a pre-trained TensorFlow model to look for Peds and look for cars, and then you could draw boxes and so we could sort of recreate exactly what you're talking about. We could run a captured video through a TensorFlow model with some locations drawn like gates, and we could do traffic counts so I could tell you turning movements from a video that we already captured using some software that's out there for free. And the pre-trained model thing is an interesting concept, right, they've taken pictures of cars and just run them through a model, and so then you can go out and you can download a TensorFlow AI model that is pre-trained vehicle identification, yeah, and that kind of. I mean we're off on a tangent, but that's where this AI stuff is headed is getting those right. It's one thing to use the off the shelf, and you're going to get heads and vehicles separated for sure, and you probably are going to get scooters now, but I'm waiting for, as far as I know, there's no publicly available model on FHWA's 13 classifications break down, and so who's going to train up that model? And then is it going to be proprietary? Are they going to keep it themselves or are they going to share it? And then, once Google or Amazon hears about it, are they just going to spin up their own folks to make it? So, just, it's fascinating to think about kind of the brains of these systems and that's kind of the chat GPT where it's forking off of people are starting to realize we need more fine tuned for our application, kind of the under the sheets. What's the brain of it? The model behind closed doors here We've been talking a lot about chat GPT. Right Like this is really cool. How can we use this? Also, how bad does it suck, right Like we. We we found a number of bugs already. Like Manning's equation. Right, there's a, there's an Imperial and an SI version of Manning's to calculate flow. If you, if you dig way into chat GPT prompts, like telling it where it is, we'll force it to consider certain things. So if I say, on a project in Minnesota you actually lock chat GPT into Imperial units because it goes oh, you're in the United States and so then it will, you it. But the problem is it doesn't recognize the 1.0 over N for SI versus 1.49 over N, the two different. There's a you know two different versions of Manning's, and so it will use everything you give it, but it will calculate it wrong because it inputs your information in, you know feet and inches, but then it uses the non-imperial version of Manning's, and so then it gives you the wrong answer. And so you're like, well, this isn't great, yeah, so we could hire an EIT, they could lie to us about what they know. And then they go to chat GPT and say you know, please size the sewer for this many homes, you know in the state of Minnesota, using Met Council standards. And you know, blah, blah, blah. And then you know, kick back. You know that's an eight inch sewer and if we're, you know, we could be in a bad place if that EIT says one positive it only needs to be an eight inch sewer when in reality it needed to be a 12 inch sewer. So for us to be able to train our own model whether you keep it proprietary or not is very important. And then running it on our own data is going to be the biggest trigger for chat GPT. Right, chat GPT works off of a snapshot of the internet and time, which is a problem because we need to be able to point it at our own data, and that exists. There are products out there that do that, but not chat GPT. So on the point cloud side of things, sort of shifted stuff back to UAV, a little bit like, well, we capture these massive point clouds. Yes, it's a data problem, but also there is AI that exists already or you can train it. So some of our point cloud products and even on the survey world, you can shift it into a profile view and you can trace out B618 curb and you can tell it how much of a buffer you want to allow it to have and then you can sort of give it a general direction. Right, this is the direction that the curb is running in. And you click go and what it will do is it will remove those points from the point cloud. It hides them. It doesn't delete them, but it hides them and it replaces that with a smart entity that is lip of curb gutter, top back of curb. Right, it just marches through your point cloud until it gets to a driveway cut. The driveway cut exceeds your buffer and it stops. It says I don't know what to do, right, and then you just click past the driveway cut, you go on and it will march through it again and replace those points with more smart entities. So there's still some human judgment involved, but person hours being taken out of putting together the survey, that's astronomical, right, right? So the slide are in most cases we are comfortable designing from that. You know you're going to want to check your Z's, because the elevation is where you're really going to get hurt if you're trying to work from that data. But this just underscores the point that, like all these people have run out and bought drones right and now they're like I can't understand why I'm not really busy. You know why is my drone company not taken off? And part of where we have been lucky is that, you know, we had this whole built in customer base right where we had access to tons and tons of data and tons and tons of pilot projects, where you know, I can go find something that's been terrestrially surveyed, topographic surveyed already, and then we can fly it with a drone and we can compare that data and we have a very good check. This is a great site. Thanks for watching from Bad� experiments. So, yeah, you can train up your staff to understand its limitations. And that's the important part of engineering is not just taking all right the output out of the computer, but saying that makes sense or it doesn't make sense. Yeah, you have to question it. You have to decide, besides water, with lighter in particular. But where have you? What is a bad site? What we're, we're? Don't you want to use your UABs for survey? So it was learned. So photogrammetry is a is a how. There's a couple of terms is a DTM, a DSM and a DEM? And these are these are technical terms. I think they come from the ASPRS, I'm not gonna harbor it there. They're like the overarching we set the standards for mapping. Okay, right, a DSM is a digital surface model, a DEM is a digital elevation model and a DTM is a digital terrain model. Those are three different products. But when we fly for photogrammetry, it's simply what we can see. So if you fly a site that is a open field but there is, you know, waist-high grasses and that grass might be moving in the wind or fold it over a little bit, if we extract an elevation model from that, that elevation model is the top of the grass as it was visible. Okay, that's a digital surface model. You're not gonna get any better than that from photogrammetry and there's no way to process it to a digital elevation model. We could go out there and we could burn the field, right? No, we've actually done this before. We did a project in Red Wing on Barn Bluff where they did a controlled burn and they were doing the controlled burn already Not for us, yeah, but they were like, hey, we're doing this controlled burn. Also. We think that at one point the indigenous peoples in this area may have farmed barn bluff that's who. We were like, okay, cool, and we came out there after the smoke cleared and we flew it, because it was at that point literally bare earth, right, and we were able to look at and this was our cultural resources staff was involved but we were able to look at the evidence of some terracing and some farming on barn bluff. Well, so we were able to see, after they burned all the weeds off you know all the tall grasses that there was some evidence of that having been farmed. And you wouldn't, you couldn't stand in there, you couldn't tell, you couldn't see the. You know the history of those furrows. But when we got above it from the aerial perspective and we flew the whole thing there, it is right, it looked like a, the beginnings of an archaeological site. I don't know if they were found anything out there, but that was a. That was a neat example, you know. So sure we can get to a digital elevation model. But that's where the lidar comes in. Is that lidar laser will penetrate vegetation? So we get we get called a hazy point cloud, right when you look at the raw data you fly over trees, you get hits on all the leaves coming down, you get hits off the branches, the trunk, and then you get hits off of maybe there's some dry leaves, there's some underbrush right growing, and then you're very the lower most, the bottom points. It's not as many millions of points as the laser shoots. But when we process it all the way down, we clean all that other stuff up and we put it on different layers, we turn it off, and so then those bottom most points, ideally, are more dense than it would have been for us to send a surveyor out, you know, and chop his way through and take a hundred foot grid. Yeah, you know, we're getting points through heavy digit vegetation that are, you know, one feet, one foot apart. So we're getting like a one by one grid rather than a hundred by hundred grid, and that's all the way through heavy vegetation. So that's where the lidar, you know, comes in into play. Yeah, is that we can penetrate? We can't penetrate all vegetation, obviously, but photogrammetry was gonna give you all of that, what was visible. We can process that lidar data into a digital elevation model much more easily than the photogrammetry data. That lidar, they said it was half a million a few years ago. Yep, I assume technology, especially with lidar going into cars more and more, is that price point coming down Exponentially not on that laser, a little a little bit. What we have seen are, you know, the, the lidar sensors being miniaturized and Reducing the quality. So, like we have had discussions internally of, do we get everybody iPhone pros right that have the lidar built in? I don't know that Apple totally knew what they were doing there. They're like, hey, how can we improve low light performance? And so they use the lidar to set the focus length on the camera. But Other developers were like, now we have access to this lidar sensor, what can we do with it? One example I like to reference is there's a you can you create an app company called polycam and they have this movement out there. I forget what it is. It's this is hashtag on Twitter, but I think it's scan the world. But they've made their product, their premium product, available in you know, essentially war-torn countries right now, and they're encouraging people to go out and download their app and scan monuments, right, scan parks, scan statues, scan everything. And so you know they're, they're encouraging people to sort of create a digital record in case things get destroyed, and so like, on the on the Other end of the spectrum, you know, which makes me feel a little entitled and safe and appreciative to be, you know, an American. I go on vacation and I use polycam to scan the lion statues outside of Edinburgh Castle. I think it's super cool, you know. And then I'm able to take that file and, like, send it to a 3d printer. So the the iPhone lidar, much more shorter range, much less accurate Mm-hmm, but you know, the best camera, the best sensor is always the one that's in your hand. Yeah, and I mean going back to the heavy vegetation, sending the surveyor out there, and you're getting a hundred foot grid. Yeah, and yes, the high end is getting a one foot grid. Are these Cheaper, miniaturized versions? Are they getting a 40 foot grid or a 50? I mean just the comparison of what's good enough. Yeah, so start to do the economics on it. So like, if you do the math, you know we're into, we're into our LiDAR sensor for something on the order of like $550,000 right right now. I could go buy a different LiDAR sensor hanging under a smaller drone. That package is about 70. They will out the gate, claim the same accuracy, but in practice it's. It's less. Hmm, you know, even when we apply all of the ground control and the survey, you know, verification shot tools, it still isn't gonna be consistently as accurate. It is better data probably than you have. Yeah, I mean it still could be much better than if you're sending out a survey crew to Economically. Yeah, get that, it's a point. Yeah, so like undeveloped sites, right, large tracks of land. If we don't have any data on that or it's changed significantly somehow, you know we'll go fly LiDAR or one of those cheaper LiDAR is actually would be a good application for this, and then you can do initial earthworks. You know you could lay out a development in an undeveloped tract of land very reliably from that lower powered LiDAR. You can use our higher powered one too, but I'm trying to. Can you put in perspective? So let's sit. Maple grove was 36 square miles, mm-hmm. So I'll say six Yep, well, well, minus a little chunk of a. So, yeah, I mean, what if a public works director wanted to have his own city flown. Point clogged the whole deal. How? What is the aspect ratio and how does that translate? Is that An operator flying it for a year, a month, ten years? How much to get their own topo done? So if it was just the flying logistics for 36 square miles With a quad rotor drone right, which is something with four propellers that flies, you're gonna be there for a couple of weeks. Okay, that that might not be the best application for that. If we were talking about Next year's recon or the next three, five years of recon's, those are a little bit more limited sites. The other thing we run into is that there are some additional rules and one of them is it's sort of a liability thing, but the FAA for a long time has said no flights over cars or people and the actual rule says sorry, what yep, no flights over, no flights over cars or people. And the phrase after it says not involved in the flight. Okay, which has been interpreted a number of different ways. But where we've landed pretty firmly on is if you're not part of the flight crew, right, you're not involved in the flight, and so for a while it was well, if we just put up a sign at one end of town that says drone flight and progress of people drive by that sign were covered, right, and they're involved in the that's. That's evolved beyond that. But what they have also come out with in the last year is they've they've classified the weights of drones. Somebody finally said, hey, why don't we come up with an ASTM standard? And a bunch of other nerds at the ASTM went away and played with some Ballistics gel and came back and said, well, we think a drone at this velocity or less and with a you know Propeller with a rotational speed of this or less, you know it doesn't cause a serious laceration and it won't kill somebody. So this is, this is these are your rules, wow, cool. So they came up with an ASTM standard for how harmful is a drone. And then FAA turned that into three classes of drones, and so the classes are under 250 grams and you I, you don't have to be able to stop the propellers. And then there's a second class that is up to so many pounds and I'm not remembering off the top of my head, but you can stop the propellers. So probably a parachute to lower the velocity as it drops and then above, that is just like that's heavy lift. You know you have to have special permission, but your airframe has been certified by someone, okay, to not fall out of the sky. So those are the three classes. Now they're saying, though, these three classes, now you can go fly over cars and people. So, like, internally, right away, there's a whole bunch of 249 gram drones on the market. We went out and bought a bunch of those. Right, okay, this is cool. These cameras aren't great, they suck a little bit, but if this is the only thing, we can fly over cars or people, okay. So now that enables us to be able to do a mapping flight or a video flight, you know, in a downtown urban situation when we couldn't before to meet the rules. And, for sure, for the simulations of just overlaying different design options For a public hearing, I assume that video quality is plenty good. Yeah, I mean, we work with it. We work with it. Those, those cameras are slowly coming up. As the you know, the under 250 gram drones get better and better. No, that's a thing. And then we have some of the like the parachute systems. Okay, for the slightly bigger drones. That was kind of a fun day. We, we were like, we absolutely like the egg drop thing. Yes, all over again. Yes, only way more expensive Failures. And we got paid to be there. Oh, so just imagine a whole group of grown men right standing over a soccer field. You know I'm I'm yelling pilot Kyle. He's not with us today. You might hear from him at some point. Yeah, he's a, he's got his masters remote sensing and yelling like I'll stop the motors. You know he's like I don't know. Oh, we're taught like we had, yeah, I had to pull up the. I had to pull up the manual online to tell him what the stick combination was to stop the motors mid-flight. It's like the cheat code, yeah, yeah, well, the cheat code you never want, right, right, it's the self-destruct code, yeah, and so we kill the motors on. The drone falls out of the sky, the parachute activates and then I don't know what we were thinking, because we had a group of guys with a tarp, like we were thinking we were gonna catch it, so, so that the drone would remain intact. Video of this, I might, yeah. I might air shoot the gym class of these guys? I might, I might, but so there was. There was just enough wind, so there was not a chance. The guys were gonna catch it with the tarp. Yeah but they tried. They ran across a soccer field, you know, chasing this drone as it falls out of the sky. And and the drone was not undamaged. I think three out of the four props shattered when it stopped the propellers. I think there was a crack in the landing gear. It not unrecoverable things, but the whole point was to protect people right right, the drone was below a certain velocity and the propellers were stopped, so there was no lacerations, nobody got seriously injured, so the parachute worked. Parachute worked great. You didn't have like a designated person to drop the the Drone on to see if the propellers cause damage or anything. No, first we so so we have the test dummy out there. Yeah, yeah, there's a couple of these. They call them UAS test sites. That, yeah, I don't. There's a long history here. Yeah, you can Google if you want. But the FAA has sponsored these places and there there's one in North Dakota, right? So they use some of the rural open-air space in North Dakota to test UAVs, test drones with different control systems. So, like up there, they have been pioneering controlling drones over the cell network. And there's another radio system that anybody who is in the aviation world called ADSB, which is where how aircraft transmit their position. But there is a radio network of ADSB antennas around around the country and so they have dabbled with using the ADSB network to control drones so you can get way belong, way beyond visual line of sight, you know, and radio connection using that, and and they've been Trying to figure out what that will look like for the next standard. Yeah, so real quickly as we get towards the end of the podcast here and that that hour went flew by, no pun intended. Like we talk about flying these drones over, and I know you can like the drones basically fly themselves. You can pretty much program them, not fly it out or stuff like that. Right, like, how far can the pilot be away from the drone? So the FAA compliant answer to that unless we are in a BVLOS which is beyond visual line of sight, means that the pilot needs to be able to see the drone unassisted, so not through binoculars. There are some caveats to that, where you could have a visual observer, a VO, that is a designated person that in most cases is also a licensed drone pilot, and you can hand off visual confirmation of the drone between the I'm gonna throw in much acronyms at you guys but the RPIC, the remote pilot in charge, is the person on the sticks. He doesn't necessarily, or he or she doesn't necessarily, need to see the drone at all times, as long as part of their team does, but you have to be within visual line of sight. Okay, so this idea of setting up a drone while you're in your staff meeting to go check something three miles away. That's not happening Under the rules as they have been, no, but we're on the cusp of that changing. You see Amazon and Domino's and whoever else dabbling with drone delivery. One of our other staff members lives in an area where they are testing drone delivery, and so he ordered wings from this place just simply to watch them be delivered and videoed the whole thing. And he said this dude drove to the end of his block flying a drone above a van and then got out and landed the drone in his front yard and he unloaded the wings. And then afterwards he went over and talked to the guy. He was like well, how is this more convenient? You drove here in a van. And he's like well, we don't have permission to do beyond line of sight yet, but you see out there, companies being granted exemptions already, and that's where Amazon. That's why Amazon has been testing delivery, not in the United States, to perfect the equipment and the science, and then they'll come here, but we see companies being licensed already to operate beyond visualized sight and power line inspection. That's exciting. What else are you excited about? Yeah, what's the future of this? Where are we going next? We talked a little bit about AI, but the sensing and extracting of data from the imagery. That's compelling to me. I see even the smallest of drones now have pretty decent sensor packages on them for avoidance. There's a thing that's going to drop for us in September called Remote ID, which is sort of the ADSB, that location information idea, but for drones, and that will place the drones into the kind of the radar world where in a small single engine aircraft you could see where drones are. They have to broadcast their position. So that's the next step in the safety net. But you hear a lot of integrating drones with general aviation. I wouldn't be surprised if someday they lift the 400 foot limitation when we figure out how to interact. But how high do you do want it to go? I mean a higher, if you could. We deal a lot with these restricted flight areas and so like in a five mile circle around MSP they take and they chop that up into little squares and each of those squares has a height restriction and so if it's below 200 feet or 100 feet we're just kind of like, yeah, that doesn't do us any good. We need to be above. Our standard mapping height is between 275 and 350 to even get a spread on the photos for it to be worthwhile. So it'd be awesome if we could be higher. But beyond visual light of sight to, the drone can fly a lot farther than we are legally able to. So that would be a big step when you can do that, probably as big as the flights over cars that people have been for us Right. Well, yeah, like Mark said, the time flew by this was great. I learned a lot. So thanks, brian, it's fun to geek out on this stuff. Thank you guys so much for having me. Oh, this has been great, and I think there probably will be some additional opportunities to talk about UAVs and some drones as some of these advancements occur too. So love it. I really particularly want to talk about someday in the future and we touched on a little bit but that marrying of that data from the UAV into our management system and how do we make that better realistically, I mean, and the logistics of file size and all the data and the set and the other. But what can you actually pull from there? That's reasonable and unusable. I think that'll be a good conversation too. Put a pin in multispectral and back yep and the other aspects of remote sensing All right, what a pin in that. That's a future geek out. All right. Well, thank you, brian. Thanks for joining us, thank you. Thank you, mike, thanks, mark, and one last thing before we go. Although we don't charge for the professional development hour you just received by listening to the podcast, the Public Works Nerds is not free If you've listened to more than one episode. The cost is that you tell one colleague about the Public Works Nerds to help us grow our audience. Thank you.