Camera versus LiDaR

LiDaR versus Cameras

The recent "Can You Fool A Self Driving Car?" video by YouTuber Mark Rober has 17.5 million views [1], is generating a lot of buzz, and is already the subject of a number of counter- and critique videos [2].

I've been asked by quite a few people what I think of it or to even post a video - so here it is.

I'm speaking from the perspective of someone who:

📡 first added an (single plane) "LiDaR" sensor to their camera-based autonomy stack back in 2004, and was amazed by the benefits it gave, especially over sonar

🚙 as someone who has worked on a number of research and commercial projects in both on- and off-road autonomous vehicles, where we used both cameras and LiDaR extensively, and

🌿 as someone who is interested in nature-inspired AI examples (the whole "humans primarily drive using vision" thing).

Key points: 

👁️ Humans do indeed provide a proof-of-concept that reliable driving can be done using primarily vision - but one aspiration for AVs is to exceed human capabilities, and in the short term we should indeed "cheat" as outrageously as possible to improve safety and reliability. Cheating includes using every sensing modality possible, and training using prodigious quantities of data and simulation, where it helps. It's not exactly "the way people learn to drive" and that's fine.

📡 LiDaR and other range sensors are active sensors - they emit energy into the environment to measure range to objects. Standard cameras are passive, and infer ranges using AI. Camera-based range estimation has improved radically, but it's still a non-explicit process. There are pointy AV challenges - like high speed merging onto, or crossing of, highways, where you likely need long range vehicle detection, recognition, range estimation and velocity estimation (using for example Doppler LiDaR).

🌧️ It's not just cameras and LiDaR - both these sensors can degrade or fail in foggy, snowing, and raining conditions - enter other range sensors like Radar that work better in these adverse conditions.

💸 A decade ago, people seemed very fixated on the (then) very high cost price of LiDaR, and cameras as a cheaper option. Although I'm sure capital cost will remain a key consideration as companies like Waymo continue to scale, I don't see (or haven't heard) any cost-driven proposals to drop LiDaR in the near future. LiDaR costs have also dropped and their capabilities have improved as well.

🧠 Finally, it's important to remember it's not just the sensing hardware - human vision does not have the capability to detect every hazard at night, in the rain, whilst driving - so prior context, knowledge, and assumptions are key here - and hard to recreate in an autonomous system. Any sensor debate shouldn't occur in a vacuum but rather in the context of how AI driving systems ingest and make decisions using that data.

🖥️ YouTube video: https://youtu.be/AGFxczr2-Yk

[1] Original video: https://youtu.be/IQJL3htsDyQ

[2] Example critique (for balance): https://teslanorth.com/2025/03/16/busted-mark-robers-misleading-tesla-test-sparks-outrage/ 

#Tesla #AutonomousVehicles #LiDaR #camera #ArtificialIntelligence #transport #robotics #driving #mobility #perception #safety