Learning physical AI from scratch. Day 1: how robots see.

Day 1. I teach robotics for a living, and here is my confession: until recently I could not have explained, cleanly, how our robot dog actually sees. I knew the parts. I could not tell the story.

So this series is me fixing that, in public. Each week I take one thing a robot does, pull it apart until I could explain it to a friend over coffee, and write it down. This week: seeing.

Seeing is three jobs, not one

Walk into a room and you do three things without noticing. You recognise stuff. You judge how far away it all is. And you keep track of where you are as you move around.

A robot cannot do all that with one sensor. It needs a different tool for each job, and the real work is getting them to agree with each other.

Cameras do the recognising

Cameras are the obvious part. They are cheap, sharp and in colour, and with an AI model behind them they can put names to what they see: person, chair, doorway.

Here is the catch. A single camera mostly guesses at distance. It can be fooled by a photo of a hallway, and it gives up in the dark. For a machine that weighs as much as an actual dog, guessing is not good enough.

Two cameras do the depth

The fix is borrowed straight from your own head. Two lenses, spaced slightly apart, see two slightly different views, and the difference between them reveals how far away things are. Your eyes have been running this trick your whole life.

It works best up close, which happens to be exactly where a walking robot needs it most. The next two metres decide where the feet go.

LiDAR is a tape measure made of light

This one is my favourite. LiDAR fires harmless laser pulses and times how long each one takes to bounce back. Light is absurdly fast, so the timing is absurdly fine, but the result is simple: an exact distance.

Now do that thousands of times a second, in every direction. You get a cloud of measured dots wrapping the whole room, like a 3D sketch where every dot is a real measurement. On the robot we teach with, a Unitree Go2, the scanner covers every direction at once and works from about five centimetres in front of its nose. That is straight from Unitree's own spec sheet, and it still surprises people.

The part that blew my mind

So the robot has three partial pictures. Software lines them up into one. Most of that work, I have learned, is deeply unglamorous: making sure every sensor agrees on what time it is and where it is bolted. Get that wrong by a few milliseconds and the robot sees a smeared world.

Then comes the trick I keep thinking about. The robot builds a map of the room while simultaneously working out where it is inside that map. Both at once, both changing constantly. Roboticists call it SLAM, and it is the same trick warehouse robots use to cross buildings that change by the minute.

It is also one of the first things students build in our course: walk the robot through a room and watch the map draw itself on screen. I have seen it dozens of times now. It still feels like a magic trick.

What still fools it

Glass. Cameras look straight through it, lasers bounce off it strangely, and the robot ends up unsure whether there is a wall there at all. Mirrors are worse. A mirror shows the robot an entire room that does not exist.

And no sensor delivers meaning. The robot can measure a door to the centimetre without knowing that a door matters more than the wall beside it. Deciding what matters is still our job, and honestly, I find that reassuring.

That is day 1. Next week: how the robot actually moves its body on purpose.

Quick answers

How does a robot dog see?
With three tools at once: cameras to recognise things, paired lenses to judge depth, and a spinning laser scanner that measures distance in every direction. Software fuses them into one map.

What is SLAM in simple terms?
Simultaneous localisation and mapping: the robot draws a map of a place while working out where it is inside that map, both at the same time.