Scaling Sidewalk Autonomy with World Models

TL;DR

• What makes sidewalks harder than roads is the constraint budget: a single RGB camera, limited compute and battery, yet the robot must handle all weather and lighting and interact in a socially compliant way with pedestrians and animals — including navigating around a dog lying on the sidewalk without stepping on its tail.
• Evaluation is the bottleneck, not model supply. Plenty of sidewalk foundation models exist (from his lab and others), but there was no fair way to compare them before deployment. Their first attempt, SidewalkBench, is built on NVIDIA Omniverse and Isaac Sim — but graphics-engine simulation still lacks visual realism, leaving a scene-to-real gap.
• The fix is to treat real video as source code: Vid2Sim (CVPR) reconstructs sidewalk footage into Gaussian splats, drops them into a physics engine, and yields data-driven simulation. RL agents trained there transfer zero-shot to the real world because there is essentially no visual gap between training and deployment.
• One video shouldn't yield only one environment: UrbanVerse extracts a scene graph from a single clip and swaps in different object instances to produce many digital cousins (as opposed to a one-to-one digital twin), alongside a released library of 100,000 correctly-scaled 3D assets.
• The payoff: FlowPilot does real-world sidewalk navigation from a single RGB camera by emitting waypoints, and the policy transfers across embodiments without fine-tuning — demonstrated on legged robots, with wheeled-legged robots and electric wheelchairs in progress.

Key Points

Sidewalk autonomy is its own problem (~01:05–01:07)

He opened with a joke about the lineup — UC Berkeley, UC San Diego, and his own UCLA: "How about having a UC AI summit?"

Most people know road autonomy, which the previous speaker had just covered: full-size vehicles running around cities. But urban environments have a second kind of autonomy he calls sidewalk autonomy. Coco Robotics runs last-mile food delivery with hundreds of robots doing this today.

Compared to road autonomy, sidewalks bring harder settings: careful navigation and collision avoidance in tight space; all weather and lighting conditions; and — because sidewalks were designed for people — the requirement to interact in a socially compliant way with dynamic agents including pedestrians and animals. His favorite clip shows a dog lying across the sidewalk, with the robot threading past without stepping on the dog's tail.

And all of it under compute and battery constraints: "we're only allowed to use a single RGB camera to do the navigation."

The bottleneck: many models, no way to evaluate before deployment (~01:07–01:09)

Recent years have produced many sidewalk navigation models, largely via imitation learning from human demonstrations — some from his lab, some from others. Trained on video demonstrations, they generalize well across conditions.

That creates the problem: given so many different sidewalk foundation models, how do you benchmark them before real-world deployment?

Their first answer was SidewalkBench, evaluating navigation policies in simulation, with environments built in NVIDIA Omniverse and Isaac Sim. It compares how models interact with people and avoid collisions. But look closely and the graphics-engine simulation still lacks visual realism — a scene-to-real gap that makes the comparison unfair.

Fix 1: real video as source code — Vid2Sim (~01:09–01:10)

Their solution is to create world simulation from real-world video: "we want to use real world videos as a source code to build up this evaluation benchmark."

Vid2Sim, published at CVPR, takes a video of walking on a sidewalk, builds a Gaussian splatting reconstruction from it, and turns those splats into a physics engine environment — a data-driven simulation in which sidewalk robots can be trained and evaluated.

The demo shows an agent trained there with reinforcement learning transferring zero-shot to the real world, for a direct reason: there is no visual gap between the training environment and the deployment. With those environments in hand, different sidewalk navigation policies can be evaluated across cases.

Fix 2: many environments from one video — UrbanVerse (~01:10–01:12)

Video reconstruction has a limitation: one video reconstructs exactly one environment. Ideally you want several variations from the same footage.

Hence UrbanVerse, which creates multiple digital cousins from a single video. He distinguishes the terms: a digital twin is one video mapping to one simulation environment; digital cousins are variations from the same source. The method uses computer vision to extract a scene graph from the input video, then plugs in different object instances to create environment variants, each usable for training and evaluation.

UrbanVerse is publicly released, together with a large-scale 3D asset library of 100,000 assets with correct scales.

They also built a simulation post-training pipeline: train the model on videos via imitation learning first, then place it in the simulated environments for post-training that improves interactivity and counterfactual reasoning.

The payoff: FlowPilot and cross-embodiment transfer (~01:12–01:13)

FlowPilot is the recent result: real-world sidewalk navigation from a single RGB camera. The model outputs waypoints and the controller executes the next one. In the demo the robot handles challenging sidewalk navigation — avoiding obstacles and interacting appropriately with pedestrians.

The direction he finds most exciting is cross-embodiment transfer: with a sidewalk navigation foundation model in hand, they can transfer the model across robot embodiments without fine-tuning. The demo shows it working well on legged robots, with ongoing work on other sidewalk platforms including wheeled-legged robots and electric wheelchairs.

His closing line: "In the future, not only can we autonomously deliver the food, we can also autonomously deliver the people on the sidewalk." Benchmark and code are released on their lab page.

Quotes

"We want to use real world videos as a source code to build up this evaluation benchmark." (~01:09)

The central metaphor of the whole methodology.

"There is no visual gap between the training environment and the deployment." (~01:10)

Why zero-shot transfer works, in one sentence.

"In the future, not only we can autonomously deliver the food, we can also autonomously deliver the people on the sidewalk." (~01:13)

Where cross-embodiment transfer is headed: from parcels to passengers.