Autonomous exploring drone

A quadrotor that takes off into an unknown city, explores it alone using nothing but its onboard camera, builds a map as it flies, avoids everything in its way, and takes plain-English orders — including "go to the blue building", where it finds the building itself.

One flight, four synchronised views. Top left is a chase camera; top right is the map being built in real time. The two panels underneath are the entire input: a 160×120 RGB frame and its depth. Nothing else — no GPS, no prior map, no floor plan. Full 90-second flight →

Everything below runs on one desktop. No cloud, no API key, no dataset.

• A real flight controller. Position → velocity → attitude → body rates → four individual motor thrusts, at 240 Hz. Not a "move the box" cheat.
• It has never been told where anything is. Obstacles come from the depth camera alone, ray-traced into an occupancy grid it builds as it goes.
• Zero collisions across a full 600-second autonomous exploration.
• Plain English, running locally. A 7B model on the same GPU turns "take us up to five metres" into a command in ~390 ms. It works with no model at all.
• It can fly to something it can only see, with the vision model choosing where and the verified controller still doing all the flying.

What the drone actually gets. The mapper reads the small RGB and depth frames ten times a second; the third frame is rendered only when a language model needs to look at something.

Below: the map it builds from that, starting from nothing.

Grey is confirmed free, red is obstacle, dark is still unknown. Rectangles are buildings; the small round blobs are trees, lamp posts and parked cars it found on its own. It stops when no reachable frontier is left.

Quick start

pip install -r requirements.txt
python main.py --auto

Three windows open: the PyBullet 3D view, the live occupancy map, and the onboard camera. Type commands into the terminal while it flies.

takeoff              land               stop
explore              go to 12 -7        orbit 6
climb to 5 metres    come home          status
what do you see      quit
go to the blue building                 fly over to that tree

Everything works with no API key at all, using a keyword parser. Point it at a language model and you get real natural language ("head over to that cluster of buildings in the northeast and circle it") plus what do you see, which sends the actual camera frame to a vision model. See Commanding it in English.

Other ways to run it:

python main.py                                   # GUI, wait for your commands
python main.py --headless -d 300 --script "takeoff;explore"
python main.py --auto --snapshot run.png         # save the map on exit

How it works

Four layers, each one running at its own rate.

Flight controller — 240 Hz. A real cascaded controller, not a "move the box" cheat. Position → velocity → attitude → body rates → four individual motor thrusts, mixed through an X-configuration mixer and applied to a PyBullet rigid body. The attitude loop is the standard SO(3) geometric controller. Hover holds to about 4 mm; waypoints track to 3 mm.

Perception — 10 Hz. A body-mounted RGB-D camera, pitched down 8°, at 160×120. That resolution is chosen for the ray tracer, not for looking at: frames sent to a vision model are rendered separately at VISION_WIDTH × VISION_HEIGHT (480×360) from the same pose, so the flight loop never pays for detail only a model needs. The mapping frame costs 6.7 ms and runs constantly; the vision frame costs 56 ms and renders only when you ask to look at something. The depth image is collapsed into a virtual laser scan (only near-horizontal rays, and only hits above 0.4 m, so the floor isn't mistaken for a wall) and ray-traced into a log-odds occupancy grid: free along the beam, occupied at the endpoint.

Exploration and planning — 2 Hz. Frontier cells are known-free cells that touch unknown space. They get clustered, scored by size − distance, and the best reachable one becomes the goal. A* over the inflated grid finds a route (unknown cells are passable but cost more), then string-pulling reduces it to a handful of waypoints. When no reachable frontier is left, the area is mapped and the drone reports and holds.

Safety — every control step. Two independent guards, because the map can be stale but the camera never is. The map guard truncates any setpoint before the first inflated obstacle and adds a repulsion push from nearby walls; the camera guard cancels forward motion outright if the live depth image shows something inside 1.3 m. A watchdog notices when the drone stops making progress, blacklists the current goal, and climbs to try to clear whatever is pinning it.

Files

File	What's in it
main.py	Entry point and the main loop
config.py	Every tunable parameter
drone.py	Rigid-body quadrotor, mixer, controller, camera
camera_backends.py	PyBullet and Colosseum/UE5 renderers behind one interface
mapping.py	Depth → virtual scan → log-odds occupancy grid
exploration.py	Frontier detection, clustering, scoring
planning.py	A*, string-pulling, waypoint following
safety.py	Reactive obstacle guards
autopilot.py	Mission state machine
commander.py	Natural-language → structured commands
cinematic.py	Offline renderer for stills and video. Never feeds perception
vision.py	Pixel + depth → a world goal the planner can fly to
world.py	Procedural city: textured buildings, roofs, trees, props
textures.py	Generates the facade/ground/roof textures on first run
viz.py	Map and camera windows
test_flight.py	Controller checks: hover, tracking, axis signs
test_mission.py	Full command sequence, collisions, exploration convergence
test_frames.py	z-up ↔ NED conversion round-trips
test_depth_agreement.py	Both renderers report the same metres
test_camera_async.py	A slow renderer never stalls the control loop
test_vision_geometry.py	Projection round-trips against the renderer
fake_colosseum.py	Stand-in RPC server so those two run without Unreal

Verified behaviour

python test_mission.py runs the whole thing headless and asserts on the results. Current numbers on the default world:

• Hover error 4 mm, waypoint error 3 mm, yaw error < 0.1°
• Every command reaches its target state; return-to-home lands within 0.6 m
• Zero collision contacts over a full 600-second autonomous exploration, including the trees, lamp posts and parked cars
• Explores until no frontiers remain, ending around 60% of the grid — which is roughly 90% of the reachable area, since a third of the grid lies outside the boundary walls and building interiors are never visible
• Deterministic: same seed, same trajectory, bit-identical occupancy grid

Headless speed depends heavily on the machine and on WORLD_DETAIL. Measured run-to-run spread on one desktop was wide enough (77–192 s of wall time for the same 600 s of simulation) that a single number would be misleading — time your own box rather than trusting a headline multiple.

Photorealistic camera (Colosseum + UE5)

The renderer is swappable. config.py picks it:

CAMERA_BACKEND = "pybullet"     # "pybullet" | "colosseum"

pybullet is the default so the repo still runs standalone with nothing else installed. colosseum pulls frames from Unreal Engine 5 over RPC instead, and falls back to the PyBullet renderer with a printed warning if it can't reach a running session, so switching it on never breaks a run.

PyBullet keeps the physics and the flight controller either way. Colosseum is only the renderer: each frame the drone's camera pose gets mirrored into Unreal with simSetVehiclePose and an image comes back. The verified controller is untouched, and the heavy renderer stays droppable.

Installing it on Windows

The quick path, if you just want frames:

1. Grab a prebuilt Blocks binary from the Colosseum releases page. No Unreal or Visual Studio needed.
2. pip install msgpack-rpc-python, then install the Python client from a Colosseum checkout with pip install -e PythonClient. It imports as airsim.
3. Run the binary, then run this sim with CAMERA_BACKEND = "colosseum".

The full path, needed for your own photorealistic levels:

1. Visual Studio 2022 with Desktop development with C++ and the Windows SDK.

2. Unreal Engine 5.2 through the Epic Games Launcher. The Colosseum main branch targets 5.2 specifically, so a newer engine may not build.

3. Clone and build the plugin from an x64 Native Tools Command Prompt for VS 2022:

   git clone https://github.com/CodexLabsLLC/Colosseum.git
   cd Colosseum
   build.cmd
   

4. Open Unreal\Environments\Blocks\Blocks.uproject with UE 5.2, let it build the plugin, and hit Play. Any UE5 level works once the AirSim plugin is in it, which is the whole point: drop in a photoscanned environment and the drone flies through that instead.

settings.json

Colosseum reads Documents\AirSim\settings.json. This repo generates the one that matches its camera:

python camera_backends.py --settings

Two details in there are worth understanding, because both fail quietly:

• FOV is horizontal. config.CAM_FOV is 75 degrees vertical, which is what mapping.py builds its per-pixel rays from. Unreal's FOV_Degrees is horizontal. At 160x120 that's 91.309 degrees, not 75. Copy the vertical number straight across and the view quietly narrows, which shows up as an occupancy grid that's subtly too small in azimuth.
• The camera sits at the vehicle origin with zero rotation. The 0.13 m forward offset and the 8 degree down-pitch stay in camera_basis(), shared by both backends, and what gets mirrored into Unreal is the camera pose rather than the airframe pose. That makes the two renderers geometrically identical by construction instead of by two sets of numbers agreeing.

Frame rate

UE5 over RPC is nowhere near the 10 Hz that TINY_RENDERER manages, so the render runs on a background thread. render_camera() drops off the latest pose and immediately returns the most recent completed frame. The 240 Hz control loop never waits on Unreal, which is the point: a blocking RPC in that loop would wreck the timing the collision-free guarantee depends on.

The consequence is that frames arrive slightly stale, so a frame has to be mapped against the pose it was rendered from, not the drone's current pose. drone.frame_basis() returns that pose and main.py uses it. Mixing the two smears the occupancy grid. For the synchronous PyBullet backend the two are identical, so nothing changes there.

COLOSSEUM_MAX_FPS caps the render thread so it can't monopolise the GPU.

GPU budget

A UE5 scene runs about 4 to 6 GB, which fits a 10 GB 3080 fine but doesn't leave room for much else. In particular, don't also load a local 7B vision model at fp16, which wants roughly 14 GB on its own. Either quantise it to 4-bit or keep the vision calls on the API, which is what commander.py already does.

What happened when it was actually run

The backend was written against the API docs and then tested against a real prebuilt Colosseum Blocks build. It works — and the trip turned up four things worth writing down, because none of them appear in the documentation.

It connects and it's fast enough. Pose mirroring, Scene + DepthPlanar capture, and the far-plane clamp all behaved: Unreal reports 65504 (float16 max) for sky, which the backend clamps to CAM_FAR exactly as designed. Frames came back in ~100 ms, which is 10 fps — precisely the CAMERA_HZ the mapper wants.

settings.json in Documents\AirSim was ignored. The running sim kept its built-in defaults: vehicle SimpleFlight instead of Drone1, and 256×144 instead of the configured 160×120 — a 16:9 frame where the mapper expects 4:3, which would have quietly broken the FOV correspondence. Passing the path explicitly is what made it load:

Blocks.exe -settings="%USERPROFILE%\Documents\AirSim\settings.json"

The camera sees the drone's own body. Because the backend mirrors the camera pose rather than the airframe pose, the vehicle origin lands exactly where the camera is and its mesh fills the frame. Fixed by sitting the body COLOSSEUM_CAM_OFFSET behind the camera and pushing the camera the same distance forward in settings.json, so the two cancel.

Capturing images crashes UE5 with Nanite on. This is the serious one:

Assertion failed: !AsyncState.bUpdateActive
  NaniteStreamingManager.cpp:1662
  Nanite::FStreamingManager::BeginAsyncUpdate()
  FDeferredShadingSceneRenderer::Render()
  UpdateSceneCaptureContentMobile_RenderThread()

AirSim grabs frames through a SceneCapture component, and that collides with Nanite's streaming manager. Forcing -dx11 avoids it — D3D11 can't run Nanite at all — but it also loses Lumen, and the scene renders essentially unlit. Keeping D3D12 and disabling Nanite by console variable survived one capture and died on the second.

So on this build the choice is stable but unlit or lit but crashes. Worth knowing before committing a weekend to it.

Checking the swap

python test_frames.py            # z-up <-> NED conversion round-trips
python test_depth_agreement.py   # both renderers report the same metres
python test_camera_async.py      # a slow renderer never stalls the control loop

None of them needs Unreal running. test_camera_async.py stands in a renderer that takes 50 ms a frame, roughly what a real UE5 scene costs, and checks two seconds of 240 Hz control still spends under a millisecond inside render_camera() in total. It also pulls the RPC out from under the camera mid-flight to confirm a dropped frame leaves the last good one in place instead of taking the drone down. test_depth_agreement.py talks to a stand-in RPC server that returns what Colosseum sends over the wire, so the real decode and pose-conversion path gets exercised; add --live to run the same assertions against an actual UE5 session with a wall 5 m in front of the player start.

Both tests carry deliberate mutations to prove they'd catch the real failures. Linearising DepthPlanar a second time turns a 5 m wall into -0.02 m, and dropping the NED flip moves the wall by 4.75 m. Either one fails the suite loudly rather than producing a plausible-looking depth image.

One measured quirk, pre-existing and not from the swap: mapping.py and safety.py trace rays through pixel centres, while TINY_RENDERER samples half a pixel off that. Square-on to a wall it cancels; at 33 degrees obliquity it's worth about 290 mm. That's still well under one 0.25 m map cell at the angles the drone flies, so it's recorded and pinned by the test rather than fixed, since changing the ray model would perturb the verified baseline. Unreal samples pixel centres, so the Colosseum backend is the one that matches what mapping.py already assumes.

Commanding it in English

The command layer is provider-agnostic. Every option except Anthropic speaks the OpenAI chat-completions format, so one backend covers all of them and switching is a config line:

LLM_PROVIDER = "ollama"   # ollama | groq | cerebras | gemini | openrouter | mistral | anthropic | auto

See what's usable right now, and why the rest aren't:

python commander.py

Provider	Cost	Key	Vision
ollama	free, local, no quota	none	with a vision model pulled
groq	free tier, very fast	GROQ_API_KEY	no
cerebras	free tier	CEREBRAS_API_KEY	no
gemini	free tier	GEMINI_API_KEY	yes
openrouter	free tier, many models	OPENROUTER_API_KEY	no
mistral	free tier	MISTRAL_API_KEY	no
anthropic	paid	ANTHROPIC_API_KEY	yes

auto walks the list, local first, and takes the first one that works. If none do, the offline keyword parser handles it and the sim runs regardless — the language model is an upgrade, never a dependency.

Running it locally with Ollama

No key, no quota, no network. Install Ollama, then pull one model:

ollama pull qwen2.5vl:7b

One model, deliberately. It handles commands, what do you see, and the coordinate grounding that visual seeking needs. An earlier setup split the work between a text model and a separate vision model, which is the obvious design and the wrong one: two 7B models don't fit in 10 GB, so every switch between a command and a look cost a 25 second VRAM swap. With one model nothing ever swaps. Plain chat models also can't return coordinates at all, which rules them out the moment vision starts choosing destinations.

Measured on an idle RTX 3080, qwen2.5vl:7b:

Command parsing	~330 ms median, 900 ms worst
Command accuracy	10/10 on varied phrasings, no fallbacks
Locating a target in frame	~1.3 s
Describing the view	~5.8 s
Cold load into VRAM	~3 s, once

Those numbers assume the GPU is yours. Sharing it with a game pushed command parsing to 18–20 s, which is why LLM_TIMEOUT is 12 s: under contention it degrades to the instant keyword parser rather than leaving you waiting. LLM_VISION_TIMEOUT is separate and much longer, since looking and locating are slower and the drone keeps flying throughout.

The GUI will starve the model if you let it

PyBullet's 3D window renders flat out and takes the whole GPU. Measured with the model resident and the sim in GUI mode:

Command latency, GPU idle	0.4 s
With the GUI sim running, shadows on	20.6 s
With the GUI sim running, shadows off	0.2 s

So GUI_SHADOWS = False is the default. Shadows are the expensive part and buy nothing at this scale; turning them off costs nothing visually and is the difference between a usable local model and one that times out on every command. The RGB/depth/segmentation preview panes are disabled for the same reason. Set GUI_SHADOWS = True if you want the prettier view and aren't running a model on the same card.

VRAM is worth knowing about too. nvidia-smi can't attribute VRAM per process on Windows, so measure it by difference: this desktop idles at 2.2 GB, and qwen2.5vl:7b takes it to 9.5 GB — about 7.2 GB for the model, not the 5.8 GB Ollama reports. The gap is KV cache and context buffers it doesn't count. On a 10 GB card that leaves little room, so if you hit trouble, a smaller model (qwen2.5vl:3b) or running the sim --headless while you drive it are the two levers.

The provider check verifies the model is actually pulled, not just that the server is up: a running Ollama with nothing in it reports exactly that instead of failing on your first command.

On local vision quality. A 7B model identifies buildings, trees and colours reliably but will invent details that aren't there, and locates small targets only approximately. If vision matters to you, Gemini's free tier is much stronger for the cost of a key. One caveat learned the hard way: don't tell a small vision model what's in the shot. An earlier prompt listed the scene contents as context and the model echoed the list straight back instead of looking at the image.

Flying to something it can only see

go to the blue building        fly over to that tree        find the lamp post

No coordinates. The vision model locates the thing in the current camera frame, and that pixel becomes a waypoint:

"go to the tall building"
   -> 'green building' at 17.5 m; flying 14.0 m to within 4 m of it   (1.4 s)
   -> flew (+6.0,-5.0) to (+12.7,+7.5), 0.0 m from goal
"go to the submarine"
   -> refused: qwen2.5vl:7b cannot see 'submarine'

The vision model only ever chooses where. It hands back a goal and stops there; the A* planner, both safety guards and the 240 Hz controller are the same verified code that flies every other waypoint. A model that misidentifies a building sends the drone to the wrong place — it cannot send it into anything, because nothing downstream of the goal changed. The seek flight above logged zero collision contacts.

How a phrase becomes a waypoint:

1. commander.locate() asks for the target's centre in normalized image coordinates, and explicitly permits "not visible" — a grounding model that always returns coordinates will happily aim at something that isn't there.
2. vision.pixel_to_world() turns that pixel plus its depth into a world point, using the same ray convention as mapping.py so the goal lands in the same frame the planner searches.
3. vision.target_to_goal() backs off VISUAL_STANDOFF metres — flying to a wall is not what "go to that building" means — holds the drone's current altitude so aiming at a rooftop doesn't command a climb, and clamps to the world bounds.
4. The result is handed to the ordinary goto path.

test_vision_geometry.py pins step 2 without needing a model or a GPU: it places an object, renders it, finds its pixel, projects back through the depth buffer, and checks it lands on the object from four viewpoints.

What it's good at, and what it isn't. Buildings work well. Small or thin targets — trees, lamp posts — get located approximately, often just off the object, which is why depth sampling widens its search rather than trusting the exact pixel. Asked for "the tall building" it will pick a building, not reliably the tallest. Treat it as a genuinely useful pointing device, not a precision instrument.

Goal updates run at roughly 1 Hz (~1.3 s per grounding call). That's fine for choosing a destination and nowhere near enough for closed-loop visual servoing — which is exactly why the fast loop stays with the planner.

Free tiers drift

Rate limits and model names on free tiers change often. If a call 404s on the model, edit model in LLM_PROVIDERS — that's a config line, not a code change. Free models also wrap JSON in prose and code fences far more than paid ones, so the parser grabs the outermost JSON object rather than trusting the whole reply, and retries once without JSON mode for endpoints that reject it.

The world

Four building archetypes (plain slab, stepped setback tower, gabled block, round tower) chosen to suit each footprint, plus trees, lamp posts and parked cars. Facades, ground, roofing and foliage are textured; textures.py generates the PNGs procedurally on first run, so there are no binary assets in the repo and the palette follows config rather than a fixed file.

The props are real collision geometry, not decals. They're placed with the same spacing rule as buildings and kept clear of the pad, and the drone still completes the mission with zero collision contacts.

Building positions and footprints are unchanged from the original box world: they come off the same RNG stream in the same order, so a given seed lays out the same city it always did. Everything cosmetic draws from a second, independent RNG. That's deliberate — it means the look changed without disturbing the flight and mapping behaviour that was already verified.

WORLD_DETAIL = "full"     # "full" | "lite"

lite keeps the same building footprints but drops the props and roof detail, which takes the world from ~150 bodies down to ~23. Rendering measures 7.5 to 11.5 ms a frame depending on detail and how much of the view is textured, so lite is worth reaching for when you're iterating and want the sim back at full speed.

Rendering it nicely

Everything the drone sees stays at 160×120 with no post-processing, because that's what it has to fly with. cinematic.py is a separate offline path used only for stills and video, and it never touches perception.

PyBullet's renderer is CPU software rasterisation — no anti-aliasing, no ambient occlusion, no post — so a fast GPU cannot help. What it does give you is a depth buffer, which is enough for the two effects that most separate a cinematic frame from a debug view:

2× supersample, box-filtered down	The only anti-aliasing available here
Sky gradient	Replaces the renderer's flat white void
Aerial perspective	Distance fades toward the sky, which is what reads as scale
Depth of field	Focal plane tracks the median hit distance as the drone moves
Exposure lift, S-curve, vignette	This renderer starts dark

About 0.5 s a frame at 384×288 and 2×, so the 450-frame video takes a few minutes. Regenerate everything with:

python tools/make_flight_video.py          # the four-panel video
python tools/make_docs_images.py           # the stills

Tuning

Most of what you'd want to change is in config.py. MAX_SPEED_XY and CRUISE_ALTITUDE change how it flies; MAP_RES trades map detail against CPU; FRONTIER_SIZE_WEIGHT shifts it between "sweep the nearby area thoroughly" and "go after the biggest unexplored region". BODY_RADIUS sets how much clearance it keeps from walls.

Things that bit me

Kept here because the bugs were more interesting than the features, and every one of them was found by measuring rather than reasoning.

The GUI was starving the model. Commands parsed in 0.4 s on an idle GPU and 20.6 s with the simulator window open — enough to blow the 12 s timeout on every command. The cause wasn't the model, the network or the prompt: PyBullet's 3D window renders flat out and takes the whole card. Shadows were the expensive part. Turning them off brought it to 0.2 s, a 100× swing from one line.

A greedy regex that only broke on success. Asked to find "a building" in a street full of them, a grounding model replies with one JSON object per building. {.*} spanned them all into invalid JSON. The obvious fix, {.*?}, breaks the nested args object in every command instead — so neither regex works, and it needed a brace-balanced scan that respects string literals. It only showed up once the model got good enough to find several targets at once.

Telling a vision model what it was looking at. The prompt helpfully listed the scene contents as context. A large model treats that as framing; a 7B model read the list and echoed it straight back instead of looking at the image. The prompt now gives the viewpoint and never the answer.

The camera the model saw wasn't the camera the drone flies with. Frames were being handed over at 160×120, nearest-upscaled 4× — a wall of hard-edged blocks, which is why early descriptions called the buildings "stacked cardboard boxes". Vision calls now render their own 480×360 frame from the same pose. The flight loop pays nothing: it still reads 160×120 at 6.7 ms, and the 56 ms vision frame renders only when something asks to look.

A half-pixel that only mattered off-axis. Chasing a 40 mm depth discrepancy turned up a pre-existing detail: mapping.py traces rays through pixel centres, while PyBullet's renderer samples half a pixel off that. Square-on to a wall it cancels; at 33° obliquity it's worth ~290 mm. Still under one 0.25 m map cell at the angles the drone flies, so it's measured and pinned by a test rather than "fixed" — changing the ray model would have perturbed a verified baseline for a sub-cell effect.

VRAM lies. Ollama reported 5.8 GB for the model; the card said otherwise. Measured by difference — desktop idle at 2.2 GB, 9.5 GB with the model loaded — it actually costs ~7.2 GB. The gap is KV cache and context buffers that don't show up in the reported figure, and on a 10 GB card that gap is the difference between working and thrashing.

Where this goes next

The interfaces are deliberately narrow, so pieces can be swapped out:

• Photorealistic camera — done, see above. camera_backends.py has both renderers behind one interface and CAMERA_BACKEND switches them.
• Learned exploration — exploration.py returns a goal; swap the frontier heuristic for a trained policy and keep the rest.
• Real hardware path — replace the controller and physics with PX4 SITL over MAVLink; the mapping, planning and command layers sit on top unchanged.
• 3D mapping — the grid is 2D at flight altitude. A voxel grid or Octomap would let it fly over things instead of around them.