Orbis Engine

Anything ever filmed, in 4D.

▶ Watch the demo

Hack the North 2026 finalist (top 12 of 350 teams) and 2nd place in the OpenAI track.

Orbis turns an ordinary video into a 4D scene: a full 3D world, plus time. The room is rebuilt in 3D, and the people and objects in it move through it as they did in the recording, in sync with the original sound. You are not locked to the camera. Put on a Quest, or use WASD in a browser, and walk anywhere while the moment plays: stand behind someone, watch a throw from the far side of the room, see it from angles the video never had.

In one scene you can also step in: walk up to a person and talk to them, or pick up the bottle they are holding, and it behaves like a physical object in your hand.

It works on phone clips and on movie shots. Built at Hack the North 2026.

How it works

flowchart LR
  A[Video] --> B[Solve cameras and depth]
  A --> C[Remove people, fill the holes]
  C --> D[Generate the room]
  B --> E[Rebuild and animate each person]
  D --> F[Fit scale, floor and placement]
  E --> F
  F --> G[Walk around in it]

Loading

Step	What happens	Built on
Cameras	Find where the camera was in every frame, plus depth	Pi3X
Clean	Mask the people out and fill the holes	Mask R-CNN, LaMa
Room	Generate a full Gaussian-splat room from the cleaned video	Marble, OpenAI vision
Correct	Train the room against the real frames (optional)	gsplat
People	Track each person and rebuild them as an animated avatar	LHM
Placement	Fit scale and floor so feet land where they stood
Viewer	Walk around in a browser or a Quest, synced to the source video	Three.js, Spark, WebXR, OpenAI Realtime

GPU stages run on Modal. Scenes are stored in S3.

What we tried first

Idea	Tried	What happened
Rebuild the room from the footage	Depth Anything 3, VGGT, Brush	Great from the camera's spot, broken everywhere else
Generate only the missing views	Stable Virtual Camera, GeoNVS, GEN3C, Lyra	A courtyard that did not exist, grey mush, a shop sign turned to gibberish
Rebuild people from depth	V-DPM, PIFuHD	Melted limbs
Refine body poses against the video	our own optimiser	Worse. One camera cannot tell how far away a hand is

A camera walking forward only records about 41° off its own path, so you cannot rebuild what was never filmed. What worked was generating the whole room, then correcting it with the footage.

Full list in the research log. Dead ends in known limits.

Run the complete local app

Install Bun, Node.js 24+, uv, and FFmpeg, then run:

bun run setup
bun run app

Open http://127.0.0.1:5399. The dashboard, source uploads, run previews, viewer, and About page now share one address. Setup installs locked dependencies and builds all three frontends. The launcher initializes or migrates a local SQLite database, keeps uploaded files and a private API token under .context/app/, and stops its child services when you press Ctrl+C. Restarting retains your uploads and run state. Private settings in .env.local and .env.author are loaded server-side; never put provider keys in VITE_ variables.

The default command saves uploads in the queue. To also enable an already configured provider worker with a finite, persistent admission allowance, use:

bun run app --worker --max-uploads 1

Configure provider access and spending controls first using pipeline setup and automatic uploads. This admits one upload across restarts; it is not a dollar cap or a promise of accepted reconstruction quality. Keep the existing shared WANDER_STAGE_LEDGER when using another checkout. The app binds only to localhost and is an operator tool; public hosting requires its own access control and worker deployment.

A run dashboard offers World preview once its generated room is available, even while people are unfinished. It opens a separate orbit/pan/zoom inspector at /world-preview.html using the same private run asset proxy. People and recorded audio are absent; scale, camera alignment, and reconstruction quality are unverified. The generation-origin view is a navigation starting point, not proof of a source-matched camera or accepted scene.

Run bun run test:e2e with the app stopped to exercise the real API, dashboard, source-video playback, pause/resume/cancel controls, and About page in Chrome/Chromium. It creates an isolated database, starts no worker, and makes no paid model calls. PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH selects a browser executable; otherwise installed Chrome or Playwright Chromium is used.

Run only the viewer

You need Bun 1.2.21+ and Chrome.

git clone --single-branch --branch main https://github.com/dtpu/OrbisEngine.git
cd OrbisEngine
bun install --frozen-lockfile
bun run demo

Open http://127.0.0.1:5399/demo.html and pick a scene.

Scenes are not in this repo. You have two ways to get one:

• Team credentials: put the read-only keys in .env.local (see .env.example).
• Your own clip: see below.

Key	Does
W A S D	Walk
Drag or click	Look around
Shift	Run
Space	Play or pause
R	Reset
M	Overhead map

Quest

1. Enable Developer Mode and plug in the headset.
2. Run adb reverse tcp:5399 tcp:5399.
3. In the Meta Browser open http://localhost:5399/demo.html?xr=1 and press Enter VR.

B on the right controller opens the scene list. The desktop page shows what the headset sees.

To talk to a person and pick up the bottle, open fourd.html?demo=elevator&interact=1&xr=1 with OPENAI_API_KEY set on the server. More options in developing.

Process your own clip

You need uv, Python 3.11 or 3.12, FFmpeg, and keys for Modal, World Labs and OpenAI in .env.author (.env.example lists them).

uv sync --locked --group inference
set -a; source .env.author; set +a
uv run --locked --group inference python -m orchestrator.doctor --clip /path/to/clip.mp4
uv run --locked --group inference scripts/run_clip.py \
  --clip /path/to/clip.mp4 --name myclip \
  --marble video --all-people --fps 12 --skip-finetune --no-publish

• The run stops once so you can check the cleaned frames. Run it again with --gate-pass to continue.
• Running the same command after a crash resumes where it stopped.
• A 10-second clip takes about 25 minutes, roughly $0.15 of Modal GPU plus 1,600 Marble credits.
• Read paid recovery before retrying a failed paid stage.
• Pipeline setup explains dependencies, credentials and read-only model-cache checks.
• Pipeline validation separates tested orchestration from fresh GPU inference and visual quality.

Clips that work: one continuous shot, 5 to 15 seconds, people at least 150 px tall, some sideways camera movement, decent light.

More

• Devpost submission: the build story, what we tried first, and how OpenAI, Codex and Devin were used.
• Developing: code map, checks, Quest options, pipeline flags, repo history.
• Research log and known limits.
• Audio, objects, interactive scene, unattended runs.

Credits

Aayan Karmali, Austin Jian, Daniel Pu and James Li.

The Tears of Steel scene uses excerpts of (CC) Blender Foundation | mango.blender.org, CC BY 3.0. Phone clips are our own footage. Movie excerpts used in testing are not redistributed here.

License

Apache-2.0. Some models the pipeline uses are research-only; see licences.