[earakely-scale]

Summary

M8.23 (FD-4971), built as the first slice of browser-recording/v1 (M8.22, FD-4970).

A run can now record the website browser with frames:

• one at the start
• one after each burst of an agent's actions settles, of the tab the agent last acted in
• one at the end

Each frame is keyed to the action-log entries it follows. The recording also keeps the log's lines between its start and its end, and a manifest. Missing frames, or a missing log, fail the run as an infrastructure failure.

This is what Online-Mind2Web's exporter (M8.46) and WebJudge (M8.47) grade from.

What changes

browser-recording/v1 (agentenv_protocol.website_browser.recording), on the browser's card, framework-only:

• Card params: max_duration_s, max_frames and frames.
• start {request_id, upload, frames?} records under the write namespace grant upload. A start again with the same request_id answers the recording it began, so a retried step joins it.
• end {recording_id, upload?, reason?} ends it, under a fresh grant when one is given, and answers its manifest. end_all {reason?} ends every open recording.
• Frames:
  • Bursts: each acting entry an agent finishes in the log (a CDP command or an MCP call, not agentenv's own) restarts a 0.5 s timer, so typing a word is one frame.
  • Which tab: the tab of the burst's last acting entry, else the active tab, as when that tab has closed. A popup or an agent's own tab is framed, not the front tab.
  • A tab behind the active one: Chrome answers a capture of it only once another capture of it arrives, and then answers both. So a capture still unanswered after 0.5 s is followed by a second, and the first answer is the frame. The tab is never brought to the front, which would change what the agent sees.
  • When: after the tab's page loads, waiting at most 5 s.
  • Format: a JPEG at quality 80, about a fifth to a tenth of a PNG. While a dialog keeps the page from painting, the frame is the last capture, a PNG.
  • Upload: each frame uploads as it's taken, under <recording_id>/frames/. Its manifest entry has kind (first, action or last), after (the entries it follows), tab, url, and error if it couldn't be taken.
• At the end: log.jsonl (the log's lines from the start's entry to the end's, actions and tab events) and manifest.json (BrowserRecordingManifest).
• Support: BrowserLog.watchers tells the recording each action as it's written. BrowserControl.frame(target, load_wait_s) captures a given tab once its page has loaded; the control now tracks each tab's commit and loadEventFired.

Steps (task_steps/browser/recording.py)

• start_browser_recording {env_id, name, frames=true, browser?} finds the browser as the other browser steps do: an env's child, or a multi env's computer's. It hands the browser a grant under agentenv-browser-recordings/.
• end_browser_recording {name} ends the recording and registers the manifest, the log and every frame as one file artifact universe.
  • Both steps fail the task by default, unlike the computer recording's best-effort steps, since a benchmark needs the frames.
  • The end fails when the recording didn't end, its log didn't upload, or a frame is missing.
• When the browser no longer knows the recording, as after its server restarted, what its namespace holds is registered instead.
• Teardown ends open browser recordings with fresh grants before envs close, and registers from its namespace any it couldn't end, as it does for computer recordings.
• Shared with the computer recording: a new module, task_steps/recordings.py, holds what both recordings' steps use. That's where a recording uploads and its grant (recording_namespace, recording_grant, refuse_no_network), the error an entry keeps (why), and teardown's two passes over the recordings a run left open (end_open, register_open).
• Rebuilt from its namespace: when the browser forgot a recording, the manifest is rebuilt from the files it uploaded. Each frame takes its index and type from its file's name, and its kind and at are None, since they aren't known.
• The episode rules list both steps as naming no agent.

Not in this slice: the rest of M8.22 goes into the same extension and steps later:

• video from CDP screencast (fps)
• network.har (M8.20)
• one recording per context (M8.15)

The end step takes name, as end_computer_recording does, rather than M8.22's sketched started_by.

Decisions (in chat with Edgar)

1. Frames come from the browser itself, after each action settles, rather than cut from the screen recording at action times, which shows only the active tab, at most 10 frames a second.
2. They're built as browser-recording/v1's first slice, rather than a browser setting delivered through data/get, since M8.22 needs the extension, its steps and its upload path anyway.

Test plan

• Unit, the SDK (test_browser_recording.py):
  • Bursts: a burst of three typing commands is one frame keyed to all three. A reading entry and agentenv's own make none. An MCP call on another tab makes a frame of that tab.
  • Ending: a burst still settling at the end gets its frame before the last.
  • Uploads: the log's lines and the manifest upload, the manifest matching what end answers.
  • Failures: a frame that can't be taken is kept with why, and counts as missing.
  • Without frames: it takes none, and still uploads the log.
  • Repeats: a start again joins its recording, and an end again answers the same manifest. An unknown recording, a bad reason and a bad grant are refused. end_all ends everything open.
  • The control: a frame waits for its tab's loadEventFired, then captures a JPEG at quality 80. A closed tab's frame is the active tab's. A tab that answers only a second capture is framed once that one is sent.
• Unit, the steps (browser_recording_test.py):
  • the grant is handed through the browser's own card, with its frames budget
  • the universe holds the manifest, the log and every frame
  • a missing frame fails the end as an infrastructure failure, unless fail_task_on_error is off
  • a recording the browser forgot registers from its namespace, and fails for its missing log
  • a browser without the extension fails the start, saying so
  • what a run left open ends at teardown with a fresh grant
  • both steps round-trip
• Results: the protocol package, tst/unit/task_step, task, drivers and bundle: 5,615 passed, 12 skipped, plus the episode rules' test.
• Integration, local Docker (Mac, arm64): the new browser computer test records while Playwright MCP types into the page, then navigates.
  • There's a first frame, frames after the browser_type and browser_navigate bursts keyed to those calls' entries, and a last frame on the page the agent ended on.
  • Every frame is a JPEG, and none are missing.
• Integration, a CDP agent's own tab behind the active one: a second new test has a CDP agent open a background tab, navigate it, change its page and navigate it again, never bringing it to the front.
  • Before the fix, one of its five frames was missing: the capture timed out.
  • With the fix, every frame is there, and the last is of the agent's tab on its last page. It passed three runs in a row.
• modal_vm, end to end: the chromium computer composed natively on a Modal VM, passed its probe there, and ran two Online-Mind2Web tasks ("find the latest MacBook Air's tech specs on apple.com") on Claude Haiku 5.5. The stores were throwaway S3 and ECR, and the agents ran locally.
  • Claude Code, Playwright MCP's tools: 14 frames, from about:blank to apple.com to the specs page, each keyed to Playwright MCP's CDP commands, which are credited to the agent's browser_* calls. The last frame shows the 15-inch tab the agent had just clicked. The log has 100 lines, none missing.
  • browser-use over CDP: 5 frames of its own tab, apple.com's MacBook Air page and then the specs page. When it closed its tab at the end, the last frames are of the tab left in front. The log has 28 lines, none missing.
  • Both recordings uploaded from the VM under their grants, were registered as file artifact universes, and failed nothing. Both sandboxes were torn down.

Docs

The user guide's browser page should cover:

• start_browser_recording and end_browser_recording
• what a recording's universe holds
• how frames are taken and keyed to the action log
• that a missing frame fails the run

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds browser recordings with frames tied to agent actions, an action log, and a manifest.

• Browser recordings tie each captured frame to the actions it follows.
• Browser recording steps save each run’s frames and log as artifacts.
• Browser and computer recordings now use the same upload and teardown helpers.

No new actionable issues were found. This review used source checks; no tests were run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Start[Start recording] --> First[Capture first frame]
  First --> Actions[Watch agent actions]
  Actions --> Burst[Wait for burst to settle]
  Burst --> Frame[Capture and upload frame]
  Frame --> Actions
  Actions --> End[End recording]
  End --> Upload[Upload last frame, log and manifest]
  Upload --> Register[Register artifact universe]
  Register --> Check[Reject missing frames or log entries]

Loading

_{Reviews (6) · Last reviewed commit: "refactor(recordings): share what compute..." · Reviewed by Greptile}