Skip to content

fleet/web mock: skip whole frames when behind, the same ones for every stream - #1346

Merged
ejc3 merged 1 commit into
mainfrom
wall/mock-switch-headroom
Oct 26, 2026
Merged

ejc3 merged 1 commit into
mainfrom
wall/mock-switch-headroom

Conversation

@ejc3

@ejc3 ejc3 commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

main's Fleet Web job failed "switching streams" on 47ec5fc (wall.spec.ts:586: after the switch back to direct, ui.shownMode stayed "aggregated" for 20 s). The same test failed on #1332's runs. This fixes the mechanism in the mock renderer.

Cause, measured on the CI runner. I ran the Fleet Web job on a throwaway branch with a CPU sampler and the switch logged every 0.5 s (run 40365237969, 4 vCPU EPYC 7763):

  • During a switch the mock serves both sets: 3 aggregates and 15 direct tiles, 18 streams, about 1.75 times the pixels of direct alone. It fell behind: about 60 renders queued, about 6 of 10 frames a second per stream.
  • Each stream got a different 6. Each lane skipped a frame when its own previous draw was still in flight, so the drops were independent per stream. Direct tiles' newest t at one sample: 186.30, 186.20, 186.20, 186.10 ...
  • The switch back to direct needs one t that all 15 direct tiles hold. With each missing 4 frames in 10 independently, none came in 22 s (log in the run: 44 samples, shown=aggregated throughout, every tile streaming).
  • The two-viewers timeout that fleet/web mock: draw the picture without per-pixel allocations #1339 addressed (11 of 15 tiles synced) is the same mechanism: a tile without the common t is drawn behind, not synced. fleet/web mock: draw the picture without per-pixel allocations #1339 bought CPU headroom; this removes the failure mode when the mock is behind anyway.

Change. When the last frame's draws have not all started (the render pool's backlog), the renderer skips the frame for every stream at once. Behind, it serves fewer frames, and every stream holds the same t, as a page drawing all tiles at one t needs.

Evidence

  • Red, unit: mock renderer, more streams than it can draw (one render thread, eight 360-high tiles at 30 fps): on main's renderer 0 of 71 frame numbers reached all eight streams; with the change every one does.
  • Red, CI: run 40365237969 (main's mock) failed the switch as above. Green, CI: run 40366842617 (only this change added): 13 switches back to direct, each within 0.6 s (the suite's one, plus the test repeated 6 times), all 49 Chromium tests passed, the mock used 1.1 to 2.1 cores of its 3.
  • Matched samples on CI (run 40372934826): the Fleet Web job three times with main's renderer and three times with this one, same commit otherwise. Main's renderer: 2 of 3 failed "switching streams" (Received: "aggregated"). This renderer: 3 of 3 passed all 56 tests.
  • This PR's own first Fleet Web run (job 121738214837) failed a different test, "no seam" (tile ec2:7, error 20.5 against 3), at test 4. I have not found a mechanism for it, so I am not calling it unrelated: it passed in every other run with this change (the three matched samples, measurement run 40366842617, and two full Chromium runs in the CI container locally), and in the earlier main runs I have logs for. If it recurs, the trace artifact of that test is the place to start.
  • Local gate: pnpm lint && pnpm typecheck && pnpm test (225) and pnpm test:e2e (56 passed).

ci.yml: the Fleet Web job's comment now says what a switch costs and how the mock behaves when behind.

Downstream impact. Test infrastructure only (the mock fleet). The page is unchanged.

…y stream

main's Fleet Web job failed "switching streams" on 47ec5fc: after the
switch back to direct, ui.shownMode stayed "aggregated" for 20 s; the
same commit's previous run, and #1332's runs, show the same intermittent
failure. Measured on the CI runner with the switch logged every 0.5 s
(run 40365237969): during the switch the mock serves the 3 aggregates
and the 15 direct tiles, 18 streams and about 1.75 times the pixels of
direct alone. It fell behind (about 60 renders queued) and every stream
got about 6 of its 10 frames a second, but each stream a different 6:
each lane skipped a frame when its own previous draw was still in
flight. The switch needs one t that all 15 direct tiles hold, and with
each missing 4 frames in 10 independently, no such t came in 22 s.
The two-viewers timeout fixed by #1339 (11 of 15 tiles synced) is the
same mechanism: a tile without the common t is drawn behind, not synced.

The renderer now skips a frame for every stream at once when the last
frame's draws have not all started (the pool's backlog). Behind, it
serves fewer frames, and all streams hold the same t. With this change
the same runner switched back to direct within 0.6 s in each of 13
switches (run 40366842617), passed all 49 Chromium tests, and the mock
used 1.1 to 2.1 cores of its 3.

- mock/test/mock.test.ts: one render thread asked for eight 360-high
  tiles at 30 fps; every frame number it sends reaches all eight
  streams. On the previous renderer: 0 of 71 frame numbers reached all
  eight.
- ci.yml: the Fleet Web job's comment says what a switch costs and how
  the mock behaves when behind.
@ejc3

ejc3 commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai

coderabbitai Bot commented Oct 26, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 39 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2d13f9d9-d04d-4dac-85d7-524603c19d2d
Commits

Reviewing files that changed from the base of the PR and between 0991437 and 5ab170c.

Files selected for processing: 3
  • .github/workflows/ci.yml
  • fleet/web/mock/renderer.ts
  • fleet/web/mock/test/mock.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T17:37:06.521876Z 5ab170c Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 5ab170c064

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ejc3
ejc3 merged commit 795be8d into main Oct 26, 2026
11 of 17 checks passed
@ejc3
ejc3 deleted the wall/mock-switch-headroom branch October 26, 2026 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant