Skip to content

test: split browser_computer_test's multi-env tests out and refresh the shard table - #622

Open
andrewpark-scaleai wants to merge 3 commits into
mainfrom
andrewpark/split-browser-multi-env-tests
Open

andrewpark-scaleai wants to merge 3 commits into
mainfrom
andrewpark/split-browser-multi-env-tests

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is a faster slow tier on the same runners, with no test weakened. On main's run 43079720062 the workflow took 870s, and the slow tier's shard 4 took 861s of that. Shard 4 ran browser_computer_test.py, which took 774s, while slow_durations.json still had it at 304s. A file's tests run on one worker of one shard, so that file set the time of the whole workflow.

What

  • browser_computer_test.py's multi-env section (browser_envs, six tests) moves unchanged to browser_multi_env_test.py. The six tests took 353s; the 22 that stay took 421s. The new file imports its helpers from browser_computer_test.py, as browser_gateway_test.py already does, and builds the module-scoped browser_computer fixture once more, about 55s.
  • slow_durations.json is rewritten with .github/scripts/slow_test_durations.py from every JUnit report of run 43079720062, with the six moved tests counted under the new file. That run skipped the installer tier, so the script left out its 59 units; their seconds are kept from the old table.
  • A comment in local-backends.yml that still said the slow tier runs serially now says it runs two workers per runner (since ci: run the slow tier's tests two at a time on each shard #600).

Expected effect

Projected from run 43079720062's test times with the new table, the longest slow shard's pytest time drops from 774s to 501s (chromium_app_test.py), and the other shards fall between 331s and 422s. The projection doesn't count the new file's extra fixture build, and the first run after the split may build the chromium image cold on the new file's shard. This PR's own CI run gives the measured numbers.

Verification

  • pytest --collect-only collects the same 28 test names across the two files that the old file had, and no node id twice across tst/.
  • The moved section is identical to the new file's body apart from one blank line.
  • ruff check passes. tst/unit/test_slow_test_durations_script.py passes.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable regression was found.

Summary

Splits six multi-environment browser tests into browser_multi_env_test.py without changing their assertions.

  • Multi-environment browser tests run from a separate file for shard balancing.

Reviews (1) · Last reviewed commit: "ci: say the slow tier runs on two worker..." · Reviewed by Greptile

… own

browser_computer_test.py took 774s on main (run 43079720062), and shard 4 of
the slow tier, which ran it, took 861s of the workflow's 870s. A file's tests
run on one worker of one shard, so the file set the tier's time.

The six tests of its multi-env section (browser_envs) move unchanged to
browser_multi_env_test.py. They took 353s; the 22 that stay took 421s. The new
file imports its helpers from browser_computer_test.py, as
browser_gateway_test.py does, and deploys the module-scoped
browser_computer fixture once more, about 55s.
The table had browser_computer_test.py at 304s; it took 774s. Rewritten with
.github/scripts/slow_test_durations.py from every JUnit report of run
43079720062, with the six moved tests counted under browser_multi_env_test.py.

That run skipped the installer tier, so the script left its 59 units out;
their seconds are kept from the old table.

Projected from that run's test times, the longest slow shard's pytest time
drops from 774s to 501s (chromium_app_test.py), and the other shards fall
between 331s and 422s.
The comment still said its tests run serially; since #600 each runner runs
its files on two pytest workers (-n 2 --dist loadfile).
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 11, 2026 19:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant