Repository navigation
test: split browser_computer_test's multi-env tests out and refresh the shard table - #622
Open
andrewpark-scaleai wants to merge 3 commits into
Open
andrewpark-scaleai wants to merge 3 commits into
andrewpark-scaleai wants to merge 3 commits into
Conversation
… own browser_computer_test.py took 774s on main (run 43079720062), and shard 4 of the slow tier, which ran it, took 861s of the workflow's 870s. A file's tests run on one worker of one shard, so the file set the tier's time. The six tests of its multi-env section (browser_envs) move unchanged to browser_multi_env_test.py. They took 353s; the 22 that stay took 421s. The new file imports its helpers from browser_computer_test.py, as browser_gateway_test.py does, and deploys the module-scoped browser_computer fixture once more, about 55s.
The table had browser_computer_test.py at 304s; it took 774s. Rewritten with .github/scripts/slow_test_durations.py from every JUnit report of run 43079720062, with the six moved tests counted under browser_multi_env_test.py. That run skipped the installer tier, so the script left its 59 units out; their seconds are kept from the old table. Projected from that run's test times, the longest slow shard's pytest time drops from 774s to 501s (chromium_app_test.py), and the other shards fall between 331s and 422s.
The comment still said its tests run serially; since #600 each runner runs its files on two pytest workers (-n 2 --dist loadfile).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The goal is a faster slow tier on the same runners, with no test weakened. On main's run 43079720062 the workflow took 870s, and the slow tier's shard 4 took 861s of that. Shard 4 ran
browser_computer_test.py, which took 774s, whileslow_durations.jsonstill had it at 304s. A file's tests run on one worker of one shard, so that file set the time of the whole workflow.What
browser_computer_test.py's multi-env section (browser_envs, six tests) moves unchanged tobrowser_multi_env_test.py. The six tests took 353s; the 22 that stay took 421s. The new file imports its helpers frombrowser_computer_test.py, asbrowser_gateway_test.pyalready does, and builds the module-scopedbrowser_computerfixture once more, about 55s.slow_durations.jsonis rewritten with.github/scripts/slow_test_durations.pyfrom every JUnit report of run 43079720062, with the six moved tests counted under the new file. That run skipped the installer tier, so the script left out its 59 units; their seconds are kept from the old table.local-backends.ymlthat still said the slow tier runs serially now says it runs two workers per runner (since ci: run the slow tier's tests two at a time on each shard #600).Expected effect
Projected from run 43079720062's test times with the new table, the longest slow shard's pytest time drops from 774s to 501s (
chromium_app_test.py), and the other shards fall between 331s and 422s. The projection doesn't count the new file's extra fixture build, and the first run after the split may build the chromium image cold on the new file's shard. This PR's own CI run gives the measured numbers.Verification
pytest --collect-onlycollects the same 28 test names across the two files that the old file had, and no node id twice acrosstst/.ruff checkpasses.tst/unit/test_slow_test_durations_script.pypasses.

Confidence Score: 5/5
The PR appears safe to merge; no actionable regression was found.Summary
Splits six multi-environment browser tests into
browser_multi_env_test.pywithout changing their assertions.Reviews (1) · Last reviewed commit: "ci: say the slow tier runs on two worker..." · Reviewed by Greptile