Skip to content

Tests: a fast tier by default, a reliable 4-minute slow tier behind --slow - #4

Merged
jries merged 6 commits into
mainfrom
claude/sleepy-sagan-7wecqq
Oct 5, 2026
Merged

jries merged 6 commits into
mainfrom
claude/sleepy-sagan-7wecqq

Conversation

@jries

@jries jries commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

The suite had grown to 1322 tests and took 11:22 on four cores. Running it in parallel hung or failed in about half of the runs.

before after
after each change: pytest tests -n auto 11:22 ~1:25
before a PR: pytest tests -n auto --slow 11:22, and in parallel it failed about half the time ~3:45, 8 of 8 green

No test was removed or weakened.

Changes

Two tiers. About 60 tests are marked slow: the tutorials, the docs-figure tests and the 11 heaviest numerical checks. They took two thirds of the time and are skipped unless --slow is given. They are skipped rather than deselected, so the summary still counts them.

The parallel hang, fixed. A GUI test that left a Qt widget in a reference cycle left it for Python's cyclic garbage collector. The collector runs in whichever thread allocates next; here that was a later test's TIFF reader thread. The widget's destructor calls QWindow::close, which waits for the GUI thread to flush window events. The GUI thread was the test, itself waiting for that fit, so the worker deadlocked. py-spy showed QWindowSystemInterface::flushWindowSystemEvents under tifffile in the reader thread. This caused:

  • test_live failing in about half of the parallel --slow runs;
  • the 300 s timeouts in test_gui_view3d and test_gui_render_axes.

The bug predates this PR: main's setup failed the same way. The fix is pyqtgraph's GarbageCollector remedy: conftest.py turns off automatic collection and collects on the main thread every 100 tests. Before the fix, about half of the parallel runs failed; after it, 8 of 8 passed. The fast tier's time is unchanged.

Balanced parallel runs. Three changes, all in conftest.py:

  • Work stealing by default, through xdist's pytest_xdist_make_scheduler hook. xdist's own scheduler gave one worker 372 s of tutorials while the other three idled after about 100 s.
  • Slow tests first when --slow is given.
  • One BLAS thread per worker. Otherwise four workers each start a thread per core and are barely faster than one.

Draft tutorials in the test. SMAPPY_TUTORIAL_DRAFT=1 takes screenshots at 1× instead of 2× and saves them as JPEG (120 ms per shot) instead of WebP (550 ms). The published tutorials are unchanged. Most of a tutorial's remaining time is the GUI really working (fits, renders, simulating its data), which this test is there to exercise.

Smaller changes:

  • pytest-xdist added to the test extra.
  • CLAUDE.md: describes the two tiers and when the slow one must run (before a PR, and before major work is called done); asks a PR to state its --slow result; and notes that automatic garbage collection is off in the tests.

Test results

--slow -n auto: 1321 passed, 4 skipped (3:39, 3:44, 3:44 in the last three runs)
-n auto: 1260 passed, 65 skipped (1:24)

🤖 Generated with Claude Code

https://claude.ai/code/session_01NjvC6By9NAqXj9KRfwWdLM

claude added 6 commits October 5, 2026 10:01
The suite took 11:22 on four cores, and 25 tests (61 with their parameters)
were two thirds of it: the twelve tutorials alone 350 s, the pages' figures
50 s, and a handful of numerical checks at 4-19 s each.  They are marked
`slow` and skipped unless `--slow` is given, so an ordinary run is 3:40
serially and about 1:10 with `-n 4`; `--slow -n 4` runs everything in 5:42,
as before a PR.

Under xdist, each worker capped OpenBLAS at one thread: four workers that
each started four took 9:36, barely better than one.

pytest-xdist joins the `test` extra.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NjvC6By9NAqXj9KRfwWdLM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NjvC6By9NAqXj9KRfwWdLM
A GUI test that drops a widget in a reference cycle left it for the cyclic
collector, which runs in whichever thread allocates next -- in the failing
runs, a live fit's TIFF reader thread.  ~QWidget closes its window, and
QWindow::close waits for the GUI thread to flush window-system events; the
GUI thread was the test, waiting for that fit, so the worker hung (py-spy:
QWindowSystemInterface::flushWindowSystemEvents under tifffile in the reader).
That was test_live failing in about half of the parallel --slow runs, and the
300 s timeouts in test_gui_view3d and test_gui_render_axes.

conftest disables automatic collection for the session and collects every
100th test on the main thread, as pyqtgraph's GarbageCollector does; the
fast tier's time is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NjvC6By9NAqXj9KRfwWdLM
Work stealing by default (conftest's pytest_xdist_make_scheduler): xdist's
own scheduler gave one worker the tutorials -- 372 s of tests against ~100 s
for each of the others -- and --slow -n 4 took 6:50; stealing takes 3:45.
With --slow the slow tests run first, so the long ones start early.

The tutorial test runs the storyboards in draft (SMAPPY_TUTORIAL_DRAFT): shots
at 1x and JPEG, a quarter of the pixels and 120 ms to encode where WebP took
550.  The published tutorials are unchanged.

CLAUDE.md: the slow tier is parallel again now that it does not hang, and
says that automatic collection is off in the tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NjvC6By9NAqXj9KRfwWdLM
@jries jries changed the title Tests: a fast tier by default, the slow one behind --slow Tests: a fast tier by default, a reliable 4-minute slow tier behind --slow Oct 5, 2026
@jries
jries merged commit 2915350 into main Oct 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants