Speed up the backend test suite, which has grown from about 5 to 20 minutes #1481

Summary

The backend test workflow on main has slowed from about 4.4 minutes in September 2025 to about 19.5 minutes now. Over the last three months the number of tests on main grew only from 627 to 760, so most of the slowdown comes from each test getting more expensive, not from having more tests. Slow CI delays every review round, and the cost compounds: new tests, including the many written by AI agents, copy the nearest existing test, so an expensive fixture pattern spreads with every PR.

This ticket collects what was measured on 2026-10-06 and suggests fixes, largest first. Each fix can be its own small PR.

What was measured

pytest reports a unittest setUp as part of each test's run time rather than as setup, so the fixture cost does not show up as "setup" in its output. The per-call numbers above come from timing the helpers on their own.

Likely causes and suggested fixes, largest first

Savings are rough estimates from the measurements above, not measured after a fix.

1. Fixtures rebuilt before every test. Most classes call setup_test_project() and create_captures() in setUp, so about one second is paid before every test, and ami/main/tests.py has 85 such calls. Move them to setUpTestData, class by class. Since Django 3.2 each test gets its own deep copy of the objects created there, and the database is rolled back after each test, so tests stay isolated. Objects that cannot be deep-copied (API clients, open files, mocks) stay in setUp. Two files in New home for algorithm results that are not species classifications #1461 went from about 26 s to under 10 s this way (commit 30cc9b7).
2. setup_test_project() registers a processing service with HTTP calls. create_test_project() calls create_processing_service(), which calls get_status() and create_pipelines() against the test ML backend; that is most of the 0.64 s. Suggestion: make it opt-in (for example setup_test_project(with_processing_service=False)), and have the tests that need pipelines ask for them. An alternative that keeps current behaviour: register pipelines from a saved copy of the minimal service's /info response, with no HTTP.
3. TransactionTestCase without available_apps. All 8 such classes are in ami/jobs/tests/test_tasks.py. After each test Django empties every table and then fires post_migrate, which re-creates permissions and content types and runs ami.users.signals.create_roles for every project. Those tests take about 4 to 6 seconds each in a full run. Where a test does not need real commits across threads or connections, switch it to TestCase (and use captureOnCommitCallbacks for on_commit hooks). Where it does, set available_apps, which limits the flush and skips post_migrate.
4. Real network retries with backoff. TestProcessingServiceLastSeen::test_get_status_updates_last_seen_for_sync_service takes 24 s and TestListEndpointHeartbeat::test_task_updates_all_project_async_services 12 s, waiting on retries to a host that does not resolve (backoff_factor=2 in ami/ml/models/processing_service.py and ami/utils/requests.py). Mock the HTTP call, or make the backoff a setting that is zero in config/settings/test.py.
5. Image-size lookups against storage. create_captures() makes captures with file paths that were never uploaded, and SourceImage.get_dimensions() asks storage for each one (1,284 failed lookups in one run). Set width and height in the fixture, or skip the lookup when the test captures are created.
6. Order-dependent tests. Under pytest, which orders tests differently, three tests fail that pass under manage.py test: ami/users/tests/test_tasks.py::test_user_count (5 users instead of 3), and DataExportTest.test_json_export_record_count and DataExportTest.test_export_record_count (occurrences_api_json). Data from another test appears to leak into them. Fixing these is also a prerequisite for running tests in parallel.

Keeping it from growing back

• Add a "Definition of Done" item to CLAUDE.md and .agents/AGENTS.md: new test classes use setUpTestData; no TransactionTestCase without a reason and available_apps; no real HTTP in tests.
• Print the slowest tests on every CI run (pytest --durations=25, or manage.py test --durations after the move to Django 5), so a new slow test is visible in the PR that adds it.
• Later, parallel test runs (manage.py test --parallel). Known trap: django-cachalot keys its cache by connection alias, so parallel runs sharing one Redis can serve each other's cached queries; give each worker its own Redis database or key prefix.

What we still need to verify

• The saving from each fix, measured with the same --durations=0 run before and after.
• Which tests depend on the processing service that setup_test_project() registers today (run the suite with it turned off and see what fails).
• Whether the TransactionTestCase classes need real transactions; the concurrency race tests probably do, the others may not.