docs(flaky-tests): add the test collections concept and migration pages by TylerJang27 · Pull Request #322 · trunk-io/docs2

TylerJang27 · 2026-09-09T15:36:51Z

Preview:

Overview

PR 1 of the Test Collections docs stack: the concept page (rewritten in place) and the new migration guide, both hidden: true, plus the CLI reference's collection flags.

Details

The stack lands bottom-up:

This PR — net-new collection pages, hidden.
Setup instructions — the ~45 framework and CI-provider pages gain the collection flag.
Stragglers — quarantining, ticketing, webhooks, PR comments, the collection/repo/variant distinction, the API reference.
Dashboard — replaced for the collections state.
Flip — remove every hidden: true and add the nav entries together.

Nothing in this PR is customer-visible except the CLI reference additions, since that page is already live.

Considerations

The concept page was a hidden stub, and wrong in six places. It said only admins can create collections (any org member can — create gates on org read, only edit/delete/quarantining are admin), used --test-collection-short-id (a Rust struct field name that was never a flag), named three of the six tabs while duplicating two of them, implied repositories are picked at creation (they're derived from uploads), and called the identifier a "short ID" where the product says Collection ID.
UI labels were taken from the source, not guessed — tab names, the Create collection dialog and button, and the four checklist titles ("Send your first upload", "Ingest your test results", "Review flake detection", "Review quarantining") are verbatim. The navigation paths are the exception: I could not walk the live product, so per CONTRIBUTING.md they're flagged rather than asserted. The concept page says "Open the Test collections view in Flaky Tests" instead of a full arrow path. Worth someone confirming the real path and tightening it.
One blank is deliberate and marked TODO. "What a new collection starts with" needs the list of seeded default monitors and quarantine settings from TRUNK-19408 — which monitors, at what thresholds, and whether auto-quarantine is among them. A reader deciding what to add first has to see what is already watching, so the section isn't publishable as prose alone. Everything else on the page is verified.
The page asserts seeding that isn't in main yet. Both new pages are hidden, so this is consistent with the plan's "written as shipped" ground rule, but it is a real dependency: the concept page is wrong if seeding doesn't ship.
The migration guide's quarantining order is the opposite of the TRD's. The TRD specified staging overrides before enabling quarantining; that was never built — the override control is disabled while the collection's setting is DISABLED — so the guide says enable-then-re-apply and carries a Warning about the window where the collection governs with an empty override set. The TRD is corrected in trunk-io/trunk2#5830.
No --no-repo anywhere, and no mention of the internal migration states — the guide says only that new organizations start on collections and existing ones migrate gradually.
Screenshots and diagrams are TODO comments, not broken <Frame>s. Four assets across the two pages, deliberately kept to two per page. They need an org with both views open, which is also the state most readers will be in.

Testing

Every internal link target confirmed to exist as a file, and the #test-collections anchor the concept page points at matches the section added to the CLI reference.
Callout types picked against CONTRIBUTING.md's decision rule, with none stacked back to back.
docs2 has no lint or formatter configuration, so there was nothing to run. Not previewed with mint dev — worth doing before this leaves draft, particularly for the <Steps> and <AccordionGroup> blocks in the migration guide, which are new to this page.

Follow-up Work

The seeded-defaults list (TRUNK-19408) has to fill the TODO before this leaves draft.
Four screenshots/diagrams to capture.
PRs 2–5 of the stack, as above.

References

trunk-io/trunk2#5830 — fixes the in-product snippets to emit --test-collection-id alongside --org-url-slug, and corrects the TRD's quarantining order
docs/trd/test-collections-trd.md in trunk2 — Phase 1, Phase 5, and the customer migration playbook
TRUNK-19408 — seeds default monitors and quarantine settings at collection creation

🤖 Generated with Claude Code

Both new pages ship `hidden: true` with no navigation entry, so they can be reviewed in full without being live. A later PR removes the flags and adds the nav entries for the whole set at once. `get-started/test-collections.mdx` existed as a hidden stub and was wrong in six places: it said only admins can create collections (any member can), used `--test-collection-short-id` (a struct field name, never a flag), named three of the six tabs while duplicating two of them, implied repositories are chosen at creation, and called the identifier a "short ID" where the product says "Collection ID". `migrate-to-test-collections.mdx` is new: the customer-facing half of the migration, written for an organization that has both views open. The CLI reference gains `--test-collection-id` and `--hide-test-collection-links` with their environment variables, plus a Test Collections section. It is an already-live page, so unlike its two neighbours here its additions are visible immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

mintlify · 2026-09-09T15:36:55Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
trunk	🟢 Ready	View Preview	Sep 9, 2026, 3:38 PM

TylerJang27 · 2026-09-09T15:51:32Z

+## Collections and repositories

-1. Navigate to **Flaky Tests** → **Collections** in the Trunk web app.
-2. Click **Create Collection**.
-3. Enter a **Name** and optional **Description**.
-4. Click **Create collection**.
+Collections and repositories are many-to-many. A repository is part of a test's identity, so the same test file in two repositories is two different test cases. That test can be uploaded to more than one collection.


This is a little opaque. We should reframe this to capabilities/examples language. For example:

Collections and repositories are many-to-many. A collection can include tests from multiple repositories, and a repository can be broken into multiple collections. This is decided by which collection you specify at upload time.
(Include the diagram below (should be SVG probs) and remove the "You never pick a collection's repositories" piece too.

TylerJang27 · 2026-09-09T15:52:19Z

+
+## What a new collection starts with
+
+A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.


Suggested change

A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.

A new collection starts with a basic set of flake-detection monitors, so detection starts working as soon as test results arrive.

TylerJang27 · 2026-09-09T15:52:40Z

+
+A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.
+
+{/* TODO: enumerate the seeded defaults — which monitors, at what thresholds, and whether auto-quarantine is on among them. Source: TRUNK-19408. A reader deciding what to add first needs to see what is already watching, so this section is not publishable as prose alone. */}


This hasn't been implemented it, so just remove this bit. It may change over time too

TylerJang27 · 2026-09-09T15:55:00Z

+4. **Review quarantining**

-Only organization admins can edit collection settings.
+The first two complete on their own once uploads are landing. The last two are decisions, and you complete them by making one — including deciding against: **Keep defaults** and **Not using this** both count as done. The checklist shows how many of the four are complete, and stops prompting you once none are outstanding.


Suggested change

The first two complete on their own once uploads are landing. The last two are decisions, and you complete them by making one — including deciding against: **Keep defaults** and **Not using this** both count as done. The checklist shows how many of the four are complete, and stops prompting you once none are outstanding.

The first two complete on their own once uploads are flowing. The rest can be configured in the app.

TylerJang27 · 2026-09-09T15:55:12Z

+<Info>
+Splitting "upload" and "ingest" into separate steps distinguishes "CI hasn't run yet" from "the upload arrived but contained no test results" — usually a test reporter that isn't writing its output file, or a path that doesn't match what the CLI was given.
+</Info>


Remove this block

TylerJang27 · 2026-09-09T15:56:20Z

+og:title: "Migrating to test collections in Trunk"
+hidden: true
+---
+[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you migrate to them at your own pace, and both views stay available while you do.


Suggested change

[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you migrate to them at your own pace, and both views stay available while you do.

[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you will need to gradually migrate to using collections, and both views stay available while you do.

TylerJang27 · 2026-09-09T15:56:54Z

+<Note>
+The first thing you'll notice is that Flaky Tests now opens on collections rather than your repositories. That's the migration starting, not a setting you lost. **Legacy view** is one click away.
+</Note>


Remove this block

TylerJang27 · 2026-09-09T15:57:44Z

+This cuts both ways, and both halves matter:
+
+- Detection thresholds you have tuned on a repository have to be re-created on the collection you want them on.
+- A new collection is not a blank slate. It [starts with seeded default monitors](./get-started/test-collections#what-a-new-collection-starts-with), so detection is running on it from its first upload.


This is AI-heavy language. Please simplify this and instead say something like "You may wish to create different monitors for each collections, such as more sensitive flakiness detection for unit tests than e2e tests"

TylerJang27 · 2026-09-09T15:58:13Z

+<Step title="Add the collection ID to one CI job">
+Add `--test-collection-id <COLLECTION_ID>` to the job's existing upload step, alongside the `--org-url-slug` it already passes. Start with one job rather than all of them.
+
+{/* TODO(diagram B): Two CI jobs, one passing the collection ID and one not, landing in different places. Light/dark pair. */}


Remove this block

TylerJang27 · 2026-09-09T15:59:18Z

+While a repository and a collection both have monitors covering the same tests, they detect independently. That's the point — it's how you compare the two before committing — but it means:
+
+- A test can be flagged in both views, at different times, according to each one's thresholds.
+- If you use webhooks, one detection can produce two events. Collection webhook events are off until you turn them on, so nothing reaches your consumers until you're ready. See [Webhooks](./webhooks/).


We're missing the details about the new setting for webhook emission for collections that Max was working on. That should be included briefly here.

…ook gating Review follow-ups on the concept and migration pages. The many-to-many section is reframed in capabilities terms and now carries the diagram as a single theme-aware SVG rather than a light/dark PNG pair — one file, with its own `prefers-color-scheme` block, so it follows the page rather than needing two assets. The seeded-defaults section no longer tries to enumerate what a new collection gets, since that isn't implemented yet and will move. Cut: the upload-vs-ingest aside, the paragraph restating the permissions table, the "you never pick a collection's repositories" paragraph, the note about Flaky Tests opening on collections, and the upload-routing diagram. The migration guide's "nothing carries over" section drops the both-halves framing for the practical point — you may well want different monitors per collection — and the collection-creation step loses the same register. New section on webhook gating while migrating: collection events are off until you switch them on from the organization's webhooks settings, it is one switch for all collection events, and consumers cannot deduplicate on `test_case.id` because the two scopes don't share an ID space. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

TylerJang27 · 2026-09-09T16:09:31Z

All eleven applied in 5d6ce5f. Notes on the four where I did something other than exactly what was written:

Diagram → one SVG, not a light/dark pair. It carries its own prefers-color-scheme block, so a single file follows the page theme and there's no second asset to keep in sync. Rendered and measured it rather than eyeballing: the mono labels clear their cards by 16px and nothing exceeds the viewBox.

"Remove this section" at test-collections.mdx:106 — I read that as the paragraph restating the permissions table, not the whole Permissions section, since the table above it carries the same information and you anchored on the sentence. The table stayed. Say the word if you meant the section.

Webhook gating — the column (testCollectionWebhooksEnabledDuringMigration) is in the schema, but no UI reads it yet, so I couldn't confirm the control's label. Per CONTRIBUTING.md I described the behaviour and its location generically ("your organization's webhooks settings") rather than inventing a label. Worth tightening once Max's UI lands. The section covers: off by default while migrating, one switch for all collection events, and the thing consumers actually get wrong — you can't deduplicate on test_case.id, because the two scopes don't share an ID space.

Diagram B removed per your comment on :53, so the upload-routing diagram is gone rather than becoming an SVG. That leaves two assets total: the SVG here, and one screenshot per page.

Also dropped the seeded-defaults enumeration entirely rather than leaving the TODO — which resolves what had been the page's one blocking gap.

…on label All four screenshots now have their `<Frame>` blocks, with light/dark pairs following the existing convention. **The PNG files themselves are not in this commit** — they were shared as images rather than files, so the six paths referenced here need dropping in before the previews render. The cross-links are two frames rather than one composite, so the header crops can land as shot without anyone compositing them. One label correction the screenshots surfaced: the collections list button is **Create Collection**, while the dialog's submit button is **Create collection**. The draft used the dialog's casing for both. Confirmed against `collections-list-client.tsx:172`. The navigation path is also tightened now that the breadcrumb is confirmed as "Collections". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Six PNGs behind the `<Frame>` blocks added in the previous commit, as light/dark pairs. Each file was matched to its slot on dimensions, then verified two more ways: mean luminance for the theme, and a pixel probe of the right-hand band for the filled Create Collection pill, which appears only in the collections-list pair and so tells the two header crops apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

max-trunk · 2026-09-09T18:07:09Z


+## Test Collections
+
+[Test collections](/flaky-tests/get-started/test-collections) group tests so they can be configured and reviewed together, with their own flake detection and quarantining. To upload into one, pass its ID with `--test-collection-id`:


So the 'hidden' test collections page will actually be immediately accessible through this link?

Yeah the in-app one will be viewable as soon as someone passes a collection id. It'll also be switchable to once the enum is in the migration state, with Dylan's recent pages.

If you're talking about those docs, those are only hidden rn and we can point people to them, but I'll make them public once we have the new org enum state updated

trunk-io · 2026-09-09T18:35:33Z

😎 Merged manually by @TylerJang27 - details.

mintlify · 2026-09-09T18:59:31Z

Docs PR opened: #328

Added a September 9 changelog post announcing Flaky Tests test collections, with index and navigation updates.

mintlify Bot deployed to staging September 9, 2026 15:38 View deployment

TylerJang27 commented Sep 9, 2026

View reviewed changes

mintlify Bot deployed to staging September 9, 2026 16:11 View deployment

mintlify Bot deployed to staging September 9, 2026 17:02 View deployment

mintlify Bot deployed to staging September 9, 2026 17:15 View deployment

TylerJang27 requested review from acatxnamedvirtue, dfrankland and max-trunk September 9, 2026 17:18

TylerJang27 mentioned this pull request Sep 9, 2026

docs(flaky-tests): add the collection id to every setup instruction #323

Draft

TylerJang27 marked this pull request as ready for review September 9, 2026 17:48

max-trunk approved these changes Sep 9, 2026

View reviewed changes

This was referenced Sep 9, 2026

docs(flaky-tests): make collections the default in the straggler copy #324

Draft

docs(flaky-tests): replace the dashboard's landing page for collections #325

Draft

TylerJang27 marked this pull request as draft September 9, 2026 18:39

TylerJang27 marked this pull request as ready for review September 9, 2026 18:39

TylerJang27 merged commit 2d419a4 into main Sep 9, 2026
3 checks passed

mintlify Bot mentioned this pull request Sep 9, 2026

Draft changelog: Flaky Tests test collections #328

Open


		## What a new collection starts with

		A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.

	A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.
	A new collection starts with a basic set of flake-detection monitors, so detection starts working as soon as test results arrive.


		A new collection is not empty. Trunk seeds it with default flake-detection monitors and default quarantining settings, so detection starts working as soon as test results arrive.

		{/* TODO: enumerate the seeded defaults — which monitors, at what thresholds, and whether auto-quarantine is on among them. Source: TRUNK-19408. A reader deciding what to add first needs to see what is already watching, so this section is not publishable as prose alone. */}

	The first two complete on their own once uploads are landing. The last two are decisions, and you complete them by making one — including deciding against: Keep defaults and Not using this both count as done. The checklist shows how many of the four are complete, and stops prompting you once none are outstanding.
	The first two complete on their own once uploads are flowing. The rest can be configured in the app.

	[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you migrate to them at your own pace, and both views stay available while you do.
	[Test collections](./get-started/test-collections) replace repositories as the way Flaky Tests is organized. New organizations start on collections. If your organization has been using Flaky Tests already, you will need to gradually migrate to using collections, and both views stay available while you do.


		## Test Collections

		[Test collections](/flaky-tests/get-started/test-collections) group tests so they can be configured and reviewed together, with their own flake detection and quarantining. To upload into one, pass its ID with `--test-collection-id`:

Uh oh!

Conversation

TylerJang27 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Considerations

Testing

Follow-up Work

References

Uh oh!

mintlify Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TylerJang27 commented Sep 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trunk-io Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

mintlify Bot commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

TylerJang27 commented Sep 9, 2026 •

edited

Loading

mintlify Bot commented Sep 9, 2026 •

edited

Loading

trunk-io Bot commented Sep 9, 2026 •

edited

Loading