test(api): stabilize official workflow lifecycle integration timeout

[seven332]

Problem and evidence

The Turbo test-api (4) job for PR #30093 failed in an Official Workflow
public-boundary integration test even though that PR contains no Turbo changes.

Observed job:

• https://github.com/vm0-ai/vm0/actions/runs/33193045577/job/98923631734
• PR head: 5c6e2f41f1248d03caf3723c8dd19c84314991e4

The failing test is
installs, guards, reconfigures, retires, and uninstalls through public boundaries
in
turbo/apps/api/src/signals/routes/__tests__/official-workflows.test.ts.
It exceeded Vitest's default 5-second test limit and completed at 5.331 seconds.
The shard otherwise passed 409 tests.

Repository inspection shows that this is one roughly 990-line scenario containing
64 sequential awaited boundaries and about 60 route-client calls. At the timeout
boundary, the log had reached the expected late-stage copy-upload failure path, so
there is no evidence that one request, promise, or lifecycle phase failed to
settle. Focused runs have varied from 3.98 seconds to 5.177 seconds, while an
earlier CI run completed the same scenario in 2.156 seconds. The supported root
cause is cumulative integration-test work with insufficient margin under shard
contention, combined with a test boundary too coarse to identify the phase when a
real regression occurs.

Decision

Keep this issue and correct its scope: stabilize and improve the diagnostic
granularity of the Official Workflow lifecycle integration coverage. Do not treat
the failure as evidence of a product lifecycle race unless a focused test exposes
one.

Constraints

• Keep coverage at public API entry points; do not replace it with unit coverage
  or direct internal-service testing.
• Do not increase the timeout or skip coverage.
• Keep each scenario independently isolated and cleanup-safe; do not make tests
  depend on state created by another test.
• Preserve the existing lifecycle behavior. Runtime changes are out of scope
  unless a focused reproduction demonstrates a product defect.

Acceptance criteria

• The oversized lifecycle scenario is reorganized into focused, independently
  diagnosable integration tests that complete within the normal 5-second budget.
• Public-boundary coverage remains for discovery and access guards, input
  validation and concurrent installation, direct-mutation guards,
  reconfiguration, active and retired copy behavior, copy failure compensation,
  uninstall, reinstall, and catalog retirement.
• Every new scenario owns its setup and cleanup and passes in repeated focused
  runs without custom timeouts or skipped assertions.
• The complete Official Workflow test file and the CI-equivalent API shard pass.
• Relevant API formatting, linting, type checking, dependency analysis, and
  tests pass.

Scope

This issue is intentionally separate from Runner GC PR #30093. It covers the
Official Workflow integration-test structure and any product defect proven while
making those public-boundary scenarios deterministic.

[seven332]

Challenge Review: Framing

Checkpoint: framing
Outcome: revise

The issue is necessary, but the original single-phase race hypothesis is not
supported. The failing scenario contains 64 sequential awaited boundaries and
about 60 route-client calls; CI timed out after reaching a late expected failure
path, and focused duration varies across the 5-second threshold.

The issue body now frames the root problem as cumulative work and coarse
diagnostic granularity. Planning should preserve all public-boundary coverage
while creating independently isolated scenarios, without increasing timeouts or
changing runtime behavior absent evidence of a product defect.

[seven332]

Research Phase

Evidence

• The observed CI body timed out at 5.331 seconds after reaching the expected
  late-stage copy archive upload failed path; the log does not identify a
  route or promise that remained pending.
• Historical measurements for the same scenario range from 806 milliseconds on
  prepared current main, through 2.156 and 3.98 seconds, to 5.177 and 5.331
  seconds.
• The failing shard passed 409 other tests.

Current structure

• The scenario occupies 990 lines, contains 64 sequential awaited boundaries,
  and makes about 60 route-client calls.
• It combines 16 behavior groups: access and discovery gates, shared-agent
  ownership, catalog projections, validation failures, concurrent installation,
  installed projections, conflicts, direct-mutation guards, pause and
  reconfiguration, active copy, failed copy compensation, uninstall and
  reinstall, zero-Blueprint lifecycle, retirement, retired copy, stale-copy
  rejection, and final lifecycle cleanup.
• The Official Workflow installations describe is sequential, and the
  file-level beforeEach installs connector-catalog state and cleans catalog
  state.
• API CI uses eight coverage-enabled shards and does not configure a custom
  testTimeout.

Existing adjacent coverage

The next integration test already covers copy publication atomicity, hidden
uncommitted copy state, rejected and late-sibling uploads, object compensation,
and Run-family absence. The oversized scenario separately covers ordinary copy
projection, disabled-automation preservation, autonomy metadata removal, empty
target state after failed copy, retired copy projection, and stale-copy
rejection.

Repository constraints

• Keep API coverage at setupApp() route/contract boundaries with real
  database behavior.
• Mock only external boundaries.
• Give every test independently owned state and cleanup; do not rely on state
  created by another test.
• Do not increase timeouts, skip assertions, or change runtime behavior without
  evidence of a product defect.

Unknowns

The available CI log has no per-request timing, so it cannot separate database
contention, coverage instrumentation, runner scheduling, or other shard load.
No individual lifecycle boundary has been shown to exceed the budget.

The complete research artifact is retained at
codex-work/research/issue-30181-official-workflow-lifecycle-timeout/research.md.

[seven332]

Options Phase

Option A: independent behavior-focused route scenarios

Replace the 990-line chain with independently owned integration scenarios for:

1. access, discovery, installation validation, and conflicts;
2. installed projections, direct-mutation guards, pause, and reconfiguration;
3. active copy projection and failed-copy cleanup;
4. uninstall, reinstall, retirement, retired copy, and stale-copy rejection.

This gives the strongest individual timeout margin and failure localization
while preserving public-boundary coverage. It repeats public setup and cleanup,
so any helper must remain narrow and justified by identical callers.

Option B: redistribute exact coverage and remove duplication

Map every assertion against adjacent Official Workflow tests, move related copy
assertions into the copy-publication scenario, and delete only route work that
proves no distinct external invariant.

This can reduce total runtime and source size, but exact coverage equivalence is
harder to prove. It may also create a new oversized copy test or leave a smaller
but still coarse lifecycle chain.

Option C: preserve one chain and optimize harness overhead

Keep the scenario intact while reusing route clients and reducing setup or
cleanup work that does not represent behavior.

This makes the smallest structural change, but it does not materially improve
failure localization and its performance margin may disappear under
coverage-enabled shard contention.

Comparison

Option A has the best timeout isolation and diagnostics with high coverage
confidence. Option B may achieve the lowest total runtime but has the highest
coverage-audit cost. Option C preserves continuity but does not satisfy the
diagnostic-granularity objective as strongly.

Open planning questions are the exact number of meaningful scenarios, whether
any copy assertions are true duplicates, and whether repeated setup justifies a
narrow helper.

The complete option analysis is retained at
codex-work/research/issue-30181-official-workflow-lifecycle-timeout/innovate.md.

[seven332]

Plan Phase

Goal and direction

Adopt Option A: replace the 990-line lifecycle test with four independently
owned public-route integration scenarios. Keep every distinct external
invariant, real database behavior, and the normal Vitest timeout. Add one narrow
helper for the common installed three-Blueprint state used by three scenarios.

Only
turbo/apps/api/src/signals/routes/__tests__/official-workflows.test.ts
is expected to change. There are no production, API contract, persistence,
deployment compatibility, migration, or CI configuration changes.

Implementation

1. Add a narrow installed-scenario helper that creates a unique active
   definition, actor, agent, feature switches, and installation through existing
   BDD/API boundaries; registers agent/catalog cleanup; and returns only the
   installed state used by callers.
2. Create an access/install scenario covering staff and ordinary discovery
   gates, public/private shared agents, zero-Blueprint install/uninstall,
   catalog projections, all existing validation failures, concurrent
   installation, duplicate installation, second-agent deletion, and ordinary
   workflow name conflict.
3. Create an installed-state scenario covering workflow/automation/workspace/
   autonomy/storage projections, direct workflow and automation guards, pause,
   reconfiguration, identity preservation, and re-enable.
4. Create an active-copy scenario covering disabled reconfigured schedule
   preservation, ordinary copied projection, removal of Official/autonomy
   metadata, injected upload failure compensation, and empty failed target
   state. Leave the adjacent copy-publication atomicity test unchanged.
5. Create a lifecycle-retirement scenario covering uninstall, missing detail,
   reinstall with new identity, catalog retirement, discovery/detail behavior,
   new-install rejection, retained access with the switch disabled, retired
   copy, stale-copy rejection, and final disable/reconfigure/enable/uninstall.
6. Audit the final diff against the original behavior map. Every test must own
   unique state and cleanup, no test may consume state created by another, and
   no timeout or skip may be added.

Validation

Run one Vitest process at a time:

1. Prettier write/check for the changed file.
2. Each new focused test at least three times sequentially.
3. The complete Official Workflow test file.
4. The coverage-enabled API shard containing the file:
   pnpm vitest run --project=api --shard=<n>/8 --coverage.enabled --coverage.reporter=json.
5. pnpm -F api lint.
6. pnpm -F api check-types.
7. pnpm knip --workspace api.

Risks and controls

• Repeated setup may increase total file runtime; keep exactly one narrow helper
  and verify the coverage shard.
• Assertion movement can lose coverage; retain an explicit before/after
  behavior map during review.
• Helper abstraction can hide behavior; expose no speculative configuration and
  keep scenario-specific operations in each test.
• Cleanup errors can leave residue; register cleanup immediately after resource
  creation and keep all identities unique.
• If a focused scenario proves a runtime defect, stop the test-only change and
  return to planning before modifying production code.

Definition of done

• Four meaningful independent tests replace the original monolith.
• All mapped behavior remains covered.
• Repeated focused runs and the coverage shard stay within the normal 5-second
  per-test budget.
• The full file, formatting, lint, type checks, and Knip pass.
• No production code, timeout, skip, contract, migration, or CI change is
  introduced.

The complete implementation plan is retained at
codex-work/research/issue-30181-official-workflow-lifecycle-timeout/plan.md.

[seven332]

Challenge Review: Plan

Checkpoint: plan
Outcome: revise

The independent-scenario direction remains the strongest option: it preserves
route integration coverage, improves timeout isolation, and avoids production
or CI changes. One material coverage gap must be corrected before
implementation.

The original lifecycle scenario has two distinct zero-Blueprint cases. The
shared public-agent case proves ownership/visibility behavior, while the later
same-agent case proves that a second Official definition with zero automations
can coexist with an already installed standard Official Workflow and then be
uninstalled independently. Folding the latter assertion into the shared-agent
case would not preserve that state combination.

Revise the Plan to keep both cases: retain the shared-agent installation in the
access/install scenario and retain the same-agent zero-Blueprint
install/assert/uninstall sequence in the lifecycle-retirement scenario. The
current Plan Phase is superseded and must be reposted before another plan
challenge.

[seven332]

Plan Phase

This Plan supersedes the earlier Plan Phase and incorporates the
plan:revise challenge.

Goal and direction

Adopt Option A: replace the 990-line lifecycle test with four independently
owned public-route integration scenarios. Keep every distinct external
invariant, real database behavior, and the normal Vitest timeout. Add one narrow
helper for the common installed three-Blueprint state used by three scenarios.

Only
turbo/apps/api/src/signals/routes/__tests__/official-workflows.test.ts
is expected to change. There are no production, contract, persistence,
deployment compatibility, migration, or CI configuration changes.

Implementation

1. Add a narrow installed-scenario helper that creates unique catalog, actor,
   agent, feature-switch, and installation state through existing public/BDD
   boundaries and registers cleanup.
2. Create an access/install scenario covering staff and ordinary discovery
   gates, shared public/private agents, the shared-agent zero-Blueprint
   install/uninstall, catalog projections, every existing validation failure,
   concurrent installation, duplicate installation, second-agent deletion, and
   ordinary workflow name conflict.
3. Create an installed-state scenario covering workflow/automation/workspace/
   autonomy/storage projections, direct workflow and automation guards, pause,
   reconfiguration, identity preservation, and re-enable.
4. Create an active-copy scenario covering disabled reconfigured schedule
   preservation, ordinary copied projection, removal of Official/autonomy
   metadata, injected upload failure compensation, and empty failed target
   state. Leave the adjacent copy-publication atomicity test unchanged.
5. Create a lifecycle-retirement scenario covering uninstall, missing detail,
   reinstall with new identity, a second zero-Blueprint definition coexisting on
   the same agent with empty automations and independent uninstall, catalog
   retirement, discovery/detail behavior, new-install rejection, retained
   access with the switch disabled, retired copy, stale-copy rejection, and
   final disable/reconfigure/enable/uninstall.
6. Audit the final diff against the original behavior map. Every test must own
   unique state and cleanup, no test may consume another test's state, and no
   timeout or skip may be added.

Validation

Run one Vitest process at a time:

1. Prettier write/check for the changed file.
2. Each new focused test at least three times sequentially.
3. The complete Official Workflow test file.
4. The coverage-enabled API shard containing the file.
5. pnpm -F api lint.
6. pnpm -F api check-types.
7. pnpm knip --workspace api.

Risks and controls

• Repeated setup may increase total file runtime; keep exactly one narrow helper
  and verify the coverage shard.
• Assertion movement can lose coverage; preserve both distinct zero-Blueprint
  combinations and maintain a before/after behavior map.
• Helper abstraction can hide behavior; expose no speculative configuration.
• Cleanup errors can leave residue; register cleanup immediately and keep all
  identities unique.
• If a focused scenario proves a runtime defect, return to planning before
  modifying production code.

Definition of done

• Four meaningful independent tests replace the original monolith.
• All mapped behavior, including both zero-Blueprint state combinations,
  remains covered.
• Repeated focused runs and the coverage shard stay within the normal 5-second
  per-test budget.
• The full file, formatting, lint, type checks, and Knip pass.
• No production code, timeout, skip, contract, migration, or CI change is
  introduced.

The revised complete plan is retained at
codex-work/research/issue-30181-official-workflow-lifecycle-timeout/plan.md.

[seven332]

Challenge Review: Plan

Checkpoint: plan
Outcome: proceed

The revised four-scenario plan is the best supported direction.

• It directly addresses both demonstrated problems: per-test timeout exposure
  and coarse failure localization.
• It preserves public route/contract integration coverage, real database
  behavior, independent ownership, and the normal timeout.
• The narrow installed-state helper has three concrete callers and introduces
  no speculative configuration.
• Both distinct zero-Blueprint combinations are now retained, closing the
  coverage gap from the superseded Plan.
• Active copy projection remains separate from the adjacent atomic-publication
  test, so neither scenario absorbs unrelated assertions.
• The change remains one independently reviewable, test-only PR with no
  production, compatibility, migration, or rollout impact.

The main residual risk is increased total file work from repeated setup. The
planned full-file and coverage-shard runs test that consequence directly, while
the independent scenario boundaries provide materially larger per-test margin.
No human trade-off or additional delivery slice is required.