Repository navigation
Deploy hardening: never leave a project half-deployed, and know what a lost deploy did - #120
Conversation
… image The runner's Docker image pinned flutterflow_cli to 0.0.39. FlutterFlow's AI snapshot now requires >=0.0.41, so every deploy failed workspace init with "This FlutterFlow AI snapshot requires flutterflow_cli >= 0.0.41, but you are running 0.0.39." This was hidden behind the stale allowedSdkPackages allowlist bug (fixed in d1dc19c but never redeployed) until that revision went live and exposed it as the next blocker. Verified: a direct probe against the redeployed runner now gets past both the SDK-package check and the CLI-version check, failing only on API key auth with a deliberately invalid test key.
…ort what landed Measured successful runner deploys took 103-215s, so the 120s browser bound fired on ordinary bundles while the server went on to succeed, leaving the user on "outcome not yet known" with step 3 (pubspec push) never run. Raise the bound to 840s (still under Cloud Run's 900s). When a provisioning response is still lost or expired, re-read the project and report which classes actually landed and what a retry will do, instead of "go check yourself".
The 840s UI bound keeps the live progress view alive on routine deploys, but when a provision response is still lost or expired the app re-reads the project and reports which classes actually landed instead of leaving the user on a dead-end. These browser tests drive the real provisioning transport against a stalled mocked runner at a millisecond bound and render the reconcile outcome through the real terminal renderer, locking in that the timeout is UNCONFIRMED — never a fabricated failure — and that the modal names the project and a next step.
… schedule a canary Cold start dominates deploy latency (103s-215s): the FlutterFlow AI SDK download and a fresh pub get run on every new instance. Pinning min-instances to 1 makes that happen once per container boot instead of per request, buying a much shorter deploy for a small always-on cost. A merged runner fix previously shipped nothing: add a workflow that deploys cloud-run/** changes on main, gated on a service-account secret so pushes before setup skip instead of failing. A daily dry-run canary to automated-test-miedro catches a broken runner before users do.
… probe
A cold runner can be silent for over a minute (SDK snapshot download,
flutter analyze), which the browser cannot distinguish from a dead
connection, and the CLI cap was a flat 5 minutes regardless of bundle size,
so a large deploy could be killed mid-write. This makes the runner's health
observable and its timeout honest:
- emit a {"event":"heartbeat"} every 15s while any CLI work runs. The
channel already drops events for non-streaming callers, and the browser
ignores unknown events, so it is backward compatible.
- scale the deploy cap to 5min + 15s/class (10 min at the 20-class maximum)
while keeping the workspace-init cap at 5 min, so even the largest bundle
stays well under Cloud Run's 900s request timeout.
- add GET /healthz returning the image git SHA (env RUNNER_GIT_SHA), the
installed flutterflow_cli, and the minimum the vendored snapshot requires,
so a merged-but-never-deployed fix is detectable in minutes.
The manifest validation (sdkPackages, dependency maps, constraints) moves
from private helpers in bin/server.dart to public functions in
lib/request_validation.dart, logic byte-for-byte unchanged, so the exact
contract - in particular that flutter_web_plugins and flutter_localizations
are accepted as package names rather than rejected by a stale allowlist -
can be pinned by tests.
…th a pubspec-only push When step 2 (the class write) succeeds but its response is lost or expires, the classes land in FlutterFlow but step 3 (the pubspec.yaml push) never runs, leaving the project unable to build. Now that commit 5231e76 re-reads the project after an unconfirmed provision, extend the terminal: when the reconcile proves every attempted class landed, offer a "Finish deploy" action that pushes the already-merged pubspec (reusing resolveProjectPubspec's output) without re-running the class write. The landed classes are excluded from the finish file map so they are never re-provisioned.
…ayload The deployed runner's _normalizeVerification hard-requires verification.sources to be an array and throws FormatException otherwise, which the handler turns into HTTP 400 before workspace init, the compile gate, or ai validate ever run. The canary omitted sources entirely, so every scheduled run was guaranteed to fail with 'verification.sources must be an array.' - a permanently-red canary that could not tell a healthy runner from a broken one, and that never actually reached the compile gate it claimed to exercise. Include the canary source as the single probe file; a non-empty sources list is also what unskips _verifyCustomCode (server.dart returns null on empty sources), so the compile gate now truly runs on every scheduled check.
…uite green The verification extraction moved request validation into lib/request_validation.dart, but the deployment unit was left two pieces short: - Dockerfile copied only pubspec.yaml and bin/, so the built image could not resolve package:ccc_ffai_runner/request_validation.dart and every endpoint (including GET /healthz) failed to start. Copy lib/ into the image so the package is complete at build time. - The node contract test read only bin/server.dart; the manifest invariants it asserts (toPubspec, the constraint/package patterns, sdkOverrides channeling, legacy-manifest skip) now live in lib/request_validation.dart, and the unavailable symbol was renamed to VerificationRequest.unavailable. Read both sources so the server-side-construction invariants are actually checked, and match the renamed symbol. Reproduced the image startup failure (import not found + Type 'VerificationRequest' not found) with the old Dockerfile layout; both resolve once lib/ is present and the server boots on 8080. Node suite back to 399/399.
test(e2e): cover the slow-runner timeout and reconcile message
runner: heartbeat, health probe, and a per-class deploy cap
fix(deploy): finish an interrupted deploy whose classes landed with a pubspec-only push
Both workflows gated the whole job with `if: secrets.X != ''`. The `secrets` context is not available in `jobs.<job_id>.if` (only github, needs, vars and inputs are), so both files were invalid and the workflows could not run at all - the canary and the merge-deploy would have silently never executed, which is the same class of failure they exist to catch. The gate is now a first step that reads the secret from env and publishes `present`; every step that needs the key skips while it does not. Behaviour is unchanged: before one-time setup runs, the jobs skip quietly and log a notice. Also passes RUNNER_GIT_SHA to the service so /healthz reports which revision is live, and documents it.
…nary infra(runner): keep-warm, CI deploy, and scheduled canary
…al step-3 push The "step-2 timeout ends with classes AND pubspec both present" criterion was only proven at the unit level (isDeployFinishable, buildFinishPushFileMap). This drives the real handlers: provision stalls past a shortened UI bound while the class has landed, reconcile arms the "Finish deploy" terminal, and running that action pushes the merged pubspec (asserted on the intercepted syncCustomCodeChanges request body), issues no second /deployCustomClasses request, and ends the terminal in a finished state.
…me counts Commit 5231e76 made every unconfirmed provision perform one extra project export (reconcileUnconfirmedProvision) to report what actually landed. Six STU-380 transport regression tests counted project exports with exact 'expect(exportCalls).toBe(N)' assertions and were each off by one (or, for the 502 test, still asserted the pre-reconcile message). Update them: - Every count after an unconfirmed provision shifts by the one reconcile re-read; confirmed/success paths add no read, so multi-provision tests (1171, 1282) shift by exactly one, not by one per provision. - The gateway-502 test now asserts the reconcile message (name of the re-read result and the re-read-before-retry advice) instead of the raw 'HTTP 502', while keeping the no-fabricated-'dropped' guard. Proved the tests still bite: with the reconcile read stubbed to a no-op in a scratch copy, exactly these six tests fail (31 passed / 6 failed); the expectations depend on the reconcile read, so removing it fails them. app.js is untouched. Full e2e suite green (262 passed).
test(e2e): finish an interrupted deploy end-to-end (pubspec pushed, class not re-provisioned)
Two failure classes from this work that nothing covered: The image copies only bin/ and pubspec.yaml by hand, and no CI builds it, so a missing COPY is invisible until a real deploy fails. It already happened once when request validation moved to lib/ and the container could not start at all. The guard reads the entrypoint's package: imports, checks each resolves under lib/, and requires bin/ and lib/ to be COPYed. The runner reports minFlutterflowCli at /healthz while the Dockerfile pins the same version separately. Drift would make /healthz assert a minimum the image does not satisfy. They must match.
The runner's request validation is the code that rejected flutter_web_plugins and failed every deploy for months. Its tests existed but only ran locally, so nothing stopped the regression returning. The image is built only at deploy time, so this is also the sole place the runner is compiled before shipping.
fix(e2e/stu380): account for the reconcile re-read in transport outcome counts
…ut a partition Two defects found by running the browser suite against the integrated branch, which neither PR had done. 1. isDeployFinishable crashed the whole deploy. The reconcile re-read can fail or time out, and it then returns the caller's original error, which carries no partition - so `reconciled.reconcilePartition.notLanded` threw a TypeError out of provisionMissingCodeFiles instead of reporting an unconfirmed deploy. An absent partition is an ordinary outcome, and "not finishable" is also the safe answer, since without a proven partition we cannot claim every class landed. 2. Reconciliation discarded the cause. It built a fresh error carrying only what the re-read found, so "HTTP 502" or "the connection dropped" vanished - the one part of the failure a later project export cannot reconstruct. The finding is now added to the cause instead of replacing it; only the generic wait/timeout wording is dropped, because the finding is what replaces it. Verified: 406 node tests, and the full browser suite - 265 passed.
…come is known The last P0 of deploy resilience: when a deploy outlives the browser's wait bound or its connection drops, the only answer the browser had was to re-read the whole project and estimate whether classes landed. This makes the outcome known rather than guessed by giving the runner a per-run record and a status endpoint the client can ask. The client generates the run id and sends it with the deploy request, which is what makes "never arrived" distinguishable from "unknown": a 404 on the status endpoint for an id the client itself minted proves the request never reached the runner, so nothing was written and a retry is safe. On an unconfirmed provision the client polls the status endpoint under a bounded budget and, when the record is definitive (clean done, or a provably pre-write failure), reports the REAL outcome instead of the reconcile estimate; a 404, a 5xx or a malformed body falls back to the existing project re-read, so an older runner keeps working untouched. The record is stored in Firestore in project low-code-connect through the REST API with an OAuth token from the GCE metadata server over dart:io - no runtime pub dependency, given the image build is already slow. The API key is never stored, only its SHA-256 digest (dependence-free, pinned to FIPS-180-4 vectors), and the status endpoint requires the caller to present the same key, so a run id alone is never a bearer capability over --allow-unauthenticated. Writing the record can never fail a deploy: each Firestore call is bounded and its failure logged, not propagated; the terminal write is awaited under a short cap so a client that lost the response reliably sees the run finish. The status endpoint's auth rejection and 404 for an unknown id, the record codec, the never-throw recorder and the client's fallback when the endpoint is absent or 404s are all unit-tested without Firestore.
feat(runner): record deploy runs and serve their status so a lost outcome is known
The catch-all could not reach the recorder: it was declared inside the try block, and a variable declared in a try block is not in scope in its catch — so an unexpected mid-run failure left the record reporting "running" forever, and a later status query answered with a state that was no longer true. That is exactly the unknown outcome the record exists to remove. The recorder is now declared before the try, assigned inside it, and closed out from the catch. Null when the throw preceded the record opening, so the catch cannot itself fail on an uninitialised variable.
…ncurrency 1 Only the write path was timeout-bounded. A lookup covers the metadata-server token fetch and the Firestore read, and if either hangs the handler never returns - and Cloud Run runs this runner at concurrency 1, so that one slot is the whole instance. Every deploy routed to it would wait behind a read nobody is waiting for. The read now has its own 5s bound, twice the write's, and an expired read becomes the 503 the browser already treats as "unknown" and answers by reconciling. The bound is injectable so a test can prove the expiry without a real 5s wait; that test fails if the bound is removed.
…utcome PR #118 made the client generate a run id, send it with the deploy request, and on an unconfirmed provision ask GET /runStatus/<runId> whether the runner recorded a definitive outcome. The mechanism is identity: the id queried must be the id the deploy carried, or the answer belongs to someone else's run. This spec captures both ids and asserts they match in every case, so a spec that stubbed the two independently would pass even if the wiring were broken. Three outcomes are driven through the real __CCC_PROVISION_CUSTOM_CLASSES__ transport with the deploy stalled past a shortened UI bound: (a) status DONE -> provision resolves as a known success, not unconfirmed (b) status FAILED at a pre-write phase (verifying) -> a real failure, nothing written (c) status 404 -> falls back to the existing reconcile and still, reports unconfirmed (backward compatibility) Neutralising the queryRunStatusWithinBudget call in a scratch app.js copy (never committed) flips all three to red on the same identity assertion, which is the bite that proves the spec would catch the wiring being removed.
test(e2e): prove the run-status wiring changes a timed-out deploy's outcome
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| finish.landed, | ||
| finish.serializedYaml, | ||
| ); | ||
| const syncMetadata = await buildApiSyncMetadata(finishFileMap, new Map()); |
There was a problem hiding this comment.
🔴 Finish push loses existing file identities
When a timed-out bundle includes existing files, finishProvisionedDeploy sends them without original checksums. buildFlutterFlowSyncMetadata also marks existing functions as additions, so the remaining sync can fail or duplicate changes.
Learn more
The normal deploy builds metadata using the project's export, which provides the original checksums for existing code files and names of existing functions. The finish action substitutes an empty map despite having just fetched a fresh project export during reconciliation. That changes the wire request for all non-provisioned bundle entries, not just the classes omitted by the finish action.
Example: A bundle contains a new GaugeModel class and an existing read_gauge.dart action. After the class lands and the runner response is lost, Finish deploy sends read_gauge.dart without original_checksum, unlike the normal step-three push.
Recommended fix: Carry the fresh files map from reconcileUnconfirmedProvision into finishDeploy and pass it to buildApiSyncMetadata, or fetch a new project snapshot immediately before finishing; preserve the normal metadata semantics for remaining files.
Was this helpful? React with 👍 or 👎 to provide feedback.
| renderCommitTerminal({ | ||
| ...finish.result, | ||
| success: false, | ||
| error: error.message, | ||
| unconfirmed: error instanceof UnconfirmedDeployError, | ||
| remoteRefusal: error.remoteRefusal === true, | ||
| }); |
There was a problem hiding this comment.
🔴 Rejected finish hides deployed classes
When FlutterFlow rejects the finish push, finishProvisionedDeploy reports FAILED despite confirmed landed classes. The failure modal says nothing was applied, hiding a project left without its dependencies.
Learn more
The finish action exists only after reconciliation confirms that every attempted class is in the project. A failed sync therefore leaves a partial deployment, even if FlutterFlow definitively refuses the sync. The FAILED terminal's guidance tells the user that nothing was applied and a full retry is safe, contradicting the actual class state.
Example: GaugeModel lands, the runner response is lost, and the user clicks Finish deploy. A 400 response to the dependency sync displays “Deploy failed” and “nothing was applied,” although GaugeModel is still in FlutterFlow.
Recommended fix: Return a PARTIAL outcome for definitive finish-push failures, retaining the sync error; reserve UNCONFIRMED for an unknown finish-push outcome and keep the landed-class context in both cases.
Was this helpful? React with 👍 or 👎 to provide feedback.
| /// Records a non-terminal progress update. Fire-and-forget: never awaited, | ||
| /// never able to stall or fail the deploy. | ||
| void recordPhase(String phase, String message) { | ||
| _phase = phase; | ||
| _write({'status': 'running', 'phase': phase, 'message': message}); | ||
| } |
There was a problem hiding this comment.
🔴 Late phase writes erase terminal outcomes
When a phase PATCH finishes after a terminal PATCH, recordPhase overwrites status with running. A lost-response client then sees an unfinished run even though the deploy already ended.
Learn more
All recorder writes use the same Firestore document. Phase writes are launched without waiting for the prior write, while recordDone and recordFailed wait only for their own PATCH. Network completion order can differ from call order, so a delayed progress write can overwrite the terminal state.
Example: The uploading phase PATCH stalls for two seconds, and the done PATCH succeeds immediately. The later uploading PATCH changes the stored status back to running; the browser polls for twelve seconds and still cannot confirm the completed deployment.
Recommended fix: Serialize writes per run and ensure terminal writes cannot be superseded by earlier phases, or use a Firestore precondition/transaction that makes terminal status irreversible. Preserve the bounded, best-effort behavior when a write fails.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (statusOutcome.definitive === "failed") { | ||
| // A provable pre-write refusal: nothing could have been written, so it | ||
| // is a real failure, not an unknown outcome. | ||
| runStatusResolved = true; | ||
| throw new Error( | ||
| statusOutcome.error || "The deploy failed before writing anything.", | ||
| ); |
There was a problem hiding this comment.
🟡 Pre-write refusal disables project caching
When run status confirms a pre-write failure, provisionMissingCodeFiles leaves the earlier uncertain-write markers intact. No class can arrive to retire them, so every later deploy re-exports the project.
Learn more
The catch handler first calls markProjectSourceDirty for all attempted classes before polling run status. A definitive pre-write failure proves none of those classes can land, but this branch exits without decrementing their uncertain credits. resolveProjectPubspec caches an export only after dirty writes clear, and absent files never clear these credits.
Example: A compile gate rejects GaugeModel after the browser times out. The status lookup returns failed at verifying. Every subsequent deploy exports the project again even though no write was possible.
Recommended fix: Retire this attempt's uncertain credits with adjustProjectSourceDirty(apiClient, missingCodeFiles, { uncertain: -1 }) before throwing; ensure a late stream does not credit this resolved attempt.
Was this helpful? React with 👍 or 👎 to provide feedback.
| for (const entry of entries) { | ||
| (remoteFiles.has(entry.path) ? landed : notLanded).push(entry); | ||
| } |
There was a problem hiding this comment.
🔴 Wrong class content qualifies for finishing
When another file appears at an attempted class path, partitionProvisionedCodeFiles counts it as landed without comparing content. Finish deploy then excludes the requested class and can claim success while its code is absent.
Learn more
A project export maps file paths to source text. The missing-class plan uses file presence at the earlier snapshot, but reconciliation uses presence alone to prove that the pending runner wrote the intended class. Another write to the same path, or any different exported content, satisfies this check and enables the finish action, which excludes the class entirely.
Example: A deploy attempts class GaugeModel { final int x; } at gauge_model.dart; before reconciliation, another editor creates gauge_model.dart with class OtherModel {}. The finish action pushes dependencies and reports GaugeModel deployed without ever writing it.
Recommended fix: Compare the exported file contents to the attempted content, accounting for any known FlutterFlow formatting, or rely on a definitive runner record that confirms the specific class and content; treat a mismatched file as unconfirmed rather than finishable.
Was this helpful? React with 👍 or 👎 to provide feedback.
| /// Caps a single deploy invocation, scaling with how many classes it must | ||
| /// upload. A fixed cap would have to be sized to the largest bundle, either | ||
| /// killing a big one or over-waiting a small one; starting at 5 minutes and | ||
| /// growing by 15s per class keeps even the maximum 20-class request at 10 | ||
| /// minutes - comfortably under Cloud Run's 900s request timeout and inside the | ||
| /// browser's 840s wait, with headroom to spare. | ||
| Duration _deployTimeout(int classCount) { | ||
| return const Duration(minutes: 5) + Duration(seconds: 15 * classCount); | ||
| } |
There was a problem hiding this comment.
🟡 Stage caps outlast the deploy request
When initialization and compilation run near their caps, _deployTimeout still gives uploading another five to ten minutes. Cloud Run ends the request at 900 seconds before the runner reports the result, leaving a valid deploy unconfirmed.
Learn more
The new cap bounds only flutterflow ai run; the same HTTP request first waits for _ensureWorkspace and _verifyCustomCode. Those phases can take five, four and four minutes, respectively, before the five-to-ten-minute upload cap starts. The Cloud Run service has a 900-second request timeout and the client stops waiting after 840 seconds, so the advertised headroom disappears when earlier stages are slow.
Example: Workspace init takes 250 seconds, pub get and analysis each take 180 seconds, and a 20-class upload takes 420 seconds. The runner needs 1030 seconds, but Cloud Run cuts it off at 900 seconds.
Recommended fix: Give the whole request a deadline based on the earliest external limit and pass its remaining time to each stage, or constrain individual stage caps so the total stays below the HTTP and client bounds.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # Keep one instance always on. Most of a deploy's 103-215s on a cold start is | ||
| # the FlutterFlow AI SDK download and a fresh `flutter pub get` after `ai init`; | ||
| # a warm instance does that once at container boot instead of per request. The | ||
| # tradeoff is a small always-on instance cost for a much shorter deploy. | ||
| MIN_INSTANCES="${MIN_INSTANCES:-1}" |
There was a problem hiding this comment.
🔍 Warm instances still initialize on first request
--min-instances keeps the container running, but _ensureWorkspace runs during its first request, not at boot. Recheck the expected first-deploy latency benefit.
Was this helpful? React with 👍 or 👎 to provide feedback.
| ), | ||
| runId: runId, | ||
| projectId: projectId, | ||
| keyHash: sha256Hex(utf8.encode(apiKey)), |
There was a problem hiding this comment.
🟥 Run records retain offline-verifiable key hashes
Every deploy stores an unsalted digest of its FlutterFlow API key in Firestore. A reader of that collection can test candidate keys offline and link the user's deploys.
Was this helpful? React with 👍 or 👎 to provide feedback.
| final query = request.uri.queryParameters['key']?.trim(); | ||
| if (query != null && query.isNotEmpty) return query; | ||
| return null; |
There was a problem hiding this comment.
🟨 Query parameters expose FlutterFlow write keys
A caller can pass a FlutterFlow API key in ?key=, putting the credential in the request URL. URL logging and copied links can retain that write-capable key.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const response = await doFetch( | ||
| runStatusUrl(provisionEndpoint, runId), | ||
| { | ||
| method: "GET", | ||
| headers: { [RUN_STATUS_KEY_HEADER]: String(apiKey || "") }, | ||
| signal: controller.signal, | ||
| }, |
There was a problem hiding this comment.
🟨 Status lookup sends keys over HTTP overrides
With an http:// runner override, fetchRunStatus sends the FlutterFlow API key in a request header without TLS. The key can be intercepted in transit.
Was this helpful? React with 👍 or 👎 to provide feedback.
CodeQL flagged the id generator's fallback as an insecure random in a security context. The id names a Firestore document, so a guessable one could be overwritten by someone writing to a run they do not own - and the id exists to tell "never arrived" apart from "unknown", which a predictable value undermines. The fallback is now crypto.getRandomValues, and where no cryptographic source exists at all it returns null instead of a weak id. A null id is the safe answer: the runner keeps no record and the deploy behaves exactly as it did before this feature, rather than recording under a guessable key.
Eleven fixes to the FlutterFlow deploy path, driven by a live bug: a stale runner image was refusing every deploy with
Unsupported SDK package: flutter_web_plugins.even when the project and the pushed code never mentioned it.The live blocker (already shipped)
The runner running in production was the 2026-09-21 image, whose hardcoded SDK allowlist omitted
flutter_web_plugins. The fix had been merged ind1dc19cbut never deployed, so every deploy failed for months. Redeploying it exposed a second hidden fault: the image pinnedflutterflow_cli0.0.39 while FlutterFlow now requires 0.0.41, which would have failed every deploy anyway. Both are fixed (e5778bc); production is on revision00018-jqf.What this branch adds
Never leave a project half-deployed
GET /runStatus/<id>answers it. A definitivedonenow makes a timed-out deploy a known success instead of a guess; a 404 proves the request never arrived. The API key is never stored — only its SHA-256 digest, and a reader must present the same key.Slowness and silence
--min-instances=1, since the SDK download dominates a cold start.Not repeating the stale deploy
GET /healthzreturns the git SHA, installedflutterflow_cli, and the required minimum. The deploy passesRUNNER_GIT_SHA.deploy-runner.ymldeploys the runner whencloud-run/**lands on main.automated-test-miedroand fails on the SDK-package rejection specifically.Test gaps
sdkPackages: ["flutter_web_plugins"]is accepted — the exact regression — now run in CI.Defects found while integrating (no single PR caught these)
if: secrets.X != ''. Thesecretscontext is not available injobs.<id>.if, so both files were invalid and neither workflow would ever have run.isDeployFinishablethrew aTypeErrorout of the deploy path whenever the reconcile re-read failed.concurrency=1one hung lookup occupies the instance's only slot.5231e76broke six existing e2e tests; repaired with the counts traced to the code path.Verification
420 node tests · 28 Dart tests + clean
dart analyze· 268/268 browser tests · deploy guard 9/9 · proxies 30/30 · accessibility · bundle leak scan clean. Several new specs were mutation-tested: the finish-deploy and run-status specs fail when the behaviour they claim to protect is removed.Consequences of merging
Merging triggers
deploy-runner.yml, which redeploys the Cloud Run runner, putting the heartbeat,/healthz, run-status and per-class timeout into production. The canary starts running daily.Known follow-ups (not done here)
Run records have no TTL — negligible cost, but expiry needs a one-off Firestore policy. The 404 "never-arrived, a retry is safe" signal is available but unused: the client still degrades to a safe re-read rather than retrying a write on the user's behalf.