feat(deploy): follow a rolling deploy's restart job, and show canary certification

[dawsontoth]

⊙ Problem

Stacked on #1824 (claude/1602-deployments-poll-policy), which stacks on #1821 → #1817 → #1812. Review and merge those first; this diff is only the two commits above #1824. The PRs for #1786 and #1787 stack on this one.

Studio deploys with restart: 'rolling'. The node that receives the deploy answers once its own part is done and hands the other nodes to a restart_service job, named in the response's restartJobId. Studio never followed that job, so a node that rejected or failed the release read as a successful deploy: the post-deploy health check reads the one node Studio reaches, and the deployment record is written only by the receiving node, so it says success while the job runs and after it fails.

From Harper 5.4 (HarperFast/harper#2981, merged in fa9b6f857, not yet tagged) a restarting deploy is also certified in a canary worker, and Studio read none of it: not the certification outcome, not a rejection's decision, and not the warning events a deploy streams when a peer installed something different (5.3.1).

❓ Your call: the false success is real today on every rolling deploy to a cluster, so following the job is warranted now; the 5.4 certification half targets an untagged Harper release and could wait for it. I kept them together because both read the same deploy result and share one reporting path, and nothing in flight supersedes either (#1841 changes the failure text, not when a deploy is done).

💡 Solution

The deploy mutation now follows the restart job before it settles. When the result carries restartJobId, it polls get_job until the job ends: a failed job rejects with a new DeployRolloutError naming each node, and a job Studio could not follow resolves as unfollowable, which no caller reports as success.

What I verified in Harper's source:

• restartJobId exists since 4.6 (release_4.6.9:components/operations.js:449), so following applies to every version that returns it. A response without one behaves exactly as before: no poll, no timer, no event.
• hdb_job is in NON_REPLICATING_SYSTEM_TABLES (harper resources/databases.ts:337), so only the receiving node can answer get_job. Central manager's cluster proxy sends every operation to the first running instance (central-manager src/resources/cluster/Cluster.js:376-379), so over the proxy the deploy and its polls reach the same node. A direct connection to a cluster's address can reach another node, which answers []; a row missing for 20 seconds counts as not-found.
• The job row is written when the job starts and when it ends (harper server/jobs/jobProcess.ts:72-87), so every node's outcome arrives at once. "Show each node as it lands" is not possible by polling; harper deploy restart=rolling exits 0 before the other nodes have taken the release harper#3058 weighs streaming it.
• Two result shapes. A 5.4 certified rollout records result.activated: [{node, certification, message}], or fails with message: {error, activated} (harper bin/restart.ts:232, :235, :258, :260). The plain restart job (4.6–5.3, and 5.4 when no canary was armed) records result.replicated, where a node the job could not reach is listed with its error and the job still completes (:363, :385, :397), so that node reads "restart not confirmed". A finished job in neither shape is unrecognized, never a success.
• certification on success is certified | uncertified | unavailable | not-requested (harper components/operations.js:817, :1073); a rejection fails the deploy with the decision object on both the streamed error event and the buffered body (:1094, :1101-1109, built by certificationFailure at :1222-1247). The copy for an unchecked release comes from Harper's own reference ("Certifying a release in a canary worker").
• warning events ship in 5.3.1 (v5.3.1:components/operations.js:945).

Gating is by what the server sends, not by a version read: certification renders only when the response carries it (only 5.4 does), and following runs only when restartJobId is present. A registration_info gate on the deploy path would hide both from roles with an operations allowlist, which Harper denies that operation (AGENTS.md, "Browse — removing an attribute").

❓ Your call: following the job changes how 4.6–5.3 clusters behave, as #1785 asks ("Following the job applies to older versions too"): the deploy now waits for the plain rolling restart, fails if its job errors, and warns about a node the job could not reach. The alternative is to follow only when the response carries certification (5.4+), which leaves older clusters exactly as they are today and keeps their false successes.

❓ Your call: I couldn't run a two-node 5.4 cluster, so the 5.4 shapes are pinned by fixtures built from Harper's source, not by a live rollout. The job-locality reasoning for direct cluster connections is also untested live.

⚖️ Alternatives

• Read the outcome from the deployment record once Harper records each node's outcome there (A rolling deploy's record says success even when another node rejects the release harper#3060), or have Harper stream the job (harper deploy restart=rolling exits 0 before the other nodes have taken the release harper#3058). Both are open, and every 4.6–5.4 cluster in the field answers only through get_job.
• Address get_job to the receiving node explicitly. The deploy response doesn't name that node, and Studio has no contract mapping a Harper node name to an instance. Detect-and-degrade covers the direct case without one.
• Keep only the health check and render the new fields. That check reads one node, which can be healthy on the release a rejecting node kept, which is the bug.

🔧 Changes

• restartJob.ts (new): readRestartJob normalizes both job shapes; followRestartJob polls with back-off (2s for a minute, then 5s, then 15s, 30-minute ceiling), stops on a 400/403, restarts its not-found grace after a failed poll (the receiving node restarts its own workers before its peers, harper bin/restart.ts:336-355), and passes and rechecks its abort signal; DeployRolloutError.
• deployComponent.ts: deployAndFollowComponent is the mutation function. onDeployComponentSubmit is unchanged, since fix(applications): explain a git deploy with no root package.json #1841 wraps it.
• deployOutcome.ts (new): the certification types and deployFailureDetails, which reads the record id and decision from a streamed, buffered or rollout failure.
• deployComponentStream.ts: the new result fields, warning events kept on the result, and one shared toDeployStreamEvent, which getDeployment.ts now uses too. streamOperation.ts passes the decision to SSEOperationError.certification in errors.ts.
• useDeploymentStream.ts: warnings and the rollout as events, and the certification, decision and record id from what the deploy answered or failed with.
• DeployProgress.tsx with describeDeployOutcome.ts (new): the certification line, a rejection's decision (copy) in place of the raw text, warnings, each other node's outcome, an unknown rollout, a link to the deployment's record, and no "verifying" badge over a rollout that already said so.
• reportDeployHealth.ts: deployNotes, and reportDeployResult for a caller that does not health-check, so both callers on both transports share one policy.
• RedeployApplicationModal.tsx: an unknown rollout settles the panel inconclusive and the modal stays open when there is something to read; a rollout failure says it did not reach every node; the buffered path reports through the shared policy with a record link.
• useImportApplication.ts: the loading toast says it is rolling out, a followed rollout is health-checked on either transport, and a rollout failure is named as one.
• useDeploymentRecordAction.ts (new): the toast action that opens a deployment's record.
• redactRelayedMessage.ts: Datadog withholds DeployRolloutError's message and stack span, which quote each node's error.
• getJob.ts: message is unknown and gains result, because from 5.3.0 a job can record a report object as its message (harper server/jobs/jobProcess.ts:86); getJob takes a signal.
• src/features/instance/applications/DESIGN.md (new), indexed from DESIGN.md: when a deploy is done, and why an unknown rollout is never a success.

Also fixed: a CSV import whose load job Harper refused with its permission report showed [object Object]; it now shows the report's text, and importData.ts reads the message the same way. The type change above forced it.

✅ Verification

• Unit and component tests, with fixtures built from the Harper lines cited above: restartJob.test.ts (both shapes, unconfirmed peers, malformed rows) and its follower tests (back-off, not-found grace and recovery, a miss run broken by failed polls, 400/403, timeout, abort before and during a poll); deployComponent.test.ts (no job means no poll and no timer; success waits for the job; a failed job rejects without deploying again; unknown resolves as unknown; streamed deploys follow the same way); deployOutcome.test.ts (the same decision off the streamed and buffered transports; warnings collected); DeployProgress.test.tsx; useDeploymentStream.test.ts; reportDeployHealth.test.ts; getJob.test.ts; redactRelayedMessage.test.ts.
• Mutation-checked the new guards: disabling the rollout-failure throw, the unconfirmed-peer read, the 400/403 stop, the not-found grace, the in-flight abort recheck, the unknown-rollout branch in the reporter, or the Datadog redaction entry each turns a test red.
• Gates, Node 24.21.0: npx vitest run (403 files, 3,786 tests at a8dc6c8, exit 0; the pre-commit hook ran it again on each commit), npx tsc -b (exit 0), npx oxlint --format stylish . (exit 0), npx dprint check (exit 0).
• End to end: not observable here. There is no 5.4 multi-node rig, and the dev server's port was shared with other sessions; nothing exercises a real rollout or CM-versus-direct job locality.
• Cross-model review: two rounds, codex (graded) and gemini. Round 1's two majors and five minors are fixed in the second commit; round 2 resolved all of them and left one minor (an unused DeployToastOptions export, un-exported in the Stage a release now and activate it later #1787 PR) and a comment nit. The Cursor legs could not run (the 1Password SSH agent refused their git fetch), and the Harper domain leg failed authentication.
• Round 3 (codex + gemini + Harper domain, the first round with adjudication), after the gemini-code-assist fix below: no finding against that fix. It raised findings against the original change, all left open here:
  • major: the rollout now keeps the deploy mutation pending for up to 30 minutes, and React Query only fires mutate()'s own onSuccess/onError while the caller is mounted. Leaving the applications page mid-rollout therefore leaves the "Redeploying…" / "Rolling out…" loading toast up and drops the outcome report (RedeployApplicationModal.tsx:155, useImportApplication.ts:86-112). The suggested fix is reporting from mutateAsync(...).then/catch or from callbacks set on useMutation.
  • minor: a rollout failure toasts twice, because the deploy mutation sets no skipGlobalErrorToast (overlaps Show a failed app import or redeploy once, and keep a failed import beside its form #1871).
  • minor: filter(isRecord) drops non-object entries of a recognized activated/replicated array, so the job could read as complete. Harper does not write that shape today.
  • nits: the unused DeployToastOptions export, narrative comments, and a pre-existing modal-reopen race.

❓ Your call: the unmount major comes from folding the follow into the mutation (mutation-spans-rollout). Fix it here by moving the outcome reporting off mutate()'s callbacks, or follow the job in a separate query?

gemini-code-assist feedback. Fixed: a failed poll did not reset missingSince, so a node restart could end a running job as not-found (a8dc6c8, with a regression test). Declined: capping consecutive failures at 5 and throwing. It would break the reject-only-on-abort contract, and 10s is shorter than a node restart. A 400/403 already stops as refused, and anything else ends timed-out.

Closes #1785

🤖 Generated with Claude Code

Related PRs: #1824 overlaps (base of this stack), #1841 overlaps (wraps onDeployComponentSubmit; same deploy error path), #1806 overlaps (adds another relayed error type to the same Datadog list), #1871 overlaps (the same double-toast on a failed redeploy or import), #1897 overlaps (same deploy-outcome toasts in RedeployApplicationModal and useImportApplication), #1874 overlaps (stacked on this branch), #1876 overlaps (stacked; extends the same outcome modules), #1898 overlaps (stacked; changes when a deploy is resent)

Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; blocked=cursor-grok(no-receipt),cursor-composer(no-receipt); declined=cursor-kimi,cursor-muse; rounds=3; full=1 @ a8dc6c8}

_{Review-Attention: study ~9m (decisions: do-less-alternative, mutation-spans-rollout, any-400-is-refused, unknown-resolves, modal-stays-open-on-warning) @ a8dc6c8}

[gemini-code-assist]

Code Review

This pull request introduces comprehensive tracking and UI visualization for application deployment rollouts and canary certifications. It updates the deployment flow to follow rolling restart jobs, parse streamed warnings, and display detailed per-node outcomes or canary rejection decisions, backed by new unit tests and updated error redaction. Feedback on the implementation highlights a potential bug in the polling loop of followRestartJob where transient network or server errors do not reset the missingSince tracker, which could prematurely trigger a 'not-found' state. It is recommended to reset this tracker in the catch block and bound consecutive failures to handle persistent issues gracefully.

[gemini-code-assist]

During rolling restarts, transient network errors or server-side 5xx errors (e.g., 502 Bad Gateway or 503 Service Unavailable) are common as nodes go up and down. Currently, if a non-deterministic error is caught, the loop continues but missingSince is not reset. This means the time spent during network/server downtime still counts towards the NOT_FOUND_GRACE_MS (20 seconds) limit. If the downtime exceeds 20 seconds, the very next successful poll that returns undefined (not found) will immediately trigger a premature unfollowable('not-found') state, even though we haven't actually had 20 seconds of successful "not found" responses.

Resetting missingSince to undefined in the catch block for non-deterministic errors ensures that the grace period only accumulates during consecutive successful polls where the job is genuinely missing.

Additionally, when implementing polling mechanisms that tolerate transient network or API errors, we should bound the number of consecutive allowed failures (e.g., to 5) so that persistent issues (such as revoked tokens or unreachable servers) are surfaced quickly rather than waiting for the entire timeout period.

Suggested change

	} catch (error) {
	signal?.throwIfAborted();
	// Anything else — a dropped connection, a 5xx while that node's own workers restart — passes.
	if (isDeterministicRejection(error)) {
	return unfollowable('refused');
	}
	}
	} catch (error) {
	signal?.throwIfAborted();
	missingSince = undefined;
	consecutiveFailures++;
	if (consecutiveFailures > 5) {
	throw error;
	}
	if (isDeterministicRejection(error)) {
	return unfollowable('refused');
	}
	}

References

1. When implementing polling mechanisms that tolerate transient network or API errors, bound the number of consecutive allowed failures (e.g., to 5) so that persistent issues (such as revoked tokens or unreachable servers) are surfaced quickly rather than waiting for the entire timeout period.

[dawsontoth]

Fixed the first half in a8dc6c8: a failed poll now resets missingSince (restartJob.ts:176), so the 20s grace only counts an unbroken run of misses. The scenario is real: the job's node restarts its own workers before its peers (harper bin/restart.ts:336-355), so a cluster address reaching another node can read one miss, then errors, and end a running job as not-found. New test: "does not count the time no node answered toward a missing job" (fails without the fix).

Declining the failure cap. A rolling restart is exactly when consecutive failures are expected, and 5 polls at 2s is 10s, shorter than a node restart. The function's contract is to reject only on abort (restartJob.ts:132); callers read an unfollowable rollout rather than catch a throw. The persistent cases it targets are already bounded: a 400/403 stops at once as refused (:172, pollUnlessForbidden.ts:33), and anything else ends timed-out (:180) with polling slowed to 15s after 5 minutes.

_{🤖 Generated by Claude Code (Anthropic Claude Opus 5.5); posted via @dawsontoth.}

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	69.77%	10959 / 15706
🔵	Statements	69.93%	11695 / 16722
🔵	Functions	63.63%	2861 / 4496
🔵	Branches	65.2%	8479 / 13004

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/features/instance/applications/components/DeployProgress/DeployProgress.tsx	84.21%	75.64%	83.33%	84.21%	56, 103-135, 281
src/features/instance/applications/components/DeployProgress/describeDeployOutcome.ts	75%	65.21%	100%	75%	32, 63-67, 79, 83
src/features/instance/applications/components/DeployProgress/useDeploymentStream.ts	76.31%	66.66%	87.5%	77.14%	83, 89, 95-101, 105-110, 122, 142
src/features/instance/applications/components/NewApplication/ImportInstructions.tsx	62.5%	100%	50%	62.5%	33, 91-128
src/features/instance/applications/components/NewApplication/schema.ts	100%	100%	100%	100%
src/features/instance/applications/hooks/useDeploymentRecordAction.ts	16.66%	0%	25%	16.66%	14-19
src/features/instance/applications/hooks/useSupportsDeploymentSSE.ts	25%	50%	33.33%	25%	24-43
src/features/instance/applications/modals/reportDeployHealth.ts	81.81%	69.23%	100%	80.95%	61-65, 70-89
src/features/instance/config/routes.ts	67.39%	100%	51.61%	67.39%	18, 25, 33, 39, 46, 52, 59, 66, 72, 79, 85, 92, 98, 105, 111
src/features/instance/config/deployments/index.tsx	79.16%	81.25%	40%	85.71%	43-44, 48, 49, 53-56
src/features/instance/config/deployments/components/DeploymentDetail.tsx	0%	0%	0%	0%	17-193
src/features/instance/config/deployments/components/deploymentStatusBadge.tsx	66.66%	0%	0%	66.66%	31
src/features/instance/config/deployments/constants/tableDefinition.tsx	33.33%	0%	0%	33.33%	9, 22-34
src/integrations/api/instance/applications/deployComponent.ts	94.44%	100%	75%	94.44%	104-106
src/integrations/api/instance/applications/deployComponentStream.ts	83.33%	61.9%	100%	83.33%	65-69
src/integrations/api/instance/applications/deployOutcome.ts	100%	92.3%	100%	100%
src/integrations/api/instance/applications/restartJob.ts	100%	96.61%	100%	100%
src/integrations/api/instance/database/getJob.ts	100%	93.75%	100%	100%
src/integrations/api/instance/database/importData.ts	0%	0%	0%	0%	28-105
src/integrations/api/instance/deployments/getDeployment.ts	18.18%	40%	28.57%	18.18%	18-22, 28, 62-78
src/integrations/api/instance/deployments/types.ts	100%	100%	100%	100%
src/integrations/api/sse/errors.ts	100%	100%	100%	100%
src/integrations/api/sse/streamOperation.ts	77.77%	75%	55.55%	78.68%	54, 128, 163-182
src/integrations/datadog/redactRelayedMessage.ts	93.75%	92.85%	100%	93.75%	69

Generated in workflow #2212 for commit 034dd9f by the Vitest Coverage Report Action