feat(config): fan cluster config saves out with replicated set_configuration on Harper 5.2+

[dawsontoth]

⊙ Problem

Saving the config editor on a cluster writes set_configuration to every RUNNING instance from the browser, one at a time, right before restarting it. Instances Studio does not restart (STOPPED, or failing their health check) never get the write and silently keep the old config. Harper can now fan the write out itself with replicated: true (HarperFast/harper#1556). That landed in 5.2.0, not "v5+" as the issue says: it is in no 5.0.x or 5.1.x tag. The write still applies nothing until each node restarts, so the one-at-a-time restart roll has to stay.

❓ Your call: is it worth keeping a second code path for this? It saves N−1 browser writes and reaches members the roll skips. The cost is that, once the write has fanned out, a Cancel leaves the new config saved but not applied on the instances that were not restarted. I think it is worth it, because the fan-out only happens after one instance has come back up on the new config. Reverting is a one-line gate change (shouldFanOutSetConfiguration).

💡 Solution

On a cluster where every live instance reports Harper ≥ 5.2.0 and at least one other live member exists, the roll changes in one place:

1. The first instance is written and restarted exactly as before. It is the canary.
2. Once it is back up on the new config, Studio sends one set_configuration with replicated: true through that instance's client. This happens even when it was the last instance to restart, so live members the roll skips still get the write.
3. A later instance skips its own write only when the fan-out's replicated[] acknowledged that exact node. Fabric names each Harper node after its CM replicationFqdn (host-manager sets NODE_HOSTNAME and REPLICATION_HOSTNAME to it). Each successful entry carries the peer's own message plus the node harper-pro's replicateOperation attaches.
4. Restarts stay strictly sequential, using the same restart_service call and recovery wait as before.

Every other case keeps the per-instance write: pre-5.2, mixed-version, or unknown-version clusters; per-instance views; local Studio; a rejected fan-out (the instance client already turns any failed peer into a rejection); a response with no usable replicated[]; and a target with no replicationFqdn.

⚠️ Look hardest: the acknowledgement check. It is what lets an instance restart without Studio writing to it. It trusts only a non-empty message string plus a non-empty node name, compared case-insensitively against replicationFqdn. The wire shape is traced from source (harper-pro replication/replicator.ts:897-941, harper server/serverHelpers/serverUtilities.ts:163-165, host-manager src/templates/compose.yaml:44-45), not observed on a live 5.2 cluster.

Also fixed (same hook):

• If loading the cluster's instances failed, Save stayed disabled, the loading toast kept spinning, and the rejection went unhandled. It now reports that nothing changed and clears the pending state. The same shape in useRestartClusterClick is filed as Restart Cluster stays disabled and its toast spins when the cluster fetch fails #1853.
• A cluster with no instances reported Success. It now reports that no instances were found.

⚖️ Alternatives

❓ Your call: the planning round (codex) returned better-alternative-exists, and I adopted it. My first plan let later instances skip their write when the number of acknowledgements covered the remaining targets. That is unsound: the origin's replication peers are not CM's restart targets, so a stale peer can stand in for a missing one. The shipped design matches by node identity, per target.

• Fan out first, then roll restarts only, the smallest change. Rejected: the new config would be saved to every node before any node had shown it can boot with it. A Cancel after the first instance fails to come back would then leave that config pending everywhere. Harper's own comment names this hazard (harper-pro#558, config/configUtils.ts:1320-1324).
• Let Harper roll the restarts too (restart_service with replicated: true). Rejected: it also writes everywhere before any restart. It runs as a job Studio does not poll, which would drop per-instance progress and Cancel. And it is no stronger at reporting a peer it could not reach (bin/restart.ts:360-364).
• A Harper-side staged config with activate/rollback would give an exact canary without the duplicate origin write. That needs a server release, so it is out of scope.

🔧 Changes

• src/hooks/useRollingConfigUpdate.tsx (best read with whitespace ignored, since the body moved into a try): planRollingConfigUpdate builds the restart targets and their replicationFqdns, as the inline code did before. Also: the fan-out with its rejection mapped to "acknowledged nobody", the per-target write skip, and the shared "may already be saved" note used in the Cancel copy and the partial-failure copy once a fan-out was attempted.
• src/integrations/api/instance/status/setConfiguration.ts: shouldFanOutSetConfiguration requires more than one live member, with every live member on ≥ 5.2.0. Terminating, terminated, and removed instances are ignored. A missing version, or a 5.2.0 prerelease, counts as unsupported.
• src/integrations/api/replication.ts: adds acknowledgedReplicationNodes, and widens a success entry's message type to the string that set_configuration peers actually send.
• DESIGN.md: records the invariant: when an instance may skip its own write, and why a count is not proof.

✅ Verification

End-to-end route: not verified against a live Harper 5.2+ cluster. No cluster harness is available here, and port 5173 belongs to another session. The Harper side is traced from source, cited above. Mechanism-level evidence:

• src/hooks/useRollingConfigUpdate.test.tsx (new, jsdom) drives the real hook with mocked transport. It covers: the canary → fan-out → acknowledged-skip order; strictly one restart in flight, held open with deferred promises; a stale peer standing in for a target, which still gets written; a missing replicationFqdn; a rejected fan-out, where the canary is still counted and no "Failed" toast appears; fan-out deferred past a failed first instance; fan-out after the last restart; both Cancel copies; mixed, missing, prerelease, and pre-5.2 versions never sending replicated; the per-instance view; an empty cluster; and a failed cluster load.
• src/integrations/api/instance/status/setConfiguration.test.ts (new) pins the gate, including exactly 5.2.0 and a single live member.
• src/integrations/api/replication.test.ts covers acknowledgements: lower-casing, failed peers, and malformed shapes (no array, node-only entries, and non-string, empty, or boolean messages).
• Fails on base: with origin/stage's hook swapped in, 8 of the 16 hook tests failed at the time, including the failed-cluster-load test, and the legacy-path tests passed.
• Mutation check: 22 single-guard mutations, each run against all three files, and all 22 went red. They covered each skip and fan-out guard, count-based coverage, the version, live-member, and prerelease gates, acknowledgement validation and case-folding, both pending notes, empty-cluster success, and the plan catch.
• Gates on the final commit: the pre-commit hook ran the full vitest suite (397 files, 3699 passed, 11 skipped), plus oxlint and dprint check --staged. Separately, npx tsc -b exited 0.

Cross-model review: three rounds (codex graded + gemini each round), after a planning round. Cursor legs could not run, because the 1Password SSH agent refused to sign their git fetch. The Harper-domain adjudicator could not run either: its Claude OAuth session had expired. So outside findings were triaged by hand. I rejected gemini's claim that wasAReleasedBeforeB is a strict comparison: it returns b >= a, and a test pins exactly 5.2.0. Its comment nits pointed at pre-existing comments that only moved under the new try.

gemini-code-assist feedback: took its one comment in 3581be6, so a cancel now breaks out of the update loop rather than continue-ing through the remaining targets. The flag never resets, so behavior is unchanged and the cancellation tests still pass. No further review round ran.

Closes #1339

🤖 Generated with Claude Code

_{🤖 Generated by Claude Code (Anthropic Claude); posted via @dawsontoth.}

Related PRs: #1867 overlaps (stacked on this branch; replaces its target planning with a shared helper), #1899 overlaps (reworks the same config-update loop in useRollingConfigUpdate.tsx)
Complexity: complicated

_{Review-Coverage: authored=claude; ran=gemini,codex; blocked=cursor-grok(no-receipt),cursor-composer(no-receipt),domain(auth); declined=cursor-kimi,cursor-muse; rounds=3; full=1 @ 3581be6}

_{Review-Attention: study ~20m (critical: replication.ts; raised: degraded review) @ 3581be6}

[gemini-code-assist]

Code Review

This pull request optimizes the rolling configuration update process for Harper clusters (version 5.2.0+). Instead of writing the configuration to each instance individually, it writes and restarts a canary instance, then fans out the configuration write to the rest of the cluster via replication. Subsequent instances skip their local configuration write if the replication fan-out successfully acknowledged them. The PR also adds comprehensive unit tests and design documentation. Feedback on the changes suggests a minor optimization in the update loop to break early when the operation is canceled.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	68.89%	10601 / 15388
🔵	Statements	69.05%	11319 / 16392
🔵	Functions	62.35%	2748 / 4407
🔵	Branches	64.25%	8147 / 12680

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/hooks/useRollingConfigUpdate.tsx	93.84%	92.68%	63.63%	93.75%	116, 203, 214, 237
src/integrations/api/replication.ts	100%	100%	100%	100%
src/integrations/api/instance/status/setConfiguration.ts	71.42%	100%	75%	66.66%	36-40

Generated in workflow #2197 for commit 3581be6 by the Vitest Coverage Report Action