[JaredP94]

Design

• Discovery announcements and public HTTP inventory are endpoint hints, never identity proofs.
• A shared bounded reader proves the exact current peer certificate over pinned cluster mTLS and rechecks authorization after decoding.
• Proof bindings include the local admission, local leaf and exact peer pin record.
• Node-info publishes broker-owned service metadata alongside its public hardware preview.
• Scanner candidates and verified peers have independent identities and lifetimes; removing a preview cannot remove a verified peer.
• Only authenticated metadata supplies PAIR service ports, model/engine inventory and scheduler telemetry.
• Broker relay delivery, scheduler snapshots and telemetry ingestion/replay recheck current provenance; worker retirement withdraws its sources.
• Explicitly user-added bare engines remain a separately labelled manual routing bypass, without PAIR identity or peer telemetry.
• Additive peerRecordId / identityVerification fields distinguish verified peers, previews and manual configurations.

Description

Adapt NVIDIA/Personal-AI-Router#179 to this fork's source-aware discovery, three inference proxies and scheduler. A peer claiming a pinned principal must prove possession of its key over cluster mTLS before its claims can establish trust or supply service decisions. Public hardware previews remain available.

The takeover audit also fixed an authorization gap for telemetryValid:false samples: unavailable readings now require the same proof as valid readings, including delivery and replay. Otherwise an unproven unavailable reading could erase authenticated GPU pressure. The scanner's engine-native replacement regression now exercises actual remote mTLS browse instead of passing through a local-self early return. Added retry-budget/lifetime and original-withdrawal-token coverage.

Version decisions, compared with v0.8.0:services/versions.json:

• nvpair-node-info: 1.3.0 → 1.4.0; additive authenticated service metadata/provenance and security behavior.
• nvpair-node-scanner: 1.2.0 → 1.3.0; additive proof-bound inventory/provenance and security behavior (main was 1.2.1).
• nvpair-manual-nodes: 1.2.1 → 1.3.0; additive authenticated/manual provenance and security behavior (main was 1.2.2).
• Broker remains at main's already-unreleased 2.5.0 (released 2.4.1), sufficient for the additive snapshot/security changes. TUI remains at main's unreleased 1.2.1 (released 1.2.0); its change consumes and labels provenance. Other existing unreleased component bumps and product/installer versions are unchanged.

The security-review follow-up is pushed as one signed-off commit per blocking finding:

1. 7022b7c — inventory proofs require an active, non-retired admission; retained pins cannot authorize fresh, pooled or cached inventory after withdrawal.
2. adde205 — local and peer certificate validity windows gate current proofs and response acceptance; expiry disqualifies cached routing inventory, telemetry ingestion and scheduler replay, including responses crossing expiry on pooled connections.
3. 6957ee6 — serialize manual trust-refresh snapshot/replay with worker callbacks and retirement; reject retired-generation status snapshots.
4. 658fc8f — atomically remove only the exact expired scanner record and order verified publication/enrichment updates with withdrawal/cache cleanup.

333f5a4 restores the original telemetry failover assertions through the authenticated fixture: the full refreshTelemetryOnce sweep, emitted identity/proof/utilization, and remembered answering address. Each finished commit was pushed immediately. 4fd864b additionally carries the existing fork-main cancellation-ownership fix (6b58e5d) with its tests, eliminating the baseline broker vet warning rather than disabling that analyzer.

Scope

Included: shared inventory/provenance, node-info, scanner, manual nodes, broker/relay/scheduler boundaries, TUI labels, generated service-contract documentation, component documentation and security/architecture guidance. The original six coherent signed-off commits replace the unpushed WIP; adapted work credits Kaylee Lubick. The security-review follow-up adds the four focused blocking fixes, restored failover coverage, and the existing fork-main vet cleanup listed above.

Follow-ups:

1. Desktop handling of peerRecordId and identityVerification, where previews must never be keyed by a claimed hostUuid.
2. Persist each manual entry's kind (PAIR peer vs bare engine) in manual-nodes.json, so a known PAIR peer never downgrades across worker restarts.

Validation

Rebase and Linux CI repair (current head 7bd921a)

Rebased the branch from a0c46f5 onto origin/main 12d00de (the merged mDNS socket-reuse work), without conflicts, in the isolated worktree /tmp/pair-work/verified-peer-ci-repair. No notice regeneration was needed. Ran all five complete module suites with go test -race ./... after rebasing and before the force-with-lease push: shared, nvpair-node-scanner, nvpair-manual-nodes, nvpair-node-info, nvpair-ui-broker — all passed.

The six Linux failures in run 41600617583 were introduced by this PR, not pre-existing failures:

• (a) TestModelsHTTPEnrichment — its public HTTP/mDNS stub supplied no pinned-mTLS identity proof; withholding model inventory is intended.
• (a) TestModelsPeriodicRefreshConvergesWithoutMDNSChange — the same unproven fixture cannot supply periodically refreshed model inventory.
• (a) TestWorkloadManagerRehydratesActiveWorkloadOnRestart — its pinned workload listener lacked authenticated node-info/service metadata, and its advertised host UUID differed from the pinned principal, so discovery correctly withheld it from broadcast peers.
• (a) TestWorkloadManagerRehydratesRecentTerminalOnRestart — the same fixture problem, not a removal of terminal rehydration.
• (b) TestWorkloadFailedOnNodeLoss — active state was accepted against a preview, then preview removal explicitly bypassed the node-loss sweep. This is a product defect.
• (a) TestWorkloadManagerOutboundBroadcast — the same unproven service-discovery fixture as the restart tests; refusing the target is intended.

Signed-off repair commits:

• 212e9f7 fixes preview withdrawal and scanner-retirement cleanup for already tracked workloads, rejects active events naming known unproven participants, and preserves jobs when a proven claim survives. Terminal reconciliation and relayed participants not yet discovered locally remain supported. Broker stays at its existing unreleased 2.5.0 bump (released 2.4.1); no extra version bumps.
• 7bd921a upgrades the end-to-end fixtures to admitted, exact-cross-pinned cluster mTLS peers with authenticated node-info identities and service maps. Model unions/per-engine attribution/loaded sets, periodic additions and final deletions, active and terminal rehydration after worker restart, outbound inference broadcasts, and verified peer-loss failure all retain their positive assertions. A plaintext lookalike remains an untrusted preview, contributes no models and receives no model read.

Local follow-up validation (macOS arm64):

• go test -race -count=1 ./... in nvpair-ui-broker — full suite passed after the product fix and again after the sensitivity check.
• New TestPreviewWithdrawalRetiresAlreadyTrackedWorkloads, TestScannerRetirementRetiresPreviewWorkloads, and TestActiveWorkloadsCannotUseUnverifiedPreview regressions failed without the fix. Temporarily removing the fix reproduced all intended failures; restoring it returned the complete broker race suite to green. A surviving verified claim is covered.
• GOOS=darwin go vet ./..., GOOS=linux go vet ./..., GOOS=windows go vet ./... in both services/tests and nvpair-ui-broker — passed.
• gofmt -l on every follow-up Go file — clean; git diff --check origin/main — passed.
• SPDX checker — 1276 checked, 0 missing.
• No cross-process suite was run locally. No desktop source or wire surface changed in this repair.

Final CI: run 41603781815, head 7bd921a5fa451a4f437e4df4494a2990644f9128, success, all six jobs passed: desktop gates (including service contracts and licenses), SPDX, Go modules on Linux/macOS, and cross-process suites on Linux/macOS.

Downloaded both diagnostic artifacts into the new empty directory /tmp/pair-review/art43-final-41603781815-EkHXtI and inspected the JSONL, without executing artifact contents. The Linux artifact records real pass events (not skips) for all six:

• TestModelsHTTPEnrichment — 16.51 s.
• TestModelsPeriodicRefreshConvergesWithoutMDNSChange — 31.51 s.
• TestWorkloadManagerRehydratesActiveWorkloadOnRestart — 11.04 s.
• TestWorkloadManagerRehydratesRecentTerminalOnRestart — 11.04 s.
• TestWorkloadFailedOnNodeLoss — 60.03 s.
• TestWorkloadManagerOutboundBroadcast — 5.03 s.

Linux reports 101 passing cases including subtests, zero failures, and one separate skip: TestScannerEvictsRecordSupersededAtItsAddress never discovered its record and is untested, not counted as passing. macOS skips these six multicast-dependent tests; Linux, not those skips, supplies their runtime evidence. Live deployment, Windows runtime, desktop launch and packaged artifacts remain unverified.

Same-address ghost-node coverage repair (head 8ff7c5e)

The Linux skip in run 41603781815 was a fixture/event-contract error, not unavailable multicast: onBrowse emits public peers as independently keyed preview:<sha> records via discovery:node-updated, while the old test awaited discovery:node-discovered under the claimed host UUID.

Signed-off follow-up commits:

• 7ee1039 adds an in-module regression for a genuinely replaced key-bearing endpoint: the original peer proves its identity over pinned mTLS, survives multicast loss, then loses its listener to an unclustered fresh identity at the exact same address/port. A transient proof failure preserves the grace period; an aged failed proof withdraws the original verified record and token despite the live public responder.
• 9a53ed6 restores TestScannerEvictsRecordSupersededAtItsAddress with the admitted, exact-cross-pinned fixture from 7bd921a, then wipes it to a fresh public identity at the same endpoint. Separate TestScannerEvictsPreviewSupersededAtItsAddress covers the public candidate's identity-mismatch liveness probe and independently keyed preview removal.
• 8ff7c5e removes the other post-multicast prerequisite skips.

No scanner product bug or runtime change was needed for this repair. Verified-peer lifetime already belongs to failed pinned-mTLS refresh/expiry, not public HTTP identity claims or TCP reachability. Disabling that existing expiry made the new in-module test fail with “a new identity at the same live endpoint kept stale verified inventory alive”; restoring it passed. Public mismatch probes still evict previews and cannot withdraw a verified peer. No version bumps: all changes are test-only.

Skip audit (never discovered, inconclusive, and t.Skip after requireMulticast) converted four sites to failures:

1. Ghost test's missing-discovery branch (both verified and preview cases now fail).
2. TestWorkloadManagerRehydratesActiveWorkloadOnRestart port guard.
3. TestWorkloadManagerRehydratesRecentTerminalOnRestart port guard.
4. startBrokerProcInCluster's workload-manager port guard, also reachable from multicast-dependent node-loss/broadcast tests.
   No other discovery-time “never discovered”/“inconclusive” self-skips were found. The environment decision in requireMulticast, explicit -short, and unrelated platform/environment guards remain.

Local validation in isolated worktree /tmp/pair-work/ghost-identity, based on origin/fix/verified-peer-inventory at 7bd921a:

• go test -race -count=1 ./... in nvpair-node-scanner — passed.
• GOOS=darwin/linux/windows go vet ./... in both services/tests and nvpair-node-scanner — passed.
• gofmt -l for both touched modules — clean; git diff --check — passed.
• SPDX checker — 1276 checked, 0 missing.
• No cross-process tests were run locally.

Latest-head CI at verification: run 41608227469, branch head 8ff7c5ec79148693fa4fc46ab25b053591764dd3, all six jobs passed (SPDX, desktop checks, Go modules on Linux/macOS, cross-process on Linux/macOS).

Downloaded the Linux diagnostic artifact into the new empty directory /tmp/pair-review/ghost43-linux-41608227469-HjJzB4 and parsed its JSONL with isolated python3 -I, without executing artifact contents:

• 103 passing cases, zero failures, ZERO skips.
• TestScannerEvictsRecordSupersededAtItsAddress — pass, 60.02 s.
• TestScannerEvictsPreviewSupersededAtItsAddress — pass, 66.01 s.

The previously skipped regression is now exercised on Linux, not counted from an inconclusive run. Later commits on this shared branch require their own latest-head CI evidence.

Risk

After a manual-nodes worker restart, an address the user added for a PAIR peer can be routed to as an unauthenticated plaintext bare engine if no PAIR endpoint answers there (e.g. if someone else answers on that address). The worker's peerSeen downgrade suppression lasts only for the configuration's in-process lifetime. Persisting entry kind is the explicit follow-up above; this PR does not claim to solve restart-time intent loss.

Mixed-cluster compatibility: upgraded consumers keep older peers without the new authenticated broker metadata as unverified previews and do not route to them or accept their scheduling inventory until upgraded. No plaintext or operator-CA compatibility downgrade is added. Older consumers retain their existing policies; upgrading one node does not retrofit this guarantee into older binaries.

Public hardware previews still expose inventory/utilization to the LAN. The desktop has not yet adopted the new independent preview identity fields; service trust/routing/scheduling enforcement is in Go, not a claim that the existing desktop projection is fully migrated. Live deployment and artifact certification remain outstanding.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.