Skip to content

Commit aed93b6

Browse files
Persist manual nodes in the broker (#60)
<!-- pair-release-intent:v1 --> ### Changelog title Manual nodes persist across restarts ### Changelog body - Manually added nodes are restored after the broker or manual-node worker restarts. ### Bumps - services: patch - nvpair-cluster-manager: none - nvpair-engine-manager: none - nvpair-errors: none - nvpair-job-scheduler: none - nvpair-manual-nodes: none - nvpair-node-info: none - nvpair-node-scanner: none - nvpair-node-settings: none - nvpair-proxy: none - nvpair-tui: none - nvpair-ui-broker: patch - nvpair-workload-manager: none <!-- /pair-release-intent:v1 --> ## Description Closes #58. Move manual-node durability into `nvpair-ui-broker` so headless TUI/service nodes keep manual peers across broker and worker restarts. Successful add/remove operations update the existing `configs/manual-nodes.json`, and a restarted manual-nodes worker is repopulated automatically. This version is rebased and adapted to the current `develop` architecture after the TUI overhaul. The obsolete Electron-owned store/replay path is removed so the broker is the single persistence owner. ## Scope - persist successful manual-node adds/removes in the broker - replay durable manual nodes into replacement workers - resolve stable discovered node IDs back to manual aliases for removal - remove Electron-owned persistence/replay logic - keep the existing JSON-RPC methods and on-disk file location - declare a patch bump for `nvpair-ui-broker` through release intent; do not edit `services/versions.json` manually ## Validation - `cd services/nvpair-ui-broker && go test ./...` — passed - `cd desktop && npm run typecheck` — passed - `cd desktop && npm run lint` — passed - `cd desktop && npm run test:unit -- modular/modular-supervisor-readiness.test.ts` — 4 passed - `cd desktop && npm run test:unit` — 39 files, 231 tests passed - `cd desktop && npm run dead-code:check` — passed with zero findings - `cd desktop && npm run service-contracts:check` — passed with no unignored drift - `node scripts/spdx-headers.mjs` — 1027 checked, 0 missing - `git diff --check` — passed ## Risk The JSON-RPC surface and persisted file schema are unchanged. Existing `manual-nodes.json` entries remain readable. The ownership change removes the duplicate Electron persistence path and makes broker/TUI-only deployments durable as well. ## Checklist - [x] I have read the Contributing Guidelines. - [x] Every commit is signed off, certifying the Developer Certificate of Origin. - [x] New or existing tests cover the change. - [x] Relevant documentation is updated. - [x] I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts. - [x] I recorded the validation commands and results above. - [x] I declared version bumps in the release-intent block above. `services/versions.json` is written by automation and is not edited in this PR. Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
1 parent c09f648 commit aed93b6

10 files changed

Lines changed: 454 additions & 225 deletions

File tree

‎desktop/docs/services-parity.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ history.
2121
| Process supervision | Complete | Electron starts only `nvpair-ui-broker`; the broker supervises all workers |
2222
| Discovery | Complete | Broker discovery snapshots drive available nodes and node state |
2323
| Node telemetry | Integrated with direct poll | Electron polls advertised `/v1/node-info` (plain HTTP); remote OS and some remote telemetry are backend-limited |
24-
| Manual nodes | Complete with local persistence | Broker owns probing and proxy registration; Electron persists entries for replay |
24+
| Manual nodes | Complete with local persistence | Broker owns probing, durable entries, restart replay, and proxy registration |
2525
| Ollama routing | Complete | Broker relay and backend scheduler drive proxy routing |
2626
| LM Studio routing | Complete | Parallel broker relay and scheduler path |
2727
| llama.cpp backend | Integrated | Desktop and TUI expose install, lifecycle, catalog pull, inventory, load/unload, endpoints, routing, and demo traffic |
@@ -487,7 +487,6 @@ provide an equivalent client-facing contract:
487487
| Responsibility | Location |
488488
| ----------------------------------------------------------- | ------------------------------------------------ |
489489
| Poll node telemetry over `/v1/node-info` | `node-info-poller.ts` |
490-
| Persist and replay manual node entries | `manual-nodes-store.ts`, `modular-supervisor.ts` |
491490
| Bridge the local node into engine proxies | `modular-supervisor.ts` |
492491
| Present optimistic engine transition state | `pending-actions.store.ts`, bridge state |
493492
| Relay the backend model catalogue to the renderer | `service-bridge/model-catalog.ts` |

‎desktop/src/electron/service-bridge/empty-handlers.ts‎

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ import {
2828

2929
import type { JsonObject, JsonValue } from './json-rpc-subprocess'
3030
import { emptyInvite, parseClusterNodes, parseInvite, parseNodeIdentity } from './cluster-json'
31-
import { removeManualNodeEntry, resolveManualNodeKey } from './manual-nodes-store'
3231

3332
type BridgeHandler<C extends WsInvokeChannel> = (
3433
payload?: WsInvokeRequest<C>
@@ -630,20 +629,10 @@ async function handleNodeRemoveMember(
630629
const selfId = state.getSelfId()
631630
const isSelfLeave = selfId !== null && payload.nodeId === selfId
632631

633-
// `payload.nodeId` is the node's UUID, but the manual-nodes store and the
634-
// broker's `node/remove` relay key a manual entry by the name it was added
635-
// with — never the UUID. Map the UUID back through the node's reachable
636-
// addresses to the persisted entry so it is actually pruned (and does not
637-
// reappear on the next replay). If it is not a manual node (no address match)
638-
// fall back to the display hostname, then the raw id; both are harmless
639-
// no-ops for a purely-discovered peer. The broker supervises
640-
// nvpair-manual-nodes and relays `node/*` verbatim.
641-
const manualKey =
642-
resolveManualNodeKey(state.getNodeAddresses(payload.nodeId)) ||
643-
state.getNodeHostname(payload.nodeId) ||
644-
payload.nodeId
645-
removeManualNodeEntry(manualKey)
646-
supervisor.callProcess('broker', 'node/remove', { id: manualKey }).catch(() => {})
632+
// The broker owns manual-node persistence and resolves a stable node UUID
633+
// back to the manual alias expected by nvpair-manual-nodes. For a purely
634+
// discovered peer this remains a harmless no-op.
635+
supervisor.callProcess('broker', 'node/remove', { id: payload.nodeId }).catch(() => {})
647636

648637
if (isSelfLeave) {
649638
// Self-departure is `cluster:leave`, not `nodes:remove` (the cluster-manager

‎desktop/src/electron/service-bridge/manual-nodes-store.ts‎

Lines changed: 0 additions & 96 deletions
This file was deleted.

‎desktop/src/electron/service-bridge/modular-state.ts‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1007,31 +1007,6 @@ class ModularBridgeState {
10071007
return this.selfId
10081008
}
10091009

1010-
/**
1011-
* The display hostname for a node key (UUID), or '' when unknown. Used as a
1012-
* last-resort manual-removal key when address matching finds no persisted
1013-
* entry (see {@link resolveManualNodeKey}).
1014-
*/
1015-
getNodeHostname(nodeId: string): string {
1016-
return this.nodes.get(nodeId)?.name ?? ''
1017-
}
1018-
1019-
/**
1020-
* Every address a node key (UUID) is known to be reachable at — its
1021-
* canonical `reachableAddress`, its discovered addresses, and its `host`.
1022-
* Used to map a UUID back to the manual-nodes store entry (keyed by the
1023-
* user-entered address) when removing a manual node. Empty for an unknown id.
1024-
*/
1025-
getNodeAddresses(nodeId: string): string[] {
1026-
const node = this.nodes.get(nodeId)
1027-
if (!node) return []
1028-
const out = new Set<string>()
1029-
if (node.reachableAddress) out.add(node.reachableAddress)
1030-
for (const address of nodeAddresses(node)) out.add(address)
1031-
if (node.host) out.add(node.host)
1032-
return Array.from(out)
1033-
}
1034-
10351010
/** The authoritative live inbound invites awaiting a PIN, oldest first. */
10361011
getPendingInvites(): Invite[] {
10371012
return Array.from(this.pendingInvites.values()).sort((a, b) => a.createdAt - b.createdAt)

‎desktop/src/electron/service-bridge/modular-supervisor.ts‎

Lines changed: 0 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,6 @@ import {
4646
isModularLogLevel,
4747
type ModularLogLevel
4848
} from '@/shared/constants/modular-runtime'
49-
import { listManualNodeEntries } from './manual-nodes-store'
5049
import {
5150
MODULAR_RUNTIME_BINARIES,
5251
modularBinaryFileName
@@ -826,29 +825,6 @@ class ModularSupervisor {
826825
return [...args, ...this.logLevelArgs()]
827826
}
828827

829-
/**
830-
* Re-add persisted manual nodes through the broker's `node/add` relay. The
831-
* broker's `nvpair-manual-nodes` loses its in-memory entries on restart (N86),
832-
* so PAIR UI owns the durable list (`manual-nodes.json`) and replays it once
833-
* the broker is ready.
834-
*/
835-
private async replayManualNodes(): Promise<void> {
836-
const entries = listManualNodeEntries()
837-
for (const entry of entries) {
838-
try {
839-
await this.callProcess('broker', 'node/add', {
840-
address: entry.address,
841-
name: entry.name
842-
})
843-
} catch (err) {
844-
log.warn({
845-
sublevel: 'manual-nodes',
846-
message: `Failed to replay manual node ${entry.address}: ${getErrorString(err)}`
847-
})
848-
}
849-
}
850-
}
851-
852828
private async onBrokerReady(): Promise<void> {
853829
const subscribe = async (method: string, label: string): Promise<void> => {
854830
try {
@@ -871,7 +847,6 @@ class ModularSupervisor {
871847

872848
await this.syncClusterIdentityToManager()
873849
await this.resolveSelfId()
874-
await this.replayManualNodes()
875850
await this.hydrateEngineManager()
876851
this.startInstalledEnginesOnFirstOpen()
877852
await this.seedClusterPeerIds()

‎desktop/tests/modular/modular-supervisor-readiness.test.ts‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,6 @@ vi.mock('@/electron/service-bridge/broadcaster', () => ({
4040
emitBridgePush: mocks.emitBridgePush
4141
}))
4242

43-
vi.mock('@/electron/service-bridge/manual-nodes-store', () => ({
44-
listManualNodeEntries: () => []
45-
}))
46-
4743
vi.mock('@/electron/service-bridge/node-info-poller', () => ({
4844
startNodeInfoPoller: vi.fn(),
4945
stopNodeInfoPoller: vi.fn()

‎services/nvpair-ui-broker/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -538,7 +538,7 @@ Any `settings/*` request is forwarded to `nvpair-node-settings` and its response
538538

539539
#### `node/add` / `node/remove` / `nodes/list` (manual nodes)
540540

541-
Relayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, tls_port?, mtls? }`) registers a user-added node and probes it; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. A `nvpair-manual-nodes` restart loses the in-memory entries because neither that worker nor the broker persists an authoritative copy, so clients must re-add manual nodes after a restart. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised.
541+
Relayed to `nvpair-manual-nodes`. `node/add` (`{ address, name?, tls_port?, mtls? }`) registers a user-added node and probes it; `node/remove` (`{ id }`) drops it; `nodes/list` returns the tracked manual nodes. Manually added nodes also surface in the shared `discovery:get-nodes` / `discovery:nodes-changed` snapshot — the broker merges `nvpair-manual-nodes`' `node/discovered|updated|removed` into the same store the scanner feeds. The broker persists successful `node/add` operations to `configs/manual-nodes.json` in the per-user data directory and replays them into a replacement `nvpair-manual-nodes` worker after startup. Successful `node/remove` operations update the same durable list. Error `-32000 "manual-nodes not available"` when no manual-nodes worker is supervised.
542542

543543
**Manual → proxy bridge.** When the broker supervises both `nvpair-manual-nodes` and a proxy, it also bridges a manual node whose engine is reachable into that proxy via `node/add-manual` (host/port from the node's per-engine status), so inference can route to it through `ollama-proxy:node/select` / `lmstudio-proxy:node/select` just like a relay-discovered node. This is per-engine: a node whose `ollama_*` status is up is bridged into `ollama-proxy` (host/port from `ollama_port`), and one whose `lmstudio_*` status is up into `lmstudio-proxy` (from `lmstudio_port`) — a node running both is bridged into both. Manual nodes are by definition the ones that never appear via the daemon's `_nvpair-node` discovery, so this explicit add is what makes them routable. The bridge tracks reachability: an engine that goes down (or a node that is removed, or whose prober crashes) is pulled back out with `node/remove-manual`. A proxy that isn't supervised → that leg is a no-op; manual nodes still appear in the discovery snapshot as before.
544544

@@ -622,7 +622,7 @@ Attach to a pre-existing endpoint:
622622
RPC (custom service, port, name, or TXT), and no way to advertise anything
623623
other than the detected engines.
624624
- **node-info control surface.** node-info is spawned and torn down with the broker, and the broker pushes it only two things over stdin: the log level, and this node's cluster principal (`nodeinfo:set-cluster-identity`, sent on spawn and on every membership or pin-set change, because node-info holds no cluster dir and so cannot read membership itself). Otherwise it's hands-off: the broker registers its port with the daemon (which enriches over plain HTTP) but doesn't pass through TLS material (`--cert` / `--key` / `--client-ca`) or a custom `--port`, and exposes no RPC to query or reconfigure it. It runs with its own defaults plus those two pushes.
625-
- **Manual-node persistence across restarts.** `nvpair-manual-nodes` keeps its entries only in memory and the broker holds no authoritative copy, so a manual-nodes crash-and-restart loses the user's manual nodes (the broker evicts the orphaned entries from the snapshot; clients must re-add them).
625+
- **Manual-node persistence across restarts.** `nvpair-manual-nodes` remains an in-memory worker; the broker owns `configs/manual-nodes.json`, clears stale live projections when the worker exits, and replays the durable entries into its replacement.
626626
- **Per-event push semantics.** `discovery:nodes-changed` always carries the full current snapshot, not a delta. For small N this is fine and lets the client treat the payload as authoritative without state reconciliation. `errors:update` is likewise a full snapshot.
627627
- **BYO-TLS node-info discovery.** The daemon enriches peers over plain HTTP per the consolidated transport policy (node-info is plain on the broker path). Reading a node-info served over operator-configured BYO-TLS / mTLS is not supported through the broker; discovery works for the plain-HTTP case (the common one).
628628
- **Per-worker log-level granularity.** `log/set-level` fans the broker's single level out to every running child, but there's no way to set a different level per worker — it's one level for the whole process tree.

0 commit comments

Comments
 (0)