From d751067fcb050e73df4a6ab29cc9a5cfa9341d6b Mon Sep 17 00:00:00 2001 From: Terve Date: Mon, 28 Sep 2026 12:01:24 -0400 Subject: [PATCH 1/2] docs(nvpair-node-scanner): add normative service specification Add services/nvpair-node-scanner/spec.md, derived from the Go source and its tests rather than the README, following the structure of the existing specs: sections 1-15, a 7.0 method and notification table, and a closing section on the process model, CLI, and build wiring. It describes the daemon as it runs today: a one-minute eviction window with per-scan probing from the third miss, inference activity as proof of life, and a grace for scans that lose every node at once; a two-second GPU telemetry loop for the scheduler; multi-address peers published as ips= and dialed in their own order; and evidence-based ranking of this host's own published addresses. It also records that the browser has no goodbye fast path, so a cleanly shut-down node is evicted at the one-minute threshold like any other. Documentation only. No Go file is touched and no version changes. Signed-off-by: Terve --- services/nvpair-node-scanner/spec.md | 424 +++++++++++++++++++++++++++ 1 file changed, 424 insertions(+) create mode 100644 services/nvpair-node-scanner/spec.md diff --git a/services/nvpair-node-scanner/spec.md b/services/nvpair-node-scanner/spec.md new file mode 100644 index 000000000..40f55ad99 --- /dev/null +++ b/services/nvpair-node-scanner/spec.md @@ -0,0 +1,424 @@ + + +# Microservice: Node Scanner (`nvpair-node-scanner`) + +## 1. Purpose + +Owns node discovery for this host. It is the single mDNS advertiser and the single mDNS browser on the machine: no other worker runs a responder or a resolver. Every local service that wants to be reachable from the LAN hands this daemon a short service key and a port (`discovery:register`); the daemon folds them into **one** `_nvpair-node._tcp` record carrying this node's identity, its LAN address, and one `key=port` TXT entry per registered service. Simultaneously it browses `_nvpair-node._tcp`, reconciles the answers into a directory of every node on the subnet (this node included), decorates each entry with the data that does not fit in a DNS record — hardware inventory from `nvpair-node-info` and the model inventory from `nvpair-engine-manager` — and emits `discovery:node-discovered` / `discovery:node-updated` / `discovery:node-removed` deltas. Separately it samples every node's GPU utilization every couple of seconds and reports it as compact `discovery:node-telemetry` for the scheduler. + +It exists so that discovery has exactly one implementation per host. One record makes this node's advertised surface atomic — a peer either sees it with all of its ports or not at all — and one browser makes "who is on the network, at what address" a single answer every consumer shares rather than a per-worker approximation. It also confines the hard parts of LAN discovery to one place: the per-interface multicast send path a multi-homed host depends on to be visible at all, the choice of which of this host's own addresses to publish, and the eviction policy that keeps a healthy node — including one too saturated by its own inference load to answer — from being dropped. `nvpair-ui-broker` relays the resulting directory to every other worker and to the desktop UI, so no other process needs a responder, a resolver, or an opinion about addressing. + +## 2. Scope + +**In scope** +- Advertise exactly one `_nvpair-node._tcp` mDNS record per host, built from the registered service set plus the daemon-stamped identity (`uuid=`, `cluster-uuid=`, `ip=`, `ips=`). +- Rank this host's own publishable addresses from evidence it observes — peer-proven addresses, interfaces facing a discovered peer, interfaces whose sends keep failing — and publish the best as `ip=` and up to four as `ips=`. +- Accept `discovery:register` / `discovery:unregister` / `discovery:update-txt` and re-advertise only on a real change to the record. +- Browse `_nvpair-node._tcp`, reconcile scans into a directory keyed by host UUID, and emit per-node deltas. +- Decorate directory entries with `/v1/node-info` inventory (GPU, CPU, memory) and `/v1/models` inventory (flat, per-engine, and per-engine loaded sets), dialing each node at every address it published until one answers as that node. +- Sample every node's `/v1/node-info` on a two-second cadence and emit compact GPU telemetry (`discovery:node-telemetry`) for the broker's scheduler feed, without touching the directory. +- Converge peer cluster membership and model inventories on one periodic sweep that is independent of mDNS change events. +- Re-resolve and re-advertise this node's identity and LAN address on demand (`discovery:reload-identity`, `discovery:reload-trust`) and on a network change. +- Annotate a browsed peer as `trusted` when this node holds a cluster pin for the principal that peer advertises, and re-derive that annotation across the whole directory on `discovery:reload-trust`. +- Evict a node only after a minute of mDNS silence during which every liveness probe also failed; keep it while it answers an `ni` / `em` probe as itself (confirmed over `/v1/node-info`), has a fresh enrichment, or has recently served inference. +- Treat a scan that loses every known node at once as this host's own failure, for a bounded number of scans. +- Drop a directory record proven superseded at its address, so a machine that comes back under a fresh identity is not carried twice. + +**Out of scope** +- Serving the directory to clients or fanning it out to workers — `nvpair-ui-broker` owns `discovery:subscribe`, the relay directory, and the client-facing `discovery:get-nodes` / `discovery:nodes-changed` snapshots (§7.7). +- Cluster identity, PIN pairing, certificate issuance, pin management, and membership — `nvpair-cluster-manager`. This daemon only *reads* the cluster directory through `nvpair-shared/clustertrust`. +- Producing the hardware inventory it fetches — `nvpair-node-info`. +- Producing the model inventory it fetches, and every model or engine lifecycle operation — `nvpair-engine-manager`. +- Manual (user-entered) nodes — `nvpair-manual-nodes` probes them and `nvpair-ui-broker` merges them into its own store (§7.7). +- Routing and target selection — `nvpair-proxy` and `nvpair-job-scheduler`. This daemon reports raw GPU utilization; turning it into scheduling pressure is the scheduler's job. +- Persisting anything. The only file this daemon can create is the fallback node id, and only indirectly through `nvpair-shared/nodeid` (§9). + +## 3. Key Use Cases + +- **Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` from its engine poller at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry. +- **Discover a peer**: a scan returns a peer's `_nvpair-node` record; `toDirectoryNode` projects TXT into a `DirectoryNode` keyed by `uuid=`, enrichment fills in hardware and models, and `discovery:node-discovered` carries the whole entry to the broker. +- **A peer changes**: a peer starts an engine (a new `ol=` key) or moves address (`ip=` / `ips=`); the order-insensitive record compare in `nvpair-shared/discovery` sees a real change, re-enrichment runs, and `discovery:node-updated` fires. A scan that returns the same addresses and TXT in a different order emits nothing. +- **A peer leaves**: it stops answering. From its third consecutive missed scan (about 15 s) the daemon dials its `ni` port and then its `em` port at each published address on **every** scan, and any answer resets the miss counter. Only after twelve consecutive misses (a full minute) with every probe failing is the entry dropped, its enrichment caches forgotten, and `discovery:node-removed` emitted. A clean shutdown multicasts an RFC 6762 goodbye (TTL 0 PTR), which clears the record from mDNS caches so it is absent from the very next scan. The browser has no goodbye fast path, though: the node still takes the miss path, its probes are refused from about 15 s on, and it is evicted at the one-minute threshold. +- **A peer is saturated by its own inference load**: its control plane is starved, so it stops answering mDNS *and* stops accepting probe connections for as long as the load runs, while still serving requests. The one-minute window absorbs most such bursts, and inference response bytes seen from it in the last 60 s (`discovery:node-activity`, relayed by the broker from whichever proxy saw them) keep it with no probe at all — the only liveness evidence that gets stronger the busier the node is. +- **This host stops hearing multicast**: a process too starved to drain its multicast socket within the scan window hears silence from every node at once. When two or more nodes are known and a scan returns none of them, no node is penalized, for up to six consecutive scans (about 30 s); after that they count normally. Independent machines do not leave the network in the same five seconds. +- **Reach a multi-homed peer**: a peer publishing a LAN address and a direct-connect link address lists both in `ips=`. Enrichment, the periodic sweep, telemetry, and the liveness probe dial each in the peer's own order, skip an answer from a different `hostUuid`, and remember per node and service which address answered so later rounds ask only that one. +- **Feed the scheduler**: every ~2 s the daemon GETs each node's `/v1/node-info`, reduces it to the highest GPU utilization plus `telemetryValid` and `msSince` (advanced by the request's own latency), and emits `discovery:node-telemetry`. The broker caches it for `nvpair-job-scheduler`. The directory is not touched. +- **A peer is wiped and reinstalled**: it mints a fresh `uuid=` but keeps its LAN address, its hostname, and its fixed service ports, so a connect-only probe would reach the new incarnation and report the old record alive forever. Two paths collapse the duplicate, and both delete only on the same proof — a `/v1/node-info` answer naming a different host. If the old record ages into the miss threshold, `identityAlive` overrules the successful TCP dial; if the new record is browsed first, `supersedingUpsert` puts the same question to the arriving record's `ni` port before the new entry is stored. +- **Model inventory converges without an mDNS event**: a peer pulls or deletes a model, or starts an engine late. Nothing about its mDNS record changes, so the browse-driven path never re-runs; the 15-second sweep re-fetches every `em` node's `/v1/models`, and only a genuinely changed inventory produces one `discovery:node-updated`. +- **A peer's cluster departure is announced but not received**: the peer leaves and re-advertises without `cluster-uuid=`, but this host never receives that record, and the liveness probe keeps the frozen entry alive precisely because the peer is still reachable. The same 15-second sweep reads `clusterUuid` from the peer's own `/v1/node-info` and clears the stale principal, so the client stops suppressing an invite the peer would now accept. +- **This node joins a cluster while running**: `nvpair-cluster-manager` writes the keypair and activates the admission, and the broker relays `discovery:reload-trust`; `reconcileIdentity` sees live membership disagree with what the registry publishes, `cluster-uuid=` appears in the record, and peers can key their pins on it. No restart. +- **A peer is paired after it was already discovered**: its mDNS record does not move when this node's pin lands, so the browse path never re-asks whether it is trusted and it would stay unroutable for the life of the process. `discovery:reload-trust` re-derives `trusted` for every directory entry against the current pin set and emits one `discovery:node-updated` per entry whose answer changed. +- **Edge case — self is never browse-driven**: Windows frequently does not loop a host's own multicast back to the browsing socket, so aging this node like a peer would evict it from its own directory. `onBrowse` discards any event whose `uuid=` matches the registry's `HostUUID`; this node's entry comes only from `publishSelf`. +- **Edge case — self enrichment behind a first-run firewall prompt**: a fresh Windows install blocks inbound to this host's own LAN address, which would blank the local node's metrics and models. Self is always dialed at `127.0.0.1` while the entry still displays the advertised LAN `ip=`, so the local card is populated regardless. +- **Edge case — identity re-stamp on a fresh host**: the scanner spawns before `nvpair-cluster-manager` and mints its own `node-id.json`, so it initially advertises a UUID that is not the cluster principal. Once the broker triggers `discovery:reload-identity`, the daemon re-resolves, moves its directory entry to the new UUID (carrying the enrichment caches across), emits `discovery:node-removed` for the stale one, and republishes. +- **Edge case — a record with no `uuid=`**: it cannot be keyed by a stable identity, so it is warned about and skipped rather than keyed by the mutable, non-unique hostname. A later `removed` for it is a no-op because it was never stored. +- **Edge case — two hosts sharing a hostname**: the browser keys by `uuid=` rather than the mDNS instance name, so they stay distinct entries. A key whose `uuid=` changes underneath it logs a collision / host-replacement warning. +- **Edge case — transient enrichment failure**: a failed `/v1/node-info` or `/v1/models` fetch reuses that node's last successful values instead of blanking the entry; a node that was never enriched simply stays un-enriched. +- **Edge case — a node removed or re-addressed mid-sweep**: an in-flight `/v1/models` result is applied only when the node still exists *and* still advertises the same `em` endpoint (`ip` + port) the fetch targeted. Otherwise it is discarded, so it can neither resurrect a removed node nor overwrite a re-addressed one. +- **Edge case — a peer that reports no `clusterUuid`**: silence is not an answer. `NodeInfoResponse.ClusterUUID` is a pointer so an absent field is distinguishable from an explicit `""`, and only the explicit value moves the annotation — reading absence as "unclustered" would mark a clustered peer invitable, which is the failure the annotation exists to prevent. A failed fetch is treated the same way. +- **Edge case — a stranger answering at a peer's address**: the membership sweep compares the `hostUuid` in the response against the entry it is refreshing and skips a mismatch rather than writing one machine's membership onto another's entry. That entry ages out through the liveness probe's identity check, which owns the eviction decision. +- **Edge case — a supersede candidate that moves during the probe**: the proof is a network read taken without the directory lock, so `upsertEvicting` re-validates each verdict against the entry actually stored. A record removed, re-addressed, renamed, or refreshed in the meantime is left alone and judged again from its new state. +- **Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only `ol` / `lm`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal. + +## 4. Open Questions / Risks + +- **Risk — mDNS staleness, by design**: an ungracefully departed node lingers for twelve missed scans (about a minute), up to six scans longer if it vanished in a scan that lost every node, and as long as it keeps answering a probe as itself. The window is long on purpose: evicting a saturated-but-working node fails its in-flight work, while a lingering departed one costs only a failed-and-retried request. A new node still appears within the next 5-second scan. Consumers that route on the directory must tolerate a target that has just gone away. +- **Risk — multicast-hostile networks**: on a segment that blocks or does not forward multicast (guest Wi-Fi, client isolation, some VPN and virtual adapters) nothing is discovered and nothing discovers this node. There is no unicast discovery fallback in this daemon; the only path to such a node is a user-entered manual node, which `nvpair-manual-nodes` probes and the broker merges (§7.7). +- **Risk — interface flapping**: VPN and dock toggles, DHCP re-lease, and sleep/wake re-IP all change the address the node should advertise. `netmon` debounces bursts for 1500 ms and only fires on an observed change, and `reloadIdentity` is idempotent for unchanged fields, so a flap costs a re-resolve and at most one re-announce. A flap that alternates between two equally-ranked addresses will alternate the advertised `ip=`. The published order is also re-ranked on every 15-second sweep as evidence arrives, so a newly discovered peer, a newly peer-proven address, or an interface whose sends start failing can move `ip=` with no interface change at all; each such move is one re-announce. +- **Risk — the responder is built once**: `mdns.NewResponder` fails when no multicast-capable IPv4 interface with a non-loopback address exists (Wi-Fi off at boot, a laptop booted undocked). The daemon logs a warning and continues browse-only, and nothing rebuilds it when an interface later appears, so the node stays unadvertised until the worker is restarted. `watchAddress` is started inside the same responder-present branch, so on such a host `ip=` also stops tracking the interface: the self entry's address is then re-derived only when the broker sends `discovery:reload-identity` or `discovery:reload-trust`. The browser is unaffected (it builds a fresh resolver per scan). +- **Risk — TXT size limit is guarded but not enforced**: `noderec.ValidateTXTSize` implements the RFC 6763 §6.1 255-byte per-string check, but nothing in the daemon calls it. With the model list moved off the record every current key is short and bounded (`v`, `uuid`, `cluster-uuid`, `ip`, `ips` — which `NodeRecord.TXT()` itself caps at four addresses — and one `key=port` per service), so no realistic overflow exists today — but a future long key would be advertised unchecked. Wiring the check into `registry.txt()` is the obvious fix. +- **Risk — hardware inventory freshness is event-driven only**: the daemon GETs each peer's `/v1/node-info` every ~2 s for telemetry and every 15 s for membership, but the telemetry pass keeps only the highest GPU utilization for the scheduler and the membership pass only `hostUuid` and `clusterUuid`; both discard `GPUs`, `cpu`, and `memory`. The directory's inventory fields are written only by a browse change and by `publishSelf`. A stable peer's GPU / CPU / memory utilization is therefore as old as its last record change even though a fresh answer was just on the wire, and neither fetch stamps `lastInfoAt`, so neither spares the liveness probe a dial. The desktop mitigates the staleness by polling each node's `/v1/node-info` itself; a headless consumer of the directory does not get live utilization. +- **Risk — eviction rests on `hostUuid` agreement between two processes**: both delete paths act on a `/v1/node-info` answer naming a different host, which assumes node-info reports exactly the `uuid=` the scanner advertises. The broker keeps them aligned by passing its resolved node id to node-info, and `nvpair-cluster-manager` adopts an existing node id rather than minting a rival one; if that invariant is ever broken the probe reads the skew as a wipe and evicts a live peer. The daemon's own safety argument also assumes a clustered peer's node-info refuses a plaintext caller, which holds only for node-info run with `--cluster-dir` — the broker deliberately does not pass it, so in the shipped configuration the probe reaches a conclusion for clustered peers too. +- **`lastSeen` semantics (open — needs a consumer decision)**: `DirectoryNode.lastSeen` is stamped when an entry is *projected*, so it advances only when the record actually changes, not on every successful scan. It is a last-change timestamp, not a liveness timestamp. A consumer that treats it as "seconds since we last heard from this node" will read a healthy, unchanging node as increasingly stale; `supersedeCandidates` uses the `supersedeMinAge` gap only as a cheap pre-probe filter for exactly that reason, never as evidence. +- **`ServiceStatus.probe` (open — needs scope)**: the field is declared and always `""` (`ProbeUnknown`). The liveness mechanism that matters — probe-before-evict — is implemented, but per-service `reachable` / `unreachable` / `inaccessible` reporting is not, because it would add one dial per service per browse. Until it lands, consumers must not read `probe` as a health signal. +- **`discovery:get-nodes` is served but unused (open)**: the daemon implements it, and it is the natural way to resynchronize a subscriber, but the broker builds its relay directory purely from the event stream and never calls it. It is exercised only by unit tests today. +- **Future — restart replay by epoch**: `noderec.ReadyParams` carries an optional `epoch` so a parent could replay registrations when it observes a new one. The daemon does not send it (its `ready` payload is `{version}` only), and the broker instead replays its registration cache unconditionally on every scanner spawn. The mechanism is documented in the shared type but not implemented here; the current behavior is correct because the broker owns the cache. +- **Future — TLS for node-info enrichment**: the `--client-cert` / `--client-key` / `--ca-bundle` scaffolding is complete and dormant. Nothing in the product passes these flags, and the choice is deliberately global (operator flags only) rather than inferred per node from `cluster-uuid=` or a TXT hint, so the transport policy stays static. +- **Discovery is unauthenticated by nature**: appearing on `_nvpair-node` is a claim, not proof. `uuid=`, `cluster-uuid=`, `ip=`, and every advertised port are self-asserted by whoever sent the packet. `trusted` is the only trust-bearing field, and it is derived locally from the pin set `nvpair-cluster-manager` owns — never from anything on the wire. + +## 5. Requirements + +**Functional** +- Accept `discovery:register`, `discovery:unregister`, `discovery:update-txt`, `discovery:get-nodes`, `discovery:reload-identity`, `discovery:reload-trust`, `discovery:set-observed-addresses`, `discovery:node-activity`, `shutdown`, and `log/set-level` from the parent as either requests (answered) or notifications. The parent sends `discovery:node-activity` as a notification. +- Ignore a `discovery:register` with an empty `service` or a zero `port`, and report no change for a re-register of an already-registered port, so an idempotent replay does not re-announce. +- Build the advertised TXT deterministically: `v`, then `uuid`, then `cluster-uuid` when set, then `ip` when set, then `ips` when there is more than one address (capped at four), then known service keys in a fixed order, then any unknown service keys sorted. +- Stamp identity centrally: `uuid=` from `nodeid.Resolve`, `cluster-uuid=` from live cluster membership, `ip=` and `ips=` from `netpick.LocalCandidates` ranked with this host's own evidence. A registering worker supplies only its key and port. +- Re-rank this host's published addresses on every peer sweep and whenever the peer-observed set changes, replacing rather than merging that set so an address peers stop reaching stops counting as proof. +- Advertise `cluster-uuid=` only while actually a cluster member, never merely because a keypair from a past membership is still on disk. +- Re-derive `trusted` for every directory entry on `discovery:reload-trust`, unconditionally rather than behind a change signal, and emit one `discovery:node-updated` per entry whose answer moved and nothing for the rest. +- Move a peer's `clusterUuid` only on its own explicit report, from a `/v1/node-info` response whose `hostUuid` matches the entry being refreshed; leave the annotation untouched on a failed fetch, an absent field, or a mismatched host. +- Delete a directory record only on proof that the machine at its address is someone else — an address, a hostname, and a `lastSeen` gap taken from an unauthenticated record may nominate, never decide. +- Key the directory by `uuid=`; skip and warn on a record without one. +- Read every service port from TXT and ignore the record's SRV port. +- Dial a node at each address it published, in its own ranked order (`netpick.Candidates`: its `ip=` / `ips=` list, then any other resolved address); accept an answer only from the node's own `hostUuid`, and remember per node and service which address answered. +- Fetch `/v1/node-info` from a node advertising `ni` over plain HTTP, and `/v1/models` from a node advertising `em`; retain the last successful values on failure. +- Sample `/v1/node-info` from every node advertising `ni` on a two-second cadence with stable per-node jitter, back off a failing remote node to 4, 8, 16, then 30 s and reset on success, never back off loopback, and emit `discovery:node-telemetry` without changing the directory. +- Evict only after twelve consecutive missed scans; from the third, probe on every scan and recover the node on any answer; skip the probe for a node with an enrichment under 10 s old or inference activity under 60 s old; and when two or more nodes are known and a scan returns none of them, count no misses for up to six consecutive scans. +- Fetch a peer's `/v1/models` only over cluster mTLS pinned to the principal it advertises; attempt nothing when no pin is held. Fetch this node's own over loopback in plaintext, in every membership state. +- Compare model inventories order-insensitively across the flat list, the per-engine map, and the per-engine loaded map, and emit at most one `discovery:node-updated` per real change. +- Emit `ready` once at startup with the build version. + +**Non-functional** +- Never let a browse event mutate this node's own directory entry. +- Serialize every mutation of this node's own entry so a registration burst and a concurrent identity or address change cannot interleave into a torn entry. +- Run each half of the peer sweep concurrently, bounded at 16 in flight, and to completion before the next tick, so overlapping sweeps are impossible and one slow peer never consumes another's timeout budget. Bound the telemetry loop at 16 in flight as well, with at most one request per node outstanding across ticks. +- Hold at most one connection per node-info origin, and queue a caller for that origin before its request timeout starts, so a caller waiting behind a slow healthy response does not time out before its own request is sent. +- Run the pre-eviction liveness probe outside the browser lock so a dead peer cannot stall readers, and gather a supersedence proof outside the directory lock, re-validating the verdict when the write lands. +- Bound every outbound fetch: 3 s for node-info enrichment (with a 1 s connect and TLS-handshake budget), 6 s for models, 1 s per liveness dial, and 1 s for the probe's own node-info read — the probe runs inside the browser's scan cycle, so it gets a tighter budget than enrichment. +- Reach the identity read only after the TCP sweep has already found the address answering, so a departed node costs no HTTP request at all. +- Survive an unreadable or absent cluster directory, an unavailable network monitor, and a responder that could not be built, in each case degrading rather than exiting. +- Log no prompts, message bodies, model payloads, certificates, or key material. + +## 6. Inputs and Outputs + +**Inputs** — JSON-RPC 2.0 frames from the parent over `stdin` (or the `--ipc` endpoint), including two evidence feeds the broker relays from other workers: `discovery:set-observed-addresses` (from `nvpair-node-info`) and `discovery:node-activity` (from `nvpair-proxy`). Plus two HTTP reads the daemon performs itself: `GET /v1/node-info` on a node's `ni` port and `GET /v1/models` on its `em` port. Multicast DNS answers for `_nvpair-node._tcp` on UDP 5353 are the third input; they are self-asserted and validated only structurally. Cluster membership and the peer pin set are read from the `--cluster-dir` subtree on every refresh. + +**Outputs** — JSON-RPC 2.0 frames to the parent over `stdout` (or the IPC endpoint): the `ready` notification, the `discovery:node-*` event stream, the `discovery:node-telemetry` samples, and responses to requests. On the wire the daemon emits its own `_nvpair-node._tcp` record — PTR, SRV, TXT, and A records with the cache-flush bit, announced twice at startup, re-announced every 60 s and on every interface change, and retired with a TTL-0 PTR goodbye on shutdown. + +`RegisterParams` — the `discovery:register` / `discovery:update-txt` payload. Transport is deliberately absent: it is derived from the service key, never advertised. + +```json +{ + "service": "em", + "port": 14322, + "txt": [] +} +``` + +`DirectoryNode` — one directory entry, carried in every `discovery:node-*` event and listed in `discovery:get-nodes`. + +```json +{ + "hostUuid": "5e1c9a4e-2f6b-4c8d-9a17-b3d0f4c81e22", + "name": "lab-desk-a", + "ip": "192.168.1.10", + "ips": ["192.168.1.10", "10.10.0.2"], + "clusterUuid": "a41f0c2d-7b93-4e50-8c6a-19d2e7f4b088", + "trusted": true, + "services": { + "ni": { "port": 14318 }, + "ol": { "port": 51377 }, + "wl": { "port": 14320 }, + "em": { "port": 14322 } + }, + "gpus": [ + { "name": "NVIDIA GeForce RTX 5090", "vram_bytes": 34359738368, "vram_used_bytes": 2147483648, "utilization_percent": 12 } + ], + "cpu": { "name": "AMD Ryzen 7 9800X3D", "cores": 8, "utilization_percent": 4 }, + "memory": { "total_bytes": 68719476736, "used_bytes": 21474836480 }, + "models": ["llama3:8b", "qwen:0.5b"], + "modelsByEngine": { "ollama": ["llama3:8b"], "lmstudio": ["qwen:0.5b"] }, + "loadedByEngine": { "ollama": ["llama3:8b"], "lmstudio": [] }, + "lastSeen": 1767225600 +} +``` + +`ip` is the node's canonical address and `ips` its full ranked list with `ip` first: for a peer, the record's own `ip=` / `ips=` followed by any other address the browse resolved for it; for this node, the locally ranked candidates. `ips` is omitted only when the node has no address at all. + +`NodeTelemetry` — the `discovery:node-telemetry` payload: + +```json +{ "hostUuid": "5e1c9a4e-2f6b-4c8d-9a17-b3d0f4c81e22", "gpuUtilizationPercent": 12, "telemetryValid": true, "msSince": 640 } +``` + +An example emitted frame — `NodeEvent` wraps the entry under `node`: + +```json +{"jsonrpc":"2.0","method":"discovery:node-discovered","params":{"node":{"hostUuid":"5e1c9a4e-2f6b-4c8d-9a17-b3d0f4c81e22","name":"lab-desk-a","ip":"192.168.1.10","trusted":false,"services":{"ni":{"port":14318}},"lastSeen":1767225600}}} +``` + +An example request and its ack: + +```json +{"jsonrpc":"2.0","id":7,"method":"discovery:register","params":{"service":"wl","port":14320}} +{"jsonrpc":"2.0","id":7,"result":{"ok":true}} +``` + +## 7. API / Interface Contract + +Three interfaces: a **local JSON-RPC interface** toward the supervising parent (newline-delimited JSON-RPC 2.0 over stdio, or a Unix socket / Windows named pipe with `--ipc`); an **mDNS interface** on UDP 5353 that advertises this node and browses for others; and an **outbound HTTP(S) client** that reads `/v1/node-info` and `/v1/models` from discovered nodes. There is no inbound HTTP listener. + +### 7.0 Methods and notifications + +Every method may arrive as a request (acked) or as a bare notification (applied silently). Unknown *requests* are rejected with `-32601` (`method not found: `); unknown notifications are logged and dropped. + +| Method | Params | Result | +|--------|--------|--------| +| `discovery:register` | `RegisterParams` — `{service, port, txt?}` | `{"ok": true}` | +| `discovery:update-txt` | `RegisterParams` — identical handling to `discovery:register` | `{"ok": true}` | +| `discovery:unregister` | `UnregisterParams` — `{service}` | `{"ok": true}` | +| `discovery:get-nodes` | `GetNodesParams` — `{service?}`; absent, empty, or unparseable returns every node | `GetNodesResult` — `{nodes: DirectoryNode[]}`, sorted by `hostUuid` | +| `discovery:reload-identity` | none | `{"ok": true}` | +| `discovery:reload-trust` | none | `{"ok": true}` | +| `discovery:set-observed-addresses` | `ObservedAddressesParams` — `{addresses: string[]}`, this host's addresses peers have reached, replacing the previous set | `{"ok": true}` | +| `discovery:node-activity` | `NodeActivityParams` — `{hostUuid, msSince}`, a peer's engine returned inference bytes `msSince` ms ago | `{"ok": true}`; the broker sends it as a notification | +| `log/set-level` | `{level}` — `debug` \| `info` \| `warn` \| `error` | `{"level": ""}` | +| `shutdown` | none | `null` | + +`discovery:node-activity` arrives while inference is streaming, so the broker sends it without an id: a request would cost a pending entry and a blocked goroutine per report for an ack nobody reads. An activity report for this node is ignored, and `msSince` is clamped to 0–60 s before use. + +Malformed `params` on `discovery:register`, `discovery:update-txt`, `discovery:unregister`, `discovery:set-observed-addresses`, or `discovery:node-activity` return `-32602` with the decode error when the frame carried an `id`, and are dropped with no reply otherwise. An invalid `log/set-level` level returns `-32602` and is warned about. A `register` / `unregister` that changes nothing is still acked `{"ok": true}` — the ack means "applied", not "changed". + +The daemon emits five notifications and never sends a request to its parent. **`discovery:node-telemetry`** carries `NodeTelemetry` for one node per successful sample and is broker-internal scheduling input, not a directory change (§7.4). **`ready`** — `{version}` — is sent once, before the mDNS responder and browser start, and is the readiness signal. **`discovery:node-discovered`**, **`discovery:node-updated`**, and **`discovery:node-removed`** each carry `NodeEvent` (`{node: DirectoryNode}`). `discovery:node-discovered` fires when an entry enters the directory; `discovery:node-updated` fires on a real change to a known entry, including a model-only or membership-only change found by the periodic sweep and a trust change found by `discovery:reload-trust`; `discovery:node-removed` fires on eviction and carries the last-known entry, and also retires a record proven superseded at its address — those removals are emitted *before* the arriving node's own event, so no consumer ever observes one machine under two identities. The one exception is the removal emitted by an identity re-stamp, which carries only `hostUuid` and `name` for the retired identity — enough for a consumer to reconcile by key. + +### 7.1 Service registration and the local record + +`registry` is this node's view of its own advertised services: a `ServiceKey → port` map plus the daemon-stamped identity. It is the only input to the advertised record. + +The canonical service keys are exactly eight, in the order they are emitted into TXT: + +| Key | Service | Port in the product | Derived transport | +|-----|---------|--------------------|-------------------| +| `ni` | `nvpair-node-info` HTTP inventory | `14318` | plain, even when clustered | +| `ol` | `nvpair-proxy` Ollama facade ingress | the facade's listen port | plain | +| `lm` | `nvpair-proxy` LM Studio facade ingress | the facade's listen port | plain | +| `er` | `nvpair-errors` cross-node ingest | `14319` | cluster mTLS when the target is clustered | +| `wl` | `nvpair-workload-manager` event ingest | `14320` | cluster mTLS when the target is clustered | +| `cl` | `nvpair-cluster-manager` pairing + trusted plane | `14321` | split channel — pairing plain, trusted endpoints mTLS | +| `em` | `nvpair-engine-manager` HTTP surface (`/v1/models`) | `14322` | plain | +| `ec` | `nvpair-engine-manager` remote control | `14323` | cluster mTLS when the target is clustered | + +Transport is **not** advertised. A consumer derives it from `ServiceKey.Transport()` plus whether the target node is clustered, because transport is global policy every same-version binary already knows rather than per-instance data. A key absent from TXT means the service is absent on that node. An unknown key parses as a service port and round-trips through the record, so a future service is carried, not dropped. + +In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` from its engine reconciler at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else. + +`register` and `unregister` report whether the record actually changed. Only on a change does the daemon call `Responder.UpdateTXT` (which swaps the TXT and re-announces immediately) and refresh its own directory entry. The refresh runs on its own goroutine so a startup burst of registrations cannot serialize the read loop behind a loopback enrichment fetch. + +### 7.2 The `_nvpair-node` record and TXT layout + +This is an internal contract between the scanner daemons on a subnet. It is deliberately **not** a client contract: the desktop and `nvpair-tui` consume the broker's reported discovery snapshots and never parse mDNS. + +- **Service type**: `_nvpair-node._tcp` in domain `local`. +- **Instance name**: the OS hostname. Display only; never an identity and never unique. +- **SRV port**: the fixed constant `14318`, non-authoritative. Consumers must ignore it and read every port from TXT; it exists only so the SRV record is well-formed and deterministic. +- **TXT**: `v=1;uuid=;cluster-uuid=;ip=;ips=,…;=…`, absent fields omitted. `v` is the schema version, for diagnostics and to disambiguate future record formats — it is not a mixed-version compatibility switch. `uuid` is the stable per-host identity (`nodeid.Resolve`: cluster-manager's `cluster/identity.json` when present, else this host's `node-id.json`, else a freshly minted and persisted UUID v4). `cluster-uuid` is the cluster principal and is present only while this node is a member. `ip` is the first of `netpick.LocalCandidates`, the host's own ranked choice, which is why a consumer prefers it over the A records; `ips` is the whole ranked list, comma-separated and capped at four, and is present only when there is more than one address. +- **Address ranking** (`netpick.LocalCandidates`): loopback, link-local, and the Docker default bridge address are never published. Every other IPv4 address is ordered by, in turn: whether it *qualifies* — not on a point-to-point interface, not a /30–/32 link only the machine on its far end can reach, and not on an interface whose mDNS sends have failed for three consecutive scans; whether a remote peer has actually reached it (relayed from `nvpair-node-info` as `discovery:set-observed-addresses`); whether a discovered peer sits on its interface's subnet; whether it is the kernel's route source address; a physical-adapter preference; an address score; and finally the address string, so the order is stable across restarts. Physical interfaces lead whenever one of their addresses qualifies, and a virtual or tunnel address is then published only if a peer has reached it, and last. Unqualified addresses are still listed, after the qualified ones. At startup no peer or send evidence exists yet, which is why the order is re-ranked on every 15-second sweep and whenever the observed set changes. +- **Announcement cadence**: two announcements about a second apart at startup, then every 60 s, plus one on every observed interface-address change and one on every TXT swap. Records carry a 120-second TTL and the cache-flush bit so a peer replaces rather than accumulates. +- **Goodbye**: on context cancellation the responder multicasts a TTL-0 PTR for the instance before returning, so mDNS caches drop the record and it is absent from peers' next scan. A PAIR peer's browser does not treat the goodbye as an eviction; the absence enters the ordinary miss path (§7.3). +- **Per-interface send path**: `grandcat/zeroconf` binds one socket to the multicast group for both directions, and Windows refuses to send from a socket bound to a group — silently, because the write error is swallowed inside the library. `nvpair-shared/mdns` and `nvpair-shared/discovery` therefore keep the library's receive behavior (a joined socket receives regardless of binding) and send every announcement, reply, and PTR query from a fresh socket bound to the interface's own address and the RFC 6762 source port 5353, with an explicit multicast interface and a TTL of 255 (falling back to 1). The same path is used on every platform; the socket's reuse options let it coexist with the long-lived receive sockets and the OS's own responder, which on macOS additionally needs `SO_REUSEPORT`. Without this a multi-homed Windows host is invisible to the LAN. Each scan's per-interface send outcome also feeds the address ranking above. +- **The model list is not in TXT.** It lives behind `em`'s `/v1/models` because it can grow past the DNS per-string limit and because `nvpair-engine-manager` already owns engine state. This keeps every TXT key short and bounded. + +### 7.3 The peer directory: freshness, eviction, and goodbye + +The browser (`nvpair-shared/discovery`) is a scan-and-diff state machine over `_nvpair-node._tcp`, configured here with the shared defaults and two options: `WithKeyFunc` keying by `uuid=` and `WithLivenessProbe` wiring the daemon's `reachable`. + +- **Cadence**: one scan every 5 s, each bounded at 3 s of listening; the first scan runs immediately. +- **Change detection**: addresses and TXT are compared as multisets, because mDNS returns a multi-homed node's records in a non-deterministic order and an order-sensitive compare would emit a spurious `updated` (and its downstream churn) on every scan. +- **Eviction**: a node absent from a scan has its miss counter bumped. Two thresholds apply. From `probeAfterMisses` (3 misses, about 15 s) the liveness probe asks about the node on **every** scan, and any positive answer resets the counter and recovers the node outright. Only at `missThresholdDefault` (12 misses, a full minute) is it evicted — so eviction rests on a run of roughly ten failed probes, not on one dial that happens to land on the deadline. The window is sized for a node saturated by its own inference load, which stops answering mDNS *and* stops accepting probe connections for as long as the load runs while still serving requests. Up to 8 probes run concurrently. +- **What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight. +- **A scan that loses everyone**: when more than one node is known and a scan returns none of them, no node is charged a miss, for up to `emptyScanGrace` (6) consecutive scans, about 30 s; after that they count normally. Independent machines do not leave a network in the same five-second window, but a process too starved to drain its multicast socket within the scan window hears silence from all of them at once — without this one saturated host would evict its whole cluster, idle peers included, in a single reconcile. With a single known node there is no "all at once" to be suspicious of, so it falls through to the ordinary threshold. +- **A connect is not an identity**: something listening only proves *something* is at that address. A machine whose appdata was wiped mints a fresh `uuid=` while keeping its address, hostname, and fixed ports, so a connect-only verdict would reset the miss counter forever and leave the fleet carrying a permanent duplicate. When the sweep does find an address answering, `identityAlive` reads `hostUuid` from `/v1/node-info` at each published address, 1 s per address. Any address naming the record's own host keeps it; the connect is overruled only when every address that answered named someone else. It asks every address rather than only the first because one wrong answer is not proof of a wipe: a direct-connect link reuses the same subnet on other pairs of machines, so a record's leading address can resolve locally to a different host entirely. Every other answer — no `ni` advertised, no `uuid=` to compare, no response inside the 1-second budget, a response omitting `hostUuid` — is inconclusive and the connect stands. Ordering the identity read after the sweep is deliberate: it only runs when the node would otherwise have been *kept*, which is the sole case where the two need telling apart. +- **Supersedence**: the same duplicate can arrive from the other direction, as a browse event for a new identity at an address the directory already holds. `supersedeCandidates` nominates a stored record sharing the arriving node's `ip` and `name`, carrying a different `uuid=`, and at least `supersedeMinAge` (300 s) older; it deletes nothing, because all three signals come from an unauthenticated record and describe the shape of a spoof exactly as well as the shape of a wipe. Each candidate is then put to the same question at the arriving record's `ni` port, and `upsertEvicting` re-validates the verdict against the entry actually stored before removing it. An unconfirmed candidate stays and is not left uncovered — when it stops being advertised the miss-threshold path asks again. +- **Identity collisions**: when the entry under a key comes back carrying a different `uuid=`, the browser warns about a possible hostname collision or host replacement. +- **Freshness**: `lastSeen` is stamped at projection time, so it tracks the last *change* rather than the last successful scan (§4). There is no TTL-based expiry in the directory itself; the miss threshold plus the liveness probe is the whole expiry mechanism, and a goodbye simply makes the node disappear from the next scan. +- **Self**: this node's entry is registry-driven, published before the browse loop starts, and immune to browse events entirely. It is also excluded from supersedence in both directions — its `lastSeen` advances only when this node republishes, so it can look arbitrarily stale and must not be displaceable by a peer claiming its address, nor displace one. When the LAN ranker has not picked an address yet, the entry falls back to `127.0.0.1` so the local node always has a dialable host. + +### 7.4 Enrichment + +Two independent decorations, both keyed by `hostUuid` and both backed by a last-known-good cache that a transient failure falls back to. A node may advertise one without the other. + +- **Hardware** — `GET http://:/v1/node-info`, 3-second timeout, plain HTTP. The response's `GPUs`, `cpu`, and `memory` land on the entry. This stays plain **even for a clustered peer**: the daemon must never infer mTLS from a node-level `cluster-uuid=`. The only way it becomes HTTPS is the operator setting `--client-cert` / `--client-key` / `--ca-bundle`, which is global and never per node. +- **Models** — `GET /v1/models` on the `em` port, 6-second timeout, deliberately longer than engine-manager's own 5-second models sweep so a peer with several engines is not cut off mid-sweep. The body is `{"models": [...], "modelsByEngine": {...}, "loadedByEngine": {...}}`; the last two are additive, so a peer that omits them yields nil maps while the flat union still enriches the node. A present engine key with `[]` is a successful empty inventory; a missing key means that engine was not running or queryable. +- **Model transport is the cluster gate.** Which models a machine holds is cluster data, so a **peer** is fetched only over mTLS pinned to the principal it advertises. No pin, no fetch — there is deliberately no plaintext fallback for a peer. **This node's own** engine-manager is dialed over loopback in plaintext in every membership state, which is what keeps a standalone machine's own model list working. +- **Dial host**: a peer is dialed at each address it published, in its own order, until one answers for the expected `hostUuid`; an answer from a different host is skipped rather than attributed to the entry. The address that answered is remembered per node and per service (`ni` and `em` can be reachable at different addresses), and later rounds ask only it until it stops answering — otherwise a multi-homed peer whose leading address this host cannot reach would cost that address's full timeout on every sweep. This node is always dialed at `127.0.0.1`, for enrichment, the periodic refresh, and telemetry, so a first-run firewall block on inbound to its own LAN address cannot blank the local entry. +- **Node-info connections**: the node-info client holds at most one connection per origin, with a 1-second connect and TLS-handshake budget. A caller for an origin waits its turn *before* its own request timeout starts, so one waiting behind a slow but healthy response does not expire before its request is sent. A response body is drained up to 64 KiB so the connection can be reused. A peer whose node-info stops answering logs one warning, and one info line when it answers again. +- **Scheduling telemetry**: independent of enrichment, every ~2 s (a stable per-node offset within ±250 ms) the daemon GETs `/v1/node-info` from each node advertising `ni`, at most 16 in flight and one outstanding per node. It keeps only the highest `utilization_percent` across `GPUs`, `telemetryValid`, and `msSince` advanced by the request's own latency, and emits them as `discovery:node-telemetry`. A remote node whose sample fails backs off to 4, 8, 16, then 30 s and resets on success; loopback never backs off. A failure emits nothing, so downstream freshness expiry handles it; a response naming another host is never attributed. Telemetry writes nothing to the directory or the enrichment caches. +- **Periodic convergence**: every 15 s the daemon first re-ranks this host's published addresses (§7.2), then runs two passes in order, membership then models, because the model fetch is pinned mTLS keyed on the peer's principal and must dial a peer that has just joined or left under its current identity. The membership pass snapshots every peer advertising `ni` with a usable address (self excluded — `publishSelf` owns the local principal) and reads `clusterUuid` from each; the model pass snapshots every node advertising `em` with a usable address. Each pass refreshes its nodes concurrently, capped at 16 in flight and run to completion before the next tick. A model result is applied under the directory's removed-or-re-addressed guard, and the enrichment cache is written *before* the directory so a concurrent browse-driven fallback whose own fetch failed cannot revert the update with a stale cached value. Only a real change emits an event, and each pass writes only its own fields — the membership pass touches `clusterUuid` and `trusted`, the model pass the three model fields, and everything else is left to the browse path. +- **Eviction clears the caches**, so a re-discovered node re-probes from scratch rather than resurrecting stale values. + +### 7.5 Network-change handling + +`nvpair-shared/netmon` is the single watcher for host addressing, with platform backends and a 10-second polling fallback; it debounces bursts for 1500 ms and notifies only on an observed change, so a duplicated or spurious OS event produces nothing. + +Two consumers react. The **responder** refreshes its per-interface address map and re-announces with the cache-flush bit, so peers drop a stale A record promptly; it keeps the last good set rather than going dark when enumeration returns nothing, and it does not re-join the multicast group on a newly appeared interface, so queries arriving there are unanswered until restart while the periodic and on-change announcements still publish on it. The **daemon** re-runs `reloadIdentity`, which re-ranks `ip=` / `ips=` from `netpick.LocalCandidates` and re-advertises if they changed — without this the advertised address would freeze at startup, and because consumers prefer `ip=` over the live A records, a stale value would misroute them. When the monitor cannot start, the daemon logs a warning and the advertised `ip=` stays static. + +`reloadIdentity` has three triggers — the broker's explicit `discovery:reload-identity` nudge, `discovery:reload-trust` by way of `reconcileIdentity`, and a network change — and one body: re-resolve `uuid=`, refresh membership, re-rank the addresses, and return early when nothing changed. Address ranking alone also re-runs on every 15-second sweep and when the peer-observed set changes, because its strongest evidence does not exist yet at startup. When `uuid=` changed it also drops the entry under the old UUID (moving its enrichment caches to the new one), emits `discovery:node-removed` for it, and republishes self under the new identity. It is serialized against `publishSelf` by the same lock. + +### 7.6 Cluster membership and trust + +The daemon holds a live `clustertrust.Mesh` over `--cluster-dir` and re-reads it rather than snapshotting, because membership is mutable at runtime: `nvpair-cluster-manager` mints the keypair, activates an admission, and writes pins whenever the user creates, joins, or leaves a cluster — in any order relative to this process starting. + +- **Membership** is a loadable identity *and* either a live admission (`admission.json` with a non-empty `clusterId` and a non-zero epoch) or at least one pinned peer. It is deliberately **not** "do I hold a keypair": a leave or removal clears the admission and the pins but keeps `node.key` / `node.crt` so the same identity can be reused on a rejoin. Advertising on keypair presence would make a departed node look clustered to the whole fleet, so peers could never invite it back. +- **`discovery:reload-trust`** is how a pin-set change reaches the daemon. `nvpair-cluster-manager` reports `cluster:trust-changed` only after the pin is on disk and the broker relays it here, so the directory this handler reads is never older than the event that prompted it. The handler refreshes the mesh and then does two things unconditionally: `reconcileIdentity` compares live membership against the `cluster-uuid=` the registry is publishing and calls `reloadIdentity` when they disagree, and `reconcileTrust` re-derives `trusted` for every directory entry. Recomputing rather than gating on a "did it change?" answer is the point — a shared one-shot change signal is consumed by whichever caller refreshes first, which is how a membership hook can silently never fire. +- **`trusted`** is set for a browsed peer when the daemon holds a pin for the `cluster-uuid=` it advertises, refreshing the pin set first. For this node's own entry the same check applies and resolves through self-trust whenever it is clustered; self is included in the reconcile pass like any other entry. `trusted` is a local statement about pins, not a claim carried on the wire, and it is the only trust-bearing field in the directory. +- **A peer's membership converges without mDNS.** A peer's principal otherwise arrives only as `cluster-uuid=` on a record read once per browse *change*, and the liveness probe deliberately keeps a still-reachable peer's entry alive, so an unreceived re-advertisement would leave a departed peer's principal in place for the life of the process — suppressing the invite that would bring it back. The 15-second membership pass reads `clusterUuid` from each peer's own `/v1/node-info`, which under the broker is served plain and so answers whether or not the two nodes share a cluster, and reaches it over TCP at the peer's published addresses, sharing nothing with multicast reception. `applyClusterIdentity` writes the principal and the trust derived from it together, since a peer that left is no longer pinned under the principal it held. +- **What membership gates**: whether `cluster-uuid=` is advertised at all, and whether a peer's `/v1/models` is fetched. It gates nothing else — a peer with no pin is still discovered, still listed, still address-resolved, and still hardware-enriched, because `ni` is plain by policy. +- **What this daemon does not do**: no key generation, no certificate handling beyond handing a pinned config to an HTTP client, no PIN handling, no membership mutation, and no cryptography of its own. All of that is `nvpair-cluster-manager`'s. + +### 7.7 The broker relay, subscriptions, and the manual-node merge + +`discovery:subscribe` is a **broker** method, not a scanner method — the daemon neither implements nor needs it. The broker reads the daemon's delta stream on the scanner peer's read pump and folds every event into two structures: + +- `relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events. +- The broker's client-facing `discoveryStore` projects each `DirectoryNode` onto its own `AvailableNode` shape and is what `discovery:get-nodes` and the `discovery:nodes-changed` push serve to the desktop and `nvpair-tui`. This is the layer clients consume; they never see `_nvpair-node`, its TXT keys, or its service keys. + +The broker also carries three feeds that are not directory events. It caches each `discovery:node-telemetry` sample as the scanner source in its telemetry cache — a scanner observation outranks the manual-node worker's for the same host — and relays the aged sample to `nvpair-job-scheduler`. It relays `nvpair-node-info`'s `nodeinfo:observed-addresses` down to this daemon as `discovery:set-observed-addresses`. And it relays each proxy's report of inference bytes from a peer down as the `discovery:node-activity` notification. + +**Manual nodes are merged there, not here.** The scanner has no manual-node concept. `nvpair-manual-nodes` probes a user-entered address, learns its real `hostUuid` from `/v1/node-info`, and the broker records that as a second *source* claim on the same `hostUuid` key. Each record tracks per-source ownership; when both claim it the scanner (mDNS) view wins, because it is the authoritative discovery record and carries `trusted`, `clusterUuid`, and per-engine models. A record is forgotten only when no source claims it, so removing a manual node never evicts a live mDNS node, and a `discovery:node-removed` that drops the scanner's claim while a manual claim survives is not treated as a departure. That final-claim distinction is what the broker uses to decide whether to fail the workloads pinned to a departed node. + +### 7.8 Versioning + +`v=1` in TXT identifies the record schema. New service keys and new `DirectoryNode` fields are **additive** and require no version bump: an unknown TXT key round-trips as a service port, and an unknown JSON field is ignored by an older consumer while `omitempty` keeps it off the wire when unset. `modelsByEngine`, `loadedByEngine`, and `ips` were added this way. A change to the meaning of an existing key, to the identity keying, or to the SRV-port-is-non-authoritative rule is a breaking change and bumps `v`; the schema is a flag-day rollout across the bundle rather than a mixed-version compatibility surface, since every binary ships together. The JSON-RPC method and notification names are frozen; new methods are additive. The binary's `main.Version` is stamped from `services/versions.json` and reported in `ready` and by `--version`. + +## 8. Dependencies + +**Upstream** +- A supervising parent that speaks the §7.0 contract over stdio or `--ipc`. In practice `nvpair-ui-broker`, which spawns the daemon with `--log-level` and, when configured, `--cluster-dir`; the daemon is otherwise supervisor-agnostic. +- `nvpair-cluster-manager`, indirectly and asynchronously: it writes the identity, admission record, and pins that the daemon reads from the cluster dir. The daemon works fine before it has ever run. + +**Downstream** +- `nvpair-ui-broker` — the only direct consumer of the event stream and the telemetry samples; it relays to every worker and to clients, and it feeds back the observed-address and node-activity evidence. +- Transitively, through the relay: `nvpair-proxy`, `nvpair-errors`, `nvpair-workload-manager`, `nvpair-cluster-manager`, `nvpair-engine-manager`, `nvpair-job-scheduler` (which also receives the GPU telemetry), and the desktop UI and `nvpair-tui`. +- Peer `nvpair-node-info` and `nvpair-engine-manager` instances, which this daemon reads over HTTP(S). + +**External** +- `nvpair-shared`: `mdns` (responder and per-interface sender), `discovery` (browser), `noderec` (record and RPC contract), `nodeid`, `netpick` (local ranking and remote candidates), `reach`, `netmon`, `clustertrust`, `jsonrpc`, `ipc`, `applog`, `appdir`. +- `github.com/grandcat/zeroconf` for the browse side, `github.com/miekg/dns` for record construction and the unicast query re-send, `golang.org/x/net/ipv4` for per-interface multicast control, `github.com/Microsoft/go-winio` for the Windows named-pipe transport. +- A multicast-capable IPv4 network with UDP 5353 reachable in both directions on the local subnet. + +## 9. Data Ownership + +**Owned** +- The in-memory registry of this node's advertised services and the identity stamped onto them. +- The in-memory peer directory (`hostUuid → DirectoryNode`) and its per-node enrichment caches, the remembered answering address per node and service, the telemetry retry state, the last inference-activity time per peer, and the current peer-observed address set. +- The advertised `_nvpair-node._tcp` record for this host. + +**Source of truth** +- Yes, for **network discovery from this node's perspective**: which nodes are present, their canonical dialable address, their advertised service ports, and their enriched hardware and model inventory. The broker's directory and store are projections of this stream. +- No, for node identity (`nvpair-cluster-manager` via `nodeid`), cluster membership and trust (`nvpair-cluster-manager`), hardware inventory (`nvpair-node-info`), model inventory (`nvpair-engine-manager`), or manual nodes (`nvpair-manual-nodes`). + +**Storage** +- Nothing durable is owned. The directory, the registry, and the caches are memory-only and rebuilt from scratch on every start. +- The one file the daemon can cause to be written is the fallback node id, `/node-id.json`, minted by `nodeid.Resolve` only when `cluster/identity.json` does not exist. It is written atomically at `0600` inside a `0700` directory, and it is superseded — not deleted — once cluster-manager's identity appears. +- It **reads** `/cluster/`: `identity.json`, `node.crt`, `node.key`, `admission.json`, and `trusted/.json`. All of those belong to `nvpair-cluster-manager`. +- `` is the per-user data directory every component shares: `%LocalAppData%\Nvidia Corporation\Personal AI Router` on Windows (deliberately non-roaming), `$XDG_CONFIG_HOME`/`~/.config/Nvidia Corporation/Personal AI Router` on Linux, `~/Library/Application Support/Nvidia Corporation/Personal AI Router` on macOS. `--cluster-dir` overrides the cluster subtree, and its parent then becomes the base the node id is resolved from, so both stay consistent under a custom root. + +## 10. Design Constraints + +- **Performance**: one 5-second scan, one 15-second peer sweep (address re-rank, then membership, then models), a ~2-second telemetry sample of every node, and a 60-second re-announce per process. A node missing from mDNS for more than about 15 s costs up to one probe dial per address per scan. Fetches are bounded per request (3 s node-info, 6 s models) and per pass (16 concurrent), and the address memory means a steady-state round asks each node one address. Telemetry is the steady cost: every node samples every other node's `/v1/node-info` about every two seconds, which at a dozen nodes is a handful of small requests per second per host. The daemon holds no lock across a network call: the liveness probe and the supersedence proof both run outside their locks, and the enrichment caches and the directory use separate locks with a defined write order. +- **Scalability**: sized for a subnet of roughly a dozen nodes; telemetry sampling grows with the square of the node count across the fleet. The concurrency cap of 16 sits above realistic LAN counts, so it bounds a pathological fan-out rather than throttling normal operation. Directory snapshots are copied and sorted per call, which is fine at this scale and would need revisiting at hundreds of nodes. +- **Reliability**: every external dependency degrades rather than fails. No cluster dir means permanently unclustered. No network monitor means a static `ip=`. No responder means browse-only (see the §4 recovery caveat). A failed enrichment reuses the last good value. A failed announcement is retried by the next cadence tick. +- **Security**: discovery is unauthenticated by design — every TXT value is a self-assertion, and presence on `_nvpair-node` is never proof of trust. The daemon implements no cryptography: it reads `nvpair-cluster-manager`'s pins through `clustertrust` and asks it for a pinned TLS config. A peer's model inventory is fetched only over pinned mTLS, with no plaintext fallback. Hardware inventory is plain HTTP even for a clustered peer, by explicit policy, and TLS is never inferred from a node's advertised `cluster-uuid=`. The daemon exposes no listening TCP port and no inbound HTTP surface; its only bound socket is the mDNS UDP group. Private key material is read only by `clustertrust` to build a TLS config and is never logged or transmitted. +- **Compliance**: TXT and directory payloads carry opaque UUIDs, an OS hostname, LAN addresses, ports, hardware model strings, and model names — no PII, no credentials, no inference content. + +## 11. Assumptions + +- A parent supervises the process and closing its `stdin` is the shutdown signal; there is no reconnect or buffering. +- Exactly one scanner runs per host. Two would advertise conflicting records under the same instance name. +- All nodes on the subnet run the same bundle version, so the record schema is uniform (`v=1` is a diagnostic, not a compatibility switch). +- The parent registers a service only when it is actually listening, and unregisters when it stops. The daemon does not verify a registered port. +- The OS hostname is available and stable enough to serve as a display label; identity never depends on it. +- The per-user config directory is readable and writable by the running user. +- `nvpair-node-info` and `nvpair-engine-manager` serve `/v1/node-info` and `/v1/models` on the ports advertised as `ni` and `em`. +- A node's `/v1/node-info` `hostUuid` names the same identity as its advertised `uuid=`. Both eviction paths read a mismatch as a replaced machine, so a skew between the two would evict a live peer. +- Loopback is never firewalled, which is what makes self enrichment reliable on a first-run Windows host. + +## 12. Failure Modes and Mitigations + +- **This node's own multicast does not loop back** (routine on Windows): the browser ages the local node like a peer and evicts it from its own directory. → Self is registry-driven and every browse event matching this node's `uuid=` is discarded, so the local entry exists from startup and cannot be evicted by a browse. +- **First-run firewall blocks inbound to this host's own LAN address**: self enrichment fails and the local card shows no metrics and no models. → Self is always dialed at `127.0.0.1`, for the registry-driven publish, the periodic model refresh, and telemetry, while still displaying the advertised LAN address. +- **A multicast gap, transient or sustained, makes a healthy node miss scans**: it would be evicted and every consumer would drop it as a routing target. → Wait a full minute of misses before evicting, and from about 15 s in probe it on every scan: keep it on an enrichment under 10 s old, and otherwise dial its `ni` and then its `em` port at each published address and recover it on any answer. +- **A node saturated by its own inference load stops answering mDNS and probes**: its control plane is starved while it keeps serving requests, so it looks departed exactly when it is busiest, and evicting it fails its in-flight work. → The one-minute window absorbs most bursts, and inference bytes seen from it in the last 60 s keep it with no probe at all. +- **This host is too starved to read its own multicast socket**: every known node vanishes from the same scan, and charging each a miss would evict the whole cluster, idle peers included. → When two or more nodes are known and a scan returns none of them, no misses are charged for up to six consecutive scans. +- **A multi-homed peer's leading address is unreachable from here** (a direct-connect link to another machine, a VPN-only address): every fetch would spend its timeout on that address first, forever. → Walk the peer's published addresses in order, skip answers from other hosts, and remember per node and service which address answered. +- **A wiped and reinstalled node duplicates itself across the fleet**: it re-binds the same address, hostname, and fixed ports under a fresh `uuid=`, so a connect-only probe reports the pre-wipe record alive on every pass and the stale entry never ages out. → Ask the machine who it is: `identityAlive` reads `hostUuid` from `/v1/node-info` at each published address and overrules the connect only when every answer names a different host, while `supersedingUpsert` puts the same question to a candidate the arriving record is claiming the address of. Neither deletes on anything weaker. +- **An mDNS record claims a live peer's address and hostname**: treating the directory's own nomination signals as evidence would turn mDNS into a removal primitive, and since the browser re-reports a node only when its record *changes*, the evicted peer might never come back. → Nominate on address, hostname, and age; delete only on a `/v1/node-info` answer naming a different host, and re-validate that verdict against the stored entry before applying it. +- **A peer leaves without a goodbye** (crash, unplug, suspend): it lingers as an apparently valid target. → The miss threshold plus the failing liveness probe evicts it within roughly a minute; the proxies and scheduler tolerate a target that has just gone away, and its scheduler telemetry goes stale and turns neutral well before that. +- **`/v1/node-info` or `/v1/models` fetch fails transiently**: naively the entry would be blanked, flickering the node's card and its routing eligibility. → Reuse the node's last successful values; write no cache entry and emit no event on failure. A node never enriched simply stays un-enriched. +- **A `/v1/models` result lands after the node was removed or re-addressed**: it could resurrect a ghost or overwrite a moved node. → `applyModels` re-checks that the node still exists and still advertises the same `em` address and port, and discards the result otherwise. +- **A record arrives with no `uuid=`**: it cannot be keyed by a stable identity. → Warn and skip it. Keying by hostname is explicitly rejected — identity is the UUID everywhere, with no fallback. +- **Two hosts share a hostname**: they would collapse into one directory entry, merging two machines' services and models. → Key by `uuid=`, and warn when a key's identity changes. +- **The scanner starts before cluster-manager on a fresh host**: it mints and advertises its own node id, so the fleet keys on an identity that is about to change. → The broker triggers `discovery:reload-identity` once cluster-manager reports its node id; the daemon adopts the principal, moves its directory entry, and emits a removal for the stale UUID so no ghost is served. +- **This node joins or leaves a cluster while the daemon runs**: a scanner that only learned membership at startup would leave the node looking unclustered to the whole fleet — skipped by every peer's workload broadcast and dropped as a routing target while still sitting in the roster. → `discovery:reload-trust` re-derives membership from disk and re-advertises `cluster-uuid=` in place, comparing against the value the registry is publishing rather than against a change signal, so a duplicated or coalesced announcement is harmless and a missed one costs nothing beyond the next announcement. +- **A peer is pinned after this node already discovered it**: its record never moves again, so the browse-time trust annotation would stay `false` for the life of the process and both `nvpair-proxy` facades would drop it as an unpinned relay peer. → The same `discovery:reload-trust` pass re-derives `trusted` for every entry, and a removal propagates by the identical route. +- **A peer's cluster departure never reaches this host over multicast**: the stale principal outlives the cluster, and a client suppresses the invite that would bring the peer back. → The 15-second membership pass reads the peer's own `clusterUuid` from `/v1/node-info` and clears it; silence (a failed fetch, an absent field, a mismatched `hostUuid`) leaves the annotation alone rather than inferring a departure. +- **The `trusted/` directory momentarily will not read** (an antivirus lock on a freshly written pin, descriptor exhaustion): replacing the pin set with an empty one would untrust every peer for a tick and drop them all out of routing. → `clustertrust` keeps the previous set when the directory exists but cannot be read, and drops a pin only on a successful read that no longer lists it — so a teardown still propagates immediately. +- **A keypair from a past membership remains on disk after a leave or removal**: gating on keypair presence would advertise the node as clustered forever, so peers could never invite it back. → Membership requires a live admission or a pinned peer, never merely a loadable identity. +- **No multicast-capable interface at startup**: `mdns.NewResponder` fails, the node advertises nothing, `watchAddress` never starts, and nothing rebuilds the responder when an interface appears. → Logged as a warning and browse-only operation continues; recovery of the advertisement requires a worker restart, and the supervisor's restart path is the mitigation (§4). +- **The network monitor cannot start**: `ip=` would freeze at its startup value and could point at an address the host no longer holds. → Logged; the responder's own address watch still refreshes the A records, the 15-second sweep still re-ranks `ip=` / `ips=`, and an explicit `discovery:reload-identity` re-derives them. +- **A multicast-hostile segment**: nothing is discovered in either direction. → Not fixable here; the user adds the peer manually and `nvpair-manual-nodes` plus the broker's store cover it. The daemon adds no unicast discovery fallback. +- **Malformed `params` on a registration frame**: a bad frame could corrupt the advertised record. → Decode first and reject with `-32602` (when the frame is a request) before touching the registry. +- **The parent exits** (`stdin` EOF): an orphaned daemon would keep advertising a node whose services are gone. → EOF ends the read loop, which cancels the daemon context; the responder sends its goodbye and the process exits cleanly. `SIGINT` / `SIGTERM` and a `shutdown` request take the same path. + +## 13. Observability + +- **Logging**: structured `slog` on stderr with an `[nvpair-node-scanner]` prefix, level set by `--log-level` (falling back to `$NVPAIR_LOG_LEVEL`, then `info`) and mutable at runtime via `log/set-level`. At warn: mDNS advertising disabled, responder exit, browse and resolver errors, a query that left no interface, a record skipped for a missing `uuid=`, an identity change under one key, an eviction for a record superseded at its address (from either delete path, with both UUIDs), a failed node event emit, and an unavailable network monitor. At warn also: a peer's node-info that has stopped answering (once per outage) and a failed telemetry emit. At info: a peer's trust or cluster membership moving, a peer's node-info answering again, this host's published addresses being re-ranked (with the new `ip` and list), log-level changes, transport selection, and clean shutdown. At debug: per-fetch enrichment failures with the target and URL, answers skipped because a different host replied at one of a node's addresses, a liveness probe skipped for recent inference activity or kept without settling identity, model enrichment skipped because a peer is unpinned, membership skipped because a different host answered at the address, per-interface mDNS send outcomes, and re-announce decisions. Never logged: private keys, certificate PEMs, `/v1/models` or `/v1/node-info` bodies, inference content of any kind. +- **Metrics**: there is **no metrics endpoint and no metrics emission** — the daemon exposes no HTTP listener at all. Operational signals must be derived from the log stream or inferred by a parent from the event stream (directory size from `discovery:get-nodes`, churn from the `discovery:node-*` rate, enrichment health from the debug failures). Adding counters for scan outcomes, evictions, probe saves, and per-endpoint fetch success would need a new surface. +- **Alerts**: none are raised by this process. The signals worth watching in the aggregated logs are a sustained "query did not leave any interface" (multicast is broken on this host), repeated eviction and rediscovery of the same UUID (a flapping peer or an over-tight threshold), a supersedence eviction that is not explained by a reinstall (an identity skew between a node's `uuid=` and its `/v1/node-info`), "mDNS advertising disabled" at startup (this node is invisible and cannot self-heal), and a peer that stays permanently unenriched (a firewalled `ni`, or a pin this node does not hold). + +## 14. Sample usage + +The broker spawns `nvpair-node-scanner --log-level info --cluster-dir /cluster` over stdio. The daemon resolves its identity, opens a live view of the cluster dir, builds a browser for `_nvpair-node._tcp` keyed by `uuid=` with the liveness probe wired, and constructs a responder for instance `lab-desk-b` on the fixed SRV port `14318`. It emits its `ready` frame carrying the version stamped into the binary at build time, then starts the responder, the address watch, its own directory entry, the browse loop, the peer sweep, and the telemetry loop. The broker, meanwhile, replays its registration cache: `discovery:register {"service":"ni","port":14318}`, then `wl`, `er`, `em`, and `cl` as those workers come up. Each changes the record, so each triggers an `UpdateTXT` re-announce and a `publishSelf` that emits `discovery:node-updated` for `lab-desk-b` with a growing `services` map. The advertised TXT settles at `v=1;uuid=;ip=192.168.1.11;ni=14318;er=14319;wl=14320;cl=14321;em=14322`. + +On the same subnet, `lab-desk-a` powers on and announces its own record. Within 5 s the browse returns it; `lab-desk-b` has never seen this key, so the reconcile yields a `discovered` event. `toDirectoryNode` parses TXT into a `DirectoryNode` keyed by `hostUuid` `5e1c9a4e-…`, named `lab-desk-a`, addressed at its `ip=` of `192.168.1.10`, with `ni`, `ol`, `wl`, and `em` ports read from TXT and the SRV port ignored. Because the record carries a `cluster-uuid=` and `lab-desk-b` holds a pin for it, the entry is annotated `trusted: true`. Enrichment then runs twice: `GET http://192.168.1.10:14318/v1/node-info` fills in the RTX 5090, the Ryzen, and 64 GiB; `GET https://192.168.1.10:14322/v1/models` — over mTLS pinned to that principal, presenting `lab-desk-b`'s own leaf — fills in `models`, `modelsByEngine`, and `loadedByEngine`. The daemon caches both against `5e1c9a4e-…` and emits `discovery:node-discovered` with the complete entry. + +The broker's scanner read pump folds that one event into both halves of its relay: `relay.Directory.Apply` upserts the node and then re-pushes **every** subscriber its full filtered set, so the Ollama facade of `nvpair-proxy` receives a `discovery:nodes` notification listing `lab-desk-a` (it matches the `ol` filter) and immediately treats it as a routing candidate, while `nvpair-workload-manager` receives its own `wl`-filtered set. In parallel the broker's client store upserts the node under the scanner source and — because the desktop has opted in via `discovery:subscribe` — pushes a fresh `discovery:nodes-changed` array of `AvailableNode`s, and the node card appears in the UI with its GPU, its models, and a trusted badge. No client ever parsed a TXT record. + +From then on the telemetry loop samples `lab-desk-a`'s `/v1/node-info` about every two seconds at the address that answered, and emits `discovery:node-telemetry` with the 5090's utilization, `telemetryValid: true`, and the sample's age. The broker caches it and relays it to `nvpair-job-scheduler`, which uses it for GPU pressure. None of those samples touches the directory. + +Some minutes later a user pulls a model on `lab-desk-a`. Its mDNS record does not change, so no browse event fires anywhere. On the next 15-second tick `lab-desk-b` re-fetches `/v1/models`, finds one added name, refreshes its cache, applies just the model fields under the re-address guard, and emits exactly one `discovery:node-updated` — which reaches `nvpair-proxy` and the UI by the same two paths. When `lab-desk-a` is later shut down cleanly, its responder multicasts a TTL-0 goodbye; it is absent from the next scan, its telemetry samples start failing and back off, and from the third miss onward every probe of its `ni` and `em` ports is refused. At the twelfth consecutive miss, a minute after it left, `lab-desk-b` drops it, forgets its caches, and emits `discovery:node-removed`. The broker drops the scanner's claim, finds no manual claim surviving, reports the node lost so its in-flight workloads are failed, and pushes the shorter set to every subscriber. + +## 15. Process model, CLI, and build wiring + +**Process model.** A single Go binary speaking newline-delimited JSON-RPC 2.0. With no `--ipc` it uses `stdin`/`stdout` (`stdout` is reserved exclusively for protocol frames; all logging goes to `stderr`); with `--ipc ` it dials a Unix domain socket or a Windows named pipe and uses that for both directions. It runs no listener: the only bound socket is the mDNS group `224.0.0.251:5353` with `SO_REUSEADDR`, so the responder, the browser's resolver, and every other process on 5353 coexist. Every other network operation is an outbound dial. In the product the broker is the parent; the binary is supervisor-agnostic and the parent hides the console window on Windows. + +**Shutdown** has three equivalent triggers: `stdin` EOF (the parent closed the pipe), `SIGINT` / `SIGTERM`, and a `shutdown` request — which is answered before the cancel so the parent sees the reply. All three cancel the root context, which stops the browse loop, the peer sweep, the telemetry loop (draining its in-flight samples), the address watch, and the responder; the responder sends its goodbye on the way out. A JSON-RPC read error that is not EOF is logged and the loop continues, so one malformed frame does not kill the process. A fatal startup problem — an invalid TLS flag combination, an unloadable cert or CA bundle, or an unreachable `--ipc` endpoint — exits non-zero before any advertising begins. + +**CLI flags** + +| Flag | Default | Meaning | +|------|---------|---------| +| `--ipc ` | _(stdio)_ | Unix socket or Windows named pipe instead of `stdin`/`stdout`. | +| `--cluster-dir ` | _(none)_ | Cluster subtree (`node.crt`/`node.key`, `admission.json`, `trusted/`). Enables `cluster-uuid=` advertising, the `trusted` annotation, and peer model fetches; its parent becomes the base the node id resolves from. | +| `--client-cert ` | _(none)_ | PEM client certificate for node-info enrichment over HTTPS. Requires `--client-key`. | +| `--client-key ` | _(none)_ | PEM private key matching `--client-cert`. Requires `--client-cert`. | +| `--ca-bundle ` | _(none)_ | PEM CA bundle for verifying node-info server certificates, added on top of the system trust store. | +| `--log-level ` | `info` | `debug` \| `info` \| `warn` \| `error`; falls back to `$NVPAIR_LOG_LEVEL`. Mutable at runtime via `log/set-level`. | +| `--version` | | Print `main.Version` and exit. | + +Setting any of the three TLS flags moves node-info enrichment onto HTTPS; setting none keeps it plain, which is the shipped configuration — the broker passes only `--log-level` and, when configured, `--cluster-dir`. `--client-cert` and `--client-key` must be given together or not at all; `--ca-bundle` alone means "trust these roots but present no client identity". No flag carries the node identity or the cluster identity, so the binary self-bootstraps under any supervisor. + +**Build wiring.** `nvpair-node-scanner` is one of the twelve shipped binaries. Its version lives at `components.nvpair-node-scanner` in `services/versions.json` and is stamped in with `-ldflags "-X main.Version=…"`; it is never hardcoded. `services/build.sh` and `services/build.bat` each build it and stage it into `services/build/bin/`, and `desktop` builds the same tree into `desktop/cli-bin/` via `npm run build:modular-binaries`. It is installed by `services/installer/nvpair-setup.nsi`, listed in `services/bom.md`, and — because it is the host's mDNS listener — needs **two** Windows firewall rules that the installer adds and removes as a pair: a program rule for the executable and an inbound `UDP 5353` rule scoped to `localsubnet`. It needs no TCP rule, since it listens on no TCP port. The desktop side marks it `needsFirewallAccess: true` in `desktop/src/shared/constants/modular-binaries.ts` with `launchOwner: 'broker'`, and `npm run service-contracts:check` verifies that flag against the macOS privileged-helper list. From bd093180ac0e3559453271ccaa8cecedea95e810 Mon Sep 17 00:00:00 2001 From: Terve Date: Mon, 12 Oct 2026 12:59:33 -0400 Subject: [PATCH 2/2] docs(nvpair-node-scanner): add the lc service key for llama.cpp #148 added ServiceLlamaCPP ("lc") to noderec and to serviceKeyOrder between lm and er, so there are nine canonical service keys, not eight. Add the lc row to the section 7.1 table and name lc wherever the spec listed only ol and lm: the broker's engine registrations in sections 3 and 7.1, the facade subscriptions in 7.7, and the inference ports the liveness probe never dials. Signed-off-by: Terve --- services/nvpair-node-scanner/spec.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/services/nvpair-node-scanner/spec.md b/services/nvpair-node-scanner/spec.md index 40f55ad99..3a78cd3e4 100644 --- a/services/nvpair-node-scanner/spec.md +++ b/services/nvpair-node-scanner/spec.md @@ -38,7 +38,7 @@ It exists so that discovery has exactly one implementation per host. One record ## 3. Key Use Cases -- **Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` from its engine poller at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry. +- **Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` / `lc` from its engine pollers at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry. - **Discover a peer**: a scan returns a peer's `_nvpair-node` record; `toDirectoryNode` projects TXT into a `DirectoryNode` keyed by `uuid=`, enrichment fills in hardware and models, and `discovery:node-discovered` carries the whole entry to the broker. - **A peer changes**: a peer starts an engine (a new `ol=` key) or moves address (`ip=` / `ips=`); the order-insensitive record compare in `nvpair-shared/discovery` sees a real change, re-enrichment runs, and `discovery:node-updated` fires. A scan that returns the same addresses and TXT in a different order emits nothing. - **A peer leaves**: it stops answering. From its third consecutive missed scan (about 15 s) the daemon dials its `ni` port and then its `em` port at each published address on **every** scan, and any answer resets the miss counter. Only after twelve consecutive misses (a full minute) with every probe failing is the entry dropped, its enrichment caches forgotten, and `discovery:node-removed` emitted. A clean shutdown multicasts an RFC 6762 goodbye (TTL 0 PTR), which clears the record from mDNS caches so it is absent from the very next scan. The browser has no goodbye fast path, though: the node still takes the miss path, its probes are refused from about 15 s on, and it is evicted at the one-minute threshold. @@ -61,7 +61,7 @@ It exists so that discovery has exactly one implementation per host. One record - **Edge case — a peer that reports no `clusterUuid`**: silence is not an answer. `NodeInfoResponse.ClusterUUID` is a pointer so an absent field is distinguishable from an explicit `""`, and only the explicit value moves the annotation — reading absence as "unclustered" would mark a clustered peer invitable, which is the failure the annotation exists to prevent. A failed fetch is treated the same way. - **Edge case — a stranger answering at a peer's address**: the membership sweep compares the `hostUuid` in the response against the entry it is refreshing and skips a mismatch rather than writing one machine's membership onto another's entry. That entry ages out through the liveness probe's identity check, which owns the eviction decision. - **Edge case — a supersede candidate that moves during the probe**: the proof is a network read taken without the directory lock, so `upsertEvicting` re-validates each verdict against the entry actually stored. A record removed, re-addressed, renamed, or refreshed in the meantime is left alone and judged again from its new state. -- **Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only `ol` / `lm`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal. +- **Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only engine keys such as `ol`, `lm`, or `lc`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal. ## 4. Open Questions / Risks @@ -208,13 +208,14 @@ The daemon emits five notifications and never sends a request to its parent. **` `registry` is this node's view of its own advertised services: a `ServiceKey → port` map plus the daemon-stamped identity. It is the only input to the advertised record. -The canonical service keys are exactly eight, in the order they are emitted into TXT: +The canonical service keys are exactly nine, in the order they are emitted into TXT: | Key | Service | Port in the product | Derived transport | |-----|---------|--------------------|-------------------| | `ni` | `nvpair-node-info` HTTP inventory | `14318` | plain, even when clustered | | `ol` | `nvpair-proxy` Ollama facade ingress | the facade's listen port | plain | | `lm` | `nvpair-proxy` LM Studio facade ingress | the facade's listen port | plain | +| `lc` | `nvpair-proxy` llama.cpp facade ingress | the facade's listen port | plain | | `er` | `nvpair-errors` cross-node ingest | `14319` | cluster mTLS when the target is clustered | | `wl` | `nvpair-workload-manager` event ingest | `14320` | cluster mTLS when the target is clustered | | `cl` | `nvpair-cluster-manager` pairing + trusted plane | `14321` | split channel — pairing plain, trusted endpoints mTLS | @@ -223,7 +224,7 @@ The canonical service keys are exactly eight, in the order they are emitted into Transport is **not** advertised. A consumer derives it from `ServiceKey.Transport()` plus whether the target node is clustered, because transport is global policy every same-version binary already knows rather than per-instance data. A key absent from TXT means the service is absent on that node. An unknown key parses as a service port and round-trips through the record, so a future service is carried, not dropped. -In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` from its engine reconciler at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else. +In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` / `lc` from its engine reconcilers at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else. `register` and `unregister` report whether the record actually changed. Only on a change does the daemon call `Responder.UpdateTXT` (which swaps the TXT and re-announces immediately) and refresh its own directory entry. The refresh runs on its own goroutine so a startup burst of registrations cannot serialize the read loop behind a loopback enrichment fetch. @@ -248,7 +249,7 @@ The browser (`nvpair-shared/discovery`) is a scan-and-diff state machine over `_ - **Cadence**: one scan every 5 s, each bounded at 3 s of listening; the first scan runs immediately. - **Change detection**: addresses and TXT are compared as multisets, because mDNS returns a multi-homed node's records in a non-deterministic order and an order-sensitive compare would emit a spurious `updated` (and its downstream churn) on every scan. - **Eviction**: a node absent from a scan has its miss counter bumped. Two thresholds apply. From `probeAfterMisses` (3 misses, about 15 s) the liveness probe asks about the node on **every** scan, and any positive answer resets the counter and recovers the node outright. Only at `missThresholdDefault` (12 misses, a full minute) is it evicted — so eviction rests on a run of roughly ten failed probes, not on one dial that happens to land on the deadline. The window is sized for a node saturated by its own inference load, which stops answering mDNS *and* stops accepting probe connections for as long as the load runs while still serving requests. Up to 8 probes run concurrently. -- **What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight. +- **What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm` / `lc`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight. - **A scan that loses everyone**: when more than one node is known and a scan returns none of them, no node is charged a miss, for up to `emptyScanGrace` (6) consecutive scans, about 30 s; after that they count normally. Independent machines do not leave a network in the same five-second window, but a process too starved to drain its multicast socket within the scan window hears silence from all of them at once — without this one saturated host would evict its whole cluster, idle peers included, in a single reconcile. With a single known node there is no "all at once" to be suspicious of, so it falls through to the ordinary threshold. - **A connect is not an identity**: something listening only proves *something* is at that address. A machine whose appdata was wiped mints a fresh `uuid=` while keeping its address, hostname, and fixed ports, so a connect-only verdict would reset the miss counter forever and leave the fleet carrying a permanent duplicate. When the sweep does find an address answering, `identityAlive` reads `hostUuid` from `/v1/node-info` at each published address, 1 s per address. Any address naming the record's own host keeps it; the connect is overruled only when every address that answered named someone else. It asks every address rather than only the first because one wrong answer is not proof of a wipe: a direct-connect link reuses the same subnet on other pairs of machines, so a record's leading address can resolve locally to a different host entirely. Every other answer — no `ni` advertised, no `uuid=` to compare, no response inside the 1-second budget, a response omitting `hostUuid` — is inconclusive and the connect stands. Ordering the identity read after the sweep is deliberate: it only runs when the node would otherwise have been *kept*, which is the sole case where the two need telling apart. - **Supersedence**: the same duplicate can arrive from the other direction, as a browse event for a new identity at an address the directory already holds. `supersedeCandidates` nominates a stored record sharing the arriving node's `ip` and `name`, carrying a different `uuid=`, and at least `supersedeMinAge` (300 s) older; it deletes nothing, because all three signals come from an unauthenticated record and describe the shape of a spoof exactly as well as the shape of a wipe. Each candidate is then put to the same question at the arriving record's `ni` port, and `upsertEvicting` re-validates the verdict against the entry actually stored before removing it. An unconfirmed candidate stays and is not left uncovered — when it stops being advertised the miss-threshold path asks again. @@ -292,7 +293,7 @@ The daemon holds a live `clustertrust.Mesh` over `--cluster-dir` and re-reads it `discovery:subscribe` is a **broker** method, not a scanner method — the daemon neither implements nor needs it. The broker reads the daemon's delta stream on the scanner peer's read pump and folds every event into two structures: -- `relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events. +- `relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm`, the llama.cpp facade for `lc` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events. - The broker's client-facing `discoveryStore` projects each `DirectoryNode` onto its own `AvailableNode` shape and is what `discovery:get-nodes` and the `discovery:nodes-changed` push serve to the desktop and `nvpair-tui`. This is the layer clients consume; they never see `_nvpair-node`, its TXT keys, or its service keys. The broker also carries three feeds that are not directory events. It caches each `discovery:node-telemetry` sample as the scanner source in its telemetry cache — a scanner observation outranks the manual-node worker's for the same host — and relays the aged sample to `nvpair-job-scheduler`. It relays `nvpair-node-info`'s `nodeinfo:observed-addresses` down to this daemon as `discovery:set-observed-addresses`. And it relays each proxy's report of inference bytes from a peer down as the `discovery:node-activity` notification.