Skip to content

Commit bd09318

Browse files
docs(nvpair-node-scanner): add the lc service key for llama.cpp
#148 added ServiceLlamaCPP ("lc") to noderec and to serviceKeyOrder between lm and er, so there are nine canonical service keys, not eight. Add the lc row to the section 7.1 table and name lc wherever the spec listed only ol and lm: the broker's engine registrations in sections 3 and 7.1, the facade subscriptions in 7.7, and the inference ports the liveness probe never dials. Signed-off-by: Terve <ntervalon@nvidia.com>
1 parent d751067 commit bd09318

1 file changed

Lines changed: 7 additions & 6 deletions

File tree

  • services/nvpair-node-scanner

‎services/nvpair-node-scanner/spec.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ It exists so that discovery has exactly one implementation per host. One record
3838

3939
## 3. Key Use Cases
4040

41-
- **Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` from its engine poller at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry.
41+
- **Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` / `lc` from its engine pollers at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry.
4242
- **Discover a peer**: a scan returns a peer's `_nvpair-node` record; `toDirectoryNode` projects TXT into a `DirectoryNode` keyed by `uuid=`, enrichment fills in hardware and models, and `discovery:node-discovered` carries the whole entry to the broker.
4343
- **A peer changes**: a peer starts an engine (a new `ol=` key) or moves address (`ip=` / `ips=`); the order-insensitive record compare in `nvpair-shared/discovery` sees a real change, re-enrichment runs, and `discovery:node-updated` fires. A scan that returns the same addresses and TXT in a different order emits nothing.
4444
- **A peer leaves**: it stops answering. From its third consecutive missed scan (about 15 s) the daemon dials its `ni` port and then its `em` port at each published address on **every** scan, and any answer resets the miss counter. Only after twelve consecutive misses (a full minute) with every probe failing is the entry dropped, its enrichment caches forgotten, and `discovery:node-removed` emitted. A clean shutdown multicasts an RFC 6762 goodbye (TTL 0 PTR), which clears the record from mDNS caches so it is absent from the very next scan. The browser has no goodbye fast path, though: the node still takes the miss path, its probes are refused from about 15 s on, and it is evicted at the one-minute threshold.
@@ -61,7 +61,7 @@ It exists so that discovery has exactly one implementation per host. One record
6161
- **Edge case — a peer that reports no `clusterUuid`**: silence is not an answer. `NodeInfoResponse.ClusterUUID` is a pointer so an absent field is distinguishable from an explicit `""`, and only the explicit value moves the annotation — reading absence as "unclustered" would mark a clustered peer invitable, which is the failure the annotation exists to prevent. A failed fetch is treated the same way.
6262
- **Edge case — a stranger answering at a peer's address**: the membership sweep compares the `hostUuid` in the response against the entry it is refreshing and skips a mismatch rather than writing one machine's membership onto another's entry. That entry ages out through the liveness probe's identity check, which owns the eviction decision.
6363
- **Edge case — a supersede candidate that moves during the probe**: the proof is a network read taken without the directory lock, so `upsertEvicting` re-validates each verdict against the entry actually stored. A record removed, re-addressed, renamed, or refreshed in the meantime is left alone and judged again from its new state.
64-
- **Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only `ol` / `lm`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal.
64+
- **Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only engine keys such as `ol`, `lm`, or `lc`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal.
6565

6666
## 4. Open Questions / Risks
6767

@@ -208,13 +208,14 @@ The daemon emits five notifications and never sends a request to its parent. **`
208208

209209
`registry` is this node's view of its own advertised services: a `ServiceKey → port` map plus the daemon-stamped identity. It is the only input to the advertised record.
210210

211-
The canonical service keys are exactly eight, in the order they are emitted into TXT:
211+
The canonical service keys are exactly nine, in the order they are emitted into TXT:
212212

213213
| Key | Service | Port in the product | Derived transport |
214214
|-----|---------|--------------------|-------------------|
215215
| `ni` | `nvpair-node-info` HTTP inventory | `14318` | plain, even when clustered |
216216
| `ol` | `nvpair-proxy` Ollama facade ingress | the facade's listen port | plain |
217217
| `lm` | `nvpair-proxy` LM Studio facade ingress | the facade's listen port | plain |
218+
| `lc` | `nvpair-proxy` llama.cpp facade ingress | the facade's listen port | plain |
218219
| `er` | `nvpair-errors` cross-node ingest | `14319` | cluster mTLS when the target is clustered |
219220
| `wl` | `nvpair-workload-manager` event ingest | `14320` | cluster mTLS when the target is clustered |
220221
| `cl` | `nvpair-cluster-manager` pairing + trusted plane | `14321` | split channel — pairing plain, trusted endpoints mTLS |
@@ -223,7 +224,7 @@ The canonical service keys are exactly eight, in the order they are emitted into
223224

224225
Transport is **not** advertised. A consumer derives it from `ServiceKey.Transport()` plus whether the target node is clustered, because transport is global policy every same-version binary already knows rather than per-instance data. A key absent from TXT means the service is absent on that node. An unknown key parses as a service port and round-trips through the record, so a future service is carried, not dropped.
225226

226-
In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` from its engine reconciler at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else.
227+
In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` / `lc` from its engine reconcilers at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else.
227228

228229
`register` and `unregister` report whether the record actually changed. Only on a change does the daemon call `Responder.UpdateTXT` (which swaps the TXT and re-announces immediately) and refresh its own directory entry. The refresh runs on its own goroutine so a startup burst of registrations cannot serialize the read loop behind a loopback enrichment fetch.
229230

@@ -248,7 +249,7 @@ The browser (`nvpair-shared/discovery`) is a scan-and-diff state machine over `_
248249
- **Cadence**: one scan every 5 s, each bounded at 3 s of listening; the first scan runs immediately.
249250
- **Change detection**: addresses and TXT are compared as multisets, because mDNS returns a multi-homed node's records in a non-deterministic order and an order-sensitive compare would emit a spurious `updated` (and its downstream churn) on every scan.
250251
- **Eviction**: a node absent from a scan has its miss counter bumped. Two thresholds apply. From `probeAfterMisses` (3 misses, about 15 s) the liveness probe asks about the node on **every** scan, and any positive answer resets the counter and recovers the node outright. Only at `missThresholdDefault` (12 misses, a full minute) is it evicted — so eviction rests on a run of roughly ten failed probes, not on one dial that happens to land on the deadline. The window is sized for a node saturated by its own inference load, which stops answering mDNS *and* stops accepting probe connections for as long as the load runs while still serving requests. Up to 8 probes run concurrently.
251-
- **What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight.
252+
- **What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm` / `lc`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight.
252253
- **A scan that loses everyone**: when more than one node is known and a scan returns none of them, no node is charged a miss, for up to `emptyScanGrace` (6) consecutive scans, about 30 s; after that they count normally. Independent machines do not leave a network in the same five-second window, but a process too starved to drain its multicast socket within the scan window hears silence from all of them at once — without this one saturated host would evict its whole cluster, idle peers included, in a single reconcile. With a single known node there is no "all at once" to be suspicious of, so it falls through to the ordinary threshold.
253254
- **A connect is not an identity**: something listening only proves *something* is at that address. A machine whose appdata was wiped mints a fresh `uuid=` while keeping its address, hostname, and fixed ports, so a connect-only verdict would reset the miss counter forever and leave the fleet carrying a permanent duplicate. When the sweep does find an address answering, `identityAlive` reads `hostUuid` from `/v1/node-info` at each published address, 1 s per address. Any address naming the record's own host keeps it; the connect is overruled only when every address that answered named someone else. It asks every address rather than only the first because one wrong answer is not proof of a wipe: a direct-connect link reuses the same subnet on other pairs of machines, so a record's leading address can resolve locally to a different host entirely. Every other answer — no `ni` advertised, no `uuid=` to compare, no response inside the 1-second budget, a response omitting `hostUuid` — is inconclusive and the connect stands. Ordering the identity read after the sweep is deliberate: it only runs when the node would otherwise have been *kept*, which is the sole case where the two need telling apart.
254255
- **Supersedence**: the same duplicate can arrive from the other direction, as a browse event for a new identity at an address the directory already holds. `supersedeCandidates` nominates a stored record sharing the arriving node's `ip` and `name`, carrying a different `uuid=`, and at least `supersedeMinAge` (300 s) older; it deletes nothing, because all three signals come from an unauthenticated record and describe the shape of a spoof exactly as well as the shape of a wipe. Each candidate is then put to the same question at the arriving record's `ni` port, and `upsertEvicting` re-validates the verdict against the entry actually stored before removing it. An unconfirmed candidate stays and is not left uncovered — when it stops being advertised the miss-threshold path asks again.
@@ -292,7 +293,7 @@ The daemon holds a live `clustertrust.Mesh` over `--cluster-dir` and re-reads it
292293

293294
`discovery:subscribe` is a **broker** method, not a scanner method — the daemon neither implements nor needs it. The broker reads the daemon's delta stream on the scanner peer's read pump and folds every event into two structures:
294295

295-
- `relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events.
296+
- `relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm`, the llama.cpp facade for `lc` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events.
296297
- The broker's client-facing `discoveryStore` projects each `DirectoryNode` onto its own `AvailableNode` shape and is what `discovery:get-nodes` and the `discovery:nodes-changed` push serve to the desktop and `nvpair-tui`. This is the layer clients consume; they never see `_nvpair-node`, its TXT keys, or its service keys.
297298

298299
The broker also carries three feeds that are not directory events. It caches each `discovery:node-telemetry` sample as the scanner source in its telemetry cache — a scanner observation outranks the manual-node worker's for the same host — and relays the aged sample to `nvpair-job-scheduler`. It relays `nvpair-node-info`'s `nodeinfo:observed-addresses` down to this daemon as `discovery:set-observed-addresses`. And it relays each proxy's report of inference bytes from a peer down as the `discovery:node-activity` notification.

0 commit comments

Comments
 (0)