You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bd09318
Browse filesBrowse the repository at this point in the historyBrowse files
docs(nvpair-node-scanner): add the lc service key for llama.cpp
#148 added ServiceLlamaCPP ("lc") to noderec and to serviceKeyOrder
between lm and er, so there are nine canonical service keys, not eight.
Add the lc row to the section 7.1 table and name lc wherever the spec
listed only ol and lm: the broker's engine registrations in sections 3
and 7.1, the facade subscriptions in 7.7, and the inference ports the
liveness probe never dials.
Signed-off-by: Terve <ntervalon@nvidia.com>
Copy file name to clipboardExpand all lines: services/nvpair-node-scanner/spec.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@ It exists so that discovery has exactly one implementation per host. One record
38
38
39
39
## 3. Key Use Cases
40
40
41
-
-**Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm` from its engine poller at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry.
41
+
-**Advertise this node**: the broker relays a `discovery:register` per local service as it starts each worker — `ni` (node-info, `14318`), `wl` (workload-manager, `14320`), `er` (errors, `14319`), `em` (engine-manager, `14322`), `ec` (engine remote control, `14323`, only when a cluster dir is configured), `cl` (cluster-manager, `14321`) — plus `ol` / `lm`/ `lc`from its engine pollers at the listen port of the matching `nvpair-proxy` facade, never the engine's own port. The registry merges each into `NodeRecord`, `Responder.UpdateTXT` re-announces, and `publishSelf` refreshes this node's own directory entry.
42
42
-**Discover a peer**: a scan returns a peer's `_nvpair-node` record; `toDirectoryNode` projects TXT into a `DirectoryNode` keyed by `uuid=`, enrichment fills in hardware and models, and `discovery:node-discovered` carries the whole entry to the broker.
43
43
-**A peer changes**: a peer starts an engine (a new `ol=` key) or moves address (`ip=` / `ips=`); the order-insensitive record compare in `nvpair-shared/discovery` sees a real change, re-enrichment runs, and `discovery:node-updated` fires. A scan that returns the same addresses and TXT in a different order emits nothing.
44
44
-**A peer leaves**: it stops answering. From its third consecutive missed scan (about 15 s) the daemon dials its `ni` port and then its `em` port at each published address on **every** scan, and any answer resets the miss counter. Only after twelve consecutive misses (a full minute) with every probe failing is the entry dropped, its enrichment caches forgotten, and `discovery:node-removed` emitted. A clean shutdown multicasts an RFC 6762 goodbye (TTL 0 PTR), which clears the record from mDNS caches so it is absent from the very next scan. The browser has no goodbye fast path, though: the node still takes the miss path, its probes are refused from about 15 s on, and it is evicted at the one-minute threshold.
@@ -61,7 +61,7 @@ It exists so that discovery has exactly one implementation per host. One record
61
61
-**Edge case — a peer that reports no `clusterUuid`**: silence is not an answer. `NodeInfoResponse.ClusterUUID` is a pointer so an absent field is distinguishable from an explicit `""`, and only the explicit value moves the annotation — reading absence as "unclustered" would mark a clustered peer invitable, which is the failure the annotation exists to prevent. A failed fetch is treated the same way.
62
62
-**Edge case — a stranger answering at a peer's address**: the membership sweep compares the `hostUuid` in the response against the entry it is refreshing and skips a mismatch rather than writing one machine's membership onto another's entry. That entry ages out through the liveness probe's identity check, which owns the eviction decision.
63
63
-**Edge case — a supersede candidate that moves during the probe**: the proof is a network read taken without the directory lock, so `upsertEvicting` re-validates each verdict against the entry actually stored. A record removed, re-addressed, renamed, or refreshed in the meantime is left alone and judged again from its new state.
64
-
-**Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only `ol` / `lm`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal.
64
+
-**Edge case — a node advertising neither `ni` nor `em`**: the liveness probe has nothing to ask, so a threshold-missed identity-only node (or one advertising only engine keys such as `ol`, `lm`, or `lc`) evicts on the ordinary path. Inference ports are never dialed as a liveness signal.
65
65
66
66
## 4. Open Questions / Risks
67
67
@@ -208,13 +208,14 @@ The daemon emits five notifications and never sends a request to its parent. **`
208
208
209
209
`registry` is this node's view of its own advertised services: a `ServiceKey → port` map plus the daemon-stamped identity. It is the only input to the advertised record.
210
210
211
-
The canonical service keys are exactly eight, in the order they are emitted into TXT:
211
+
The canonical service keys are exactly nine, in the order they are emitted into TXT:
212
212
213
213
| Key | Service | Port in the product | Derived transport |
@@ -223,7 +224,7 @@ The canonical service keys are exactly eight, in the order they are emitted into
223
224
224
225
Transport is **not** advertised. A consumer derives it from `ServiceKey.Transport()` plus whether the target node is clustered, because transport is global policy every same-version binary already knows rather than per-instance data. A key absent from TXT means the service is absent on that node. An unknown key parses as a service port and round-trips through the record, so a future service is carried, not dropped.
225
226
226
-
In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm` from its engine reconciler at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else.
227
+
In practice all registrations come from `nvpair-ui-broker`, which caches them (`relay.RegistrationCache`) and pushes each down as an id-bearing `discovery:register` when it starts the owning worker: `ni` when node-info spawns, `wl`, `er`, `em` likewise, `ec` only when a cluster dir is configured, `cl` when cluster-manager spawns, and `ol` / `lm`/ `lc`from its engine reconcilers at the matching `nvpair-proxy` facade's listen port — never the engine's own port — with an `unregister` whenever the engine goes unhealthy or its facade is down. Because the broker owns the cache, it replays the whole set on every scanner spawn, so a scanner restart needs no reconnect logic anywhere else.
227
228
228
229
`register` and `unregister` report whether the record actually changed. Only on a change does the daemon call `Responder.UpdateTXT` (which swaps the TXT and re-announces immediately) and refresh its own directory entry. The refresh runs on its own goroutine so a startup burst of registrations cannot serialize the read loop behind a loopback enrichment fetch.
229
230
@@ -248,7 +249,7 @@ The browser (`nvpair-shared/discovery`) is a scan-and-diff state machine over `_
248
249
-**Cadence**: one scan every 5 s, each bounded at 3 s of listening; the first scan runs immediately.
249
250
-**Change detection**: addresses and TXT are compared as multisets, because mDNS returns a multi-homed node's records in a non-deterministic order and an order-sensitive compare would emit a spurious `updated` (and its downstream churn) on every scan.
250
251
-**Eviction**: a node absent from a scan has its miss counter bumped. Two thresholds apply. From `probeAfterMisses` (3 misses, about 15 s) the liveness probe asks about the node on **every** scan, and any positive answer resets the counter and recovers the node outright. Only at `missThresholdDefault` (12 misses, a full minute) is it evicted — so eviction rests on a run of roughly ten failed probes, not on one dial that happens to land on the deadline. The window is sized for a node saturated by its own inference load, which stops answering mDNS *and* stops accepting probe connections for as long as the load runs while still serving requests. Up to 8 probes run concurrently.
251
-
-**What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight.
252
+
-**What the probe asks**, cheapest first. A `/v1/node-info` enrichment newer than 10 s (`reachableInfoFreshness`) keeps the node with no dial. So do inference response bytes from it within the last 60 s (`activityFreshness`), reported by the broker as `discovery:node-activity` from whichever proxy saw them — the only liveness signal that gets stronger the busier the node is. Otherwise it dials the node's `ni` port and then its `em` port at each of its published addresses in order, with a 1-second timeout each, and stops at the first that connects; nothing answering is the ordinary departure and is settled without any HTTP. Inference ports (`ol` / `lm` / `lc`) are never dialed — they front the proxies, and probe traffic there contributed to accept-queue exhaustion. A node advertising neither `ni` nor `em` has nothing to probe and evicts on the threshold alone, as does one with no resolvable address. The probe runs outside the browser lock, and the node stays routable while it is in flight.
252
253
-**A scan that loses everyone**: when more than one node is known and a scan returns none of them, no node is charged a miss, for up to `emptyScanGrace` (6) consecutive scans, about 30 s; after that they count normally. Independent machines do not leave a network in the same five-second window, but a process too starved to drain its multicast socket within the scan window hears silence from all of them at once — without this one saturated host would evict its whole cluster, idle peers included, in a single reconcile. With a single known node there is no "all at once" to be suspicious of, so it falls through to the ordinary threshold.
253
254
- **A connect is not an identity**: something listening only proves *something* is at that address. A machine whose appdata was wiped mints a fresh `uuid=` while keeping its address, hostname, and fixed ports, so a connect-only verdict would reset the miss counter forever and leave the fleet carrying a permanent duplicate. When the sweep does find an address answering, `identityAlive` reads `hostUuid` from `/v1/node-info` at each published address, 1 s per address. Any address naming the record's own host keeps it; the connect is overruled only when every address that answered named someone else. It asks every address rather than only the first because one wrong answer is not proof of a wipe: a direct-connect link reuses the same subnet on other pairs of machines, so a record's leading address can resolve locally to a different host entirely. Every other answer — no `ni` advertised, no `uuid=` to compare, no response inside the 1-second budget, a response omitting `hostUuid` — is inconclusive and the connect stands. Ordering the identity read after the sweep is deliberate: it only runs when the node would otherwise have been *kept*, which is the sole case where the two need telling apart.
254
255
-**Supersedence**: the same duplicate can arrive from the other direction, as a browse event for a new identity at an address the directory already holds. `supersedeCandidates` nominates a stored record sharing the arriving node's `ip` and `name`, carrying a different `uuid=`, and at least `supersedeMinAge` (300 s) older; it deletes nothing, because all three signals come from an unauthenticated record and describe the shape of a spoof exactly as well as the shape of a wipe. Each candidate is then put to the same question at the arriving record's `ni` port, and `upsertEvicting` re-validates the verdict against the entry actually stored before removing it. An unconfirmed candidate stays and is not left uncovered — when it stops being advertised the miss-threshold path asks again.
@@ -292,7 +293,7 @@ The daemon holds a live `clustertrust.Mesh` over `--cluster-dir` and re-reads it
292
293
293
294
`discovery:subscribe` is a **broker** method, not a scanner method — the daemon neither implements nor needs it. The broker reads the daemon's delta stream on the scanner peer's read pump and folds every event into two structures:
294
295
295
-
-`relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events.
296
+
-`relay.Directory`, keyed by `hostUuid`, is the internal fan-out. A worker subscribes with `discovery:subscribe` and `SubscribeParams` (`{services?: ServiceKey[]}`; an empty list means all nodes) and receives `discovery:nodes` — `{nodes: DirectoryNode[]}`, its **full filtered set**, re-sent on every directory change. Consumers replace their set from each push rather than applying deltas, so a dropped or reordered push cannot leave them drifted. Today each `nvpair-proxy` engine facade subscribes for its own key — the Ollama facade for `ol`, the LM Studio facade for `lm`, the llama.cpp facade for `lc` — `nvpair-errors` for `er`, `nvpair-workload-manager` for `wl`, `nvpair-cluster-manager` for `cl`, and `nvpair-engine-manager` for `ec`. This directory is fed **only** by scanner events.
296
297
- The broker's client-facing `discoveryStore` projects each `DirectoryNode` onto its own `AvailableNode` shape and is what `discovery:get-nodes` and the `discovery:nodes-changed` push serve to the desktop and `nvpair-tui`. This is the layer clients consume; they never see `_nvpair-node`, its TXT keys, or its service keys.
297
298
298
299
The broker also carries three feeds that are not directory events. It caches each `discovery:node-telemetry` sample as the scanner source in its telemetry cache — a scanner observation outranks the manual-node worker's for the same host — and relays the aged sample to `nvpair-job-scheduler`. It relays `nvpair-node-info`'s `nodeinfo:observed-addresses` down to this daemon as `discovery:set-observed-addresses`. And it relays each proxy's report of inference bytes from a peer down as the `discovery:node-activity` notification.
0 commit comments