Repository navigation
Commit d90590f
authored
Retry failed scheduler priority notifications (#122)
## Description
Closes #121.
Record a priority snapshot as emitted only after its JSON-RPC
notification write succeeds. Previously a failed write updated the
delivery cache first, so ordinary reconciliation suppressed retries and
scheduler status incorrectly showed the failed update as delivered.
The change keeps notification I/O outside the state mutex and retains
the existing recompute serialization. Each engine remains independent: a
successful delivery is not repeated because another engine failed.
Ranking policy, wire fields, retry cadence and receiver behavior are
unchanged.
## Release intent
<!-- pair-release-intent:v1 -->
### Changelog title
Retry failed scheduler priority notifications
### Changelog body
The scheduler retries changed priorities after a failed local
notification write and keeps its status aligned with the last successful
delivery.
### Bumps
- services: patch
- nvpair-cluster-manager: none
- nvpair-engine-manager: none
- nvpair-errors: none
- nvpair-job-scheduler: patch
- nvpair-manual-nodes: none
- nvpair-node-info: none
- nvpair-node-scanner: none
- nvpair-node-settings: none
- nvpair-proxy: none
- nvpair-tui: none
- nvpair-ui-broker: none
- nvpair-workload-manager: none
<!-- /pair-release-intent:v1 -->
## Scope
Only the scheduler delivery bookkeeping, its regression tests and its
README are changed. The scheduler binary and services aggregate receive
patch intent. No version files are edited by hand.
The broker, shared wire types, proxies, desktop and terminal interface
need no contract changes. Existing cross-process scheduler tests verify
the emitted frames and ranking behavior. The open engine-addition PRs
modify the engine list in the same source file, not this delivery path.
## Validation
- Five new regression scenarios fail before the fix and pass afterward.
They cover initial write failure, error and zero-byte retry paths,
retention of an older successful snapshot, and failed forced-refresh
timestamps.
- The complete scheduler package passes with race detection on macOS and
Linux arm64: 36 top-level tests and 11 subtests, no failures or skips.
- Four existing cross-process scheduler tests pass, driving real built
binaries through stdin/stdout. They cover workload updates, telemetry,
restart replay and a 50-job synthetic burst. The harness builds the
service binaries but the selected tests start only the scheduler.
- All 16 shared JSON-RPC tests pass with race detection on macOS.
- `go vet`, the scheduler build, `gofmt`, repository SPDX-header checks
and `git diff --check` pass. The release-intent description validates
locally.
- Linux tests run in an isolated container with networking disabled.
App-data paths are isolated under temporary directories.
```sh
cd services/nvpair-job-scheduler
go test -race -count=1 ./...
go vet ./...
go build .
# From services/tests:
go test -race -count=1 -run '^TestScheduler' ./...
# From services/shared:
go test -race -count=1 ./jsonrpc
# From the repository root:
node scripts/spdx-headers.mjs
```
Tested with Go 1.27.1. The desktop gates, whole cross-process suite,
live engines and multi-node integration were not run. No test or runtime
guard was disabled. These tests cover failures before any frame bytes
are delivered; they do not claim recovery from partial-frame corruption.
## Risk
Low, localized state-ordering correction. No new dependency, network
listener, persistent data, protocol field or service boundary. The
previous snapshot remains authoritative when a new write fails. The
issue was reported on 25 September 2026 and rechecked for assignments
and competing fixes before submission.
## Checklist
- [x] I have read the contributing guidelines.
- [x] Every commit is signed off.
- [x] Regression and existing tests cover the change.
- [x] Relevant documentation is updated.
- [x] The diff and commit contents contain no credentials, private data
or generated artifacts.
- [x] Validation commands and limitations are recorded above.
- [x] Version bumps are declared in the release-intent block;
automation-owned version files are unchanged.
Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>1 parent 603fb0c commit d90590f
3 files changed
Lines changed: 120 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | 139 | | |
143 | 140 | | |
144 | 141 | | |
| |||
149 | 146 | | |
150 | 147 | | |
151 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
152 | 153 | | |
153 | 154 | | |
154 | 155 | | |
| |||
0 commit comments