Skip to content

Make the macmon backoff test deterministic - #46

Merged
JaredP94 merged 1 commit into
mainfrom
test/macmon-backoff-flake
Nov 3, 2026
Merged

JaredP94 merged 1 commit into
mainfrom
test/macmon-backoff-flake

Conversation

@JaredP94

@JaredP94 JaredP94 commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Description

Make TestMacmonSupervisorBackoffDoublesCapsAndResets deterministic on loaded macOS runners. Runs 41643032848 and 41586628573 failed because process-start gaps included scheduler latency and exceeded narrow wall-clock upper bounds, not because the scheduled backoff exceeded its cap.

Record the actual backoff waits and assert the exact sequence 50, 100, 150, 150, 50, 100 ms. Six real shell children still exercise execution and parsing; only the fifth emits a good line. This proves the initial minimum, doubling, first and repeated capped waits, reset after good output, and doubling again after reset. The only real-time bound is a loose 10-second context deadline.

No user-facing runtime behavior changes. The unexported waitBackoff config seam defaults to the existing waitMacmon context-aware timer. The compiled output changes, but node-info already has an unreleased minor bump to 1.4.0 versus 1.3.0 at v0.8.0; retain that sufficient bump per fork practice.

Scope

Only services/nvpair-node-info/macmon_supervisor.go and its test change. No wire contracts, dependencies, packaging, or documentation behavior changes.

Timing assertion audit (grep of node-info tests for time.Since, time.Now, time.After, elapsed/gap/deadline):

  • Comparable remaining upper bound: macmon_supervisor_test.go:205, TestMacmonSupervisorRestartsOnIntervalChange, restart must take at most 500 ms. It covers policy-restart latency, not backoff values, so it is unchanged.
  • Other asynchronous deadlines, not backoff-gap assertions: stats_darwin_test.go:79 (TestDarwinCollectorPublishesAndStops, 250 ms); stats_darwin_test.go:219,234 (TestDarwinCollectorPublishesSystemStatsWhileGPUBlocks, 250 ms publication and shutdown); shared waitFor/supervisor cleanup (2 s), sensor collector shutdown (3 s). All unchanged.

Validation

On macOS arm64, in services/nvpair-node-info, all passed:

  • go test -race -count=20 -run TestMacmonSupervisor ./...
  • go test -race ./...
  • go test -count=1 ./...
  • gofmt -l macmon_supervisor.go macmon_supervisor_test.go (empty)
  • GOOS=darwin go vet ./...
  • GOOS=linux go vet ./...
  • GOOS=windows go vet ./...

At repository root: node scripts/spdx-headers.mjs (0 missing) and git diff --check passed.

Mutation checks, each run with go test -race -count=1 -run '^TestMacmonSupervisorBackoffDoublesCapsAndResets$' ./..., failed as required and were then restored before the passing verification above:

  • Replace capped assignment with doubling: waits 3/4 became 200/400 ms, expected 150/150 ms.
  • Reset to BackoffMax instead of BackoffMin: waits 5/6 became 150/150 ms, expected 50/100 ms.

Independent inline code review reported no findings.

CI run 41660517582, attempt 2: all six jobs passed (desktop, SPDX, Go modules on Linux/macOS, cross-process tests on Linux/macOS). Attempt 1's macOS Go job never acquired a hosted runner and had no executed steps; GitHub reported runner capacity constraints. Rerunning only that job succeeded without code changes.

Risk

Low: the default timer, cancellation, child lifecycle, and backoff arithmetic are unchanged. The injection is private and configured before running the supervisor. Linux/Windows are cross-vetted, not runtime-tested locally.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated (no documented behavior changed).
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I checked affected component versions: retain node-info's existing unreleased 1.4.0 bump as described above.

Process-start gaps include scheduler latency, so narrow upper bounds made the macmon backoff test flaky on loaded macOS runners. Inject a private backoff wait function with the existing context-aware timer as its production default and assert exact delays around real child output instead.

Cover repeated capped waits and doubling again after a good line resets the delay. Retain the existing unreleased node-info 1.4.0 bump relative to v0.8.0. Independent cap and reset mutations fail the revised test.

Signed-off-by: Jared Ping <13906915+JaredP94@users.noreply.github.com>
@JaredP94
JaredP94 merged commit e12beb8 into main Nov 3, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant