Skip to content

Admission lanes: light jobs stop waiting behind heavy ones - #116

Merged
addisdev merged 1 commit into
mainfrom
ci/admission-lanes
Oct 23, 2026
Merged

addisdev merged 1 commit into
mainfrom
ci/admission-lanes

Conversation

@addisdev

Copy link
Copy Markdown
Owner

CI throughput plan (CI-OPTIMIZATION-PLAN-2026-10.md) Phase 0.1, 0.2, 0.6 and 3.3. Owner decision WB-617: heavy 4, Simulator 2, light up to 12.

What changes

Lanes in the admission hook (FLEET_ADMIT_LANES=off|observe|enforce, default off)

  • Each job is simulator (existing runner rule; also takes a heavy slot), heavy or light, in this order: Simulator runner, FLEET_ADMIT_HEAVY_PATTERNS, FLEET_ADMIT_LIGHT_PATTERNS (space-separated repo:job shell globs on GITHUB_JOB, repo with or without owner), then the dashboard's GET /api/job-class (median of the job's last 30 ran_s from the admission log's release lines; light under FLEET_ADMIT_SHORT_JOB_S, ≥ 5 samples), else heavy. The hook caches the answer in .admission/job-class/ for 300 s and gives the daemon 3 s.
  • Limits: FLEET_ADMIT_HEAVY_MAX_CONCURRENT (4), FLEET_ADMIT_LIGHT_MAX_CONCURRENT (12; the dead FLEET_ADMIT_SHORT_MAX_CONCURRENT is its alias), Simulator unchanged, under FLEET_ADMIT_TOTAL_MAX_CONCURRENT (default FLEET_ADMIT_MAX_CONCURRENT).
  • FIFO per lane: the oldest light waiter is never behind an older heavy one. Waiter files are named <ts>-<pid>-<light|heavy>-<runner> so the lockless check finds its own line's head from one listing; old-format names read as heavy. Mutex, reaping, cancel polling, disk floors, timeout semantics and the strict Simulator rule are unchanged.
  • observe: the single lane decides exactly as today; every row also records lane, lane_source, lane counts and limits, lane_would_hold, lane_reason and lane_wait_s (when the lanes would first have admitted a held job). off writes byte-for-byte the old line.
  • Hold reasons name the lane (4 heavy job(s) already running, at the heavy limit of 4, an older job is waiting for the next heavy slot); the legacy reasons keep their words, so the lake's fct_admission_hold classifier still files them.

Dashboard: /api/job-class, /api/load-by-busy, 11 lane columns on admission_events (addColumn) plus an index on (repo, job, ts), a lanes block in /api/admission, lane rows and a lane column on the Capacity tab.

cockpit load: host load1 p50/p90 (and p90/core) by busy-runner count over --days N, from host_samples.

Phase 0.6: cleanup.sh erases an idle runner's _work/<repo>/DerivedData-*, spm-*, .build-cache-* with nothing written in FLEET_CLEANUP_RUNNER_CACHE_DAYS (7), all of them under pressure (never a building runner's, never _work/_*, never a git checkout), and writes any missing _work/.metadata_never_index. register.sh (and so mirror-runners.sh) already wrote it; ephemeral-runner.sh now does too.

Tests

  • hooks/tests/run.sh: 150 pass locally (55 new: classification incl. cache and fallback, heavy/light/total limits, per-lane FIFO, Simulator counts as heavy, legacy alias, observe vs enforce, lockless per-lane check).
  • scripts/test-cleanup.sh: 77 pass (15 new).
  • Node: new job-class.test.js, lane cases in admission.test.js, route tests for both endpoints; related suites pass.
  • Swift LoadTableTests run in CI only (laptop under load).

Rollout

Merge, deploy to ultra in observe (single lane keeps deciding at 8), review the observe data, then FLEET_ADMIT_LANES=enforce.

🤖 Generated with Claude Code

….1, 0.2, 0.6, 3.3)

One host-wide FIFO made a 5-second job wait behind 12-minute Xcode builds;
on ultra load rises ~8 per concurrent job on 30 cores and 74 % of jobs
finish in under 90 s.

- hooks: FLEET_ADMIT_LANES=off|observe|enforce. Each job is simulator
  (runner rule, also takes a heavy slot), heavy or light: HEAVY/LIGHT
  patterns (repo:job shell globs on GITHUB_JOB), then the dashboard's
  /api/job-class (median of recent ran_s from the admission log, cached
  in .admission/job-class), else heavy. Limits HEAVY (4), LIGHT (12;
  SHORT_MAX_CONCURRENT is the alias), TOTAL (default MAX_CONCURRENT),
  FIFO per lane with the lockless check reading the line from the
  waiter's name. observe keeps the single lane deciding and logs
  lane_would_hold / lane_reason / lane_wait_s; off logs exactly the old
  line. Hold reasons name the lane; legacy reasons unchanged.
- dashboard: /api/job-class, /api/load-by-busy, lane columns on
  admission_events, lanes block in /api/admission and the Capacity tab.
- cockpit load: load1 p50/p90 by busy runners.
- cleanup.sh: erase idle runners' _work/<repo>/DerivedData-*, spm-*,
  .build-cache-* unused 7 days (all under pressure); write missing
  Spotlight markers; ephemeral runners get the marker too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@addisdev
addisdev merged commit d4f9b89 into main Oct 23, 2026
11 checks passed
@addisdev
addisdev deleted the ci/admission-lanes branch October 23, 2026 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant