From 71f30a0b1c7129136375bcc68f164808ef623de5 Mon Sep 17 00:00:00 2001 From: Taylor Addison Date: Thu, 22 Oct 2026 23:37:18 -0400 Subject: [PATCH] Admission lanes: light jobs stop waiting behind heavy ones (CI plan 0.1, 0.2, 0.6, 3.3) One host-wide FIFO made a 5-second job wait behind 12-minute Xcode builds; on ultra load rises ~8 per concurrent job on 30 cores and 74 % of jobs finish in under 90 s. - hooks: FLEET_ADMIT_LANES=off|observe|enforce. Each job is simulator (runner rule, also takes a heavy slot), heavy or light: HEAVY/LIGHT patterns (repo:job shell globs on GITHUB_JOB), then the dashboard's /api/job-class (median of recent ran_s from the admission log, cached in .admission/job-class), else heavy. Limits HEAVY (4), LIGHT (12; SHORT_MAX_CONCURRENT is the alias), TOTAL (default MAX_CONCURRENT), FIFO per lane with the lockless check reading the line from the waiter's name. observe keeps the single lane deciding and logs lane_would_hold / lane_reason / lane_wait_s; off logs exactly the old line. Hold reasons name the lane; legacy reasons unchanged. - dashboard: /api/job-class, /api/load-by-busy, lane columns on admission_events, lanes block in /api/admission and the Capacity tab. - cockpit load: load1 p50/p90 by busy runners. - cleanup.sh: erase idle runners' _work//DerivedData-*, spm-*, .build-cache-* unused 7 days (all under pressure); write missing Spotlight markers; ephemeral runners get the marker too. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 28 ++ cleanup.sh | 60 +++- cockpit/Sources/CockpitCore/LoadTable.swift | 61 ++++ cockpit/Sources/cockpit/main.swift | 11 + .../CockpitCoreTests/LoadTableTests.swift | 30 ++ dashboard/fleetd.js | 39 ++ dashboard/lib/admission.js | 90 ++++- dashboard/lib/admit-gate.js | 2 + dashboard/lib/db.js | 13 + dashboard/lib/job-class.js | 97 +++++ dashboard/lib/load-table.js | 45 +++ dashboard/public/capacity.js | 43 ++- dashboard/test/admission.test.js | 49 +++ dashboard/test/admit-gate.test.js | 6 +- dashboard/test/job-class.test.js | 125 +++++++ dashboard/test/routes.test.js | 18 + docs/admission-and-scaling.md | 86 ++++- docs/api.md | 32 +- docs/cockpit/cli.md | 22 ++ docs/configuration.md | 10 + docs/operations.md | 7 + docs/security-hardening.md | 2 + examples/fleet.env.second-host | 4 +- fleet.env.example | 34 ++ hooks/common.sh | 337 +++++++++++++++++- hooks/job-completed.sh | 6 + hooks/job-started.sh | 119 ++++++- hooks/tests/run.sh | 282 +++++++++++++++ scripts/ephemeral-runner.sh | 4 + scripts/test-cleanup.sh | 38 ++ 30 files changed, 1670 insertions(+), 30 deletions(-) create mode 100644 cockpit/Sources/CockpitCore/LoadTable.swift create mode 100644 cockpit/Tests/CockpitCoreTests/LoadTableTests.swift create mode 100644 dashboard/lib/job-class.js create mode 100644 dashboard/lib/load-table.js create mode 100644 dashboard/test/job-class.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 4203a72..c3e6521 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,34 @@ version numbers (`v0.1.0`, `v0.1.1`, etc.). ### Added +- **Admission lanes (`FLEET_ADMIT_LANES`).** One host-wide FIFO made a + 5-second job wait behind 12-minute Xcode builds: on ultra load rose about 8 + per concurrent job on 30 cores, and 74 % of jobs finish in under 90 s. Each + job is now `light`, `heavy` or `simulator` (which also takes a heavy slot), + by the Simulator runner rule, `FLEET_ADMIT_HEAVY_PATTERNS` / + `FLEET_ADMIT_LIGHT_PATTERNS` (`repo:job` shell patterns), then the + dashboard's history (`GET /api/job-class`: light when the job's recent + median run is under `FLEET_ADMIT_SHORT_JOB_S`, cached by the hook), else + heavy. Limits `FLEET_ADMIT_HEAVY_MAX_CONCURRENT` (4) and + `FLEET_ADMIT_LIGHT_MAX_CONCURRENT` (12; `FLEET_ADMIT_SHORT_MAX_CONCURRENT`, + set in fleet.env but read by nothing until now, is its alias) under + `FLEET_ADMIT_TOTAL_MAX_CONCURRENT`, with the waiting line kept per lane. + `observe` keeps the single lane deciding and logs what the lanes would have + done (`lane_would_hold`, `lane_reason`, `lane_wait_s`); `enforce` lets them + decide; `off` (default) is the old behaviour, byte for byte in the log. + Admission rows carry the lane, the Capacity tab shows per-lane counts and + the lanes' 24 h tally. See + [Lanes](docs/admission-and-scaling.md#lanes). +- **`cockpit load` and `GET /api/load-by-busy`**: host load1 p50/p90 by number + of busy runners from the per-minute host samples, to re-tune the heavy + limit when the job mix changes. +- **`cleanup.sh` erases warm build caches nobody uses.** The kit is moving iOS + builds to per-runner `_work//DerivedData-*` and `spm-*` so they are + incremental; cleanup now removes an idle runner's (and `.build-cache-*`) + after `FLEET_CLEANUP_RUNNER_CACHE_DAYS` (7) without a build, and all of them + under disk pressure. It also writes the Spotlight marker + `_work/.metadata_never_index` for any runner missing it, and + `scripts/ephemeral-runner.sh` writes it as `register.sh` already did. - **`shims/ps`: a finishing job no longer pins a core for minutes.** At "Complete job" the GitHub runner checks every process that started during the job for its `RUNNER_TRACKING_ID`; macOS has no `/proc`, so it spawns `ps e -p -o command` once per process. On a busy shared host that is diff --git a/cleanup.sh b/cleanup.sh index a7ffcdb..569d483 100755 --- a/cleanup.sh +++ b/cleanup.sh @@ -16,9 +16,12 @@ # - never `simctl shutdown all`, never kill CoreSimulatorService. Those close # simulators the user is working in. `delete unavailable` only removes devices # whose runtime is already gone, which nothing can be using. -# - never touch a runner's checkouts or build caches in _work; wiping them -# makes every job re-clone and recompile from cold. The one exception is -# the Playwright tool cache, and only under disk pressure. +# - never touch a runner's checkouts in _work, and never a warm build cache +# a job used this week: wiping them makes every job re-clone and recompile +# from cold. Warm caches (_work//DerivedData-*, spm-*, +# .build-cache-*) go after FLEET_CLEANUP_RUNNER_CACHE_DAYS without use, or +# all of an idle runner's under disk pressure, as do its Playwright +# browsers. # - never touch a simulator or Android AVD that is not a runner's own # ci- (or shots-) device. set -uo pipefail @@ -39,6 +42,10 @@ done DERIVED="$HOME/Library/Developer/Xcode/DerivedData" DERIVED_AGE_DAYS=7 +# Per-runner warm build caches the kit keeps beside each checkout. A week +# without a build reading them means the next one rebuilds anyway. +RUNNER_CACHE_DAYS="${FLEET_CLEANUP_RUNNER_CACHE_DAYS:-7}" +case "$RUNNER_CACHE_DAYS" in '' | *[!0-9]*) RUNNER_CACHE_DAYS=7 ;; esac DIAG_AGE_DAYS=14 PW_LOCK_AGE_HOURS=6 @@ -396,6 +403,53 @@ for d in "$ROOT"/*/; do done say " ${n} directories" +# --------------------------------------------------------------------------- +# Warm build caches in each runner's _work//: DerivedData-, +# spm-* (SwiftPM checkouts) and .build-cache-*, which the kit's iOS and +# SwiftPM jobs keep per runner so a build is incremental (CI plan Phase 1.4). +# Each one is a few GB and per runner, so across 80 runners they are the disk. +# One whose newest file (two levels down, where xcodebuild and SwiftPM write on +# every build) is older than RUNNER_CACHE_DAYS goes; under pressure, all of an +# idle runner's go. Never a building runner's, never under _work/_*/ (the +# runner's own _temp, _tool, _actions), and never a directory that is a git +# checkout (a repo that happens to be named spm-something). +# --------------------------------------------------------------------------- +say "==> runner build caches unused for ${RUNNER_CACHE_DAYS}d$([ "$PRESSURE" = 1 ] && echo ' (under pressure: all of them, on idle runners)')" +n=0 +for d in "$ROOT"/*/; do + d="${d%/}" + [ -f "$d/.runner" ] && [ -d "$d/_work" ] || continue + building "$(runner_name "$d")" && continue + for cache in "$d"/_work/*/DerivedData* "$d"/_work/*/spm* "$d"/_work/*/.build-cache*; do + [ -d "$cache" ] || continue + case "$cache" in "$d"/_work/_*) continue ;; esac + [ -e "$cache/.git" ] && continue + if [ "$PRESSURE" != 1 ] \ + && [ -n "$(find "$cache" -maxdepth 2 -mtime -"$RUNNER_CACHE_DAYS" -print -quit 2>/dev/null)" ]; then + continue + fi + say " $(du -sh "$cache" 2>/dev/null | cut -f1) ${cache#"$ROOT"/}" + run rm -rf "$cache" + n=$((n + 1)) + done +done +say " ${n} directories" + +# --------------------------------------------------------------------------- +# Spotlight stays out of every runner's build tree. register.sh writes the +# marker for a new runner; this catches one created any other way (copied, +# restored, registered by hand). Cheap and idempotent: one test per runner. +# --------------------------------------------------------------------------- +n=0 +for d in "$ROOT"/*/; do + d="${d%/}" + [ -f "$d/.runner" ] && [ -d "$d/_work" ] || continue + [ -e "$d/_work/.metadata_never_index" ] && continue + if [ "$APPLY" = "1" ]; then touch "$d/_work/.metadata_never_index" 2>/dev/null || continue; fi + n=$((n + 1)) +done +[ "$n" -gt 0 ] && say "==> Spotlight exclusion written for ${n} runner work tree(s)$([ "$APPLY" = 1 ] || echo ' [dry-run]')" + # --------------------------------------------------------------------------- # Under pressure only: each idle runner's Playwright browsers. They are 0.5-1.1 GB # per web runner (13 GB on runner-host) and the next job that needs them diff --git a/cockpit/Sources/CockpitCore/LoadTable.swift b/cockpit/Sources/CockpitCore/LoadTable.swift new file mode 100644 index 0000000..c01af16 --- /dev/null +++ b/cockpit/Sources/CockpitCore/LoadTable.swift @@ -0,0 +1,61 @@ +import Foundation + +/// GET /api/load-by-busy: host load1 p50/p90 by the number of busy runners, +/// from the dashboard's per-minute host samples. `cockpit load` prints it so +/// the admission lanes' heavy limit can be re-tuned when the job mix changes +/// (actions-runners docs/admission-and-scaling.md, "Tuning the lanes"). +public struct LoadTable: Codable, Sendable, Equatable { + public struct Row: Codable, Sendable, Equatable { + public var busy: Int + public var n: Int + public var p50: Double? + public var p90: Double? + } + + public struct Limits: Codable, Sendable, Equatable { + public var heavy: Int? + public var light: Int? + public var total: Int? + } + + public struct Lanes: Codable, Sendable, Equatable { + public var mode: String? + public var limits: Limits? + } + + public var days: Int + public var host: String? + public var samples: Int + public var cores: Int? + public var rows: [Row] + public var lanes: Lanes? + + public static func decode(_ data: Data) throws -> LoadTable { + try JSONDecoder().decode(LoadTable.self, from: data) + } + + /// The table as the command line prints it. + public func text() -> String { + func num(_ v: Double?) -> String { v.map { String(format: "%.1f", $0) } ?? "–" } + func pad(_ s: String, _ w: Int) -> String { String(repeating: " ", count: max(0, w - s.count)) + s } + var out = ["\(host ?? "host"): load1 by busy runners, last \(days) day(s), \(samples) sample(s)" + + (cores.map { ", \($0) cores" } ?? "")] + if rows.isEmpty { + out.append(" no samples in the window") + } else { + out.append(" busy samples load1 p50 load1 p90" + (cores != nil ? " p90/core" : "")) + for r in rows { + var line = " " + pad(String(r.busy), 4) + " " + pad(String(r.n), 7) + + " " + pad(num(r.p50), 9) + " " + pad(num(r.p90), 9) + if let cores, cores > 0 { line += " " + pad(num(r.p90.map { $0 / Double(cores) }), 8) } + out.append(line) + } + } + if let l = lanes, let mode = l.mode { + let lim = l.limits + out.append("lanes \(mode): heavy \(lim?.heavy.map(String.init) ?? "–"), light \(lim?.light.map(String.init) ?? "–"), total \(lim?.total.map(String.init) ?? "–")") + } + out.append("busy counts runners GitHub shows busy, including jobs held by admission.") + return out.joined(separator: "\n") + } +} diff --git a/cockpit/Sources/cockpit/main.swift b/cockpit/Sources/cockpit/main.swift index eab4a56..de65884 100644 --- a/cockpit/Sources/cockpit/main.swift +++ b/cockpit/Sources/cockpit/main.swift @@ -21,6 +21,8 @@ commands: --recover (with --pr): recover this PR's orphaned runs (force-cancel, then re-run) instead of stopping; recorded in the dashboard's action log top the host's top CPU users, naming Spotlight when it is the culprit + load host load1 p50/p90 by number of busy runners (--days N, default 7), + to re-tune the admission lanes' heavy limit sentinel check the host out of band (SSH, GitHub, the other machine) — what the app does when the dashboard does not answer brief a markdown incident brief of the current state @@ -426,6 +428,15 @@ case "top": for l in top.lines.prefix(10) { print(String(format: " %5.1f%% %@", l.cpu, (l.command as NSString).lastPathComponent)) } } exit(top.spotlightCPU >= 50 ? 1 : 0) +case "load": + if opts.fixture != nil { fail("load asks the live dashboard", code: 64) } + let days = opts.flags["days"].flatMap(Int.init) ?? 7 + let (client, link) = await connect(opts) + let raw: Data + do { raw = try await client.get("/api/load-by-busy", query: ["days": String(days)]) } catch { link.close(); fail(error) } + link.close() + if opts.json { printRaw(raw); exit(0) } + do { print(try LoadTable.decode(raw).text()) } catch { fail("the dashboard's answer did not parse: \(error)", code: 3) } case "brief": let src = await obtain(opts) print(IncidentBrief.markdown(verdict: src.verdict, glance: src.glance, route: src.route, connection: "cli")) diff --git a/cockpit/Tests/CockpitCoreTests/LoadTableTests.swift b/cockpit/Tests/CockpitCoreTests/LoadTableTests.swift new file mode 100644 index 0000000..216c551 --- /dev/null +++ b/cockpit/Tests/CockpitCoreTests/LoadTableTests.swift @@ -0,0 +1,30 @@ +import Foundation +import Testing +@testable import CockpitCore + +@Suite struct LoadTableTests { + let json = """ + {"days":7,"host":"ultra","since":0,"samples":6,"cores":30, + "rows":[{"busy":0,"n":2,"p50":3,"p90":4},{"busy":2,"n":4,"p50":12,"p90":30}], + "lanes":{"mode":"observe","limits":{"heavy":4,"light":12,"total":16}}} + """ + + @Test func decodesTheDaemonAnswer() throws { + let t = try LoadTable.decode(Data(json.utf8)) + #expect(t.rows.count == 2) + #expect(t.rows[1] == LoadTable.Row(busy: 2, n: 4, p50: 12, p90: 30)) + #expect(t.lanes?.limits?.heavy == 4) + } + + @Test func printsOneLinePerBusyCount() throws { + let text = try LoadTable.decode(Data(json.utf8)).text() + #expect(text.contains("ultra: load1 by busy runners, last 7 day(s), 6 sample(s), 30 cores")) + #expect(text.contains(" 2 4 12.0 30.0 1.0")) + #expect(text.contains("lanes observe: heavy 4, light 12, total 16")) + } + + @Test func toleratesAnOlderDaemonWithoutLanesOrCores() throws { + let t = try LoadTable.decode(Data(#"{"days":1,"samples":0,"rows":[]}"#.utf8)) + #expect(t.text().contains("no samples in the window")) + } +} diff --git a/dashboard/fleetd.js b/dashboard/fleetd.js index 9aedd30..12c2473 100644 --- a/dashboard/fleetd.js +++ b/dashboard/fleetd.js @@ -35,6 +35,8 @@ import { deriveGroups } from './lib/groups.js'; import { Backfill } from './lib/backfill.js'; import { RunSettler } from './lib/settle.js'; import { RunStatus } from './lib/run-status.js'; +import { createJobClass } from './lib/job-class.js'; +import { loadByBusy } from './lib/load-table.js'; import { analytics, repoDetail } from './lib/analytics.js'; import { buildActions, ActionError, ACTION_ARGS } from './lib/actions.js'; import { handleApi, ApiError, API_PREFIX } from './lib/api.js'; @@ -314,6 +316,13 @@ let alertRun = Promise.resolve(); // append to. This process never makes an admission decision itself — a job's // ability to start must not depend on the dashboard being alive. const admission = createAdmission({ db, logPath: CONFIG.admissionLog, warn, hostId: CONFIG.replicaId }); +// Admission lanes ask a job's class here (hooks/common.sh admit_classify). The +// threshold is the hook's own FLEET_ADMIT_SHORT_JOB_S, sent with the request; +// this host's fleet.env stands in for a hook that does not send it. +const jobClass = createJobClass({ + db, + shortJobS: () => admissionSettings({ root: CONFIG.root }).shortJobS, +}); const sampleShm = createShmSampler({ logPath: CONFIG.shmLog, lowFree: lowFreeFromEnv() }); // host_samples rows from before host_id existed are this host's own vitals. try { @@ -3253,6 +3262,36 @@ const server = http.createServer(async (req, res) => { } } + // The lane hooks/job-started.sh should put a job in: { class, p50Ms, n }. + // Read-only, from this database's own history; cheap and cached. + if (url.pathname === '/api/job-class') { + try { + return json(res, jobClass.lookup({ + repo: url.searchParams.get('repo'), + job: url.searchParams.get('job'), + shortS: url.searchParams.get('short_s'), + })); + } catch (err) { + if (!err.status) warn('job-class api:', err.message); + return json(res, { error: err.message }, err.status ?? 500); + } + } + + // Host load1 p50/p90 by busy runners, for tuning the heavy-lane limit + // (`cockpit load`). + if (url.pathname === '/api/load-by-busy') { + try { + return json(res, { + ...loadByBusy(db, { days: url.searchParams.get('days') ?? 7, hostId: CONFIG.replicaId }), + cores: os.cpus().length, + lanes: admission.summary().lanes, + }); + } catch (err) { + warn('load-by-busy api:', err.message); + return json(res, { error: err.message }, 500); + } + } + if (url.pathname === '/api/run-status') { try { return json(res, await runStatus.lookup(url.searchParams.get('repo'), url.searchParams.get('run'))); diff --git a/dashboard/lib/admission.js b/dashboard/lib/admission.js index 3161ea6..321e927 100644 --- a/dashboard/lib/admission.js +++ b/dashboard/lib/admission.js @@ -52,9 +52,20 @@ function rowFrom(line) { e.owner_kind || e.owner || null, e.owner_pid || null, e.reason || null, + // Lanes, present only on lines from a hook with FLEET_ADMIT_LANES on. + e.lanes || null, + e.lane || null, + e.lane_source || null, + num(e.heavy_busy), num(e.light_busy), + num(e.heavy_limit), num(e.light_limit), num(e.total_limit), + e.lane_would_hold === true ? 1 : e.lane_would_hold === false ? 0 : null, + num(e.lane_wait_s), + e.lane_reason || null, ]; } +const num = (v) => (v != null && v !== '' && Number.isFinite(Number(v)) ? Number(v) : null); + /** * @param {object} opts * @param {string} [opts.hostId] - this host's id; stamped on locally ingested rows. @@ -65,8 +76,11 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) const insert = db.prepare(` INSERT INTO admission_events (ts, event, mode, runner, repo, run_id, job, waited_s, ran_s, busy, limit_n, owner_kind, - owner_pid, reason, host_id) - VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`); + owner_pid, reason, + lanes, lane, lane_source, heavy_busy, light_busy, + heavy_limit, light_limit, total_limit, + lane_would_hold, lane_wait_s, lane_reason, host_id) + VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`); // Rows from before host_id existed were all written by this host's hooks. if (hostId) { try { @@ -85,7 +99,8 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) // live rather than after the fact, because an unexplained pause before a // job's first step is otherwise invisible. latestPerRunner: db.prepare(` - SELECT runner, repo, event, ts, reason, busy, limit_n, host_id + SELECT runner, repo, event, ts, reason, busy, limit_n, host_id, lane, lanes, + lane_would_hold, lane_reason FROM admission_events WHERE id IN ( SELECT MAX(id) FROM admission_events WHERE runner IS NOT NULL GROUP BY host_id, runner @@ -95,8 +110,21 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) FROM admission_events WHERE ts >= ? GROUP BY event`), recent: db.prepare(` SELECT ts, event, mode, runner, repo, job, waited_s, ran_s, busy, limit_n, - owner_kind, owner_pid, reason, host_id + owner_kind, owner_pid, reason, host_id, + lanes, lane, lane_source, heavy_busy, light_busy, heavy_limit, light_limit, + total_limit, lane_would_hold, lane_wait_s, lane_reason FROM admission_events ORDER BY ts DESC LIMIT ?`), + // The newest line from a hook with lanes on: their mode and limits. + lastLanes: db.prepare(` + SELECT ts, lanes, heavy_limit, light_limit, total_limit FROM admission_events + WHERE lanes IS NOT NULL AND ${LOCAL} ORDER BY id DESC LIMIT 1`), + // What the lanes did, or in observe would have done, over a window. + laneCountsSince: db.prepare(` + SELECT event, lane, lane_would_hold AS would, COUNT(*) AS n, + SUM(COALESCE(waited_s, 0)) AS waited, SUM(COALESCE(lane_wait_s, 0)) AS lane_waited, + SUM(CASE WHEN lane_wait_s IS NOT NULL THEN 1 ELSE 0 END) AS lane_wait_n + FROM admission_events WHERE ts >= @since AND lanes IS NOT NULL AND ${LOCAL} + GROUP BY event, lane, lane_would_hold`), // Restricted to rows that actually carry a mode. Reporting the newest row // unconditionally would let one event written without one — an older hook, // a hand-appended line — read as "installed but switched off" while the @@ -131,6 +159,52 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) // on the slow loop. let installed = { installed: 0, total: 0, checkedAt: null }; + // Lanes on this host: their mode and limits from the newest line that has + // them, jobs running and waiting per lane from each runner's newest event + // (a runner whose newest line admitted a job is running it), and, over 24 h, + // what the lanes did or would have done. null while no hook has lanes on. + function lanesSummary(latest, since) { + const last = stmt.lastLanes.get(local); + if (!last) return null; + const isLocal = (r) => r.host_id == null || r.host_id === hostId; + const running = { heavy: 0, light: 0, simulator: 0 }; + const waiting = { heavy: 0, light: 0, simulator: 0 }; + for (const r of latest) { + if (!isLocal(r) || !r.lane || !(r.lane in running)) continue; + if (['admitted', 'timeout', 'observed', 'would-hold'].includes(r.event)) running[r.lane] += 1; + else if (r.event === 'held') waiting[r.lane] += 1; + } + const day = { + // observe: held by the single lane, where the lanes would have admitted + // at once (lane_wait_s 0) or sooner; and admitted by it where the lanes + // would have held. + heldLanesWouldAdmit: 0, admittedLanesWouldHold: 0, + // observe: summed waits of jobs that finished waiting, under each policy. + waitedS: 0, laneWaitS: 0, finishedWaits: 0, + byLane: {}, + }; + for (const row of stmt.laneCountsSince.all({ since, local: local.local })) { + const lane = row.lane ?? 'unknown'; + day.byLane[lane] ??= {}; + day.byLane[lane][row.event] = (day.byLane[lane][row.event] ?? 0) + row.n; + if (row.event === 'held' && row.would === 0) day.heldLanesWouldAdmit += row.n; + if ((row.event === 'admitted' || row.event === 'observed') && row.would === 1) day.admittedLanesWouldHold += row.n; + if ((row.event === 'admitted' || row.event === 'timeout') && row.lane_wait_n > 0) { + day.waitedS += row.waited ?? 0; + day.laneWaitS += row.lane_waited ?? 0; + day.finishedWaits += row.lane_wait_n; + } + } + return { + mode: last.lanes, + limits: { heavy: last.heavy_limit, light: last.light_limit, total: last.total_limit }, + running, + waiting, + last24h: day, + lastDecisionAt: last.ts, + }; + } + return { /** * Reads only the bytes appended since the last pass, so this stays cheap on @@ -217,8 +291,8 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) if (row.event === 'admitted' || row.event === 'timeout') heldSeconds += row.waited ?? 0; } const last = stmt.last.get(local); - const waiting = stmt.latestPerRunner - .all() + const latest = stmt.latestPerRunner.all(); + const waiting = latest .filter((r) => r.event === 'held') .map((r) => ({ runner: r.runner, @@ -228,6 +302,9 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) busy: r.busy, limit: r.limit_n, host: r.host_id ?? hostId, + lane: r.lane ?? null, + laneWouldHold: r.lane_would_hold == null ? null : r.lane_would_hold === 1, + laneReason: r.lane_reason ?? null, })); // Mode and limit per host, newest decision each. A fleet where one host // enforces 2 and another 6 is not described by a single number. @@ -239,6 +316,7 @@ export function createAdmission({ db, logPath, warn = () => {}, hostId = null }) if (w.host && byHost[w.host]) byHost[w.host].waiting += 1; } return { + lanes: lanesSummary(latest, since), // Reported from the events rather than from configuration. fleet.env is // read by the hooks inside a job, not by this process, so the events are // the only honest source for what mode is actually in force. diff --git a/dashboard/lib/admit-gate.js b/dashboard/lib/admit-gate.js index 98b2885..d7df15c 100644 --- a/dashboard/lib/admit-gate.js +++ b/dashboard/lib/admit-gate.js @@ -54,6 +54,8 @@ export function admissionSettings({ root, env = process.env, read = readFileSync mode: mode === 'observe' || mode === 'enforce' ? mode : 'off', minFreeDiskGb: wholeOr(v('FLEET_ADMIT_MIN_FREE_DISK_GB'), 40), minPlainFreeGb: wholeOr(v('FLEET_ADMIT_MIN_PLAIN_FREE_GB'), 15), + // Lanes: a job with a recent median run under this many seconds is light. + shortJobS: wholeOr(v('FLEET_ADMIT_SHORT_JOB_S'), 90) || 90, }; } diff --git a/dashboard/lib/db.js b/dashboard/lib/db.js index 28c4b90..c5199d2 100644 --- a/dashboard/lib/db.js +++ b/dashboard/lib/db.js @@ -405,6 +405,19 @@ export function openDb(path) { addColumn(db, 'admission_events', 'host_id', 'TEXT'); addColumn(db, 'host_samples', 'host_id', 'TEXT'); + // Admission lanes (FLEET_ADMIT_LANES in hooks/common.sh). Written only by a + // hook with lanes on; NULL everywhere else. `lanes` is the policy mode + // (observe/enforce), `lane` the job's lane, lane_would_hold/lane_reason/ + // lane_wait_s what the lanes decided or, in observe, would have. + for (const [col, type] of [ + ['lanes', 'TEXT'], ['lane', 'TEXT'], ['lane_source', 'TEXT'], + ['heavy_busy', 'INTEGER'], ['light_busy', 'INTEGER'], + ['heavy_limit', 'INTEGER'], ['light_limit', 'INTEGER'], ['total_limit', 'INTEGER'], + ['lane_would_hold', 'INTEGER'], ['lane_wait_s', 'INTEGER'], ['lane_reason', 'TEXT'], + ]) addColumn(db, 'admission_events', col, type); + // /api/job-class reads a job's recent releases by repo and job id. + db.exec('CREATE INDEX IF NOT EXISTS idx_admission_repo_job ON admission_events(repo, job, ts DESC)'); + // Which branch the repo considers canonical. Needed to tell "this finding is // on the branch you would open a PR against" from "this finding is on a branch // that merely still gets pushes". diff --git a/dashboard/lib/job-class.js b/dashboard/lib/job-class.js new file mode 100644 index 0000000..6aaaa53 --- /dev/null +++ b/dashboard/lib/job-class.js @@ -0,0 +1,97 @@ +// A job's admission lane, from how long it has been running lately. +// +// hooks/job-started.sh puts every job in a lane (FLEET_ADMIT_LANES): `light` +// jobs are short and cheap and get many slots, `heavy` ones (Xcode, Gradle, +// Playwright, pytest + dbt) few. A pattern in fleet.env decides first; for the +// rest the hook asks here, with the repo and GITHUB_JOB, the job's YAML id. +// +// The history that matches that key exactly is the hook's own: every release +// line in the admission log carries the repo, the job id and ran_s, how long +// the job held its slot, which leaves the admission hold out. GitHub's job rows +// carry the display name instead ("ci / test", "Migration filenames"), which +// for a reusable workflow ends in the id and otherwise may share nothing with +// it, and their duration includes the hold. So the admission log is read +// first, and the jobs table only when it has too few releases (a job new to +// the hook, or a display name passed by hand), matched on the whole name or +// its last " / " part. +// +// Too few samples either way is `unknown`, which the hook treats as heavy: a +// heavy job let into the light lane is what overloads the host. + +const REPO = /^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/; +const DEFAULTS = { minSamples: 5, maxSamples: 30, windowDays: 14, cacheMs: 60_000, shortJobS: 90 }; + +function median(values) { + if (!values.length) return null; + const s = [...values].sort((a, b) => a - b); + const mid = Math.floor(s.length / 2); + return s.length % 2 ? s[mid] : Math.round((s[mid - 1] + s[mid]) / 2); +} + +const likeEscape = (s) => s.replace(/[\\%_]/g, (c) => `\\${c}`); + +/** + * @param {object} opts + * @param {import('node:sqlite').DatabaseSync} opts.db + * @param {() => number} [opts.now] + * @param {() => number} [opts.shortJobS] - the host's FLEET_ADMIT_SHORT_JOB_S, + * used when the hook does not send its own. + */ +export function createJobClass({ db, now = () => Date.now(), shortJobS = () => DEFAULTS.shortJobS, ...opts } = {}) { + const cfg = { ...DEFAULTS, ...opts }; + const stmt = { + released: db.prepare(` + SELECT ran_s FROM admission_events + WHERE event = 'released' AND repo = ? AND job = ? AND ts >= ? AND ran_s IS NOT NULL + ORDER BY ts DESC LIMIT ?`), + jobs: db.prepare(` + SELECT duration_ms FROM jobs + WHERE repo = ? AND (name = ? OR name LIKE ? ESCAPE '\\') + AND conclusion = 'success' AND duration_ms IS NOT NULL AND started_at >= ? + ORDER BY started_at DESC LIMIT ?`), + }; + const cache = new Map(); + + return { + /** + * @returns {{ repo: string, job: string, class: 'light'|'heavy'|'unknown', + * p50Ms: number|null, n: number, source: string, shortJobS: number }} + * @throws {Error} with .status 400 for a malformed query + */ + lookup({ repo, job, shortS } = {}) { + repo = String(repo ?? ''); + job = String(job ?? '').trim(); + if (!REPO.test(repo) || !job || job.length > 200) { + throw Object.assign(new Error('repo (owner/name) and job required'), { status: 400 }); + } + const threshold = /^[0-9]{1,5}$/.test(String(shortS ?? '')) && Number(shortS) > 0 + ? Number(shortS) : shortJobS(); + const key = `${repo}\n${job}\n${threshold}`; + const at = now(); + const hit = cache.get(key); + if (hit && at - hit.at < cfg.cacheMs) return hit.value; + + // A display name like "ci / test" ends in the id the hook would send. + const id = job.includes(' / ') ? job.split(' / ').pop().trim() : job; + const since = at - cfg.windowDays * 86_400_000; + let samples = stmt.released.all(repo, id, since, cfg.maxSamples).map((r) => Number(r.ran_s) * 1000); + let source = 'admission'; + if (samples.length < cfg.minSamples) { + const fromJobs = stmt.jobs + .all(repo, job, `% / ${likeEscape(id)}`, new Date(since).toISOString(), cfg.maxSamples) + .map((r) => Number(r.duration_ms)); + if (fromJobs.length > samples.length) { + samples = fromJobs; + source = 'jobs'; + } + } + const n = samples.length; + const p50Ms = median(samples); + const cls = n < cfg.minSamples ? 'unknown' : p50Ms < threshold * 1000 ? 'light' : 'heavy'; + const value = { repo, job, class: cls, p50Ms, n, source: n ? source : 'none', shortJobS: threshold }; + cache.set(key, { at, value }); + if (cache.size > 2000) cache.clear(); + return value; + }, + }; +} diff --git a/dashboard/lib/load-table.js b/dashboard/lib/load-table.js new file mode 100644 index 0000000..650f4f4 --- /dev/null +++ b/dashboard/lib/load-table.js @@ -0,0 +1,45 @@ +// Host load by the number of busy runners, from host_samples (one row a +// minute). This is the table the heavy-lane limit was set from (plan §1.4, +// 2026-10-22: ultra's load1 rose about 8 per concurrent job on 30 cores), kept +// as a live view so the limit can be re-tuned when the job mix changes. +// +// busy_runners counts runners GitHub shows busy, which includes jobs the +// admission hook is holding: a held job burns no CPU, so on a host that holds +// a lot the low buckets read high. Read the table with the hold count beside it. + +const pct = (sorted, p) => { + if (!sorted.length) return null; + const i = Math.min(sorted.length - 1, Math.max(0, Math.ceil((p / 100) * sorted.length) - 1)); + return Math.round(sorted[i] * 10) / 10; +}; + +/** + * @param {import('node:sqlite').DatabaseSync} db + * @param {object} opts + * @param {number} [opts.days=7] + * @param {string|null} [opts.hostId] - this host's id; rows with no host are its own + * @param {number} [opts.now] + * @returns {{ days: number, host: string|null, samples: number, since: number, + * rows: Array<{ busy: number, n: number, p50: number|null, p90: number|null }> }} + */ +export function loadByBusy(db, { days = 7, hostId = null, now = Date.now() } = {}) { + const d = Math.min(Math.max(Number(days) || 7, 1), 90); + const since = now - d * 86_400_000; + const rows = db.prepare(` + SELECT busy_runners AS busy, load1 FROM host_samples + WHERE ts >= ? AND load1 IS NOT NULL AND busy_runners IS NOT NULL + AND (host_id IS NULL OR host_id = ?)`).all(since, hostId ?? ''); + const by = new Map(); + for (const r of rows) { + const b = Math.max(0, Math.round(Number(r.busy))); + if (!by.has(b)) by.set(b, []); + by.get(b).push(Number(r.load1)); + } + const out = [...by.entries()] + .sort((a, b) => a[0] - b[0]) + .map(([busy, loads]) => { + loads.sort((a, b) => a - b); + return { busy, n: loads.length, p50: pct(loads, 50), p90: pct(loads, 90) }; + }); + return { days: d, host: hostId, since, samples: rows.length, rows: out }; +} diff --git a/dashboard/public/capacity.js b/dashboard/public/capacity.js index d68fb70..34fc4ec 100644 --- a/dashboard/public/capacity.js +++ b/dashboard/public/capacity.js @@ -311,20 +311,25 @@ function admissionPanel(s) { ) : null, + lanesBlock(a.lanes, mins), + waiting.length ? h('div', {}, h('h4', { text: `Waiting now (${waiting.length})` }), h('table', { class: 'grid' }, h('thead', {}, h('tr', {}, h('th', { scope: 'col', text: 'runner' }), h('th', { scope: 'col', text: 'repo' }), + h('th', { scope: 'col', text: 'lane' }), h('th', { scope: 'col', text: 'waiting' }), h('th', { scope: 'col', text: 'why' }) )), h('tbody', {}, waiting.map((w) => h('tr', { class: 'is-warn' }, h('td', { text: w.runner ?? '–' }), h('td', { text: (w.repo ?? '').split('/').pop() || '–' }), + h('td', { class: 'mono', text: w.lane ?? '–' }), h('td', { text: elapsed(w.since) }), - h('td', { class: 'muted', text: w.reason ?? '–' }) + h('td', { class: 'muted', text: + (w.reason ?? '–') + (w.laneWouldHold === false ? ' (the lanes would admit it)' : '') }) ) )) )) @@ -339,6 +344,42 @@ function admissionPanel(s) { ); } +// Admission lanes (FLEET_ADMIT_LANES): light, heavy and Simulator jobs each +// have their own limit and their own line. In observe the single lane still +// decides and these rows say what the lanes would have done instead. +function lanesBlock(l, mins) { + if (!l) return null; + const lim = l.limits ?? {}; + const run = l.running ?? {}; + const wait = l.waiting ?? {}; + const day = l.last24h ?? {}; + const sims = run.simulator ?? 0; + const line = (running, limit, waiting) => + `${running}${limit != null ? ` of ${limit}` : ''} running` + (waiting ? `, ${waiting} waiting` : ''); + const observing = l.mode === 'observe'; + return h('div', {}, + h('h4', { text: `Lanes — ${observing ? 'observing (the single lane decides)' : l.mode}` }), + h('dl', { class: 'kv' }, + h('dt', { text: 'heavy' }), + h('dd', { text: line((run.heavy ?? 0) + sims, lim.heavy, (wait.heavy ?? 0) + (wait.simulator ?? 0)) + + (sims ? ` (${sims} of them Simulator jobs)` : '') }), + h('dt', { text: 'light' }), + h('dd', { text: line(run.light ?? 0, lim.light, wait.light ?? 0) }), + h('dt', { text: 'total cap' }), + h('dd', { text: lim.total != null ? `${lim.total} job(s)` : '–' }), + observing ? h('dt', { text: 'held, lanes would admit (24h)' }) : null, + observing ? h('dd', { text: String(day.heldLanesWouldAdmit ?? 0) }) : null, + observing ? h('dt', { text: 'admitted, lanes would hold (24h)' }) : null, + observing ? h('dd', { text: String(day.admittedLanesWouldHold ?? 0) }) : null, + observing && day.finishedWaits + ? h('dt', { text: 'wait, single lane vs lanes (24h)' }) : null, + observing && day.finishedWaits + ? h('dd', { text: `${mins(day.waitedS ?? 0)} vs ${mins(day.laneWaitS ?? 0)} over ${day.finishedWaits} held job(s)` }) + : null + ) + ); +} + function autoscalePanel(s) { const a = s.autoscale ?? {}; const state = !a.enabled ? 'off' : a.dryRun ? 'dry run' : 'live'; diff --git a/dashboard/test/admission.test.js b/dashboard/test/admission.test.js index d6b72b1..db7b284 100644 --- a/dashboard/test/admission.test.js +++ b/dashboard/test/admission.test.js @@ -201,3 +201,52 @@ describe('countInstalled', () => { assert.ok(admission.summary().hooks.checkedAt > 0); }); }); + +describe('lanes', () => { + const lane = (o) => line({ lanes: 'observe', lane: 'heavy', lane_source: 'pattern', heavy_busy: 1, + light_busy: 0, heavy_limit: 4, light_limit: 12, total_limit: 16, lane_would_hold: false, + lane_wait_s: null, lane_reason: '', ...o }); + + test('a line from a hook with lanes off has no lane summary', () => { + const { admission } = fresh([line({ event: 'admitted' })]); + assert.equal(admission.summary().lanes, null); + }); + + test('lane fields are stored as written', () => { + const { db } = fresh([lane({ event: 'held', runner: 'r1', lane_would_hold: true, + lane_reason: '4 heavy job(s) already running, at the heavy limit of 4', lane_wait_s: 0 })]); + const r = db.prepare('SELECT * FROM admission_events').get(); + assert.equal(r.lanes, 'observe'); + assert.equal(r.lane, 'heavy'); + assert.equal(r.lane_source, 'pattern'); + assert.equal(r.lane_would_hold, 1); + assert.equal(r.lane_wait_s, 0); + assert.equal(r.total_limit, 16); + assert.equal(r.lane_reason, '4 heavy job(s) already running, at the heavy limit of 4'); + }); + + test('the summary counts running and waiting jobs per lane, and what the lanes would have done', () => { + const { admission } = fresh([ + lane({ event: 'admitted', runner: 'h1', lane: 'heavy' }), + lane({ event: 'admitted', runner: 'h2', lane: 'heavy', lane_would_hold: true }), + lane({ event: 'admitted', runner: 'l1', lane: 'light' }), + lane({ event: 'admitted', runner: 's1', lane: 'simulator' }), + lane({ event: 'held', runner: 'l2', lane: 'light', reason: '8 job(s) already running, at the limit of 8', + lane_would_hold: false, lane_wait_s: 0 }), + lane({ event: 'admitted', runner: 'l3', lane: 'light', waited_s: 40, lane_wait_s: 0 }), + lane({ event: 'admitted', runner: 'x', lane: 'light' }), + lane({ event: 'released', runner: 'x', lane: 'light' }), + ]); + const s = admission.summary(); + assert.equal(s.lanes.mode, 'observe'); + assert.deepEqual(s.lanes.limits, { heavy: 4, light: 12, total: 16 }); + assert.deepEqual(s.lanes.running, { heavy: 2, light: 2, simulator: 1 }); + assert.deepEqual(s.lanes.waiting, { heavy: 0, light: 1, simulator: 0 }); + assert.equal(s.lanes.last24h.heldLanesWouldAdmit, 1); + assert.equal(s.lanes.last24h.admittedLanesWouldHold, 1); + assert.equal(s.lanes.last24h.waitedS, 40); + assert.equal(s.lanes.last24h.laneWaitS, 0); + assert.equal(s.waiting[0].lane, 'light'); + assert.equal(s.waiting[0].laneWouldHold, false); + }); +}); diff --git a/dashboard/test/admit-gate.test.js b/dashboard/test/admit-gate.test.js index 986ee6c..663367e 100644 --- a/dashboard/test/admit-gate.test.js +++ b/dashboard/test/admit-gate.test.js @@ -22,7 +22,7 @@ describe('parseEnvFile', () => { describe('admissionSettings', () => { test('defaults match hooks/common.sh: off, 40 GB, 15 GB', () => { - assert.deepEqual(admissionSettings({ root: '/r', env: {}, read: noFile }), { mode: 'off', minFreeDiskGb: 40, minPlainFreeGb: 15 }); + assert.deepEqual(admissionSettings({ root: '/r', env: {}, read: noFile }), { mode: 'off', minFreeDiskGb: 40, minPlainFreeGb: 15, shortJobS: 90 }); }); test('fleet.env is sourced over the environment, as the hook sources it', () => { @@ -31,12 +31,12 @@ describe('admissionSettings', () => { env: { FLEET_ADMIT_MODE: 'observe', FLEET_ADMIT_MIN_FREE_DISK_GB: '50', FLEET_ADMIT_MIN_PLAIN_FREE_GB: '20' }, read: fileOf('FLEET_ADMIT_MODE=enforce\nFLEET_ADMIT_MIN_FREE_DISK_GB=100\n'), }); - assert.deepEqual(s, { mode: 'enforce', minFreeDiskGb: 100, minPlainFreeGb: 20 }); + assert.deepEqual(s, { mode: 'enforce', minFreeDiskGb: 100, minPlainFreeGb: 20, shortJobS: 90 }); }); test('a typo is the default, never the most expensive reading', () => { const s = admissionSettings({ root: '/r', env: {}, read: fileOf('FLEET_ADMIT_MODE=enforec\nFLEET_ADMIT_MIN_FREE_DISK_GB=1OO\n') }); - assert.deepEqual(s, { mode: 'off', minFreeDiskGb: 40, minPlainFreeGb: 15 }); + assert.deepEqual(s, { mode: 'off', minFreeDiskGb: 40, minPlainFreeGb: 15, shortJobS: 90 }); }); }); diff --git a/dashboard/test/job-class.test.js b/dashboard/test/job-class.test.js new file mode 100644 index 0000000..0278c04 --- /dev/null +++ b/dashboard/test/job-class.test.js @@ -0,0 +1,125 @@ +import { test, describe, after } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { openDb } from '../lib/db.js'; +import { createJobClass } from '../lib/job-class.js'; +import { loadByBusy } from '../lib/load-table.js'; + +const NOW = Date.parse('2026-10-22T20:00:00Z'); +const dirs = []; +after(() => { for (const d of dirs) rmSync(d, { recursive: true, force: true }); }); + +function fresh() { + const dir = mkdtempSync(join(tmpdir(), 'job-class-')); + dirs.push(dir); + return openDb(join(dir, 'test.db')); +} + +// The admission log's release lines, which carry the job id and ran_s. +function releases(db, repo, job, secs, { ageMs = 3600_000 } = {}) { + const ins = db.prepare(`INSERT INTO admission_events (ts, event, repo, job, ran_s, host_id) + VALUES (?, 'released', ?, ?, ?, 'ultra')`); + secs.forEach((s, i) => ins.run(NOW - ageMs - i * 1000, repo, job, s)); +} +let jobId = 1; +function jobRows(db, repo, name, secs, { conclusion = 'success' } = {}) { + const ins = db.prepare(`INSERT INTO jobs (id, run_id, repo, name, conclusion, started_at, duration_ms) + VALUES (?, 1, ?, ?, ?, ?, ?)`); + secs.forEach((s, i) => ins.run(jobId++, repo, name, conclusion, + new Date(NOW - 3600_000 - i * 1000).toISOString(), s * 1000)); +} + +describe('job class', () => { + test('a job whose recent median run is under the threshold is light', () => { + const db = fresh(); + releases(db, 'acme/site', 'words', [10, 12, 14, 11, 13, 200]); + const c = createJobClass({ db, now: () => NOW }); + const r = c.lookup({ repo: 'acme/site', job: 'words' }); + assert.equal(r.class, 'light'); + assert.equal(r.n, 6); + assert.equal(r.p50Ms, 12500); + assert.equal(r.source, 'admission'); + }); + + test('a long job is heavy, and the hook can send its own threshold', () => { + const db = fresh(); + releases(db, 'acme/mobile', 'test', [500, 533, 610, 480, 520]); + const c = createJobClass({ db, now: () => NOW }); + assert.equal(c.lookup({ repo: 'acme/mobile', job: 'test' }).class, 'heavy'); + assert.equal(c.lookup({ repo: 'acme/mobile', job: 'test', shortS: '900' }).class, 'light'); + assert.equal(c.lookup({ repo: 'acme/mobile', job: 'test', shortS: 'banana' }).shortJobS, 90); + }); + + test('too few runs is unknown, which the hook treats as heavy', () => { + const db = fresh(); + releases(db, 'acme/mobile', 'new', [3, 4]); + const r = createJobClass({ db, now: () => NOW }).lookup({ repo: 'acme/mobile', job: 'new' }); + assert.equal(r.class, 'unknown'); + assert.equal(r.n, 2); + }); + + test('a job never seen has no samples at all', () => { + const r = createJobClass({ db: fresh(), now: () => NOW }).lookup({ repo: 'a/b', job: 'x' }); + assert.deepEqual([r.class, r.n, r.p50Ms, r.source], ['unknown', 0, null, 'none']); + }); + + test('releases older than the window do not count', () => { + const db = fresh(); + releases(db, 'a/b', 'old', [5, 5, 5, 5, 5], { ageMs: 30 * 86_400_000 }); + assert.equal(createJobClass({ db, now: () => NOW }).lookup({ repo: 'a/b', job: 'old' }).class, 'unknown'); + }); + + test('a display name falls back to GitHub job rows, by its last part too', () => { + const db = fresh(); + jobRows(db, 'acme/mobile', 'ios / test', [540, 600, 580, 560, 610]); + const c = createJobClass({ db, now: () => NOW }); + const r = c.lookup({ repo: 'acme/mobile', job: 'ios / test' }); + assert.deepEqual([r.class, r.source, r.n], ['heavy', 'jobs', 5]); + // The id alone finds the same rows through "% / test". + assert.equal(c.lookup({ repo: 'acme/mobile', job: 'test' }).source, 'jobs'); + }); + + test('only successful job rows count, and LIKE wildcards in a name are literal', () => { + const db = fresh(); + jobRows(db, 'a/b', 'ci / a_b', [5, 5, 5, 5, 5], { conclusion: 'failure' }); + jobRows(db, 'a/b', 'ci / axb', [5, 5, 5, 5, 5]); + assert.equal(createJobClass({ db, now: () => NOW }).lookup({ repo: 'a/b', job: 'a_b' }).n, 0); + }); + + test('a malformed query is a 400', () => { + const c = createJobClass({ db: fresh(), now: () => NOW }); + assert.throws(() => c.lookup({ repo: 'nope', job: 'x' }), (e) => e.status === 400); + assert.throws(() => c.lookup({ repo: 'a/b', job: '' }), (e) => e.status === 400); + }); + + test('answers are cached for a minute', () => { + const db = fresh(); + let t = NOW; + const c = createJobClass({ db, now: () => t }); + assert.equal(c.lookup({ repo: 'a/b', job: 'j' }).class, 'unknown'); + releases(db, 'a/b', 'j', [1, 1, 1, 1, 1]); + assert.equal(c.lookup({ repo: 'a/b', job: 'j' }).class, 'unknown'); + t += 61_000; + assert.equal(c.lookup({ repo: 'a/b', job: 'j' }).class, 'light'); + }); +}); + +describe('load by busy runners', () => { + test('p50 and p90 of load1 per busy-runner count, this host only', () => { + const db = fresh(); + const ins = db.prepare('INSERT INTO host_samples (ts, load1, busy_runners, host_id) VALUES (?,?,?,?)'); + let ts = NOW - 3600_000; + for (const l of [10, 12, 14, 30]) ins.run(ts++, l, 2, 'ultra'); + for (const l of [3, 4]) ins.run(ts++, l, 0, null); + ins.run(ts++, 99, 2, 'other-host'); + ins.run(NOW - 30 * 86_400_000, 500, 2, 'ultra'); + const t = loadByBusy(db, { days: 7, hostId: 'ultra', now: NOW }); + assert.equal(t.samples, 6); + assert.deepEqual(t.rows, [ + { busy: 0, n: 2, p50: 3, p90: 4 }, + { busy: 2, n: 4, p50: 12, p90: 30 }, + ]); + }); +}); diff --git a/dashboard/test/routes.test.js b/dashboard/test/routes.test.js index 1925c3e..1dce959 100644 --- a/dashboard/test/routes.test.js +++ b/dashboard/test/routes.test.js @@ -338,6 +338,24 @@ test('GET /metrics exposes queue and collector gauges', async () => { assert.match(body, /fleet_autoscale_deficit/); }); +test('GET /api/job-class answers without auth, unknown for a job never seen', async () => { + const bad = await fetch(url('/api/job-class?repo=nope&job=x')); + assert.equal(bad.status, 400); + const r = await fetch(url('/api/job-class?repo=acme%2Fsite&job=words&short_s=90')); + assert.equal(r.status, 200); + const body = await r.json(); + assert.deepEqual([body.class, body.n, body.shortJobS], ['unknown', 0, 90]); +}); + +test('GET /api/load-by-busy returns rows, the window and the core count', async () => { + const r = await fetch(url('/api/load-by-busy?days=3')); + assert.equal(r.status, 200); + const body = await r.json(); + assert.equal(body.days, 3); + assert.ok(Array.isArray(body.rows)); + assert.ok(body.cores > 0); +}); + test('GET /api/queue-history validates run ids and returns bounded rows', async () => { const bad = await fetch(url('/api/queue-history?runId=not-a-number')); assert.equal(bad.status, 400); diff --git a/docs/admission-and-scaling.md b/docs/admission-and-scaling.md index 2c7740f..8f2efe0 100644 --- a/docs/admission-and-scaling.md +++ b/docs/admission-and-scaling.md @@ -80,7 +80,8 @@ both starting at once. When a job starts on any runner, `hooks/job-started.sh` runs first. In `enforce` mode: -1. It joins a host-wide FIFO waiter queue under an atomic directory mutex. +1. It joins a host-wide FIFO waiter queue under an atomic directory mutex + (one queue per [lane](#lanes) when lanes are enforced). 2. It reaps waiter files whose hook process has exited and slot files whose owning `Runner.Worker` has exited. 3. The oldest waiter checks live admitted slots and free disk. Later waiters @@ -114,6 +115,83 @@ the jobs wait in GitHub's queue, where the wait is not charged to the job. A hold that still ends in a timeout is classed `admission-hold` and shown as not your code. +### Lanes + +One host-wide limit treats a 5-second grep like a 12-minute Xcode build. On +ultra (30 cores, October 2026) load1 rose about 8 per concurrent job, 74 % of +jobs finished in under 90 s, and with one FIFO line the short ones waited +behind the long ones. `FLEET_ADMIT_LANES` gives each kind of job its own limit +and its own line: + +| Lane | Which jobs | Limit | +|---|---|---| +| `simulator` | runners `FLEET_SIMULATOR_RUNNERS` selects (with `FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT` > 0) | `FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT`, strict as before, **and** a heavy slot | +| `heavy` | `FLEET_ADMIT_HEAVY_PATTERNS`, jobs whose recent median run is `FLEET_ADMIT_SHORT_JOB_S` or more, and anything not known | `FLEET_ADMIT_HEAVY_MAX_CONCURRENT` (4) | +| `light` | `FLEET_ADMIT_LIGHT_PATTERNS`, and jobs whose recent median run is under `FLEET_ADMIT_SHORT_JOB_S` (90 s) | `FLEET_ADMIT_LIGHT_MAX_CONCURRENT` (12) | + +All under `FLEET_ADMIT_TOTAL_MAX_CONCURRENT` (default +`FLEET_ADMIT_MAX_CONCURRENT`), and the disk floors, as before. + +**Classification**, once per job before it waits, first match wins: + +1. The Simulator runner rule. +2. `FLEET_ADMIT_HEAVY_PATTERNS`, then `FLEET_ADMIT_LIGHT_PATTERNS`: + space-separated `repo:job` shell patterns, quoted in `fleet.env`. `job` is + `GITHUB_JOB`, the YAML id of the job that runs (inside a reusable workflow, + its own key: `test`, not `ios / test`); the repo may be written with or + without its owner. `"*:e2e acme/app:test site:*"`. +3. The dashboard's history, `GET /api/job-class` (see the [API](api.md)): the + median of the job's last 30 runs in 14 days, from the admission log's own + release lines (`ran_s`, the time it held a slot). Light under + `FLEET_ADMIT_SHORT_JOB_S` with at least 5 runs, heavy at or over it, + `unknown` with fewer. The answer is cached in `.admission/job-class/` for + `FLEET_ADMIT_CLASS_CACHE_S` (300 s), so a push that starts twenty jobs asks + once per job name. +4. Otherwise heavy: `unknown`, a daemon that does not answer in 3 s, or no + daemon URL. A heavy job let into the light lane is what overloads the host; + a light job in the heavy lane only waits a little longer. + +A job's duration says nothing about its CPU: a 70-second Gradle build is short +and heavy. Name such jobs in `FLEET_ADMIT_HEAVY_PATTERNS`. + +**Per-lane FIFO.** Each line is FIFO on its own: the oldest light job is never +behind an older heavy one, and a later job in a line still leaves the mutex to +the oldest one in it. A waiter file is named +`---` so the lockless check reads its line from +the directory listing; a name from before lanes reads as heavy. Simulator jobs +wait in the heavy line, and one held by the Simulator limit is skipped, as in +the single line. Mutex, reaping, cancel polling, the disk floors and the +timeout action are unchanged; the heavy and light limits are relaxed by +`FLEET_ADMIT_TIMEOUT_ACTION=admit` like the host limit, and the Simulator limit +stays strict. + +**Reasons.** The total cap, the Simulator limit and the disk floors keep their +words. The lanes add `4 heavy job(s) already running, at the heavy limit of 4`, +`12 light job(s) already running, at the light limit of 12` and `an older job +is waiting for the next heavy slot` (or `light slot`), which the lake files with +the old ones as `concurrency limit` and `queue order`. + +**Modes.** + +- `off` (default): no lane logic at all; the log line is the one it always was. +- `observe`: the single lane decides, exactly as with lanes off. Every line also + carries `lanes`, `lane`, `lane_source` (`runner`, `pattern`, `history`, + `unknown`, `fallback`), `heavy_busy`, `light_busy`, the three limits, and the + lanes' own answer: `lane_would_hold` (true/false, null when it could not be + worked out that poll), `lane_reason`, and for a wait `lane_wait_s`, how far + into the wait the lanes would first have admitted the job. Set + `FLEET_ADMIT_TOTAL_MAX_CONCURRENT` to the total you mean to enforce, so the + lanes are judged at it while the single lane keeps its own limit. +- `enforce`: the lanes decide. The log's `limit` is then the total cap. + +**Rolling out.** Observe for a day, then read the Capacity tab's lane rows (held +by the single lane where the lanes would admit, admitted where they would hold, +summed waits under each) and `cockpit load` (host load1 p50/p90 by busy +runners): the heavy limit is right when load at that many heavy jobs stays near +the core count. Then set `FLEET_ADMIT_LANES=enforce`. Going back is +`FLEET_ADMIT_LANES=off`; nothing restarts, because the hooks read `fleet.env` +for every job. + ### Installing the hooks ```bash @@ -186,6 +264,12 @@ FLEET_ADMIT_MAX_WAIT_S=600 FLEET_ADMIT_TIMEOUT_ACTION=admit FLEET_ADMIT_CANCEL_POLL_S=30 FLEET_ADMIT_MIN_FREE_DISK_GB=40 +# Lanes (see above) +FLEET_ADMIT_LANES=observe +FLEET_ADMIT_TOTAL_MAX_CONCURRENT=16 +FLEET_ADMIT_HEAVY_MAX_CONCURRENT=4 +FLEET_ADMIT_LIGHT_MAX_CONCURRENT=12 +FLEET_ADMIT_SHORT_JOB_S=90 ``` See [Configuration](configuration.md) for all admission variables. diff --git a/docs/api.md b/docs/api.md index 1dc3bcc..2da16b4 100644 --- a/docs/api.md +++ b/docs/api.md @@ -431,7 +431,13 @@ Recent autoscale decisions. ### `GET /api/admission?limit=60` -Job admission decisions. `limit` defaults to 60, max 500. +Job admission decisions. `limit` defaults to 60, max 500. With +[lanes](admission-and-scaling.md#lanes) on, rows carry `lanes`, `lane`, +`lane_source`, `heavy_busy`, `light_busy`, `heavy_limit`, `light_limit`, +`total_limit`, `lane_would_hold`, `lane_wait_s` and `lane_reason`, waiting +entries carry `lane`, and the summary has a `lanes` block: `mode`, `limits`, +`running` and `waiting` per lane, and over 24 h `heldLanesWouldAdmit`, +`admittedLanesWouldHold`, `waitedS` against `laneWaitS` and `byLane`. ### `GET /api/run-status?repo=owner/name&run=123` @@ -441,6 +447,30 @@ this to notice a run that ended while they waited: runner LaunchAgents set `SessionCreate`, so `gh` inside a job cannot read the keychain token. 400 for a malformed query, 404 for a repo this fleet does not serve. +### `GET /api/job-class?repo=owner/name&job=&short_s=90` + +`{ "repo", "job", "class": "light"|"heavy"|"unknown", "p50Ms", "n", "source", "shortJobS" }`: +the [admission lane](admission-and-scaling.md#lanes) for a job, which +`hooks/job-started.sh` asks for every job its patterns do not name. `job` is +`GITHUB_JOB` (the job's YAML id); a display name such as `ios / test` also +works. The median is over the job's newest 30 releases in the last 14 days +from the admission log (`ran_s`, which leaves the hold out), or, with fewer +than 5 of those, its successful GitHub job rows matched by name or by +`% / ` (whose durations include any hold). `light` when the median is +under `short_s` seconds (the hook's `FLEET_ADMIT_SHORT_JOB_S`; the host's +`fleet.env` when omitted); `unknown` with fewer than 5 samples, which the hook +treats as heavy. `source` is `admission`, `jobs` or `none`. Cached for 60 s. +400 for a malformed query. No auth, like `/api/run-status`. + +### `GET /api/load-by-busy?days=7` + +`{ "days", "host", "since", "samples", "cores", "rows": [{ "busy", "n", "p50", "p90" }], "lanes" }`: +this host's load1 p50 and p90 per number of busy runners over the window +(1–90 days), from the per-minute `host_samples`. `busy` counts runners GitHub +shows busy, which includes jobs the admission hook is holding. `lanes` is the +`lanes` block of `/api/admission` (null while lanes are off). `cockpit load` +prints it. + ### `GET /api/simulate` (POST) Replay historical job data against a hypothetical runner configuration. diff --git a/docs/cockpit/cli.md b/docs/cockpit/cli.md index 04f8af2..81bd47e 100644 --- a/docs/cockpit/cli.md +++ b/docs/cockpit/cli.md @@ -321,6 +321,28 @@ Spotlight is using 150% CPU — it is indexing, most likely the runner work tree `{ verdict, spotlightCPU, top }`. Exit `1` when Spotlight is using 50% CPU or more, `3` when `ps` could not be run, else `0`. +## `cockpit load` + +Host load by how many runners were busy, from the dashboard's per-minute host +samples (`/api/load-by-busy`): load1 p50 and p90 for each busy-runner count, +and p90 per core. This is the table the [admission lanes](../admission-and-scaling.md#lanes)' +heavy limit is set from; re-read it when the job mix changes. + +```console +$ cockpit load --days 7 +ultra: load1 by busy runners, last 7 day(s), 9841 sample(s), 30 cores + busy samples load1 p50 load1 p90 p90/core + 0 2210 3.0 8.0 0.3 + 2 890 12.0 27.0 0.9 + 4 640 28.0 53.0 1.8 +lanes observe: heavy 4, light 12, total 16 +busy counts runners GitHub shows busy, including jobs held by admission. +``` + +`--days N` (1 to 90, default 7) sets the window. `--json` prints the daemon's +answer. Busy runners include jobs the admission hook is holding, which use no +CPU, so on a host that holds a lot the low rows read high. + ## `cockpit brief` A markdown incident brief of the current state: the verdict, its evidence and diff --git a/docs/configuration.md b/docs/configuration.md index d3da12f..3228aa8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -70,6 +70,7 @@ Used by `cleanup.sh` and `cleanupctl.sh`. See | Variable | Default | What it does | |---|---|---| | `FLEET_CLEANUP_PRESSURE_GB` | `FLEET_ADMIT_MIN_FREE_DISK_GB` + 20 | Below this much usable disk, `--auto` acts on every tick and takes the pressure steps too. | +| `FLEET_CLEANUP_RUNNER_CACHE_DAYS` | `7` | Erase an idle runner's warm build caches (`_work//DerivedData-*`, `spm-*`, `.build-cache-*`) unused this long; under pressure, all of them. | | `FLEET_CLEANUP_SIM_MAX_GB` | `3` | Erase an idle runner's `ci-` simulator past this size. | | `FLEET_CLEANUP_SIM_PRESSURE_GB` | `1` | ...and past this under pressure. | | `FLEET_CLEANUP_FULL_EVERY_S` | `86400` | `--auto` runs a full pass at least this often. | @@ -316,6 +317,15 @@ judge the host by the limits the hooks actually enforce. After changing any of t | `FLEET_ADMIT_POLL_S` | `5` | How often a held job re-checks. | | `FLEET_ADMIT_MUTEX_TRIES` | `50` | Mutex spin attempts per wait-loop tick in `enforce` mode. | | `FLEET_ADMIT_SLOT_TTL_S` | `21600` | Slot lease TTL in seconds. | +| `FLEET_ADMIT_LANES` | `off` | `off` / `observe` / `enforce`: [admission lanes](admission-and-scaling.md#lanes). `observe` keeps the single lane deciding and logs what the lanes would have decided. | +| `FLEET_ADMIT_TOTAL_MAX_CONCURRENT` | `FLEET_ADMIT_MAX_CONCURRENT` | The total cap the lanes apply. | +| `FLEET_ADMIT_HEAVY_MAX_CONCURRENT` | `4` | Heavy jobs at once; Simulator jobs count here too. | +| `FLEET_ADMIT_LIGHT_MAX_CONCURRENT` | `12` | Light jobs at once. `FLEET_ADMIT_SHORT_MAX_CONCURRENT` is accepted as an alias. | +| `FLEET_ADMIT_HEAVY_PATTERNS` | (none) | Space-separated `repo:job` shell patterns that are always heavy. `job` is `GITHUB_JOB`, the job's YAML id; `repo` may omit the owner. Checked before the light patterns. | +| `FLEET_ADMIT_LIGHT_PATTERNS` | (none) | The same, always light. | +| `FLEET_ADMIT_SHORT_JOB_S` | `90` | A job the patterns do not name is light when its recent median run (`/api/job-class`) is under this many seconds, with at least 5 runs; otherwise heavy. | +| `FLEET_ADMIT_CLASS_CACHE_S` | `300` | How long the hook reuses the daemon's answer for one repo and job. | +| `FLEET_ADMIT_CLASS_URL` | `/api/job-class` beside `FLEET_ADMIT_STATUS_URL` | Where the hook asks a job's class. Empty, or an empty `FLEET_ADMIT_STATUS_URL`, skips it: unnamed jobs are then heavy. | Eligible enforced waiters are FIFO. A Simulator waiter blocked by its resource-specific limit does not prevent an unrelated waiter from using free diff --git a/docs/operations.md b/docs/operations.md index 0380adc..7d0dbb4 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -127,6 +127,13 @@ What it cleans: never candidates) - Unavailable simulators (`xcrun simctl delete unavailable`) - Runner `_diag` logs older than 14 days +- Each idle runner's warm build caches beside its checkout + (`_work//DerivedData-*`, `spm-*`, `.build-cache-*`) with nothing + written in `FLEET_CLEANUP_RUNNER_CACHE_DAYS` (7); under disk pressure, all of + them. Never a building runner's, never under the runner's own `_work/_*`, + never a git checkout. +- The Spotlight marker `_work/.metadata_never_index` for any runner missing it + (`register.sh` writes it for every new runner) - Playwright browser cache sizes (reported every run) - Stale Playwright `__dirlock` files older than 6 hours (skipped while an install is in flight) diff --git a/docs/security-hardening.md b/docs/security-hardening.md index 105deaf..9bed351 100644 --- a/docs/security-hardening.md +++ b/docs/security-hardening.md @@ -171,6 +171,8 @@ The following routes do not require a bearer token, by design: - `GET /api/runner?name=X` — runner detail (log tail is redacted) - `GET /api/analytics`, `/api/billing`, `/api/autoscale`, `/api/admission` - `GET /api/run-status` — one run's status, limited to repos this fleet serves +- `GET /api/job-class` — a job's admission lane and median run time, from this database +- `GET /api/load-by-busy` — host load by busy-runner count, from this database - `GET /api/actions` — action catalogue (knowing which buttons exist ≠ pressing them) - All static files under `/public/` diff --git a/examples/fleet.env.second-host b/examples/fleet.env.second-host index 36e9244..d5a6f92 100644 --- a/examples/fleet.env.second-host +++ b/examples/fleet.env.second-host @@ -44,8 +44,10 @@ FLEET_CEILING=6 FLEET_ADMIT_MODE=enforce FLEET_ADMIT_MAX_CONCURRENT=6 FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT=2 +FLEET_ADMIT_LANES=observe +FLEET_ADMIT_HEAVY_MAX_CONCURRENT=3 +FLEET_ADMIT_LIGHT_MAX_CONCURRENT=8 FLEET_ADMIT_SHORT_JOB_S=90 -FLEET_ADMIT_SHORT_MAX_CONCURRENT=4 FLEET_ADMIT_MAX_WAIT_S=600 FLEET_ADMIT_TIMEOUT_ACTION=hold FLEET_ADMIT_CANCEL_POLL_S=30 diff --git a/fleet.env.example b/fleet.env.example index e1c50e5..b3371cb 100644 --- a/fleet.env.example +++ b/fleet.env.example @@ -138,6 +138,39 @@ # FLEET_SIMULATOR_RUNNERS patterns disables it. #FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT=0 +# LANES. With one host-wide limit a 5-second job waits behind 12-minute Xcode +# builds. Lanes put each job in `light`, `heavy` or `simulator` (a heavy job on +# a FLEET_SIMULATOR_RUNNERS runner) and give each lane its own limit and its +# own FIFO line, under a total cap. See docs/admission-and-scaling.md#lanes. +# off — the default: one lane, exactly as before. +# observe — the single lane still decides; every admission log line also +# records what the lanes would have decided (lane_would_hold, +# lane_reason, lane_wait_s). Run this first. +# enforce — the lanes decide. +#FLEET_ADMIT_LANES=observe +# The total cap the lanes apply (default FLEET_ADMIT_MAX_CONCURRENT). Its own +# setting so observe can judge the lanes at the total they will enforce while +# the single lane keeps its own limit; enforce is then FLEET_ADMIT_LANES alone. +#FLEET_ADMIT_TOTAL_MAX_CONCURRENT=16 +# Heavy jobs at once (Simulator jobs count here too). +#FLEET_ADMIT_HEAVY_MAX_CONCURRENT=4 +# Light jobs at once. FLEET_ADMIT_SHORT_MAX_CONCURRENT is accepted as an alias. +#FLEET_ADMIT_LIGHT_MAX_CONCURRENT=12 +# Classification, first match wins: the Simulator runner rule, these patterns +# (space-separated `repo:job` shell globs; repo with or without its owner; job +# is GITHUB_JOB, the job's YAML id, `test` rather than `ios / test`), then the +# dashboard's history (/api/job-class: light when the job's recent median run +# is under FLEET_ADMIT_SHORT_JOB_S), else heavy. Quote the lists. +#FLEET_ADMIT_HEAVY_PATTERNS="*:e2e acme/app:test site:test" +#FLEET_ADMIT_LIGHT_PATTERNS="*:gate *:changes" +#FLEET_ADMIT_SHORT_JOB_S=90 +# How long the hook reuses the dashboard's answer for one repo:job. +#FLEET_ADMIT_CLASS_CACHE_S=300 +# Where it asks; defaults to /api/job-class beside FLEET_ADMIT_STATUS_URL, and +# an empty FLEET_ADMIT_STATUS_URL skips it too (every job is then heavy unless +# a pattern says otherwise). +#FLEET_ADMIT_CLASS_URL=http://127.0.0.1:7878/api/job-class + # How long a job may be held before the timeout action below. A held job is IN # PROGRESS as far as GitHub is concerned, so this counts against its own # `timeout-minutes`. @@ -416,6 +449,7 @@ # Disk cleanup (cleanup.sh --auto). Below the pressure line it also erases CI # simulators over the smaller size and idle runners' Playwright browsers. # FLEET_CLEANUP_PRESSURE_GB=60 # default: FLEET_ADMIT_MIN_FREE_DISK_GB + 20 +# FLEET_CLEANUP_RUNNER_CACHE_DAYS=7 # erase a runner's _work//DerivedData-*, spm-*, .build-cache-* unused this long # FLEET_CLEANUP_SIM_MAX_GB=3 # erase a runner's ci- simulator past this # FLEET_CLEANUP_SIM_PRESSURE_GB=1 # ...and past this under pressure # FLEET_CLEANUP_FULL_EVERY_S=86400 # full pass at least this often diff --git a/hooks/common.sh b/hooks/common.sh index 3d1031e..c0a92b9 100644 --- a/hooks/common.sh +++ b/hooks/common.sh @@ -121,6 +121,72 @@ ADMIT_MUTEX_TRIES="$(admit_int "${FLEET_ADMIT_MUTEX_TRIES:-}" 50)" # live after a reboot recycled the number. ADMIT_SLOT_TTL="$(admit_int "${FLEET_ADMIT_SLOT_TTL_S:-}" 21600)" +# LANES. One host-wide limit treats a 5-second grep like a 12-minute Xcode +# build: on ultra (30 cores) load rises about 8 per concurrent job, 74 % of jobs +# finish in under 90 s, and with one FIFO the short ones queued behind the long +# ones. Lanes classify each job as `light`, `heavy` or `simulator` (a heavy job +# on a runner FLEET_SIMULATOR_RUNNERS selects) and limit each lane on its own, +# under FLEET_ADMIT_MAX_CONCURRENT as the total cap, with the waiting line kept +# per lane: the oldest light job is never behind an older heavy one. +# +# off — no lane logic at all (the default): today's single lane. +# observe — the single lane still decides; every decision also records what +# the lanes would have decided, so the limits can be checked first. +# enforce — the lanes decide. +case "${FLEET_ADMIT_LANES:-off}" in + observe) ADMIT_LANES=observe ;; + enforce) ADMIT_LANES=enforce ;; + *) ADMIT_LANES=off ;; +esac +# The total cap the lanes apply, FLEET_ADMIT_MAX_CONCURRENT unless set. Its own +# setting so that in observe the lanes can be judged at the total they will +# enforce, while the single lane keeps deciding at its own limit; switching to +# enforce is then FLEET_ADMIT_LANES alone. +ADMIT_TOTAL_MAX="$(admit_int "${FLEET_ADMIT_TOTAL_MAX_CONCURRENT:-}" "$ADMIT_MAX")" +[ "$ADMIT_TOTAL_MAX" -lt 1 ] && ADMIT_TOTAL_MAX=1 +# A simulator job counts against the heavy limit too. +ADMIT_HEAVY_MAX="$(admit_int "${FLEET_ADMIT_HEAVY_MAX_CONCURRENT:-}" 4)" +[ "$ADMIT_HEAVY_MAX" -lt 1 ] && ADMIT_HEAVY_MAX=1 +# FLEET_ADMIT_SHORT_MAX_CONCURRENT is the name this setting had in fleet.env +# before anything read it; still honoured. +ADMIT_LIGHT_MAX="$(admit_int "${FLEET_ADMIT_LIGHT_MAX_CONCURRENT:-${FLEET_ADMIT_SHORT_MAX_CONCURRENT:-}}" 12)" +[ "$ADMIT_LIGHT_MAX" -lt 1 ] && ADMIT_LIGHT_MAX=1 +# A job whose recent median run is under this many seconds is light. +ADMIT_SHORT_JOB_S="$(admit_int "${FLEET_ADMIT_SHORT_JOB_S:-}" 90)" +# How long the dashboard's answer for one repo:job is reused. A burst of a +# dozen jobs from one push then costs the daemon one request per job name. +ADMIT_CLASS_TTL="$(admit_int "${FLEET_ADMIT_CLASS_CACHE_S:-}" 300)" +ADMIT_CLASSES="$ADMIT_STATE/job-class" +# Where the job's class is asked, next to /api/run-status by default: an empty +# FLEET_ADMIT_STATUS_URL (a host whose daemon is elsewhere) skips this too. +if [ -n "${FLEET_ADMIT_CLASS_URL+x}" ]; then + ADMIT_CLASS_URL="$FLEET_ADMIT_CLASS_URL" +elif [ -n "${FLEET_ADMIT_STATUS_URL+x}" ]; then + case "$FLEET_ADMIT_STATUS_URL" in + '') ADMIT_CLASS_URL="" ;; + */api/run-status) ADMIT_CLASS_URL="${FLEET_ADMIT_STATUS_URL%/api/run-status}/api/job-class" ;; + *) ADMIT_CLASS_URL="http://127.0.0.1:${FLEET_PORT:-7878}/api/job-class" ;; + esac +else + ADMIT_CLASS_URL="http://127.0.0.1:${FLEET_PORT:-7878}/api/job-class" +fi + +# The limit a log line reports: the total cap that is actually deciding. +if [ "$ADMIT_LANES" = enforce ]; then ADMIT_LOG_LIMIT="$ADMIT_TOTAL_MAX"; else ADMIT_LOG_LIMIT="$ADMIT_MAX"; fi + +# Set by admit_classify (job-started.sh) or read back from the slot +# (job-completed.sh). Empty while lanes are off. +ADMIT_LANE="" +ADMIT_LANE_SOURCE="" +# The lane policy's view of the current decision, for the log. VIEWED is 0 when +# it was not worked out (lanes off, or a poll that could not take the mutex). +ADMIT_HEAVY_BUSY=0 +ADMIT_LIGHT_BUSY=0 +ADMIT_LANE_BLOCKER="" +ADMIT_LANE_VIEWED=0 +# observe: seconds into a hold at which the lanes would first have admitted. +ADMIT_LANE_WAIT="" + # Resolved by admit_resolve_owner when a slot is claimed. Declared here so the # log line is well-formed for decisions taken before any claim, such as a hold. ADMIT_OWNER_PID="" @@ -186,7 +252,7 @@ admit_log() { # waited_s is time this job spent waiting for a slot; ran_s is how long a # released slot was occupied. Separate fields because they are separate # quantities — the release used to report its occupancy as a wait. - printf '{"ts":%s,"event":"%s","mode":"%s","runner":"%s","repo":"%s","run":"%s","job":"%s","waited_s":%s,"ran_s":%s,"busy":%s,"limit":%s,"owner_kind":"%s","owner_pid":"%s","reason":"%s"}\n' \ + printf '{"ts":%s,"event":"%s","mode":"%s","runner":"%s","repo":"%s","run":"%s","job":"%s","waited_s":%s,"ran_s":%s,"busy":%s,"limit":%s,"owner_kind":"%s","owner_pid":"%s","reason":"%s"%s}\n' \ "$(admit_now)" \ "$(admit_json_str "$event")" \ "$(admit_json_str "$ADMIT_MODE")" \ @@ -194,13 +260,35 @@ admit_log() { "$(admit_json_str "${GITHUB_REPOSITORY:-}")" \ "$(admit_json_str "${GITHUB_RUN_ID:-}")" \ "$(admit_json_str "${GITHUB_JOB:-}")" \ - "${waited:-0}" "${ran:-0}" "${busy:-0}" "$ADMIT_MAX" \ + "${waited:-0}" "${ran:-0}" "${busy:-0}" "$ADMIT_LOG_LIMIT" \ "$(admit_json_str "$ADMIT_OWNER_KIND")" \ "$(admit_json_str "$ADMIT_OWNER_PID")" \ "$(admit_json_str "$reason")" \ + "$(admit_lane_json)" \ >> "$ADMIT_LOG" 2>/dev/null || true } +# The lane fields of a log line, with their leading comma, or nothing while +# lanes are off: an off host writes exactly the line it always has. +# lane_reason is the lane policy's own reason text. In enforce it is the hold's +# reason; in observe it is what the lanes would have said (empty: admit now). +admit_lane_json() { + [ "$ADMIT_LANES" = off ] && return 0 + local would=null wait=null + if [ "$ADMIT_LANE_VIEWED" = 1 ]; then + if [ -n "$ADMIT_LANE_BLOCKER" ]; then would=true; else would=false; fi + fi + case "$ADMIT_LANE_WAIT" in '' | *[!0-9]*) ;; *) wait="$ADMIT_LANE_WAIT" ;; esac + printf ',"lanes":"%s","lane":"%s","lane_source":"%s","heavy_busy":%s,"light_busy":%s,"heavy_limit":%s,"light_limit":%s,"total_limit":%s,"lane_would_hold":%s,"lane_wait_s":%s,"lane_reason":"%s"' \ + "$ADMIT_LANES" \ + "$(admit_json_str "$ADMIT_LANE")" \ + "$(admit_json_str "$ADMIT_LANE_SOURCE")" \ + "${ADMIT_HEAVY_BUSY:-0}" "${ADMIT_LIGHT_BUSY:-0}" \ + "$ADMIT_HEAVY_MAX" "$ADMIT_LIGHT_MAX" "$ADMIT_TOTAL_MAX" \ + "$would" "$wait" \ + "$(admit_json_str "$ADMIT_LANE_BLOCKER")" +} + # mkdir is the portable atomic test-and-set. macOS ships no flock(1), and this # needs no dependency beyond coreutils behaviour that has been stable for # decades. @@ -327,10 +415,10 @@ admit_claim_slot() { local key="$1" mkdir -p "$ADMIT_SLOTS" 2>/dev/null || return 1 admit_resolve_owner - printf 'pid=%s\nowner=%s\nts=%s\nrunner=%s\nrepo=%s\nrun=%s\njob=%s\n' \ + printf 'pid=%s\nowner=%s\nts=%s\nrunner=%s\nrepo=%s\nrun=%s\njob=%s\nlane=%s\n' \ "$ADMIT_OWNER_PID" "$ADMIT_OWNER_KIND" "$(admit_now)" \ "${RUNNER_NAME:-}" \ - "${GITHUB_REPOSITORY:-}" "${GITHUB_RUN_ID:-}" "${GITHUB_JOB:-}" \ + "${GITHUB_REPOSITORY:-}" "${GITHUB_RUN_ID:-}" "${GITHUB_JOB:-}" "$ADMIT_LANE" \ > "$ADMIT_SLOTS/$key" 2>/dev/null || return 1 return 0 } @@ -349,10 +437,18 @@ admit_join_waiters() { [ -n "$ADMIT_WAITER" ] && [ -f "$ADMIT_WAITER" ] && return 0 mkdir -p "$ADMIT_WAITERS" 2>/dev/null || return 1 stamp="$(printf '%020d-%010d' "$(admit_now)" "$$")" - ADMIT_WAITER="$ADMIT_WAITERS/$stamp-$key" - printf 'pid=%s\nts=%s\nrunner=%s\nrepo=%s\nrun=%s\njob=%s\n' \ + # With lanes on, the lane's line is in the name (after the fixed-width + # stamp), so the lockless check can find the head of its own line from one + # directory listing. The stamp still comes first: sorting by name is still + # arrival order across every line. + if [ "$ADMIT_LANES" = off ]; then + ADMIT_WAITER="$ADMIT_WAITERS/$stamp-$key" + else + ADMIT_WAITER="$ADMIT_WAITERS/$stamp-$(admit_lane_line "$ADMIT_LANE")-$key" + fi + printf 'pid=%s\nts=%s\nrunner=%s\nrepo=%s\nrun=%s\njob=%s\nlane=%s\n' \ "$$" "$(admit_now)" "${RUNNER_NAME:-}" "${GITHUB_REPOSITORY:-}" \ - "${GITHUB_RUN_ID:-}" "${GITHUB_JOB:-}" \ + "${GITHUB_RUN_ID:-}" "${GITHUB_JOB:-}" "$ADMIT_LANE" \ > "$ADMIT_WAITER" 2>/dev/null || { ADMIT_WAITER=""; return 1; } } @@ -489,6 +585,233 @@ admit_waiting_behind_older() { return 0 } +# --------------------------------------------------------------------------- +# Lanes (FLEET_ADMIT_LANES). Every function here returns 0 unless it is a test +# used in a condition: the runner runs this hook under `bash -e`, and a failed +# command substitution there would exit the hook, whose EXIT trap then admits +# the job without a slot. +# --------------------------------------------------------------------------- + +# The waiting line a lane belongs to. A simulator job waits in the heavy line, +# because it takes a heavy slot as well as a Simulator one. +admit_lane_line() { + case "${1:-}" in + light) printf 'light' ;; + *) printf 'heavy' ;; + esac +} + +# Sets ADMIT_WAITER_LINE from a waiter file's name: <20-digit ts>-<10-digit +# pid>--. A name from before lanes (no line) reads as heavy, the +# safe side, and so does anything else unexpected. +ADMIT_WAITER_LINE="" +admit_waiter_line_of() { + local rest="${1:32}" + case "${rest%%-*}" in + light) ADMIT_WAITER_LINE=light ;; + *) ADMIT_WAITER_LINE=heavy ;; + esac + return 0 +} + +# True when this job matches one of a space- (or comma-) separated list of +# `repo:job` shell patterns. The job is GITHUB_JOB, the id (YAML key) of the +# job that runs, which inside a reusable workflow is that workflow's own key +# (`test`, not `ios / test`). The repo may be written with or without its +# owner: `acme/app:test`, `app:test`, `*:e2e` and `site:*` all work. +admit_job_matches() { + local list="${1:-}" pattern full short + local -a patterns + [ -n "$list" ] || return 1 + full="${GITHUB_REPOSITORY:-}:${GITHUB_JOB:-}" + short="${GITHUB_REPOSITORY##*/}:${GITHUB_JOB:-}" + list="${list//,/ }" + read -r -a patterns <<< "$list" + for pattern in "${patterns[@]}"; do + # shellcheck disable=SC2254 + case "$full" in $pattern) return 0 ;; esac + # shellcheck disable=SC2254 + case "$short" in $pattern) return 0 ;; esac + done + return 1 +} + +# The dashboard's class for this repo:job (light, heavy or unknown), from its +# own history of how long the job runs; cached for ADMIT_CLASS_TTL seconds in +# the state directory. Returns non-zero, printing nothing, when there is no +# answer: no URL, no curl, or the daemon is down or too old to know the route. +admit_class_from_daemon() { + local repo="${GITHUB_REPOSITORY:-}" job="${GITHUB_JOB:-}" key file now ts cls body tmp + [ -n "$repo" ] && [ -n "$job" ] || return 1 + [ -n "$ADMIT_CLASS_URL" ] && command -v curl >/dev/null 2>&1 || return 1 + key="$(printf '%s__%s' "$repo" "$job" | tr -c 'A-Za-z0-9._-' '_')" + file="$ADMIT_CLASSES/$key" + now="$(admit_now)" + if [ -f "$file" ]; then + ts="$(sed -n 's/^ts=//p' "$file" 2>/dev/null | head -1)" + cls="$(sed -n 's/^class=//p' "$file" 2>/dev/null | head -1)" + case "$ts" in + '' | *[!0-9]*) ;; + *) + if [ $((now - ts)) -lt "$ADMIT_CLASS_TTL" ] && [ -n "$cls" ]; then + printf '%s' "$cls" + return 0 + fi + ;; + esac + fi + body="$(curl -fsS --max-time 3 -G "$ADMIT_CLASS_URL" \ + --data-urlencode "repo=$repo" --data-urlencode "job=$job" \ + --data-urlencode "short_s=$ADMIT_SHORT_JOB_S" 2>/dev/null)" || return 1 + cls="$(printf '%s' "$body" | sed -n 's/.*"class":"\([a-z]*\)".*/\1/p' | head -1)" + case "$cls" in + light | heavy | unknown) ;; + *) return 1 ;; + esac + # Written aside and renamed, so a concurrent reader never sees half a file. + mkdir -p "$ADMIT_CLASSES" 2>/dev/null || true + tmp="$file.$$" + if printf 'class=%s\nts=%s\n' "$cls" "$now" > "$tmp" 2>/dev/null; then + mv -f "$tmp" "$file" 2>/dev/null || rm -f "$tmp" 2>/dev/null + fi + printf '%s' "$cls" + return 0 +} + +# Sets ADMIT_LANE and ADMIT_LANE_SOURCE. In order: the Simulator runner rule, +# FLEET_ADMIT_HEAVY_PATTERNS, FLEET_ADMIT_LIGHT_PATTERNS, the dashboard's +# history; anything it cannot place is heavy, because a heavy job let into the +# light lane is what overloads the host, and a light job in the heavy lane only +# waits a little longer. +admit_classify() { + local cls="" + ADMIT_LANE="" + ADMIT_LANE_SOURCE="" + [ "$ADMIT_LANES" = off ] && return 0 + if admit_is_simulator_job; then + ADMIT_LANE=simulator + ADMIT_LANE_SOURCE=runner + return 0 + fi + if admit_job_matches "${FLEET_ADMIT_HEAVY_PATTERNS:-}"; then + ADMIT_LANE=heavy + ADMIT_LANE_SOURCE=pattern + return 0 + fi + if admit_job_matches "${FLEET_ADMIT_LIGHT_PATTERNS:-}"; then + ADMIT_LANE=light + ADMIT_LANE_SOURCE=pattern + return 0 + fi + cls="$(admit_class_from_daemon)" || cls="" + case "$cls" in + light) ADMIT_LANE=light ; ADMIT_LANE_SOURCE=history ;; + heavy) ADMIT_LANE=heavy ; ADMIT_LANE_SOURCE=history ;; + unknown) ADMIT_LANE=heavy ; ADMIT_LANE_SOURCE=unknown ;; + *) ADMIT_LANE=heavy ; ADMIT_LANE_SOURCE=fallback ;; + esac + return 0 +} + +# Sets ADMIT_HEAVY_BUSY and ADMIT_LIGHT_BUSY from the slot files, given the +# total ($1, from admit_live_slots, which has just reaped the dead ones). A slot +# written without a lane (lanes off, or a hook from before lanes) is heavy. +admit_lane_counts() { + local total="${1:-0}" light + light="$(grep -l '^lane=light$' "$ADMIT_SLOTS"/* 2>/dev/null | wc -l | tr -d ' ')" + case "$light" in '' | *[!0-9]*) light=0 ;; esac + case "$total" in '' | *[!0-9]*) total=0 ;; esac + ADMIT_LIGHT_BUSY="$light" + ADMIT_HEAVY_BUSY=$((total - light)) + [ "$ADMIT_HEAVY_BUSY" -lt 0 ] && ADMIT_HEAVY_BUSY=0 + return 0 +} + +# Why the lanes would hold this job on capacity alone, or empty. The total cap +# and the Simulator limit keep the reasons they have always had, word for word: +# the lake classifies holds by this text. +admit_lane_capacity_blocker() { + local busy="${1:-0}" simulator_busy="${2:-0}" + if [ "$busy" -ge "$ADMIT_TOTAL_MAX" ]; then + printf '%s job(s) already running, at the limit of %s' "$busy" "$ADMIT_TOTAL_MAX" + return 0 + fi + if [ "$ADMIT_LANE" = simulator ] && admit_simulator_blocked "$simulator_busy"; then + printf '%s Simulator job(s) already running, at the limit of %s' \ + "$simulator_busy" "$ADMIT_SIMULATOR_MAX" + return 0 + fi + if [ "$ADMIT_LANE" = light ]; then + if [ "$ADMIT_LIGHT_BUSY" -ge "$ADMIT_LIGHT_MAX" ]; then + printf '%s light job(s) already running, at the light limit of %s' \ + "$ADMIT_LIGHT_BUSY" "$ADMIT_LIGHT_MAX" + fi + return 0 + fi + if [ "$ADMIT_HEAVY_BUSY" -ge "$ADMIT_HEAVY_MAX" ]; then + printf '%s heavy job(s) already running, at the heavy limit of %s' \ + "$ADMIT_HEAVY_BUSY" "$ADMIT_HEAVY_MAX" + fi + return 0 +} + +# The hold reason for a job behind an older one in its own line. Starts like +# the single-lane reason so the lake files both as queue order. +admit_lane_queue_reason() { + printf 'an older job is waiting for the next %s slot' "$(admit_lane_line "$ADMIT_LANE")" +} + +# The per-lane FIFO, under the mutex: true when this waiter is the oldest one +# in its own line that could take a slot. As in the single line, a Simulator +# job held by the Simulator limit is skipped rather than allowed to block the +# heavy jobs behind it. $2=noreap skips the reaping a caller has just done. +admit_waiter_is_first_in_lane() { + local simulator_busy="${1:-0}" f name first="" mine runner + [ -n "$ADMIT_WAITER" ] || return 1 + [ "${2:-}" = noreap ] || admit_reap_waiters + mine="$(admit_lane_line "$ADMIT_LANE")" + for f in "$ADMIT_WAITERS"/*; do + [ -f "$f" ] || continue + name="${f##*/}" + admit_waiter_line_of "$name" + [ "$ADMIT_WAITER_LINE" = "$mine" ] || continue + if [ "$ADMIT_SIMULATOR_MAX" -gt 0 ] && [ "$simulator_busy" -ge "$ADMIT_SIMULATOR_MAX" ]; then + runner="$(sed -n 's/^runner=//p' "$f" 2>/dev/null | head -1)" + admit_runner_matches "$runner" && continue + fi + if [ -z "$first" ] || [ "$name" \< "${first##*/}" ]; then + first="$f" + fi + done + [ "$first" = "$ADMIT_WAITER" ] +} + +# The lockless check of admit_waiting_behind_older, per line: true when a live +# waiter ahead of this one is in this job's own line. Anything it cannot vouch +# for falls through to the locked check, as there. +admit_waiting_behind_older_in_lane() { + local name first="" mine pid runner + [ -n "$ADMIT_WAITER" ] && [ -f "$ADMIT_WAITER" ] || return 1 + mine="$(admit_lane_line "$ADMIT_LANE")" + while IFS= read -r name; do + [ -n "$name" ] || continue + admit_waiter_line_of "$name" + if [ "$ADMIT_WAITER_LINE" = "$mine" ]; then + first="$name" + break + fi + done < <(LC_ALL=C ls -1 "$ADMIT_WAITERS" 2>/dev/null) + [ -n "$first" ] || return 1 + [ "$ADMIT_WAITERS/$first" = "$ADMIT_WAITER" ] && return 1 + pid="$(sed -n 's/^pid=//p' "$ADMIT_WAITERS/$first" 2>/dev/null | head -1)" + [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null || return 1 + if [ "$ADMIT_SIMULATOR_MAX" -gt 0 ]; then + runner="$(sed -n 's/^runner=//p' "$ADMIT_WAITERS/$first" 2>/dev/null | head -1)" + admit_runner_matches "$runner" && return 1 + fi + return 0 +} + # Returns the GitHub Actions run status string, or empty when unknown. # # The dashboard daemon is asked first. Runner LaunchAgents set SessionCreate, diff --git a/hooks/job-completed.sh b/hooks/job-completed.sh index 0ace51b..c1a2881 100755 --- a/hooks/job-completed.sh +++ b/hooks/job-completed.sh @@ -71,6 +71,11 @@ if [ -f "$SLOT" ]; then '' | *[!0-9]*) ;; *) HELD=$(($(admit_now) - TS)) ;; esac + # The lane this job ran in, for the release line: the dashboard's job-class + # history and the per-lane running counts are read from these. + if [ "$ADMIT_LANES" != off ]; then + ADMIT_LANE="$(sed -n 's/^lane=//p' "$SLOT" 2>/dev/null | head -1)" || ADMIT_LANE="" + fi fi admit_free_slot "$KEY" @@ -78,6 +83,7 @@ admit_free_slot "$KEY" BUSY=0 if admit_lock; then BUSY="$(admit_live_slots)" + [ "$ADMIT_LANES" = off ] || admit_lane_counts "$BUSY" admit_unlock fi diff --git a/hooks/job-started.sh b/hooks/job-started.sh index 7c16d72..a70c3d7 100755 --- a/hooks/job-started.sh +++ b/hooks/job-started.sh @@ -57,10 +57,10 @@ ROOT="${FLEET_ROOT:-$(cd "$HERE/.." && pwd)}" KEY="$(admit_key)" -# Why the job would be held right now, or empty if there is room. Callers must +# Why the single lane would hold the job on capacity, or empty. Callers must # hold the mutex, because the count it reads is only meaningful under it. -admit_blocker() { - local busy="$1" simulator_busy="$2" disk +admit_capacity_blocker() { + local busy="$1" simulator_busy="$2" if [ "$busy" -ge "$ADMIT_MAX" ]; then printf '%s job(s) already running, at the limit of %s' "$busy" "$ADMIT_MAX" return 0 @@ -70,6 +70,12 @@ admit_blocker() { "$simulator_busy" "$ADMIT_SIMULATOR_MAX" return 0 fi + return 0 +} + +# The disk floors, shared by both policies. +admit_disk_blocker() { + local disk # `|| disk=""`: production runs this under bash -e, where a failed read # would exit the hook, and its EXIT trap admits the job without a slot. disk="$(admit_free_disk_gb)" || disk="" @@ -85,6 +91,68 @@ admit_blocker() { return 0 } +# Works out both policies' answer for this moment, under the mutex. Sets +# ADMIT_GATE_BLOCKER (what decides: the lanes in lanes-enforce, the single lane +# otherwise; empty = room) and, with lanes on, ADMIT_LANE_BLOCKER (the lanes' +# answer, logged beside the decision). The per-lane and single-lane queue +# order are the caller's, because only a waiter has a place in a line. +ADMIT_GATE_BLOCKER="" +admit_decide() { + local busy="$1" simulator_busy="$2" legacy="" lane="" disk="" + [ "$ADMIT_LANES" = enforce ] || legacy="$(admit_capacity_blocker "$busy" "$simulator_busy")" + if [ "$ADMIT_LANES" != off ]; then + admit_lane_counts "$busy" + lane="$(admit_lane_capacity_blocker "$busy" "$simulator_busy")" + fi + if { [ "$ADMIT_LANES" != enforce ] && [ -z "$legacy" ]; } \ + || { [ "$ADMIT_LANES" != off ] && [ -z "$lane" ]; }; then + disk="$(admit_disk_blocker)" + fi + [ -n "$legacy" ] || legacy="$disk" + [ -n "$lane" ] || lane="$disk" + if [ "$ADMIT_LANES" = enforce ]; then + ADMIT_GATE_BLOCKER="$lane" + else + ADMIT_GATE_BLOCKER="$legacy" + fi + if [ "$ADMIT_LANES" != off ]; then + ADMIT_LANE_BLOCKER="$lane" + ADMIT_LANE_VIEWED=1 + fi + return 0 +} + +# observe-lanes only: the lanes' answer for a waiter the single line keeps +# behind an older job, read without the mutex (slots are counted, not reaped, +# and the disk is re-read at most every 30 s). It is a log field, not a +# decision, and it stops being worked out once the lanes would have admitted. +ADMIT_DISK_AT=-1000 +ADMIT_DISK_REASON="" +admit_lane_view_lockless() { + local total simulator_busy now + total="$(ls -1 "$ADMIT_SLOTS" 2>/dev/null | wc -l | tr -d ' ')" + simulator_busy="$(admit_live_simulator_slots)" + admit_lane_counts "$total" + ADMIT_LANE_BLOCKER="$(admit_lane_capacity_blocker "$total" "$simulator_busy")" + if [ -z "$ADMIT_LANE_BLOCKER" ]; then + now="$(admit_now)" + if [ $((now - ADMIT_DISK_AT)) -ge 30 ]; then + ADMIT_DISK_REASON="$(admit_disk_blocker)" + ADMIT_DISK_AT="$now" + fi + ADMIT_LANE_BLOCKER="$ADMIT_DISK_REASON" + fi + if [ -z "$ADMIT_LANE_BLOCKER" ] && admit_waiting_behind_older_in_lane; then + ADMIT_LANE_BLOCKER="$(admit_lane_queue_reason)" + fi + ADMIT_LANE_VIEWED=1 + return 0 +} + +# Network, so outside the mutex and once per job: the lane cannot change while +# the job waits. +admit_classify + # observe: take the slot so the count stays honest, never delay anything, and # record whether enforcing would have held this job. Mirrors autoscaleDryRun — # the log has to look right before the mechanism is allowed to cost a build. @@ -95,7 +163,8 @@ if [ "$ADMIT_MODE" = "observe" ]; then if admit_lock; then BUSY="$(admit_live_slots)" SIMULATOR_BUSY="$(admit_live_simulator_slots)" - BLOCKER="$(admit_blocker "$BUSY" "$SIMULATOR_BUSY")" + admit_decide "$BUSY" "$SIMULATOR_BUSY" + BLOCKER="$ADMIT_GATE_BLOCKER" admit_claim_slot "$KEY" admit_unlock else @@ -136,18 +205,54 @@ while :; do echo "fleet: job cancelled while waiting — releasing this runner" exit 0 fi - if admit_waiting_behind_older; then + # Only the oldest waiter in a line can be admitted, so a later one decides + # nothing under the mutex (see admit_waiting_behind_older). With lanes + # enforced the line is the job's own lane, not the host. + BEHIND=0 + if [ "$ADMIT_LANES" = enforce ]; then + if admit_waiting_behind_older_in_lane; then + BEHIND=1 + BLOCKER="$(admit_lane_queue_reason)" + ADMIT_LANE_BLOCKER="$BLOCKER" + ADMIT_LANE_VIEWED=1 + fi + elif admit_waiting_behind_older; then + BEHIND=1 BLOCKER="an older job is waiting for the next host slot" + if [ "$ADMIT_LANES" = observe ] && [ -z "$ADMIT_LANE_WAIT" ]; then + admit_lane_view_lockless + [ -n "$ADMIT_LANE_BLOCKER" ] || ADMIT_LANE_WAIT="$WAITED" + fi + fi + if [ "$BEHIND" -eq 1 ]; then + : elif admit_lock; then admit_join_waiters "$KEY" || true BUSY="$(admit_live_slots)" SIMULATOR_BUSY="$(admit_live_simulator_slots)" admit_simulator_blocked "$SIMULATOR_BUSY" && STRICT_LIMIT=1 - BLOCKER="$(admit_blocker "$BUSY" "$SIMULATOR_BUSY")" - if [ -z "$BLOCKER" ] \ + admit_decide "$BUSY" "$SIMULATOR_BUSY" + BLOCKER="$ADMIT_GATE_BLOCKER" + if [ "$ADMIT_LANES" = enforce ]; then + if [ -z "$BLOCKER" ] \ + && ! admit_waiter_is_first_in_lane "$SIMULATOR_BUSY"; then + BLOCKER="$(admit_lane_queue_reason)" + ADMIT_LANE_BLOCKER="$BLOCKER" + fi + elif [ -z "$BLOCKER" ] \ && ! admit_waiter_is_first_eligible "$SIMULATOR_BUSY"; then BLOCKER="an older job is waiting for the next host slot" fi + # observe-lanes: would the job's own line have let it go? And from when. + if [ "$ADMIT_LANES" = observe ]; then + # noreap: the single line's own check has just reaped, or had no need + # to (the host was full, which holds the lanes too). + if [ -z "$ADMIT_LANE_BLOCKER" ] \ + && ! admit_waiter_is_first_in_lane "$SIMULATOR_BUSY" noreap; then + ADMIT_LANE_BLOCKER="$(admit_lane_queue_reason)" + fi + [ -n "$ADMIT_LANE_BLOCKER" ] || [ -n "$ADMIT_LANE_WAIT" ] || ADMIT_LANE_WAIT="$WAITED" + fi if [ -z "$BLOCKER" ]; then admit_claim_slot "$KEY" admit_leave_waiters diff --git a/hooks/tests/run.sh b/hooks/tests/run.sh index 6a2f2a5..05adf68 100755 --- a/hooks/tests/run.sh +++ b/hooks/tests/run.sh @@ -742,6 +742,288 @@ wait_for "1" "events admitted" 10 ok "it is admitted once the head is gone and a slot frees" "$(events admitted)" "1" end_jobs +# --------------------------------------------------------------------------- +# Lanes (FLEET_ADMIT_LANES). Runners named h-* run heavy jobs and l-* light +# ones, by pattern, unless a case says otherwise. +# --------------------------------------------------------------------------- +lanes_setup() { + # $1 = FLEET_ADMIT_LANES, $2 = total (FLEET_ADMIT_MAX_CONCURRENT), + # $3 = heavy limit, $4 = light limit + setup enforce "$2" 30 1 + { + echo "FLEET_ADMIT_LANES=$1" + echo "FLEET_ADMIT_HEAVY_MAX_CONCURRENT=$3" + echo "FLEET_ADMIT_LIGHT_MAX_CONCURRENT=$4" + echo 'FLEET_ADMIT_HEAVY_PATTERNS="acme/h-*:*"' + echo 'FLEET_ADMIT_LIGHT_PATTERNS="l-*:build"' + } >> "$ROOT/fleet.env" +} +# The newest log line for a runner and event. +row() { grep "\"event\":\"$2\"" "$LOG" | grep "\"runner\":\"$1\"" | tail -1; } +# Matching lines in a file that may not exist (grep -c then prints nothing). +count_in() { + local n + n="$(grep -c "$1" "$2" 2>/dev/null)" + printf '%s' "${n:-0}" +} +# A field of one log line: a string's text, or a bare number/true/false/null. +field() { + case "$1" in + *"\"$2\":\""*) printf '%s' "$1" | sed -n "s/.*\"$2\":\"\([^\"]*\)\".*/\1/p" ;; + *) printf '%s' "$1" | sed -n "s/.*\"$2\":\([^,}]*\).*/\1/p" ;; + esac +} + +# Classifies the job of runner $1 with the case's fleet.env plus extra +# assignments in $2, and prints lane/source. +classify() { + FLEET_ROOT="$ROOT" RUNNER_NAME="$1" GITHUB_REPOSITORY="acme/$1" GITHUB_JOB="${3:-build}" \ + PATH="$BIN:$PATH" FAKE_CURL_LOG="$ROOT/curl-calls" bash -c ' + ROOT="$FLEET_ROOT"; '"${2:-:}"' + . "'"$HOOKS"'/common.sh" + admit_classify; echo "$ADMIT_LANE/$ADMIT_LANE_SOURCE"; trap - EXIT' +} + +echo "== lanes off: no lane logic, the log line is unchanged ==" +setup enforce 2 4 1 +cat > "$BIN/curl" <> "$ROOT/curl-calls" +exit 7 +EOF +chmod +x "$BIN/curl" +echo 'FLEET_ADMIT_HEAVY_PATTERNS="*:*"' >> "$ROOT/fleet.env" +start alpha +ok "admitted" "$(events admitted)" "1" +ok "no lane fields" "$(grep -c '"lane"' "$LOG")" "0" +ok "nobody asked the daemon for a class" "$(count_in job-class "$ROOT/curl-calls")" "0" +ok "nothing is classified" "$(classify alpha)" "/" + +echo "== classification: Simulator runner, patterns, history, fallback ==" +setup enforce 4 4 1 +{ + echo 'FLEET_ADMIT_LANES=enforce' + echo 'FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT=1' + echo 'FLEET_SIMULATOR_RUNNERS=*-ios' + echo 'FLEET_ADMIT_HEAVY_PATTERNS="acme/h-*:* *:e2e"' + echo 'FLEET_ADMIT_LIGHT_PATTERNS="l-one:build,acme/h-x:lint"' + echo 'FLEET_ADMIT_CLASS_URL=http://fleet.test/api/job-class' +} >> "$ROOT/fleet.env" +# The daemon: words is light, test is heavy, new has too few runs to say. +cat > "$BIN/curl" <<'EOF' +#!/usr/bin/env bash +echo "$*" >> "$FAKE_CURL_LOG" +case "$*" in + *fleet.test/api/job-class*job=words*) printf '{"class":"light","p50Ms":12000,"n":40}' ;; + *fleet.test/api/job-class*job=test*) printf '{"class":"heavy","p50Ms":533000,"n":59}' ;; + *fleet.test/api/job-class*job=new*) printf '{"class":"unknown","p50Ms":null,"n":2}' ;; + *) exit 7 ;; +esac +EOF +chmod +x "$BIN/curl" +ok "a Simulator runner is the simulator lane" "$(classify app-ios)" "simulator/runner" +ok "a heavy pattern with the owner" "$(classify h-api)" "heavy/pattern" +ok "a heavy pattern on the job alone" "$(classify web '' e2e)" "heavy/pattern" +ok "a light pattern without the owner" "$(classify l-one)" "light/pattern" +ok "heavy patterns win over light ones" "$(classify h-x '' lint)" "heavy/pattern" +ok "the daemon's history says light" "$(classify docs-site '' words)" "light/history" +ok "the daemon's history says heavy" "$(classify mobile '' test)" "heavy/history" +ok "too few runs to say is heavy" "$(classify fresh '' new)" "heavy/unknown" +ok "an unreachable daemon is heavy" "$(classify other '' gone)" "heavy/fallback" +: > "$ROOT/curl-calls" +ok "the answer is cached" "$(classify docs-site '' words)" "light/history" +ok "...so the daemon was not asked again" "$(grep -c 'job=words' "$ROOT/curl-calls")" "0" + +echo "== an empty FLEET_ADMIT_STATUS_URL means no daemon to ask for a class ==" +setup enforce 4 4 1 +echo 'FLEET_ADMIT_LANES=enforce' >> "$ROOT/fleet.env" +cat > "$BIN/curl" <<'EOF' +#!/usr/bin/env bash +echo "$*" >> "$FAKE_CURL_LOG" +printf '{"class":"light","p50Ms":1000,"n":40}' +EOF +chmod +x "$BIN/curl" +ok "heavy, as a fallback" "$(classify docs-site '' words)" "heavy/fallback" +ok "curl was not called" "$(count_in job-class "$ROOT/curl-calls")" "0" + +echo "== lanes enforce: the heavy limit holds a heavy job and names the lane ==" +lanes_setup enforce 10 2 5 +start_bg h-1 +start_bg h-2 +wait_for "2" "events admitted" 10 +start_bg h-3 +wait_for "1" "events held" 10 +ok "third heavy job held" "$(events held)" "1" +ok "the reason names the heavy lane" \ + "$(field "$(row h-3 held)" reason)" "2 heavy job(s) already running, at the heavy limit of 2" +ok "the held row carries its lane" "$(field "$(row h-3 held)" lane)" "heavy" +ok "...and the lane counts" "$(field "$(row h-3 held)" heavy_busy)" "2" +ok "the decided limit is the total" "$(field "$(row h-3 held)" limit)" "10" + +echo "== lanes enforce: a light job passes an older heavy waiter ==" +BEFORE=$(date +%s) +start l-1 +AFTER=$(date +%s) +ok "light job admitted" "$(field "$(row l-1 admitted)" lane)" "light" +ok "...at once" "$([ $((AFTER - BEFORE)) -le 3 ] && echo prompt || echo slow)" "prompt" +ok "the heavy waiter still waits" "$(events admitted)" "3" +complete h-1 +wait_for "4" "events admitted" 15 +ok "the heavy waiter took the freed heavy slot" "$(field "$(row h-3 admitted)" lane)" "heavy" +ok "a release carries the lane it freed" "$(field "$(row h-1 released)" lane)" "heavy" +end_jobs + +echo "== lanes enforce: the light limit, and FIFO within a lane ==" +lanes_setup enforce 10 2 1 +start_bg l-1 +wait_for "1" "events admitted" 10 +start_bg l-2 +sleep 1 +start_bg l-3 +wait_for "2" "events held" 10 +ok "light limit reason" "$(field "$(row l-2 held)" reason)" "1 light job(s) already running, at the light limit of 1" +start h-1 +ok "a heavy job is not held by the light line" "$(field "$(row h-1 admitted)" lane)" "heavy" +complete l-1 +wait_for "3" "events admitted" 15 +ok "the oldest light waiter went first" \ + "$(grep '"event":"admitted"' "$LOG" | grep '"lane":"light"' | tail -1 | sed -n 's/.*"runner":"\([^"]*\)".*/\1/p')" "l-2" +end_jobs + +echo "== lanes enforce: the total cap still applies ==" +lanes_setup enforce 2 5 5 +start_bg h-1 +start_bg l-1 +wait_for "2" "events admitted" 10 +start_bg l-2 +wait_for "1" "events held" 10 +ok "held at the total, in the old words" "$(field "$(row l-2 held)" reason)" "2 job(s) already running, at the limit of 2" +end_jobs + +echo "== lanes enforce: a Simulator job counts as heavy ==" +lanes_setup enforce 10 1 5 +{ + echo 'FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT=2' + echo 'FLEET_SIMULATOR_RUNNERS=*-ios' +} >> "$ROOT/fleet.env" +start_bg a-ios +wait_for "1" "events admitted" 10 +ok "the Simulator job's lane" "$(field "$(row a-ios admitted)" lane)" "simulator" +start_bg h-1 +wait_for "1" "events held" 10 +ok "a heavy job waits for the Simulator job's heavy slot" \ + "$(field "$(row h-1 held)" reason)" "1 heavy job(s) already running, at the heavy limit of 1" +end_jobs + +echo "== lanes enforce: unknown jobs fall back to heavy ==" +lanes_setup enforce 10 1 5 +echo 'FLEET_ADMIT_HEAVY_PATTERNS=' >> "$ROOT/fleet.env" +start_bg x-1 +wait_for "1" "events admitted" 10 +start_bg x-2 +wait_for "1" "events held" 10 +ok "an unclassified job is heavy" "$(field "$(row x-2 held)" lane)" "heavy" +ok "...and says why" "$(field "$(row x-2 held)" lane_source)" "fallback" +end_jobs + +echo "== legacy alias: FLEET_ADMIT_SHORT_MAX_CONCURRENT is the light limit ==" +setup enforce 10 30 1 +{ + echo 'FLEET_ADMIT_LANES=enforce' + echo 'FLEET_ADMIT_SHORT_MAX_CONCURRENT=1' + echo 'FLEET_ADMIT_LIGHT_PATTERNS="*:*"' +} >> "$ROOT/fleet.env" +start_bg l-1 +wait_for "1" "events admitted" 10 +start_bg l-2 +wait_for "1" "events held" 10 +ok "short max reads as the light limit" "$(field "$(row l-2 held)" light_limit)" "1" +ok "...and holds" "$(field "$(row l-2 held)" reason)" "1 light job(s) already running, at the light limit of 1" +echo 'FLEET_ADMIT_LIGHT_MAX_CONCURRENT=3' >> "$ROOT/fleet.env" +start l-3 +ok "the new name wins over the alias" "$(field "$(row l-3 admitted)" light_limit)" "3" +end_jobs + +echo "== lanes observe: the single lane decides, the lanes are recorded ==" +# Single lane: 1 at a time. Lanes: 1 heavy, 3 light, 5 in all. +setup enforce 1 30 1 +{ + echo 'FLEET_ADMIT_LANES=observe' + echo 'FLEET_ADMIT_TOTAL_MAX_CONCURRENT=5' + echo 'FLEET_ADMIT_HEAVY_MAX_CONCURRENT=1' + echo 'FLEET_ADMIT_LIGHT_MAX_CONCURRENT=3' + echo 'FLEET_ADMIT_HEAVY_PATTERNS="h-*:*"' + echo 'FLEET_ADMIT_LIGHT_PATTERNS="l-*:*"' +} >> "$ROOT/fleet.env" +start_bg h-1 +wait_for "1" "events admitted" 10 +ok "the lanes would have admitted it too" "$(field "$(row h-1 admitted)" lane_would_hold)" "false" +start_bg l-1 +wait_for "1" "events held" 10 +ok "the single lane held the light job" "$(field "$(row l-1 held)" reason)" "1 job(s) already running, at the limit of 1" +ok "...where the lanes would not have" "$(field "$(row l-1 held)" lane_would_hold)" "false" +ok "...from the start of the wait" "$(field "$(row l-1 held)" lane_wait_s)" "0" +ok "the row says which policy decided" "$(field "$(row l-1 held)" lanes)" "observe" +ok "the single lane's limit is the one reported" "$(field "$(row l-1 held)" limit)" "1" +start_bg h-2 +wait_for "2" "events held" 10 +ok "the lanes would have held a second heavy job" "$(field "$(row h-2 held)" lane_would_hold)" "true" +ok "...for the heavy limit" "$(field "$(row h-2 held)" lane_reason)" "1 heavy job(s) already running, at the heavy limit of 1" +complete h-1 +wait_for "2" "events admitted" 15 +ok "the single lane admitted its oldest waiter" \ + "$(grep '"event":"admitted"' "$LOG" | tail -1 | sed -n 's/.*"runner":"\([^"]*\)".*/\1/p')" "l-1" +ok "it waited under the single lane" \ + "$([ "$(field "$(row l-1 admitted)" waited_s)" -ge 1 ] && echo waited || echo instant)" "waited" +ok "the lanes would have let it in at once" "$(field "$(row l-1 admitted)" lane_wait_s)" "0" +end_jobs + +echo "== lanes observe with admission observe: would-hold carries the lane view ==" +setup observe 5 +{ + echo 'FLEET_ADMIT_LANES=enforce' + echo 'FLEET_ADMIT_HEAVY_MAX_CONCURRENT=1' + echo 'FLEET_ADMIT_HEAVY_PATTERNS="*:*"' +} >> "$ROOT/fleet.env" +start alpha; start beta +ok "the lanes' heavy limit is what would have held" \ + "$(field "$(row beta would-hold)" reason)" "1 heavy job(s) already running, at the heavy limit of 1" + +echo "== the lockless check, per lane ==" +setup enforce 1 30 1 +echo 'FLEET_ADMIT_LANES=enforce' >> "$ROOT/fleet.env" +W="$ROOT/.admission/waiters" +mkdir -p "$W" +behind_lane() { + # $1 = this waiter's file name, $2 = its lane + FLEET_ROOT="$ROOT" bash -c ' + ROOT="$FLEET_ROOT"; set -a; . "$ROOT/fleet.env"; set +a + . "'"$HOOKS"'/common.sh" + ADMIT_LANE='"$2"' + ADMIT_WAITER="$ADMIT_WAITERS/'"$1"'" + admit_waiting_behind_older_in_lane && echo behind || echo decide + trap - EXIT' +} +sleep 300 >/dev/null 2>&1 & +HEAD_PID=$! +printf 'pid=%s\nrunner=h-1\nlane=heavy\n' "$HEAD_PID" > "$W/00000000000000000001-0000000001-heavy-h-1" +printf 'pid=%s\nrunner=l-1\nlane=light\n' "$$" > "$W/00000000000000000002-0000000002-light-l-1" +printf 'pid=%s\nrunner=h-2\nlane=heavy\n' "$$" > "$W/00000000000000000003-0000000003-heavy-h-2" +printf 'pid=%s\nrunner=l-2\nlane=light\n' "$$" > "$W/00000000000000000004-0000000004-light-l-2" +ok "a light waiter is not behind an older heavy one" \ + "$(behind_lane 00000000000000000002-0000000002-light-l-1 light)" "decide" +ok "a heavy waiter is behind the older heavy one" \ + "$(behind_lane 00000000000000000003-0000000003-heavy-h-2 heavy)" "behind" +ok "a light waiter is behind the older light one" \ + "$(behind_lane 00000000000000000004-0000000004-light-l-2 light)" "behind" +printf 'pid=%s\nrunner=old\n' "$HEAD_PID" > "$W/00000000000000000000-0000000000-old-runner" +ok "a waiter from before lanes counts as heavy" \ + "$(behind_lane 00000000000000000002-0000000002-light-l-1 light)" "decide" +ok "...and is ahead of the heavy ones" \ + "$(behind_lane 00000000000000000001-0000000001-heavy-h-1 heavy)" "behind" +kill "$HEAD_PID" 2>/dev/null +wait "$HEAD_PID" 2>/dev/null + echo echo "passed $PASS, failed $FAIL" [ "$FAIL" -eq 0 ] diff --git a/scripts/ephemeral-runner.sh b/scripts/ephemeral-runner.sh index 4e55fc4..de65b64 100755 --- a/scripts/ephemeral-runner.sh +++ b/scripts/ephemeral-runner.sh @@ -171,6 +171,10 @@ LABELS="${LABEL:+--labels $LABEL}" --name "$RUNNER_NAME" $LABELS \ --work _work --unattended --ephemeral >/dev/null +# Spotlight stays out of the build tree, as register.sh does for every runner. +mkdir -p _work +touch _work/.metadata_never_index + echo "==> waiting for one job (Ctrl-C to give up and clean up)" # Foreground, no service install. run.sh exits after the single job completes, # and the trap then removes the directory. Installing a LaunchAgent would outlive diff --git a/scripts/test-cleanup.sh b/scripts/test-cleanup.sh index c3c6669..f78b344 100755 --- a/scripts/test-cleanup.sh +++ b/scripts/test-cleanup.sh @@ -281,6 +281,44 @@ ok "kept a busy runner's self-update" "$(gone "$T/web/_work/_update")" "kept" ok "kept a self-update under six hours old" "$(gone "$T/android/_work/_update")" "kept" { kill "$EMU"; wait "$EMU"; } 2>/dev/null +echo "== warm build caches: a week unused goes, all of an idle runner's under pressure ==" +setup 30 +echo 120 > "$T/free" +echo "RL-web" > "$T/busy" +for r in ios web android; do + mkdir -p "$T/$r/_work/$r/DerivedData-App/Build" "$T/$r/_work/$r/spm-App" \ + "$T/$r/_work/$r/.build-cache-x" "$T/$r/_work/$r/$r/.git" +done +# ios: everything stale. android: DerivedData used yesterday. web: building. +for c in "$T"/*/_work/*/DerivedData-App "$T"/*/_work/*/spm-App "$T"/*/_work/*/.build-cache-x; do + touch -t 202601010000 "$c/state" + touch -t 202601010000 "$c" +done +touch -t 202601010000 "$T"/*/_work/*/DerivedData-App/Build +touch "$T/android/_work/android/DerivedData-App/Build" +# A checkout named like a cache, and a cache under the runner's own _temp. +mkdir -p "$T/ios/_work/spm-tools/spm-tools/.git" "$T/ios/_work/_temp/DerivedData" +touch -t 202601010000 "$T/ios/_work/spm-tools/spm-tools" "$T/ios/_work/_temp/DerivedData" +rm -f "$T/web/_work/.metadata_never_index" +clean > "$T/out" +ok "a dry run keeps the caches" "$(gone "$T/ios/_work/ios/DerivedData-App")" "kept" +ok "a dry run writes no Spotlight marker" "$(gone "$T/web/_work/.metadata_never_index")" "removed" +clean --apply > "$T/out" +ok "removed a week-old DerivedData" "$(gone "$T/ios/_work/ios/DerivedData-App")" "removed" +ok "removed a week-old SwiftPM cache" "$(gone "$T/ios/_work/ios/spm-App")" "removed" +ok "removed a week-old build cache" "$(gone "$T/ios/_work/ios/.build-cache-x")" "removed" +ok "kept a DerivedData built in this week" "$(gone "$T/android/_work/android/DerivedData-App")" "kept" +ok "but removed that runner's stale SwiftPM cache" "$(gone "$T/android/_work/android/spm-App")" "removed" +ok "kept a building runner's caches" "$(gone "$T/web/_work/web/DerivedData-App")" "kept" +ok "never touched a checkout" "$(gone "$T/ios/_work/ios/ios")" "kept" +ok "never touched a git checkout named like a cache" "$(gone "$T/ios/_work/spm-tools/spm-tools")" "kept" +ok "never touched the runner's own _temp" "$(gone "$T/ios/_work/_temp/DerivedData")" "kept" +ok "wrote the missing Spotlight marker" "$(gone "$T/web/_work/.metadata_never_index")" "kept" +echo 45 > "$T/free" +clean --apply > "$T/out" +ok "under pressure, an idle runner's recent cache goes too" "$(gone "$T/android/_work/android/DerivedData-App")" "removed" +ok "...but never a building runner's" "$(gone "$T/web/_work/web/DerivedData-App")" "kept" + echo echo "passed $PASS, failed $FAIL" [ "$FAIL" -eq 0 ]