Admission lanes: light jobs stop waiting behind heavy ones

CHANGELOG.md

@@ -10,6 +10,34 @@ version numbers (`v0.1.0`, `v0.1.1`, etc.).
 
 ### Added
 
+- **Admission lanes (`FLEET_ADMIT_LANES`).** One host-wide FIFO made a
+  5-second job wait behind 12-minute Xcode builds: on ultra load rose about 8
+  per concurrent job on 30 cores, and 74 % of jobs finish in under 90 s. Each
+  job is now `light`, `heavy` or `simulator` (which also takes a heavy slot),
+  by the Simulator runner rule, `FLEET_ADMIT_HEAVY_PATTERNS` /
+  `FLEET_ADMIT_LIGHT_PATTERNS` (`repo:job` shell patterns), then the
+  dashboard's history (`GET /api/job-class`: light when the job's recent
+  median run is under `FLEET_ADMIT_SHORT_JOB_S`, cached by the hook), else
+  heavy. Limits `FLEET_ADMIT_HEAVY_MAX_CONCURRENT` (4) and
+  `FLEET_ADMIT_LIGHT_MAX_CONCURRENT` (12; `FLEET_ADMIT_SHORT_MAX_CONCURRENT`,
+  set in fleet.env but read by nothing until now, is its alias) under
+  `FLEET_ADMIT_TOTAL_MAX_CONCURRENT`, with the waiting line kept per lane.
+  `observe` keeps the single lane deciding and logs what the lanes would have
+  done (`lane_would_hold`, `lane_reason`, `lane_wait_s`); `enforce` lets them
+  decide; `off` (default) is the old behaviour, byte for byte in the log.
+  Admission rows carry the lane, the Capacity tab shows per-lane counts and
+  the lanes' 24 h tally. See
+  [Lanes](docs/admission-and-scaling.md#lanes).
+- **`cockpit load` and `GET /api/load-by-busy`**: host load1 p50/p90 by number
+  of busy runners from the per-minute host samples, to re-tune the heavy
+  limit when the job mix changes.
+- **`cleanup.sh` erases warm build caches nobody uses.** The kit is moving iOS
+  builds to per-runner `_work/<repo>/DerivedData-*` and `spm-*` so they are
+  incremental; cleanup now removes an idle runner's (and `.build-cache-*`)
+  after `FLEET_CLEANUP_RUNNER_CACHE_DAYS` (7) without a build, and all of them
+  under disk pressure. It also writes the Spotlight marker
+  `_work/.metadata_never_index` for any runner missing it, and
+  `scripts/ephemeral-runner.sh` writes it as `register.sh` already did.
 - **`shims/ps`: a finishing job no longer pins a core for minutes.** At "Complete job" the GitHub
   runner checks every process that started during the job for its `RUNNER_TRACKING_ID`; macOS has no
   `/proc`, so it spawns `ps e -p <pid> -o command` once per process. On a busy shared host that is

cleanup.sh

@@ -16,9 +16,12 @@
 #   - never `simctl shutdown all`, never kill CoreSimulatorService. Those close
 #     simulators the user is working in. `delete unavailable` only removes devices
 #     whose runtime is already gone, which nothing can be using.
-#   - never touch a runner's checkouts or build caches in _work; wiping them
-#     makes every job re-clone and recompile from cold. The one exception is
-#     the Playwright tool cache, and only under disk pressure.
+#   - never touch a runner's checkouts in _work, and never a warm build cache
+#     a job used this week: wiping them makes every job re-clone and recompile
+#     from cold. Warm caches (_work/<repo>/DerivedData-*, spm-*,
+#     .build-cache-*) go after FLEET_CLEANUP_RUNNER_CACHE_DAYS without use, or
+#     all of an idle runner's under disk pressure, as do its Playwright
+#     browsers.
 #   - never touch a simulator or Android AVD that is not a runner's own
 #     ci- (or shots-) device.
 set -uo pipefail
@@ -39,6 +42,10 @@ done
 
 DERIVED="$HOME/Library/Developer/Xcode/DerivedData"
 DERIVED_AGE_DAYS=7
+# Per-runner warm build caches the kit keeps beside each checkout. A week
+# without a build reading them means the next one rebuilds anyway.
+RUNNER_CACHE_DAYS="${FLEET_CLEANUP_RUNNER_CACHE_DAYS:-7}"
+case "$RUNNER_CACHE_DAYS" in '' | *[!0-9]*) RUNNER_CACHE_DAYS=7 ;; esac
 DIAG_AGE_DAYS=14
 PW_LOCK_AGE_HOURS=6
 
@@ -396,6 +403,53 @@ for d in "$ROOT"/*/; do
 done
 say "    ${n} directories"
 
+# ---------------------------------------------------------------------------
+# Warm build caches in each runner's _work/<repo>/: DerivedData-<scheme>,
+# spm-* (SwiftPM checkouts) and .build-cache-*, which the kit's iOS and
+# SwiftPM jobs keep per runner so a build is incremental (CI plan Phase 1.4).
+# Each one is a few GB and per runner, so across 80 runners they are the disk.
+# One whose newest file (two levels down, where xcodebuild and SwiftPM write on
+# every build) is older than RUNNER_CACHE_DAYS goes; under pressure, all of an
+# idle runner's go. Never a building runner's, never under _work/_*/ (the
+# runner's own _temp, _tool, _actions), and never a directory that is a git
+# checkout (a repo that happens to be named spm-something).
+# ---------------------------------------------------------------------------
+say "==> runner build caches unused for ${RUNNER_CACHE_DAYS}d$([ "$PRESSURE" = 1 ] && echo ' (under pressure: all of them, on idle runners)')"
+n=0
+for d in "$ROOT"/*/; do
+  d="${d%/}"
+  [ -f "$d/.runner" ] && [ -d "$d/_work" ] || continue
+  building "$(runner_name "$d")" && continue
+  for cache in "$d"/_work/*/DerivedData* "$d"/_work/*/spm* "$d"/_work/*/.build-cache*; do
+    [ -d "$cache" ] || continue
+    case "$cache" in "$d"/_work/_*) continue ;; esac
+    [ -e "$cache/.git" ] && continue
+    if [ "$PRESSURE" != 1 ] \
+      && [ -n "$(find "$cache" -maxdepth 2 -mtime -"$RUNNER_CACHE_DAYS" -print -quit 2>/dev/null)" ]; then
+      continue
+    fi
+    say "    $(du -sh "$cache" 2>/dev/null | cut -f1)  ${cache#"$ROOT"/}"
+    run rm -rf "$cache"
+    n=$((n + 1))
+  done
+done
+say "    ${n} directories"
+
+# ---------------------------------------------------------------------------
+# Spotlight stays out of every runner's build tree. register.sh writes the
+# marker for a new runner; this catches one created any other way (copied,
+# restored, registered by hand). Cheap and idempotent: one test per runner.
+# ---------------------------------------------------------------------------
+n=0
+for d in "$ROOT"/*/; do
+  d="${d%/}"
+  [ -f "$d/.runner" ] && [ -d "$d/_work" ] || continue
+  [ -e "$d/_work/.metadata_never_index" ] && continue
+  if [ "$APPLY" = "1" ]; then touch "$d/_work/.metadata_never_index" 2>/dev/null || continue; fi
+  n=$((n + 1))
+done
+[ "$n" -gt 0 ] && say "==> Spotlight exclusion written for ${n} runner work tree(s)$([ "$APPLY" = 1 ] || echo ' [dry-run]')"
+
 # ---------------------------------------------------------------------------
 # Under pressure only: each idle runner's Playwright browsers. They are 0.5-1.1 GB
 # per web runner (13 GB on runner-host) and the next job that needs them

cockpit/Sources/CockpitCore/LoadTable.swift

@@ -0,0 +1,61 @@
+import Foundation
+
+/// GET /api/load-by-busy: host load1 p50/p90 by the number of busy runners,
+/// from the dashboard's per-minute host samples. `cockpit load` prints it so
+/// the admission lanes' heavy limit can be re-tuned when the job mix changes
+/// (actions-runners docs/admission-and-scaling.md, "Tuning the lanes").
+public struct LoadTable: Codable, Sendable, Equatable {
+    public struct Row: Codable, Sendable, Equatable {
+        public var busy: Int
+        public var n: Int
+        public var p50: Double?
+        public var p90: Double?
+    }
+
+    public struct Limits: Codable, Sendable, Equatable {
+        public var heavy: Int?
+        public var light: Int?
+        public var total: Int?
+    }
+
+    public struct Lanes: Codable, Sendable, Equatable {
+        public var mode: String?
+        public var limits: Limits?
+    }
+
+    public var days: Int
+    public var host: String?
+    public var samples: Int
+    public var cores: Int?
+    public var rows: [Row]
+    public var lanes: Lanes?
+
+    public static func decode(_ data: Data) throws -> LoadTable {
+        try JSONDecoder().decode(LoadTable.self, from: data)
+    }
+
+    /// The table as the command line prints it.
+    public func text() -> String {
+        func num(_ v: Double?) -> String { v.map { String(format: "%.1f", $0) } ?? "–" }
+        func pad(_ s: String, _ w: Int) -> String { String(repeating: " ", count: max(0, w - s.count)) + s }
+        var out = ["\(host ?? "host"): load1 by busy runners, last \(days) day(s), \(samples) sample(s)"
+            + (cores.map { ", \($0) cores" } ?? "")]
+        if rows.isEmpty {
+            out.append("  no samples in the window")
+        } else {
+            out.append("  busy  samples  load1 p50  load1 p90" + (cores != nil ? "  p90/core" : ""))
+            for r in rows {
+                var line = "  " + pad(String(r.busy), 4) + "  " + pad(String(r.n), 7)
+                    + "  " + pad(num(r.p50), 9) + "  " + pad(num(r.p90), 9)
+                if let cores, cores > 0 { line += "  " + pad(num(r.p90.map { $0 / Double(cores) }), 8) }
+                out.append(line)
+            }
+        }
+        if let l = lanes, let mode = l.mode {
+            let lim = l.limits
+            out.append("lanes \(mode): heavy \(lim?.heavy.map(String.init) ?? "–"), light \(lim?.light.map(String.init) ?? "–"), total \(lim?.total.map(String.init) ?? "–")")
+        }
+        out.append("busy counts runners GitHub shows busy, including jobs held by admission.")
+        return out.joined(separator: "\n")
+    }
+}

cockpit/Sources/cockpit/main.swift

@@ -21,6 +21,8 @@ commands:
                       --recover (with --pr): recover this PR's orphaned runs (force-cancel,
                       then re-run) instead of stopping; recorded in the dashboard's action log
   top                 the host's top CPU users, naming Spotlight when it is the culprit
+  load                host load1 p50/p90 by number of busy runners (--days N, default 7),
+                      to re-tune the admission lanes' heavy limit
   sentinel            check the host out of band (SSH, GitHub, the other machine) —
                       what the app does when the dashboard does not answer
   brief               a markdown incident brief of the current state
@@ -426,6 +428,15 @@ case "top":
         for l in top.lines.prefix(10) { print(String(format: "  %5.1f%%  %@", l.cpu, (l.command as NSString).lastPathComponent)) }
     }
     exit(top.spotlightCPU >= 50 ? 1 : 0)
+case "load":
+    if opts.fixture != nil { fail("load asks the live dashboard", code: 64) }
+    let days = opts.flags["days"].flatMap(Int.init) ?? 7
+    let (client, link) = await connect(opts)
+    let raw: Data
+    do { raw = try await client.get("/api/load-by-busy", query: ["days": String(days)]) } catch { link.close(); fail(error) }
+    link.close()
+    if opts.json { printRaw(raw); exit(0) }
+    do { print(try LoadTable.decode(raw).text()) } catch { fail("the dashboard's answer did not parse: \(error)", code: 3) }
 case "brief":
     let src = await obtain(opts)
     print(IncidentBrief.markdown(verdict: src.verdict, glance: src.glance, route: src.route, connection: "cli"))

cockpit/Tests/CockpitCoreTests/LoadTableTests.swift

@@ -0,0 +1,30 @@
+import Foundation
+import Testing
+@testable import CockpitCore
+
+@Suite struct LoadTableTests {
+    let json = """
+    {"days":7,"host":"ultra","since":0,"samples":6,"cores":30,
+     "rows":[{"busy":0,"n":2,"p50":3,"p90":4},{"busy":2,"n":4,"p50":12,"p90":30}],
+     "lanes":{"mode":"observe","limits":{"heavy":4,"light":12,"total":16}}}
+    """
+
+    @Test func decodesTheDaemonAnswer() throws {
+        let t = try LoadTable.decode(Data(json.utf8))
+        #expect(t.rows.count == 2)
+        #expect(t.rows[1] == LoadTable.Row(busy: 2, n: 4, p50: 12, p90: 30))
+        #expect(t.lanes?.limits?.heavy == 4)
+    }
+
+    @Test func printsOneLinePerBusyCount() throws {
+        let text = try LoadTable.decode(Data(json.utf8)).text()
+        #expect(text.contains("ultra: load1 by busy runners, last 7 day(s), 6 sample(s), 30 cores"))
+        #expect(text.contains("     2        4       12.0       30.0       1.0"))
+        #expect(text.contains("lanes observe: heavy 4, light 12, total 16"))
+    }
+
+    @Test func toleratesAnOlderDaemonWithoutLanesOrCores() throws {
+        let t = try LoadTable.decode(Data(#"{"days":1,"samples":0,"rows":[]}"#.utf8))
+        #expect(t.text().contains("no samples in the window"))
+    }
+}

dashboard/fleetd.js

@@ -35,6 +35,8 @@ import { deriveGroups } from './lib/groups.js';
 import { Backfill } from './lib/backfill.js';
 import { RunSettler } from './lib/settle.js';
 import { RunStatus } from './lib/run-status.js';
+import { createJobClass } from './lib/job-class.js';
+import { loadByBusy } from './lib/load-table.js';
 import { analytics, repoDetail } from './lib/analytics.js';
 import { buildActions, ActionError, ACTION_ARGS } from './lib/actions.js';
 import { handleApi, ApiError, API_PREFIX } from './lib/api.js';
@@ -314,6 +316,13 @@ let alertRun = Promise.resolve();
 // append to. This process never makes an admission decision itself — a job's
 // ability to start must not depend on the dashboard being alive.
 const admission = createAdmission({ db, logPath: CONFIG.admissionLog, warn, hostId: CONFIG.replicaId });
+// Admission lanes ask a job's class here (hooks/common.sh admit_classify). The
+// threshold is the hook's own FLEET_ADMIT_SHORT_JOB_S, sent with the request;
+// this host's fleet.env stands in for a hook that does not send it.
+const jobClass = createJobClass({
+  db,
+  shortJobS: () => admissionSettings({ root: CONFIG.root }).shortJobS,
+});
 const sampleShm = createShmSampler({ logPath: CONFIG.shmLog, lowFree: lowFreeFromEnv() });
 // host_samples rows from before host_id existed are this host's own vitals.
 try {
@@ -3253,6 +3262,36 @@ const server = http.createServer(async (req, res) => {
     }
   }
 
+  // The lane hooks/job-started.sh should put a job in: { class, p50Ms, n }.
+  // Read-only, from this database's own history; cheap and cached.
+  if (url.pathname === '/api/job-class') {
+    try {
+      return json(res, jobClass.lookup({
+        repo: url.searchParams.get('repo'),
+        job: url.searchParams.get('job'),
+        shortS: url.searchParams.get('short_s'),
+      }));
+    } catch (err) {
+      if (!err.status) warn('job-class api:', err.message);
+      return json(res, { error: err.message }, err.status ?? 500);
+    }
+  }
+
+  // Host load1 p50/p90 by busy runners, for tuning the heavy-lane limit
+  // (`cockpit load`).
+  if (url.pathname === '/api/load-by-busy') {
+    try {
+      return json(res, {
+        ...loadByBusy(db, { days: url.searchParams.get('days') ?? 7, hostId: CONFIG.replicaId }),
+        cores: os.cpus().length,
+        lanes: admission.summary().lanes,
+      });
+    } catch (err) {
+      warn('load-by-busy api:', err.message);
+      return json(res, { error: err.message }, 500);
+    }
+  }
+
   if (url.pathname === '/api/run-status') {
     try {
       return json(res, await runStatus.lookup(url.searchParams.get('repo'), url.searchParams.get('run')));