shims/ps: a finishing job no longer pins a core for minutes

.github/workflows/ci.yml

@@ -145,6 +145,8 @@ jobs:
         run: bash scripts/test-preflight-tools.sh
       - name: runner-path.sh — one job PATH per runner
         run: bash scripts/test-runner-path.sh
+      - name: shims/ps — the runner's orphan scan without a ps per process
+        run: bash scripts/test-ps-shim.sh
 
   # ---------------------------------------------------------------------------
   # Offline release check (no network, checks tracked files for private data)

.gitignore

@@ -22,6 +22,10 @@
 !README.md
 !*.sh
 
+# shims/ps (why: see the file). The one file, not the folder.
+!shims/
+!shims/ps
+
 # GitHub Actions CI and community files. All paths inside .github/ are listed
 # explicitly — a catch-all !.github/**/* would also un-ignore any secrets or
 # workflow outputs that landed there accidentally.

CHANGELOG.md

@@ -10,6 +10,18 @@ version numbers (`v0.1.0`, `v0.1.1`, etc.).
 
 ### Added
 
+- **`shims/ps`: a finishing job no longer pins a core for minutes.** At "Complete job" the GitHub
+  runner checks every process that started during the job for its `RUNNER_TRACKING_ID`; macOS has no
+  `/proc`, so it spawns `ps e -p <pid> -o command` once per process. On a busy shared host that is
+  hundreds of spawns per job, quadratic in load: on ultra (2026-10-22) 81,867 spawns over 400 jobs, a
+  3-minute "Complete job" after a 1-minute job, ~15 Runner.Workers at ~96 % CPU at once (load ~240 on 30
+  cores), and zero orphans ever killed. The scan cannot be disabled from a workflow (`process.clean` is a
+  runner variable GitHub never sends). `shims/ps` answers exactly that call with "could not read" (exit 1,
+  the outcome the runner already reached) and passes every other `ps` to `/bin/ps`;
+  `scripts/install-ps-shim.sh --apply` links it into the runners' first PATH directory
+  (`/opt/homebrew/bin`), effective on the next spawn with no restarts. `FLEET_PS_SHIM=off` restores the
+  scan. Tests: `scripts/test-ps-shim.sh` (in CI).
+
 - **Parking: held time becomes queue time, not job time.** A held job is in
   progress as far as GitHub is concerned, so the admission hook's wait counts
   against `timeout-minutes`, and the hook cannot see that limit to budget

docs/reference/scripts.md

@@ -37,12 +37,14 @@ without `--repair` — have no `--apply` because there is nothing to guard.
 | `scripts/ephemeral-runner.sh` | One job in a fresh directory, then delete it | **yes** |
 | `scripts/reap-ephemeral.sh` | Remove ephemeral directories a crash left behind | **yes** |
 | `scripts/install-hooks.sh` | Point every runner at the job hooks | **yes** |
+| `scripts/install-ps-shim.sh` | Put `shims/ps` first on the runners' PATH so the runner's orphan scan stops spawning `ps` per process | **yes** (one symlink; `--remove`) |
 | `scripts/release-check.sh` | Fail if anything host-specific reached a tracked file | read-only except `--install-hook` |
 | `scripts/check-docs.sh` | Verify the docs are internally consistent | read-only |
 | `scripts/test-drain.sh` | Shell tests for drain and resume | runs against a temporary fleet |
 | `scripts/test-ephemeral.sh` | Shell tests for the ephemeral reaper | runs against a temporary fleet |
 | `scripts/test-cleanup.sh` | Shell tests for disk cleanup | runs against a temporary fleet |
 | `scripts/test-health.sh` | Shell tests for `health.sh` | runs against a temporary fleet |
+| `scripts/test-ps-shim.sh` | Shell tests for `shims/ps` and its installer | a temporary PATH directory |
 | `scripts/test-preflight-tools.sh` | Shell tests for the CLI-tool checks | stub PATH and a temporary database |
 | `scripts/infer-checks.py` | Work out which preflight checks this fleet needs | read-only |
 | `scripts/check-tools.sh` | Check the CLI tools in `scripts/cli-tools.txt` on the runners' PATH | read-only |
@@ -942,6 +944,26 @@ that a building or slot-holding runner keeps its simulator and browsers, that
 host-wide steps wait for an idle fleet, that a dry run deletes nothing, and
 that two runs never overlap. Exits non-zero if any test fails.
 
+### `scripts/install-ps-shim.sh`
+
+Stops the GitHub runner's orphan scan from pinning a core when a job finishes. At
+"Complete job" the runner checks every process that started during the job for its
+`RUNNER_TRACKING_ID`; macOS has no `/proc`, so it spawns `ps e -p <pid> -o command` once per
+process — hundreds per job on a busy host (ultra 2026-10-22: 81,867 spawns over 400 jobs, a
+p90 of 143 s per job, zero orphans ever killed). The scan cannot be switched off from a
+workflow. `shims/ps` answers exactly that call with exit 1 (the runner's own "could not read"
+path) and passes every other `ps` call to `/bin/ps`; this links it into the runners' first
+PATH directory, which takes effect on the next spawn with no restart. `FLEET_PS_SHIM=off`
+restores the scan for one shell or job.
+
+Source: [`scripts/install-ps-shim.sh`](https://github.com/addisdev/actions-runners/blob/main/scripts/install-ps-shim.sh).
+
+```bash
+scripts/install-ps-shim.sh                  # show what would change
+scripts/install-ps-shim.sh --apply          # link it
+scripts/install-ps-shim.sh --remove --apply # unlink it
+```
+
 ### `scripts/runner-path.sh`
 
 Gives every runner on this host the same job PATH. A job's PATH comes from the

scripts/install-ps-shim.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Put shims/ps first on the runners' PATH, so the runner's orphan scan stops
+# spawning /bin/ps once per process (see shims/ps for why and what it costs).
+#
+#   scripts/install-ps-shim.sh            # show what would change
+#   scripts/install-ps-shim.sh --apply    # link it
+#   scripts/install-ps-shim.sh --remove --apply
+#
+# It links `ps` in the FIRST directory of the runners' .path (every runner here
+# starts with /opt/homebrew/bin), which takes effect on the next process the
+# runner starts: no runner restart, no job interrupted. It refuses when the
+# runners disagree on that first directory, or when a real file (not our link)
+# named ps is already there. The link passes every other ps call to /bin/ps.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+[ -f "$HERE/fleet.env" ] && . "$HERE/fleet.env"
+ROOT="${FLEET_ROOT:-$HERE}"
+SHIM="${FLEET_PS_SHIM_SOURCE:-$HERE/shims/ps}"
+APPLY=0; REMOVE=0
+for arg in "$@"; do
+  case "$arg" in
+    --apply) APPLY=1 ;;
+    --remove) REMOVE=1 ;;
+    -h|--help) sed -n '2,13p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 1 ;;
+  esac
+done
+[ -x "$SHIM" ] || { echo "missing or not executable: $SHIM" >&2; exit 1; }
+
+shopt -s nullglob
+firsts=()
+for p in "$ROOT"/*/.path; do firsts+=("$(cut -d: -f1 < "$p")"); done
+[ "${#firsts[@]}" -gt 0 ] || { echo "no runners under $ROOT (no */.path)" >&2; exit 1; }
+DIR="$(printf '%s\n' "${firsts[@]}" | sort -u)"
+[ "$(printf '%s\n' "$DIR" | wc -l | tr -d ' ')" = 1 ] || {
+  echo "runners disagree on the first PATH directory:" >&2; printf '  %s\n' $DIR >&2; exit 1; }
+TARGET="$DIR/ps"
+echo "${#firsts[@]} runners; first PATH directory: $DIR"
+
+if [ "$REMOVE" -eq 1 ]; then
+  if [ -L "$TARGET" ] && [ "$(readlink "$TARGET")" = "$SHIM" ]; then
+    [ "$APPLY" -eq 1 ] && rm "$TARGET" && echo "removed $TARGET" || echo "would remove $TARGET"
+  else
+    echo "nothing to remove at $TARGET"
+  fi
+  exit 0
+fi
+if [ -e "$TARGET" ] && ! { [ -L "$TARGET" ] && [ "$(readlink "$TARGET")" = "$SHIM" ]; }; then
+  echo "refusing: $TARGET exists and is not this shim" >&2; exit 1
+fi
+if [ -L "$TARGET" ]; then echo "already linked: $TARGET -> $SHIM"; exit 0; fi
+if [ "$APPLY" -eq 1 ]; then ln -s "$SHIM" "$TARGET" && echo "linked $TARGET -> $SHIM"; else echo "would link $TARGET -> $SHIM"; fi

scripts/test-ps-shim.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# Tests for shims/ps and scripts/install-ps-shim.sh.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SHIM="$HERE/shims/ps"
+T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT
+fail() { echo "FAIL: $*" >&2; exit 1; }
+
+# 1. The runner's exact orphan-scan call: exit 1, nothing printed, and no /bin/ps.
+out="$("$SHIM" e -p $$ -o command 2>&1)" && fail "the runner's call exited 0"
+[ -z "$out" ] || fail "the runner's call printed: $out"
+
+# 2. Everything else is /bin/ps, byte for byte.
+[ "$("$SHIM" -p $$ -o pid=)" = "$(/bin/ps -p $$ -o pid=)" ] || fail "ps -p -o pid= differs from /bin/ps"
+[ "$("$SHIM" e -p $$ -o pid=)" = "$(/bin/ps e -p $$ -o pid=)" ] || fail "a near miss (-o pid=) was not passed through"
+a=0; "$SHIM" e -p notapid -o command >/dev/null 2>&1 || a=$?; b=0; /bin/ps e -p notapid -o command >/dev/null 2>&1 || b=$?
+[ "$a" = "$b" ] || fail "a non-numeric pid was not passed through ($a vs $b)"
+
+# 3. FLEET_PS_SHIM=off restores the runner's own scan.
+FLEET_PS_SHIM=off "$SHIM" e -p $$ -o command >/dev/null || fail "FLEET_PS_SHIM=off did not pass through"
+
+# 4. It is cheap: 200 runner calls (one job's worth on a busy host) well under 5 s.
+s=$(date +%s); for _ in $(seq 200); do "$SHIM" e -p 1 -o command || true; done; e=$(date +%s)
+[ $((e - s)) -lt 5 ] || fail "200 calls took $((e - s)) s"
+
+# 5. Installer: dry run changes nothing; --apply links into the runners' first PATH dir; idempotent; --remove.
+mkdir -p "$T/bin" "$T/r1" "$T/r2"
+printf '%s:/usr/bin:/bin\n' "$T/bin" > "$T/r1/.path"; printf '%s:/usr/bin:/bin\n' "$T/bin" > "$T/r2/.path"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" | grep -q "would link" || fail "dry run did not say what it would do"
+[ ! -e "$T/bin/ps" ] || fail "dry run linked"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null
+[ "$(readlink "$T/bin/ps")" = "$SHIM" ] || fail "--apply did not link the shim"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply | grep -q "already linked" || fail "second --apply was not idempotent"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --remove --apply >/dev/null
+[ ! -e "$T/bin/ps" ] || fail "--remove did not unlink"
+
+# 6. Refuses a real ps it does not own, and runners that disagree on PATH.
+printf '#!/bin/sh\n' > "$T/bin/ps"; chmod +x "$T/bin/ps"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null 2>&1 && fail "overwrote a real ps"
+rm "$T/bin/ps"; printf '/usr/bin:/bin\n' > "$T/r2/.path"
+FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null 2>&1 && fail "linked although runners disagree on PATH"
+echo "ps shim: all checks passed"

shims/ps

@@ -0,0 +1,28 @@
+#!/bin/bash
+# `ps`, minus the one call that pinned a CPU core per finishing CI job.
+#
+# At "Complete job" the GitHub runner looks for orphans: every process that
+# started during the job is checked for the job's RUNNER_TRACKING_ID. Linux
+# reads /proc/<pid>/environ; macOS has no /proc, so the runner spawns
+# `ps e -p <pid> -o command` once PER PROCESS (actions/runner,
+# src/Runner.Common/ProcessExtensions.cs). On a shared host every other job's
+# processes started "during" this job, so it is hundreds of spawns per job, a
+# Runner.Worker at ~96 % CPU for minutes, and quadratic in how busy the host
+# is. Measured on ultra 2026-10-22: 81,867 spawns over the last 400 jobs, a
+# 3-minute "Complete job" on a job whose steps took one minute, ~15 workers at
+# full CPU at once (host load ~240 on 30 cores) — and ZERO orphans killed. The
+# fleet's own hooks reap what jobs leave behind (simulators, Gradle, shm).
+#
+# The scan cannot be turned off from a workflow: it is gated on a runner
+# variable (`process.clean`) GitHub never sends to self-hosted jobs. So this
+# answers exactly the runner's call, `ps e -p <digits> -o command`, with "could
+# not read" (exit 1, no output), which the runner treats as "not this job's
+# process" — the same outcome it reached on every one of those 81,867 calls,
+# without the spawn of /bin/ps. Everything else goes straight to /bin/ps.
+#
+# FLEET_PS_SHIM=off restores the runner's own scan (e.g. to hunt a real orphan).
+if [ "$#" -eq 5 ] && [ "$1" = e ] && [ "$2" = -p ] && [ "$4" = -o ] && [ "$5" = command ] \
+   && [[ "$3" =~ ^[0-9]+$ ]] && [ "${FLEET_PS_SHIM:-on}" != off ]; then
+  exit 1
+fi
+exec /bin/ps "$@"