diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 92bec66..28f47a6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -145,6 +145,8 @@ jobs: run: bash scripts/test-preflight-tools.sh - name: runner-path.sh — one job PATH per runner run: bash scripts/test-runner-path.sh + - name: shims/ps — the runner's orphan scan without a ps per process + run: bash scripts/test-ps-shim.sh # --------------------------------------------------------------------------- # Offline release check (no network, checks tracked files for private data) diff --git a/.gitignore b/.gitignore index 87079fc..6f25c0e 100644 --- a/.gitignore +++ b/.gitignore @@ -22,6 +22,10 @@ !README.md !*.sh +# shims/ps (why: see the file). The one file, not the folder. +!shims/ +!shims/ps + # GitHub Actions CI and community files. All paths inside .github/ are listed # explicitly — a catch-all !.github/**/* would also un-ignore any secrets or # workflow outputs that landed there accidentally. diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9b88b..4203a72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,18 @@ version numbers (`v0.1.0`, `v0.1.1`, etc.). ### Added +- **`shims/ps`: a finishing job no longer pins a core for minutes.** At "Complete job" the GitHub + runner checks every process that started during the job for its `RUNNER_TRACKING_ID`; macOS has no + `/proc`, so it spawns `ps e -p -o command` once per process. On a busy shared host that is + hundreds of spawns per job, quadratic in load: on ultra (2026-10-22) 81,867 spawns over 400 jobs, a + 3-minute "Complete job" after a 1-minute job, ~15 Runner.Workers at ~96 % CPU at once (load ~240 on 30 + cores), and zero orphans ever killed. The scan cannot be disabled from a workflow (`process.clean` is a + runner variable GitHub never sends). `shims/ps` answers exactly that call with "could not read" (exit 1, + the outcome the runner already reached) and passes every other `ps` to `/bin/ps`; + `scripts/install-ps-shim.sh --apply` links it into the runners' first PATH directory + (`/opt/homebrew/bin`), effective on the next spawn with no restarts. `FLEET_PS_SHIM=off` restores the + scan. Tests: `scripts/test-ps-shim.sh` (in CI). + - **Parking: held time becomes queue time, not job time.** A held job is in progress as far as GitHub is concerned, so the admission hook's wait counts against `timeout-minutes`, and the hook cannot see that limit to budget diff --git a/docs/reference/scripts.md b/docs/reference/scripts.md index 2bdb20b..7c5fd76 100644 --- a/docs/reference/scripts.md +++ b/docs/reference/scripts.md @@ -37,12 +37,14 @@ without `--repair` — have no `--apply` because there is nothing to guard. | `scripts/ephemeral-runner.sh` | One job in a fresh directory, then delete it | **yes** | | `scripts/reap-ephemeral.sh` | Remove ephemeral directories a crash left behind | **yes** | | `scripts/install-hooks.sh` | Point every runner at the job hooks | **yes** | +| `scripts/install-ps-shim.sh` | Put `shims/ps` first on the runners' PATH so the runner's orphan scan stops spawning `ps` per process | **yes** (one symlink; `--remove`) | | `scripts/release-check.sh` | Fail if anything host-specific reached a tracked file | read-only except `--install-hook` | | `scripts/check-docs.sh` | Verify the docs are internally consistent | read-only | | `scripts/test-drain.sh` | Shell tests for drain and resume | runs against a temporary fleet | | `scripts/test-ephemeral.sh` | Shell tests for the ephemeral reaper | runs against a temporary fleet | | `scripts/test-cleanup.sh` | Shell tests for disk cleanup | runs against a temporary fleet | | `scripts/test-health.sh` | Shell tests for `health.sh` | runs against a temporary fleet | +| `scripts/test-ps-shim.sh` | Shell tests for `shims/ps` and its installer | a temporary PATH directory | | `scripts/test-preflight-tools.sh` | Shell tests for the CLI-tool checks | stub PATH and a temporary database | | `scripts/infer-checks.py` | Work out which preflight checks this fleet needs | read-only | | `scripts/check-tools.sh` | Check the CLI tools in `scripts/cli-tools.txt` on the runners' PATH | read-only | @@ -942,6 +944,26 @@ that a building or slot-holding runner keeps its simulator and browsers, that host-wide steps wait for an idle fleet, that a dry run deletes nothing, and that two runs never overlap. Exits non-zero if any test fails. +### `scripts/install-ps-shim.sh` + +Stops the GitHub runner's orphan scan from pinning a core when a job finishes. At +"Complete job" the runner checks every process that started during the job for its +`RUNNER_TRACKING_ID`; macOS has no `/proc`, so it spawns `ps e -p -o command` once per +process — hundreds per job on a busy host (ultra 2026-10-22: 81,867 spawns over 400 jobs, a +p90 of 143 s per job, zero orphans ever killed). The scan cannot be switched off from a +workflow. `shims/ps` answers exactly that call with exit 1 (the runner's own "could not read" +path) and passes every other `ps` call to `/bin/ps`; this links it into the runners' first +PATH directory, which takes effect on the next spawn with no restart. `FLEET_PS_SHIM=off` +restores the scan for one shell or job. + +Source: [`scripts/install-ps-shim.sh`](https://github.com/addisdev/actions-runners/blob/main/scripts/install-ps-shim.sh). + +```bash +scripts/install-ps-shim.sh # show what would change +scripts/install-ps-shim.sh --apply # link it +scripts/install-ps-shim.sh --remove --apply # unlink it +``` + ### `scripts/runner-path.sh` Gives every runner on this host the same job PATH. A job's PATH comes from the diff --git a/scripts/install-ps-shim.sh b/scripts/install-ps-shim.sh new file mode 100755 index 0000000..7663bf9 --- /dev/null +++ b/scripts/install-ps-shim.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# Put shims/ps first on the runners' PATH, so the runner's orphan scan stops +# spawning /bin/ps once per process (see shims/ps for why and what it costs). +# +# scripts/install-ps-shim.sh # show what would change +# scripts/install-ps-shim.sh --apply # link it +# scripts/install-ps-shim.sh --remove --apply +# +# It links `ps` in the FIRST directory of the runners' .path (every runner here +# starts with /opt/homebrew/bin), which takes effect on the next process the +# runner starts: no runner restart, no job interrupted. It refuses when the +# runners disagree on that first directory, or when a real file (not our link) +# named ps is already there. The link passes every other ps call to /bin/ps. +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +[ -f "$HERE/fleet.env" ] && . "$HERE/fleet.env" +ROOT="${FLEET_ROOT:-$HERE}" +SHIM="${FLEET_PS_SHIM_SOURCE:-$HERE/shims/ps}" +APPLY=0; REMOVE=0 +for arg in "$@"; do + case "$arg" in + --apply) APPLY=1 ;; + --remove) REMOVE=1 ;; + -h|--help) sed -n '2,13p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown argument: $arg" >&2; exit 1 ;; + esac +done +[ -x "$SHIM" ] || { echo "missing or not executable: $SHIM" >&2; exit 1; } + +shopt -s nullglob +firsts=() +for p in "$ROOT"/*/.path; do firsts+=("$(cut -d: -f1 < "$p")"); done +[ "${#firsts[@]}" -gt 0 ] || { echo "no runners under $ROOT (no */.path)" >&2; exit 1; } +DIR="$(printf '%s\n' "${firsts[@]}" | sort -u)" +[ "$(printf '%s\n' "$DIR" | wc -l | tr -d ' ')" = 1 ] || { + echo "runners disagree on the first PATH directory:" >&2; printf ' %s\n' $DIR >&2; exit 1; } +TARGET="$DIR/ps" +echo "${#firsts[@]} runners; first PATH directory: $DIR" + +if [ "$REMOVE" -eq 1 ]; then + if [ -L "$TARGET" ] && [ "$(readlink "$TARGET")" = "$SHIM" ]; then + [ "$APPLY" -eq 1 ] && rm "$TARGET" && echo "removed $TARGET" || echo "would remove $TARGET" + else + echo "nothing to remove at $TARGET" + fi + exit 0 +fi +if [ -e "$TARGET" ] && ! { [ -L "$TARGET" ] && [ "$(readlink "$TARGET")" = "$SHIM" ]; }; then + echo "refusing: $TARGET exists and is not this shim" >&2; exit 1 +fi +if [ -L "$TARGET" ]; then echo "already linked: $TARGET -> $SHIM"; exit 0; fi +if [ "$APPLY" -eq 1 ]; then ln -s "$SHIM" "$TARGET" && echo "linked $TARGET -> $SHIM"; else echo "would link $TARGET -> $SHIM"; fi diff --git a/scripts/test-ps-shim.sh b/scripts/test-ps-shim.sh new file mode 100755 index 0000000..84837d3 --- /dev/null +++ b/scripts/test-ps-shim.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# Tests for shims/ps and scripts/install-ps-shim.sh. +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SHIM="$HERE/shims/ps" +T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT +fail() { echo "FAIL: $*" >&2; exit 1; } + +# 1. The runner's exact orphan-scan call: exit 1, nothing printed, and no /bin/ps. +out="$("$SHIM" e -p $$ -o command 2>&1)" && fail "the runner's call exited 0" +[ -z "$out" ] || fail "the runner's call printed: $out" + +# 2. Everything else is /bin/ps, byte for byte. +[ "$("$SHIM" -p $$ -o pid=)" = "$(/bin/ps -p $$ -o pid=)" ] || fail "ps -p -o pid= differs from /bin/ps" +[ "$("$SHIM" e -p $$ -o pid=)" = "$(/bin/ps e -p $$ -o pid=)" ] || fail "a near miss (-o pid=) was not passed through" +a=0; "$SHIM" e -p notapid -o command >/dev/null 2>&1 || a=$?; b=0; /bin/ps e -p notapid -o command >/dev/null 2>&1 || b=$? +[ "$a" = "$b" ] || fail "a non-numeric pid was not passed through ($a vs $b)" + +# 3. FLEET_PS_SHIM=off restores the runner's own scan. +FLEET_PS_SHIM=off "$SHIM" e -p $$ -o command >/dev/null || fail "FLEET_PS_SHIM=off did not pass through" + +# 4. It is cheap: 200 runner calls (one job's worth on a busy host) well under 5 s. +s=$(date +%s); for _ in $(seq 200); do "$SHIM" e -p 1 -o command || true; done; e=$(date +%s) +[ $((e - s)) -lt 5 ] || fail "200 calls took $((e - s)) s" + +# 5. Installer: dry run changes nothing; --apply links into the runners' first PATH dir; idempotent; --remove. +mkdir -p "$T/bin" "$T/r1" "$T/r2" +printf '%s:/usr/bin:/bin\n' "$T/bin" > "$T/r1/.path"; printf '%s:/usr/bin:/bin\n' "$T/bin" > "$T/r2/.path" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" | grep -q "would link" || fail "dry run did not say what it would do" +[ ! -e "$T/bin/ps" ] || fail "dry run linked" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null +[ "$(readlink "$T/bin/ps")" = "$SHIM" ] || fail "--apply did not link the shim" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply | grep -q "already linked" || fail "second --apply was not idempotent" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --remove --apply >/dev/null +[ ! -e "$T/bin/ps" ] || fail "--remove did not unlink" + +# 6. Refuses a real ps it does not own, and runners that disagree on PATH. +printf '#!/bin/sh\n' > "$T/bin/ps"; chmod +x "$T/bin/ps" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null 2>&1 && fail "overwrote a real ps" +rm "$T/bin/ps"; printf '/usr/bin:/bin\n' > "$T/r2/.path" +FLEET_ROOT="$T" "$HERE/scripts/install-ps-shim.sh" --apply >/dev/null 2>&1 && fail "linked although runners disagree on PATH" +echo "ps shim: all checks passed" diff --git a/shims/ps b/shims/ps new file mode 100755 index 0000000..d4e5992 --- /dev/null +++ b/shims/ps @@ -0,0 +1,28 @@ +#!/bin/bash +# `ps`, minus the one call that pinned a CPU core per finishing CI job. +# +# At "Complete job" the GitHub runner looks for orphans: every process that +# started during the job is checked for the job's RUNNER_TRACKING_ID. Linux +# reads /proc//environ; macOS has no /proc, so the runner spawns +# `ps e -p -o command` once PER PROCESS (actions/runner, +# src/Runner.Common/ProcessExtensions.cs). On a shared host every other job's +# processes started "during" this job, so it is hundreds of spawns per job, a +# Runner.Worker at ~96 % CPU for minutes, and quadratic in how busy the host +# is. Measured on ultra 2026-10-22: 81,867 spawns over the last 400 jobs, a +# 3-minute "Complete job" on a job whose steps took one minute, ~15 workers at +# full CPU at once (host load ~240 on 30 cores) — and ZERO orphans killed. The +# fleet's own hooks reap what jobs leave behind (simulators, Gradle, shm). +# +# The scan cannot be turned off from a workflow: it is gated on a runner +# variable (`process.clean`) GitHub never sends to self-hosted jobs. So this +# answers exactly the runner's call, `ps e -p -o command`, with "could +# not read" (exit 1, no output), which the runner treats as "not this job's +# process" — the same outcome it reached on every one of those 81,867 calls, +# without the spawn of /bin/ps. Everything else goes straight to /bin/ps. +# +# FLEET_PS_SHIM=off restores the runner's own scan (e.g. to hunt a real orphan). +if [ "$#" -eq 5 ] && [ "$1" = e ] && [ "$2" = -p ] && [ "$4" = -o ] && [ "$5" = command ] \ + && [[ "$3" =~ ^[0-9]+$ ]] && [ "${FLEET_PS_SHIM:-on}" != off ]; then + exit 1 +fi +exec /bin/ps "$@"