Repository navigation
shims/ps: a finishing job no longer pins a core for minutes - #115
Merged
Merged
Conversation
On macOS the GitHub runner's orphan scan spawns `ps e -p <pid> -o command` once for every process that started during the job. On a busy shared host that is hundreds of spawns per job: ultra 2026-10-22, 81,867 spawns over 400 jobs, ~15 Runner.Workers at ~96% CPU (load ~240/30 cores), 3-minute "Complete job" phases, and zero orphans killed. The scan's switch (`process.clean`) is a runner variable GitHub never sends, so shims/ps answers exactly that call with exit 1 (the runner's existing "could not read" outcome) and passes all other ps calls to /bin/ps. install-ps-shim.sh links it first on the runners' PATH; FLEET_PS_SHIM=off restores the scan. Test in CI, mutation-checked. .gitignore allows exactly shims/ps (the file is an allowlist). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-10-22 ~15
Runner.Workerprocesses on ultra sat at ~96% CPU each (load ~240 on 30 cores). Cause, from the runner source and the workers'_diaglogs: at "Complete job" the GitHub runner's orphan scan checks every process that started during the job for itsRUNNER_TRACKING_ID. macOS has no/proc, so it spawnsps e -p <pid> -o commandonce per process (actions/runnersrc/Runner.Common/ProcessExtensions.cs). On a shared host every other job's processes count, so the cost grows with load.Measured on ultra (last 400 worker logs):
psspawns (~200 per job), 55,704 successful readsci / dashboard— steps took 1 min, "Complete job" took 3 min 14 s (517 inspections)The scan can't be switched off from a workflow: it's gated on a runner variable,
process.clean, that GitHub never sends to self-hosted jobs (the job message's variable list has none of the repo's).What
shims/ps: answers exactly the runner's call (ps e -p <digits> -o command) with exit 1 and no output — the runner's existing "could not read this process" path, i.e. "not this job's", which is what all 81,867 calls concluded anyway. Every otherpscall isexec /bin/ps "$@".FLEET_PS_SHIM=offrestores the scan.scripts/install-ps-shim.sh [--apply|--remove]: links it into the runners' first PATH directory (every runner's.pathstarts with/opt/homebrew/bin), effective on the next spawn — no runner restarts, no job interrupted. Dry run by default; refuses a realpsit doesn't own and runners that disagree on PATH.scripts/test-ps-shim.sh(in CI): runner call exits 1 silently; near misses and other calls byte-identical to/bin/ps; the off switch; 200 calls < 5 s; installer dry-run/apply/idempotent/remove/refusals. Mutation: passing the runner call through fails the test..gitignore(an allowlist) gains exactlyshims/ps.Rollout
Merge,
git pullon ultra,scripts/install-ps-shim.sh --apply, then watchRunner.WorkerCPU at job ends.🤖 Generated with Claude Code