scripts.md

Scripts

Every command-line entry point in the fleet, with the flags its code actually parses.

One convention runs through all of them: anything that deletes, deregisters or rewrites is dry-run by default and needs --apply. Run it once, read what it says it would do, then run it again. The other half of that convention is that each script states what it refuses to do — cleanup.sh --apply will not run its host-wide steps while a job is building and skips a building runner's own simulator and browsers (a job held by the admission hook is waiting, not building, and does not count; the dry run always runs), deregister.sh will not remove a runner mid-job or leave a repo with no CI, the reaper will not touch a directory with a live process in it. Those refusals are the interesting part of the behaviour, so they are documented here alongside the flags.

Scripts that only read — status.sh, runs.sh, preflight.sh, health.sh without --repair — have no --apply because there is nothing to guard.

Summary

Script	What it does	Dry run by default
register.sh	Register a runner for one repo	no — creating a runner is reversible
status.sh	Every runner, its status, and what the fleet costs idle	read-only
health.sh	launchd + GitHub state per runner; --repair restarts dead services	read-only without --repair
healthctl.sh	Install a LaunchAgent that runs health.sh --repair on a timer	n/a — subcommands
cleanupctl.sh	Install a LaunchAgent that runs cleanup.sh --apply --auto every 15 minutes	n/a — subcommands
runs.sh	What is building across every repo, and on which machine	read-only
cleanup.sh	Erase idle CI simulators and AVDs (keeping runners' snapshot AVDs), stop orphaned CI emulators; prune stale DerivedData, dead simulators, old _diag, superseded runner versions	yes
preflight.sh	Check a host against what the workflows assume	read-only, installs nothing
scripts/deregister.sh	Remove one named runner completely	yes
scripts/drain-runner.sh	Stop a runner gracefully, or resume it	no — every state it writes is reversible
scripts/restart-runner.sh	Restart one runner service by directory name	no
scripts/host-drain.sh	Switch a host off or on (--off/--on: every runner stops taking jobs, busy ones finish first), or pause new-runner placement (--drain/--resume)	no — reversible
scripts/drain-stop-when-idle.sh	Stop a drained runner once its job ends	internal; called by the completion hook
scripts/ephemeral-runner.sh	One job in a fresh directory, then delete it	yes
scripts/reap-ephemeral.sh	Remove ephemeral directories a crash left behind	yes
scripts/install-hooks.sh	Point every runner at the job hooks	yes
scripts/install-ps-shim.sh	Put shims/ps first on the runners' PATH so the runner's orphan scan stops spawning ps per process	yes (one symlink; --remove)
scripts/release-check.sh	Fail if anything host-specific reached a tracked file	read-only except --install-hook
scripts/check-docs.sh	Verify the docs are internally consistent	read-only
scripts/test-drain.sh	Shell tests for drain and resume	runs against a temporary fleet
scripts/test-ephemeral.sh	Shell tests for the ephemeral reaper	runs against a temporary fleet
scripts/test-cleanup.sh	Shell tests for disk cleanup	runs against a temporary fleet
scripts/test-health.sh	Shell tests for health.sh	runs against a temporary fleet
scripts/test-ps-shim.sh	Shell tests for shims/ps and its installer	a temporary PATH directory
scripts/test-preflight-tools.sh	Shell tests for the CLI-tool checks	stub PATH and a temporary database
scripts/infer-checks.py	Work out which preflight checks this fleet needs	read-only
scripts/check-tools.sh	Check the CLI tools in scripts/cli-tools.txt on the runners' PATH	read-only
dashboard/fleetctl.sh	Install, run, inspect, back up and restore the dashboard daemon	n/a — subcommands
dashboard/agentctl.sh	Install, run and inspect the fleet agent on an agent Mac	n/a — subcommands
dashboard/bin/fleet.mjs	The fleet command line for /api/v1, built from the daemon's OpenAPI document	reads; actions preview first and need --yes
install.sh	One-command setup for coordinator or agent role	n/a — runs preflight then fleetctl/agentctl
dashboard/watch/watchctl.sh	Install, run and inspect the watchdog	n/a — subcommands

Fleet management

register.sh

Registers a self-hosted runner for one repository: downloads the runner tarball, verifies its checksum, configures it, and installs and starts a LaunchAgent. Source: register.sh.

./register.sh owner/repo [extra-label...]
RUNNER_INSTANCE=2 ./register.sh owner/repo [extra-label...]

Examples:

./register.sh owner/project-ios xcode-16.3
./register.sh owner/project-web ci playwright

It parses no flags. The repository is the first positional argument; every argument after it is an extra label. Multiple labels are passed to config.sh as a single comma-separated --labels value (ci,playwright).

Variable	Effect
RUNNER_INSTANCE	Which instance this is. 1 (the default) keeps the bare directory name; anything else appends -<n>, so instance 2 of app-ios lives in app-ios-2.
FLEET_ROOT	Where runners are registered. Defaults to the script's own directory, so a clone somewhere else registers into itself rather than silently building a second fleet.
RUNNER_TOKEN	A registration token supplied by the caller instead of being minted with gh. See the SSH note in Get started.

What it refuses to do:

• It will not register on top of an existing runner. If the target directory already has a .runner file it exits and prints the commands to remove that runner first, along with the RUNNER_INSTANCE value that would add another one alongside it.
• It will not use a tarball whose checksum does not match, and the checksum is verified on every run rather than only on download — a truncated or swapped cache is otherwise found by a runner that behaves strangely.

Two details worth knowing. The runner's .env is written with an explicit PATH because a LaunchAgent does not inherit a login shell's, and ANDROID_HOME is written only when ~/Library/Android/sdk exists at registration time. And the job hook lines are written for every new runner but are inert: both hooks exit immediately unless FLEET_ADMIT_MODE is set in fleet.env.

When gh cannot authenticate locally, the post-registration status check is skipped rather than reported as a failure — svc.sh install and svc.sh start are the real evidence that it worked.

status.sh

One line per runner: repo, status, runner name, followed by the fleet's idle cost in resident memory and its size on disk. Source: status.sh.

./status.sh

No flags. Reads FLEET_ROOT from the environment or fleet.env. Results are deduplicated by repository rather than by directory, because a repo with a second runner has two directories that name the same repo.

health.sh

Asks whether every runner service is actually alive and whether GitHub agrees. Source: health.sh.

./health.sh           # report only
./health.sh --repair  # also restart any service that is loaded but dead

Flag	Effect
--repair	Restart any service whose launchd state is not running, or whose live local listener is reported offline by GitHub. Recognised only as the first argument.

It exists because no runner LaunchAgent sets KeepAlive. runsvc.sh supervises Runner.Listener, so a crashed listener recovers on its own; a crashed or OOM-killed RunnerService.js does not, and the only symptom is one repo's jobs queueing forever while every other repo looks fine.

Exit status is non-zero if any runner is DEAD, NOT-LOADED, offline, or unknown to the GitHub API.

What it refuses to do:

• It will not revive a drained runner. A runner with a .drain file is reported as drained and skipped before any launchd or GitHub check, including under --repair. A repair that undoes a deliberate drain is worse than no repair, and it would also set a non-zero exit that reads as "the fleet is unhealthy" when nothing is wrong.

healthctl.sh

Installs a LaunchAgent that runs health.sh --repair on a timer so a dead RunnerService.js is restarted within about a minute instead of whenever someone notices one repo queueing forever. Source: healthctl.sh.

./healthctl.sh install     # write the LaunchAgent and load it
./healthctl.sh status      # is it loaded, and what did the last sweep say
./healthctl.sh logs [n]    # tail the repair log (default 60 lines)
./healthctl.sh restart
./healthctl.sh uninstall
./healthctl.sh run         # one foreground sweep, for debugging

Subcommand	What it does
install	Write the periodic health-repair LaunchAgent plist, load it, and report status.
uninstall	Unload the agent and delete the plist.
start / stop	Load or unload the agent.
restart	Unload, load, and report status.
status	The default. launchd state, interval, last log line, sweep count.
logs [n]	Tail the repair log. Default 60 lines.
run	Run one repair sweep in the foreground via the same wrapper launchd uses.

The plist uses StartInterval (default 60 seconds, from FLEET_HEALTH_INTERVAL in fleet.env) and RunAtLoad. It does not set KeepAlive — this job is periodic, not resident. scripts/health-repair-launchd.sh holds a lock so a slow sweep cannot overlap the next tick.

What it refuses to do:

• It will not edit runner LaunchAgents. Only <FLEET_LABEL_PREFIX>.fleet-health is written under ~/Library/LaunchAgents/. GitHub runner plists from register.sh are untouched.
• It will not revive drained runners. Repair behaviour is delegated entirely to health.sh --repair, which skips any runner with a .drain file.

Logs default to logs/fleet-health-repair.log under FLEET_ROOT. The wrapper resolves gh once at install time into the plist's PATH, because launchd gets a minimal environment.

runs.sh

A cross-repo view of what is building, which GitHub does not offer — each repo has its own Actions tab, so past a handful of repos there is no single page that answers "is anything running, and where". Source: runs.sh.

./runs.sh            # one snapshot
./runs.sh --watch    # refresh every 15s until interrupted
./runs.sh --host     # group by machine instead of by repo
./runs.sh --refresh  # rediscover which repos have CI, then report

Flag	Effect
--watch	Redraw every 15 seconds until interrupted.
--host	Attribute runs to a host rather than grouping by repo.
--refresh	Rediscover which repos have Actions before reporting, ignoring the cache TTL.
-h, --help	Print the usage header.

Unrecognised arguments are ignored rather than rejected.

Runs are attributed to a host, not just to a runner, because after a migration the same repo has runners on two machines. The host is read straight off the runner name, since register.sh names every runner <LocalHostName>-<repo>. Runs assigned to a runner but still parked in the admission hook are shown as waiting_host, not in_progress; GitHub's API calls both states in-progress even though the workflow has not reached its first step.

On a machine with no runner directories of its own — a laptop watching a fleet hosted elsewhere — it falls back to discovery:

Variable	Effect
FLEET_REPOS	An explicit repo list, which always wins over discovery. Bare names get FLEET_OWNER prepended.
FLEET_OWNER	The account to discover repos for. Defaults to whoever gh is logged in as.
FLEET_REPO_CACHE	Where the discovered list is cached. Defaults to .fleet-repos beside the script.
FLEET_REPO_CACHE_TTL	Cache lifetime in seconds. Default 21600 (six hours).
FLEET_REPO_LIMIT	How many repos gh repo list returns. Default 200.

Discovery keeps only repos that actually have workflows, and a failed discovery never overwrites a cache that still works.

cleanup.sh

Reclaims the disk that CI quietly eats. Source: cleanup.sh.

./cleanup.sh                  # dry run — prints what it WOULD delete, touches nothing
./cleanup.sh --apply          # actually delete
./cleanup.sh --apply --auto   # the LaunchAgent's every-15-minutes run

Flag	Effect
--apply	Perform the deletions.
--auto	Exit after one df unless free disk is under the pressure line or a day has passed since the last full run.

Variable	Default	Meaning
FLEET_CLEANUP_PRESSURE_GB	admission floor + 20	Below this, the pressure steps run too and --auto acts at once
FLEET_CLEANUP_SIM_MAX_GB	3	Erase a runner's ci- simulator past this size
FLEET_CLEANUP_SIM_PRESSURE_GB	1	...and past this under pressure
FLEET_CLEANUP_FULL_EVERY_S	86400	--auto runs a full pass at least this often
FLEET_CLEANUP_AVD_KEEP_DAYS	14	Keep a runner's snapshot AVD until this long without a boot
FLEET_CLEANUP_AVD_KEEP_GB	30	Snapshot AVDs in all; over it, least recently used snapshots go
FLEET_CLEANUP_EMU_ORPHAN_MIN_S	300	Age at which an unowned CI emulator is stopped

Host-wide steps, only while no job is building: DerivedData directories older than 7 days, simulators whose runtime is no longer installed, and stale Playwright __dirlock entries older than 6 hours when no browser install is running.

Per-runner steps, skipping only the runners that are building: each runner's own ci-<runner> <platform> <version> simulator (the kit's ios-ci creates one per runner, and recreates it if it is gone) is erased once it is shut down and past the size limit, and a ci- device whose runner no longer exists is deleted. Under pressure, idle runners' Playwright browsers in _work/_tool/ms-playwright are removed; the next job that needs them downloads them again.

Runners' Android emulators, first of the deleting steps. A plain ci-<runner> (or shots-<runner>) AVD, which android-emulator-runner recreates every job, goes once its runner is idle and it is 30 minutes untouched. A snapshot AVD, ci-<runner>-a<api>-<target>-<profile> (the launch kit's emulator_snapshot), is kept while its runner exists and has booted it in the last FLEET_CLEANUP_AVD_KEEP_DAYS: deleting it would make that runner's next job boot three times instead of once. Over FLEET_CLEANUP_AVD_KEEP_GB, idle runners' snapshots (snapshots/, not the AVD) go least recently used first; below the admission floor every idle runner's snapshot goes before anything else. A building runner's AVD, one an emulator has open, and one written in the last 30 minutes are never touched.

Every tick, even a quiet --auto one: a CI emulator (-avd ci-…/shots-…) whose parent is launchd, older than FLEET_CLEANUP_EMU_ORPHAN_MIN_S, whose runner runs no job (no Runner.Worker under its directory) or no longer exists is stopped as hooks/emulator-control.sh does it.

Always: runner _diag logs older than 14 days and runner versions older than the linked one (a self-update leaves the previous pair behind). It prints free disk before and after.

"Building" is a runner GitHub reports busy that the admission hook is not holding, or a runner holding a live admission slot. A lock keeps two runs from overlapping; one older than 2 hours is taken over.

What it refuses to do:

• Nothing host-wide while any runner is building. Deleting DerivedData out from under a live xcodebuild produces a failure that looks like a code problem and is not reproducible afterwards.
• It never touches a simulator outside CI. Only devices named ci-… are erased, only while shut down, and never runs simctl shutdown all or kills CoreSimulatorService. simctl delete unavailable removes only devices whose runtime is already gone.
• It never touches a runner's checkouts or build caches in _work. Wiping them makes every job re-clone and recompile from cold. The one exception is the Playwright tool cache, under pressure only.

Dry run is the default because the build host is usually somebody's laptop too, and everything above is shared with their interactive Xcode.

cleanupctl.sh

Installs the LaunchAgent that runs cleanup.sh --apply --auto every FLEET_CLEANUP_INTERVAL seconds (900), so the disk is reclaimed before the admission floor holds the fleet. Source: cleanupctl.sh.

./cleanupctl.sh install     # write the LaunchAgent and load it
./cleanupctl.sh status      # is it loaded, and what did the last run say
./cleanupctl.sh logs [n]    # tail the cleanup log (default 60 lines)
./cleanupctl.sh restart
./cleanupctl.sh uninstall
./cleanupctl.sh run         # one cleanup.sh --apply --auto in the foreground

Subcommand	What it does
install	Write <FLEET_LABEL_PREFIX>.runner-cleanup.plist, bootstrap it into gui/<uid>, and report status.
uninstall	Boot the agent out and delete the plist.
start / stop	Bootstrap or boot out the agent.
restart	Boot out, bootstrap, and report status.
status	The default. launchd state and run count, interval, pressure line, last run's free-disk lines.
logs [n]	Tail the cleanup log. Default 60 lines.
run	One cleanup.sh --apply --auto in the foreground with the agent's PATH.

The plist has no RunAtLoad; the first tick comes one interval after login. cleanup.sh keeps its own lock, so ticks never overlap.

preflight.sh

Checks a prospective runner host against what the fleet's workflows actually assume. Every check corresponds to something a workflow does, so a miss is not a warning — it is a red build later, usually with a diagnostic that points somewhere other than the real cause. Source: preflight.sh.

./preflight.sh            # check only what the workflows actually need
./preflight.sh --all      # check everything this script knows how to check
./preflight.sh --explain  # show what was inferred, and stop

Flag	Effect
--all	Skip inference and run every check.
--explain	Print what was inferred from the workflows, then exit 0 without checking.
-h, --help	Print the usage header.

An unknown argument exits 2. Any miss exits non-zero.

Which checks apply is inferred from the workflow YAML the dashboard has already fetched into SQLite, read via scripts/infer-checks.py. Only files mentioning self-hosted are consulted: a job pinned to ubuntu-latest runs on GitHub's hardware and implies nothing about this Mac. With no readable database it says so and checks everything, which is the safe direction to fail in.

Plain CLI tools — shellcheck, jq, make, deno, ruby — are listed once in scripts/cli-tools.txt and checked by scripts/check-tools.sh on the PATH register.sh gives each runner, not the caller's. Tools marked always are checked whatever the inference says, because workflows reach them through a Makefile or script the YAML never shows: a repo's make lint failed on a host with make: shellcheck: No such file or directory. Tools marked infer are checked when a self-hosted workflow runs them outside a comment. Add a line there when a workflow starts depending on a new one.

Preflight also warns, without failing, when the runners' python3 is PEP 668 "externally managed" (Homebrew Python): a workflow's bare pip install fails there, so it must install into a venv.

Variable	Effect
FLEET_DB	The dashboard database to infer from. Defaults to dashboard/fleet.db beside the script.

What it refuses to do:

• It installs nothing. It reports and exits; every miss comes with the command that would fix it.
• It will not eval the inference output. The KEY=value lines from infer-checks.py are filtered against a known key list and a value pattern before being sourced, so a tampered or buggy script cannot run arbitrary code.

The Node check is unconditional rather than workflow-derived, because the dashboard's node:sqlite requirement is architectural. The gh check requires authentication, not merely installation. The network checks — one default route, one primary resolver, and five successful resolutions of codeload.github.com — are there because a VPN alongside the LAN makes DNS a race, and the resulting job failure names no cause and points at GitHub rather than at the network.

Runner lifecycle

scripts/join-host.sh

Joins this Mac to an existing fleet as a second runner host in one command: preflight (macOS, Node, a clone, fleet.env agent settings, the coordinator answering, and the toolchains jobs use, including every CLI tool in scripts/cli-tools.txt), then the agent, the mirrored runners and the health-repair timer. Dry run unless --apply. Start from examples/fleet.env.second-host; see Federation. Source: scripts/join-host.sh.

scripts/join-host.sh                                  # check everything, change nothing
scripts/join-host.sh --apply --only peer-api,widget-web # pilot a few repos
scripts/join-host.sh --apply --install-tools          # also brew-install what is missing

--install-tools installs the missing tools that have a Homebrew formula (gh, jq, node, openjdk@21, and from cli-tools.txt shellcheck, deno). Tools macOS ships, such as make, are reported with their own fix instead.

scripts/mirror-runners.sh

Registers runners on this host that copy the coordinator's, one per distinct label set a repo's runners carry, so GitHub can hand that repo's jobs to either host. Skips label sets that need a label this host does not advertise (FLEET_HOST_LABELS), Simulator runners when there is no Xcode, and repos in FLEET_MIRROR_SKIP_REPOS; runners already here are left alone. --tokens-from reads owner/repo token lines, for registration tokens minted on a machine whose gh works (over SSH it cannot read the keychain). Source: scripts/mirror-runners.sh.

scripts/mirror-runners.sh                    # the plan
scripts/mirror-runners.sh --apply --only peer-api
while read -r repo; do printf '%s %s\n' "$repo" "$(gh api -X POST repos/$repo/actions/runners/registration-token --jq .token)"; done < repos.txt \
  | ssh build-mac-2 'cd ~/actions-runners && scripts/mirror-runners.sh --apply --tokens-from -'

scripts/deregister.sh

Removes one runner from this host completely: stops the service, uninstalls its LaunchAgent, deregisters it from GitHub, deletes its directory. Source: scripts/deregister.sh.

scripts/deregister.sh <dir-name>                    # say what would happen
scripts/deregister.sh <dir-name> --apply            # do it
scripts/deregister.sh <dir-name> --apply --force    # ignore the last-runner guard

Flag	Effect
--apply	Perform the removal. Without it, the four commands it would run are printed in order.
--force	Allow removing the only runner a repo has.
-h, --help	Print the usage header.

The argument is the runner's directory name under the fleet root — app-ios, or app-ios-2 for a second instance — which is what register.sh created and what the dashboard shows. RUNNER_TOKEN may supply the removal token when gh cannot mint one locally.

This replaced teardown.sh, and the replacement is the whole point. teardown.sh had one selector, --keep <dir,dir,…>: to remove a single runner you had to name every other runner, and an empty or mistyped keep-list removed the entire fleet. The dashboard had to reconstruct the keep-list on every call to work around it. Naming the target directly cannot fail that way — the worst typo removes nothing, because the name will not match a directory.

What it refuses to do:

• It refuses an argument that is not a plain directory name. Anything containing /, or . or .. on its own, is rejected outright, so no argument can escape the fleet root however the script is called.
• It refuses a runner that is executing a job. Mid-job is the one state where this is actively destructive: the job dies and GitHub reports it as a lost runner rather than as somebody's decision, which reads like an infrastructure fault to whoever finds the red build.
• It refuses to leave a repo with no runner unless --force is passed. It counts siblings first and prints the count. This is why the autoscaler only ever removes second and later instances.
• It will not delete the directory if deregistration failed. If no remove-token can be minted, or config.sh remove fails, the directory is left in place and the command to retry is printed. Deleting it would orphan the registration on GitHub, where it shows as an offline runner that nothing on this host can remove.

scripts/drain-runner.sh

Marks one runner as draining or drained, and stops or restarts its service. Source: scripts/drain-runner.sh.

scripts/drain-runner.sh <dir-name>            # show current drain state
scripts/drain-runner.sh <dir-name> --drain    # stop if idle, mark draining if busy
scripts/drain-runner.sh <dir-name> --stop     # stop the service immediately
scripts/drain-runner.sh <dir-name> --resume   # clear the marker and start the service

Flag	Effect
(none)	Print the drain state and change nothing.
--drain	Idle: write a drained marker and stop the service. Busy: write a draining marker plus a .drain-stop flag, and let the completion hook stop it after the job.
--stop	Write a drained marker and stop the service now, busy or not.
--resume	Remove both markers and start the service.
-h, --help	Print the usage header.

Not dry-run by default, unlike the destructive scripts: every state it writes is undone by --resume. Note that with no flag at all it only reports — draining is always something you ask for explicitly.

It refuses an argument that is not a plain directory name, and a directory with no .runner file, on the same reasoning as deregister.sh.

Two behaviours are worth knowing. A drained runner is not deregistered from GitHub: it stays on the roster and continues to show as online until it misses enough heartbeats, because GitHub has no temporary-disable API. And draining a busy runner depends on ACTIONS_RUNNER_HOOK_JOB_COMPLETED being configured — without the hook the intent is recorded and the dashboard shows it, but the service is not stopped until someone drains the runner again while it is idle. See Operations.

scripts/drain-stop-when-idle.sh

Stops a runner's service once it is no longer executing a job. Source: scripts/drain-stop-when-idle.sh.

scripts/drain-stop-when-idle.sh <runner-dir> [timeout-seconds]

Two positional arguments, no flags. The timeout defaults to 3600 seconds.

Called detached by hooks/job-completed.sh when a .drain-stop flag is present, and never in the foreground from a hook. The reason is specific: the completion hook runs as a child of Runner.Worker, and the worker has not yet reported the job's result when the hook runs. svc.sh stop unloads the LaunchAgent, tearing down the listener and the worker under it — so stopping from inside the hook kills the process that was about to say "this job passed", and the job shows on GitHub as a lost runner: a red build whose steps all succeeded, caused by the drain rather than by the code.

Because it starts under the worker, it looks for the worker with pgrep -a: macOS pgrep leaves the caller's ancestors out of its match otherwise, and the worker is one. Without -a the first poll found no worker and stopped the service mid-hook, which left the job in_progress on GitHub until the run was force-cancelled (fixed 2026-10-13; see cockpit why's orphaned jobs). It ignores SIGTERM around the stop, since unloading the LaunchAgent signals the runner's whole process group, this script included.

What it refuses to do:

• It will not stop a runner whose drain was cancelled. The .drain-stop flag is checked before the wait and again after it, so a --resume during the wait wins.
• It will not wait forever. On reaching the timeout it exits, leaving the drain marker showing draining and the flag in place so the next completed job tries again. A wedged job is exactly the sort of thing someone is draining a runner to deal with.

On success it stops the service, removes the flag, and rewrites the marker from draining to drained, which is what the dashboard reads to stop showing the operation as in progress.

scripts/ephemeral-runner.sh

Registers a one-job runner in a fresh directory, waits for the job, and then removes the directory. Source: scripts/ephemeral-runner.sh.

scripts/ephemeral-runner.sh owner/repo [extra-label]           # show what it would do
scripts/ephemeral-runner.sh owner/repo [extra-label] --apply

Flag	Effect
--apply	Actually create the runner and wait for a job. Without it, the plan is printed and nothing is created.
-h, --help	Print the header comment as help.

The repo is positional; any other non-flag argument is taken as the extra label. An unknown --prefixed argument exits 1.

GitHub's --ephemeral makes the runner take exactly one job and then deregister itself. GitHub removes the registration; nothing removes the directory, which is the part that matters — a fleet that creates ephemeral runners and never cleans up leaves a 121 MB unpacked runner plus a full checkout behind for every job it ran. So this script owns the whole lifecycle and runs in the foreground: a backgrounded version cannot clean up after itself if the caller goes away.

The runner version and checksum are read out of register.sh rather than duplicated, and the tarball comes from the same cache the persistent runners use.

What it refuses to do:

• It refuses a repo that is not owner/name.
• It refuses to delete a path that does not look like one it created. The EXIT/INT/TERM trap only removes directories under .ephemeral/, and says so if handed anything else.
• It does not use --replace. An ephemeral runner's name is unique by construction, and --replace on a colliding name would evict a runner somebody else is using.

Before deleting, the trap tries to remove the registration from GitHub, so an interrupted run does not leave a permanently offline runner for someone to clean up by hand.

Not for routine CI: unpacking the runner and cloning from scratch costs minutes per job. See Ephemeral runners.

scripts/reap-ephemeral.sh

The backstop for ephemeral directories nothing owns any more. Source: scripts/reap-ephemeral.sh.

scripts/reap-ephemeral.sh                          # show what would be removed
scripts/reap-ephemeral.sh --apply
scripts/reap-ephemeral.sh --apply --min-age-hours 1

Flag	Effect
--apply	Remove the candidates.
--min-age-hours <n>	Minimum directory age to consider. Default 2. Must be a whole number, or the script exits 1.
-h, --help	Print the header comment as help.

ephemeral-runner.sh already cleans up after itself, but from a trap — and a trap cannot run if the process was SIGKILLed, if the machine lost power, or if the OOM reaper took it, which on a 16 GB machine running Xcode is a real event. Each of those leaves a directory holding an unpacked runner and a full checkout that nothing else will ever remove.

What it refuses to do:

• It refuses a directory with a live process in it. This is the check that makes the script safe to run from a timer: without it, a sweep during a 40-minute build would delete the build.
• It refuses a directory younger than the minimum age. Registration takes a while, and there is a window where the directory exists and run.sh has not started yet.
• It refuses to delete a path outside .ephemeral/.
• A missing .ephemeral directory is not an error — it exits 0 saying nothing has run an ephemeral runner on this host.

When a reaped directory still has a .runner file, its registration is removed from GitHub first, so it does not linger there as a permanently offline runner. Scheduling is covered in Ephemeral runners.

Hooks

scripts/install-hooks.sh

Points every registered runner at the fleet's job hooks by rewriting the ACTIONS_RUNNER_HOOK_JOB_STARTED and ACTIONS_RUNNER_HOOK_JOB_COMPLETED lines in each runner's .env. Source: scripts/install-hooks.sh.

scripts/install-hooks.sh                    # show what would change
scripts/install-hooks.sh --apply            # write it
scripts/install-hooks.sh --apply --restart
scripts/install-hooks.sh --remove --apply

Flag	Effect
--apply	Write the changes. Without it nothing is written.
--remove	Strip the hook lines instead of adding them.
--restart	Stop and start each changed runner's service, so the new .env is definitely picked up.
-h, --help	Print the header comment as help.

An unknown argument exits 1. Runners already pointing at the right hooks are counted as skipped rather than rewritten.

The hooks serve two features that behave differently, and the script says so before the dry-run exit rather than after it, because an operator planning a rollout is exactly who needs to know:

• Job admission is inert until FLEET_ADMIT_MODE is set in fleet.env. Rolling a hook out to a fleet and turning it on are two decisions, and the second one is the one that can cost a build.
• Draining a busy runner needs the completion hook and works as soon as it is installed, with no opt-in. It does nothing until somebody asks for a drain, and only acts on the runner they asked about.

What it refuses to do:

• It refuses to restart a runner that is mid-job. Those are collected and named at the end, with the suggestion to re-run --restart when they are idle.
• It will not leave a truncated .env behind. Each file is rewritten via a temp file in the same directory and moved into place, because a half-written .env strips the runner's PATH and breaks every job on it.

hooks/job-started.sh

Runs before a job's first step. This is the only place in the fleet where code runs before a job is allowed to begin, which is what makes admission control possible at all. Source: hooks/job-started.sh.

Before admission, runners named in FLEET_MUTE_RUNNERS acquire a host-audio lease. The first lease remembers the Mac's current output-mute state and mutes it; a detached guardian follows the job's Runner.Worker so cancellation or a crash cannot leave the host muted indefinitely.

When FLEET_SIMULATOR_CLEANUP=1, runners named in FLEET_SIMULATOR_RUNNERS also acquire a simulator lease. The first overlapping job records every already-booted device as a baseline. After the final job ends, only devices absent from that baseline are shut down. A detached guardian releases the lease and performs the same cleanup if Runner.Worker exits without reaching the completion hook.

Nothing in the fleet throttles execution — there is no scheduler, every registered runner listens independently, and if 27 of them are offered work in the same second, 27 jobs start. The dashboard's ceiling only refuses to add a runner; it cannot stop a burst across runners that already exist. This hook can.

Behaviour by FLEET_ADMIT_MODE:

Mode	What the hook does
off (default)	Exits immediately. Anything unrecognised also means off, because a misspelled mode reading as enforce would be the most expensive interpretation of a typo.
observe	Takes a slot so the count stays honest, never delays anything, and logs whether enforcing would have held this job.
enforce	Queues eligible waiters FIFO until there is room. At FLEET_ADMIT_MAX_WAIT_S, host and disk limits either admit or continue holding according to FLEET_ADMIT_TIMEOUT_ACTION; the Simulator limit remains strict.

A job is held when an older eligible waiter has priority, the live slot count has reached FLEET_ADMIT_MAX_CONCURRENT, a matching Apple runner has reached FLEET_ADMIT_SIMULATOR_MAX_CONCURRENT, or free disk is below FLEET_ADMIT_MIN_FREE_DISK_GB. A Simulator waiter blocked on that narrower limit does not prevent an unrelated job from using otherwise-free host capacity. The Simulator limit requires matching FLEET_SIMULATOR_RUNNERS patterns; zero or no matches disables it. While held, the hook polls GitHub every FLEET_ADMIT_CANCEL_POLL_S; a completed or cancelled run returns immediately so its assigned runner does not remain trapped in Set up runner. Every decision is appended as one NDJSON line to dashboard/logs/admission.ndjson, which fleetd ingests on its slow tick — a log file rather than a direct SQLite write, because the daemon holds that database open and a second writer appearing from inside a CI job is a race nobody wants to debug. Defaults are listed in Configuration.

What it refuses to do:

• It can never fail a job. A non-zero exit from a job-started hook fails the job before its first step, so an admission controller that errored would turn every build on the host red at once. There is no set -e and there is a trap 'exit 0' EXIT: every unexpected condition means "let the job run".
• Enforce mode keeps the limit under mutex contention. A contended mutex retries on the next poll tick instead of bypassing the concurrency count. observe mode may still claim a slot without the lock so dry-run counts stay useful during bursts.
• It will not ignore cancellation. A sleeping hook may not receive the runner's cancellation signal, so it polls the run — via gh when present, or curl with GITHUB_TOKEN / GH_TOKEN — and returns when GitHub reports it complete.
• Strict timeout is explicit. The default remains fail-open for compatibility. Hosts where overload is worse than queueing can set FLEET_ADMIT_TIMEOUT_ACTION=hold; the job's own timeout still ends the run, and cancellation polling releases the runner afterwards.

Slots are files rather than a pgrep count, because the hook is itself a child of the Runner.Worker about to run the job — five held jobs counting worker processes would each see the other four plus themselves, all conclude the host is full, and all wait out the timeout together. Each slot records the PID of the worker that owns it, so a job killed with SIGKILL has its slot reaped by the next admission instead of leaking concurrency permanently.

hooks/audio-control.sh

Internal helper used by both job hooks. Selected runners acquire one lease per job; the first lease remembers and mutes host output, and the final release restores the original state. A detached guardian releases the lease if its Runner.Worker exits before the completion hook. Source: hooks/audio-control.sh.

hooks/simulator-control.sh

Internal helper used by both job hooks when simulator cleanup is enabled. It records the devices booted before the first overlapping selected job, then shuts down only newly booted devices after the final selected job ends. Each lease has a detached Runner.Worker guardian, so cancellation and abrupt worker failure follow the same cleanup path. Source: hooks/simulator-control.sh.

hooks/emulator-control.sh

Internal helper used by the job-completed hook (complete) and sourced by cleanup.sh. It stops an Android emulator the job left running on its runner's own AVD (ci-<runner>…, shots-<runner>…) the way a quick-boot snapshot survives: the console's adb emu kill, then SIGTERM, then SIGKILL; a save-capable boot that needed SIGKILL has its snapshot removed, since it may have been mid-save. Night QA's fleet-explore-* and any other AVD are never touched. FLEET_EMULATOR_CLEANUP=0 turns the hook path off. Source: hooks/emulator-control.sh.

hooks/tests/emulator.sh   # stand-in emulators, fake adb; no real emulator is touched

hooks/shm-reap.sh

Internal helper used by both job hooks. Removes System V shared memory segments a killed Postgres cluster left behind: owned by the runner user, attached by no process, creator PID gone. Columns are read from the ipcs -m -o -p header by name, and an unreadable header removes nothing. A sweep that removed something, or left fewer than FLEET_SHM_LOW_FREE of kern.sysv.shmmni free, writes one line to dashboard/logs/shm.ndjson and one to the job log. See Shared memory variables. Source: hooks/shm-reap.sh.

hooks/tests/shm.sh   # fake ipcs/ipcrm; no real segment is touched

hooks/job-completed.sh

Runs after a job's last step. Its main job is to give back the slot job-started.sh took, so the next queued job can start. Source: hooks/job-completed.sh.

It first releases any host-audio lease for this runner. Output returns to the mute state that existed before the first selected job only after the last selected job finishes.

It then releases the runner's simulator lease. The final selected job shuts down the simulators CI introduced while preserving devices that were already booted before the selected jobs began.

It also handles drain, and does so before the admission-mode check, because a drain has nothing to do with admission and must work on a fleet that never turned admission on. It identifies the runner's directory from RUNNER_WORKSPACE (falling back to GITHUB_WORKSPACE for older runner versions), and if a .drain-stop flag is there it launches scripts/drain-stop-when-idle.sh detached and returns immediately.

The slot release records how long the slot was occupied and logs it as ran_s, which is the job's execution time as the host saw it — GitHub's own duration includes the queue.

Same rule as the started hook: it can never fail a job. A non-zero exit here marks a job failed after its steps have already succeeded, which is the most confusing possible outcome — a green build reported red by its own cleanup. The PID check on each slot means a leaked slot is reclaimed even if this hook never runs at all; this is the fast path, not the only path.

Maintenance and release

scripts/release-check.sh

Fails if anything host- or account-specific reached a tracked file. Source: scripts/release-check.sh.

scripts/release-check.sh                # every tracked file
scripts/release-check.sh --staged       # only what is about to be committed
scripts/release-check.sh --list         # print what it looks for, and stop
scripts/release-check.sh --install-hook # run --staged on every commit
scripts/release-check.sh --offline      # skip the gh lookups, check the rest

Flag	Effect
--staged	Check the staged content from the index rather than the working tree. The point of a pre-commit hook is to judge what is about to be committed, which is not always what is on disk.
--list	Print the derived patterns and exit 0.
--install-hook	Write .git/hooks/pre-commit to run --staged. The only thing this script writes.
--offline	Accept that gh could not be reached and check only what is derivable from this machine.
-h, --help	Print the usage header.

An unknown argument exits 2.

What it looks for is derived from the machine — serial number, username, LocalHostName, ComputerName, the gh login, the repos this host serves, and every repo the account owns — never from a list kept in the file. A hardcoded list of forbidden strings would go stale the moment a repo was renamed, and could not ship in a public repo without publishing the very strings it was hiding. Patterns shorter than 4 characters are dropped, as are generic CI usernames (runner, ubuntu, admin, and similar) that would otherwise match almost every line in a codebase about GitHub Actions runners. This repo's own owner and name are stripped from each line before matching, so the README's own git clone line is not reported every time.

What it refuses to do:

• It refuses to report a pass it did not actually perform. If no patterns could be derived it exits 1 rather than printing a green line.
• It refuses to run when gh could not list the account's repos, unless --offline says the caller knows. Those names are most of what it checks for, so a pass without them would be meaningless. A refusal is recoverable; a false pass is what the script exists to prevent.

Matching uses grep -F, so a repo name containing a . cannot become a wildcard that matches a name it merely resembles.

scripts/check-docs.sh

Verifies the documentation is internally consistent. Safe to run offline; no flags. Source: scripts/check-docs.sh.

scripts/check-docs.sh

Four checks, and it exits non-zero if any fails:

1. Every internal Markdown link in a tracked .md file resolves to a file that exists. Paths in .github/ are resolved from the repo root, because that is how GitHub renders them. No file is exempt. One used to be, and the exemption outlived the reason for it — a file excluded from a link check is a file whose links rot silently.
2. Every tracked script appears on this page. The check used to be the other way round, against a hand-maintained array of twelve names, which caught the direction that never breaks — a documented script is deleted — and missed the one that always does. Nine scripts had reached the tree undocumented by the time anyone looked.
3. Every FLEET_* and RUNNER_INSTANCE variable in fleet.env.example is mentioned in docs/configuration.md.
4. Example workflows target self-hosted and pin actions to a full SHA.

A short list of files are named as deliberate exceptions to check 2 — the sourced hook libraries (hooks/common.sh, hooks/free-disk.sh) and the hook test helpers under hooks/tests/ — because each is sourced by something else rather than run by a person. They are listed individually rather than matched by a pattern, so that adding one is a decision somebody makes in a diff.

There is deliberately no check that every script named on this page exists, because this page names teardown.sh, which was removed and is described here as history, and config.sh, svc.sh and runsvc.sh, which ship inside each runner directory and are never tracked. A check that has to special-case those is a check that gets silenced rather than fixed.

scripts/test-drain.sh

Shell tests for drain, resume, and unsafe-path handling. No flags. Source: scripts/test-drain.sh.

scripts/test-drain.sh

It builds a fake fleet in a temporary directory — real .runner files and a stub svc.sh that records what it was asked to do instead of talking to launchd — and drives the real scripts against it with FLEET_ROOT pointed at the temp directory. Nothing touches the live fleet.

The stub is the design: the thing worth testing is that drain-runner.sh calls stop at the right moment and does not call it at the wrong one, and a stub that appends to a log file answers exactly that, against the real script rather than a copy of its logic. Most of the assertions are about refusals — that a busy runner is not interrupted, that health.sh --repair does not revive a drained runner, that ../escape and a/b are rejected. Exits non-zero if any test fails.

scripts/test-cleanup.sh

Shell tests for cleanup.sh. No flags. Source: scripts/test-cleanup.sh.

scripts/test-cleanup.sh

It copies the real cleanup.sh into a temporary fleet with three runners and stubs df, xcrun and gh, so free disk, the simulator list and GitHub's busy runners are whatever each case says. It checks that --auto is a no-op with room to spare, that only idle, shut-down, over-size ci- devices are erased, that a building or slot-holding runner keeps its simulator and browsers, that host-wide steps wait for an idle fleet, that a dry run deletes nothing, and that two runs never overlap. Exits non-zero if any test fails.

scripts/install-ps-shim.sh

Stops the GitHub runner's orphan scan from pinning a core when a job finishes. At "Complete job" the runner checks every process that started during the job for its RUNNER_TRACKING_ID; macOS has no /proc, so it spawns ps e -p <pid> -o command once per process — hundreds per job on a busy host (ultra 2026-10-22: 81,867 spawns over 400 jobs, a p90 of 143 s per job, zero orphans ever killed). The scan cannot be switched off from a workflow. shims/ps answers exactly that call with exit 1 (the runner's own "could not read" path) and passes every other ps call to /bin/ps; this links it into the runners' first PATH directory, which takes effect on the next spawn with no restart. FLEET_PS_SHIM=off restores the scan for one shell or job.

Source: scripts/install-ps-shim.sh.

scripts/install-ps-shim.sh                  # show what would change
scripts/install-ps-shim.sh --apply          # link it
scripts/install-ps-shim.sh --remove --apply # unlink it

scripts/runner-path.sh

Gives every runner on this host the same job PATH. A job's PATH comes from the runner's .path, which config.sh fills from whatever shell registered it, so older runners differ (on runner-host: six PATHs, 17 runners with no Homebrew). register.sh now writes FLEET_RUNNER_PATH itself; this rewrites .path and .env's PATH for runners registered before that, only when idle, and restarts each. Dry run unless --apply. Source: scripts/runner-path.sh.

scripts/runner-path.sh            # which runners differ
scripts/runner-path.sh --apply    # rewrite the idle ones and restart them

scripts/test-runner-path.sh

Tests scripts/runner-path.sh against throwaway runner directories: a dry run changes nothing, --apply rewrites .path and .env and restarts only the runner that differed. Source: scripts/test-runner-path.sh.

bash scripts/test-runner-path.sh

scripts/test-health.sh

Shell tests for health.sh. No flags. Source: scripts/test-health.sh.

scripts/test-health.sh

It builds a fake fleet in a temporary directory and puts stub gh, launchctl and plutil first on PATH, then runs the real health.sh. It checks that GitHub not answering is reported but not counted as unhealthy, that a runner missing from a real answer is not-registered and is a fault, that offline is a fault and --repair restarts it, and that a dead launchd job is still a fault while GitHub cannot be asked.

scripts/test-ephemeral.sh

Shell tests for the ephemeral runner reaper. No flags. Source: scripts/test-ephemeral.sh.

scripts/test-ephemeral.sh

Same approach and the same emphasis: the reaper deletes directories, so the tests that matter are the ones proving it refuses to — a directory with a live process in it, one younger than the minimum age, one protected by a raised --min-age-hours. It also checks that ephemeral-runner.sh creates nothing without --apply and rejects a repo that is not owner/name.

scripts/infer-checks.py

Works out which of preflight.sh's checks this fleet's workflows actually need. Source: scripts/infer-checks.py.

python3 scripts/infer-checks.py <path-to-fleet.db>

One positional argument, no flags. Called by preflight.sh; rarely run by hand.

It reads workflow YAML out of the dashboard's workflow_files table, keeps only files mentioning self-hosted, and prints shell assignments — NEED_XCODE, NEED_XCODEGEN, NEED_WATCHOS, NEED_SIM, NEED_NODE, NEED_DOCKER, NEED_ANDROID, NEED_POSTGRES, PG_VERSIONS, INFERRED_FROM. Postgres versions are whatever the workflows name, never a default.

It is a separate file rather than a heredoc inside preflight.sh because /bin/bash on macOS is still 3.2, which mis-parses a heredoc inside command substitution. It opens the database read-only, since the dashboard daemon is very likely writing to it, and it exits non-zero printing nothing when there is no usable data — the caller then checks everything, which is the safe direction to fail in.

Matching is substring matching on lowercased YAML rather than a YAML parse, because the interesting facts live inside run: shell blocks, where a parser gives you one long string and no more structure than this does.

Dashboard daemons

Each of the three control scripts generates its LaunchAgent plist rather than shipping one, because the paths are user- and version-specific and a hardcoded /Users/<someone> fails silently a month later. All three take the LaunchAgent label prefix from FLEET_LABEL_PREFIX (default com.runner-fleet), read from fleet.env — changing it renames the agent, and a mismatch makes a daemon that is running perfectly well look absent.

dashboard/fleetctl.sh

Installs, runs and inspects the dashboard daemon. Source: dashboard/fleetctl.sh.

Subcommand	What it does
install	Install the locked PostgreSQL driver when HA is configured, write the LaunchAgent plist, load it, and report status.
uninstall	Unload the agent and delete the plist.
start / stop	Load or unload the agent.
restart	Unload, load, and report status.
status	The default. launchd PID and last exit status, plus the daemon's /api/health summary.
logs [n]	Tail the daemon log. Default 60 lines.
run	Run fleetd.js in the foreground, for debugging.
token	Print the control token. Exits 1 if the daemon has not generated one yet.
label	Print the launchd service label used by this host.
agent-token	Print (and create if missing) the legacy shared agent token.
agent-token --host <id>	Generate or rotate a host-scoped token; only its SHA-256 hash is retained.
leader	Print this replica's HA role and the elected leader.
pair	Create a single-use five-minute pairing code for a remote browser without exposing the master token.
devices	List paired browser devices and their last-seen times.
revoke <key-or-name>	Revoke one paired browser token.
remote status	Show bind mode, reachable URLs, the public MCP URL, Tailscale Serve state, and firewall state.
remote lan on|off	Enable or disable LAN binding and restart the daemon when running.
remote tailscale on|off	Configure or reset Tailscale Serve for the dashboard.
remote funnel on|off	Publish only the MCP listener on Tailscale Funnel port 8443, and set FLEET_PUBLIC_PORT/FLEET_PUBLIC_URL. See Chat tools.
remote cloudflare on <hostname>|off	The same through a Cloudflare Tunnel: create or reuse the fleet-mcp tunnel, route the hostname, and run it as a LaunchAgent. off stops it and keeps the tunnel.
backup	Write a consistent SQLite backup with sqlite3 .backup. Safe while the daemon is running.
restore <path>	Replace the live database from an explicit backup path. Refuses while the daemon is running.

Anything else prints the usage header and exits 1.

Database backup and restore

cd dashboard
./fleetctl.sh backup
./fleetctl.sh stop
./fleetctl.sh restore backups/fleet-20260322-143000.db
./fleetctl.sh start

Backups land in dashboard/backups/ by default (FLEET_BACKUP_DIR), are timestamped (fleet-YYYYMMDD-HHMMSS.db), and are written with mode 0600. backup uses SQLite's online .backup command so the copy is consistent even while fleetd holds the database open in WAL mode. restore requires an explicit path — there is no "restore latest" shortcut — and copies the current database aside as fleet.db.rollback-YYYYMMDD-HHMMSS before overwriting it.

What it refuses to do:

• restore while the daemon is running. A live restore against an open WAL database is how you get a corrupted store and a daemon that looks fine while every query lies. Stop the daemon first.
• restore without a backup path. Passing nothing prints usage and exits 1.

Run a restore drill after upgrades: back up, stop, restore into a throwaway copy, verify schema_migrations, then start normally. See Configuration.

Dismissing an alert has no subcommand here: it is an × on the row in the dashboard's Alerts tab, which is where you are when an alert is annoying you. See Dismissing alerts.

The plist sets KeepAlive, unlike the runner plists — health.sh exists precisely because theirs do not, and a monitoring daemon that dies quietly is worse than no monitoring daemon. FLEET_PORT (default 7878) is baked into the plist along with the resolved node path and gh's directory, because launchd gets a minimal PATH.

The control token is deliberately not served to the page: read access and the right to restart runners are different things. Paste it into the Control tab once and the browser keeps it. Note that run over SSH will fail to get a GitHub token — see the SSH note in Get started.

dashboard/agentctl.sh

Installs, runs and inspects the fleet agent daemon (agent.js) on an agent Mac. Source: dashboard/agentctl.sh.

Subcommand	What it does
install	Write the LaunchAgent plist, load it, and report status.
uninstall	Unload and delete the plist.
start / stop	Load or unload the plist.
restart	Unload, reload, show status.
status	launchd state plus whether the coordinator is reachable.
logs [n]	Tail the agent log. Default 60 lines.
run	Run agent.js in the foreground, for debugging.

Requires FLEET_COORDINATOR, FLEET_AGENT_TOKEN, and FLEET_HOST_NAME in fleet.env before running install. The agent token is written to a mode-0600 file and referenced from the plist rather than embedded in EnvironmentVariables where it would be readable by any local user via defaults read.

See Federation for the full multi-host setup guide.

dashboard/host-token.mjs

Generates or rotates a host-scoped fleet-agent token and prints the plaintext token once. The token store keeps only its SHA-256 hash. fleetctl.sh agent-token --host <id> is the normal operator-facing wrapper. Source: dashboard/host-token.mjs.

node dashboard/host-token.mjs <stable-host-id>

dashboard/bin/fleet.mjs

The fleet command line: /api/v1 from a shell, in one file with no dependencies. Its commands are read from the daemon's /api/v1/openapi.json on each run, one per route. FLEET_URL and FLEET_TOKEN say where and as whom. See API Reference. Source: dashboard/bin/fleet.mjs.

fleet status
fleet why acme/comet-web
fleet wait acme/comet-web --pr 96 --timeout 30m
fleet run runner.drain --name build-host-comet-web-2 --yes

dashboard/scripts/migrate-sqlite-to-pg.mjs

One-shot migration that streams every SQLite table into PostgreSQL as a checksummed JSONB archive. It reads FLEET_DB and requires FLEET_DATABASE_URL; the source database is opened read-only and rerunning the command replaces each table's archived rows transactionally. Source: dashboard/scripts/migrate-sqlite-to-pg.mjs.

cd dashboard
FLEET_DATABASE_URL=postgres://... node scripts/migrate-sqlite-to-pg.mjs

dashboard/scripts/scrub-snapshot.mjs

Turns a live /api/state snapshot into a test fixture that is safe to commit. Every string passes through a map of real-to-replacement names that you keep outside the repository; free text people wrote (commit messages, run titles, actors, alert bodies) is replaced wholesale. It refuses to write the fixture if a home directory, an e-mail address or a GitHub URL for another owner survives. Source: dashboard/scripts/scrub-snapshot.mjs.

cd dashboard
curl -s localhost:7878/api/state > /tmp/live.json
node scripts/scrub-snapshot.mjs /tmp/live.json test/fixtures/snapshots/new.json --map ~/scrub-map.json

dashboard/scripts/settle-stale-jobs.mjs

One-off repair for job rows that froze before their run completed (no conclusion, completed_at or duration under a completed run). The daemon now settles these itself through lib/settle.js, a few hundred per backfill pass; this clears the existing history in one go. fetch runs wherever gh is signed in (over SSH the runner host's gh cannot read its keychain), re-fetches each job by id (one worker unless --concurrency says otherwise), reads the rate limit from each response's X-RateLimit-* headers and sleeps to the reset below --floor (default 2500, because the quota is per user and the daemon shares it), backs off on the secondary rate limit, and resumes from its output file. sql emits updates guarded by conclusion IS NULL, so it never overwrites a row the daemon settled meanwhile. Source: dashboard/scripts/settle-stale-jobs.mjs.

cd dashboard
ssh ultra "/usr/bin/sqlite3 -json ~/actions-runners/dashboard/fleet.db \"$(node scripts/settle-stale-jobs.mjs query)\"" > stale.json
node scripts/settle-stale-jobs.mjs fetch --in stale.json --out fetched.ndjson
node scripts/settle-stale-jobs.mjs sql --in fetched.ndjson | ssh ultra /usr/bin/sqlite3 ~/actions-runners/dashboard/fleet.db

dashboard/scripts/mirror-plan.mjs

Prints the plan scripts/mirror-runners.sh acts on, one tab-separated line per runner (register, present or skip with its reason). The logic is dashboard/lib/mirror.js; host facts come from fleet.env and Xcode is detected by running xcodebuild -version. Source: dashboard/scripts/mirror-plan.mjs.

node dashboard/scripts/mirror-plan.mjs --coordinator http://coordinator-mac:7878 --root ~/actions-runners

dashboard/scripts/make-glance-fixtures.mjs

Writes one /api/glance payload per scenario in dashboard/test/fixtures/scenarios.js into cockpit/Sources/CockpitCore/Fixtures/, so the macOS cockpit is tested against exactly what the daemon produces for each recorded incident. It also writes Fixtures/v1/, a set of /api/v1 answers made by the route handler. --check exits non-zero when any fixture is out of date; CI runs it. Source: dashboard/scripts/make-glance-fixtures.mjs.

node dashboard/scripts/make-glance-fixtures.mjs

dashboard/scripts/make-skill.mjs

Fills the generated blocks of cockpit/skills/fleet-cockpit/SKILL.md (the table of ways in, the order to ask in, the verdict table and the rules) from dashboard/lib/guide.js. The MCP server's instructions come from the same functions. The prose between the blocks is edited by hand. --check exits non-zero when the skill is out of date; CI runs it. Source: dashboard/scripts/make-skill.mjs.

node dashboard/scripts/make-skill.mjs

scripts/host-sentinel.sh

Runs on a machine other than the runner host and posts to ntfy when the host stops answering for SENTINEL_FAILS consecutive minutes, and again when it is back — the one failure the host cannot report itself. Configuration is ~/.config/fleet-sentinel.env; --install writes a LaunchAgent that probes every 60 seconds, --test-notify sends a test, --uninstall removes it. A message that could not be delivered is retried on the next probe. Source: scripts/host-sentinel.sh.

scripts/test-sentinel.sh

Tests for host-sentinel.sh with a scripted probe and delivery: announce after the threshold, once; announce recovery, once; retry an undelivered message. Source: scripts/test-sentinel.sh.

install.sh

One-command setup for a coordinator or agent host. Runs preflight checks, then delegates to fleetctl.sh install or agentctl.sh install. When PostgreSQL HA is configured, fleetctl.sh installs the locked database driver. Source: install.sh.

./install.sh coordinator   # install the dashboard daemon on this machine
./install.sh agent         # install the fleet agent on this machine

Built-in checks: macOS 14+, arm64 (warns if not), Node.js 22.5+, git clone present, gh authenticated (coordinator only), and required fleet.env variables set.

dashboard/watch/watchctl.sh

Installs, runs and inspects the fleet watchdog. Source: dashboard/watch/watchctl.sh.

Subcommand	What it does
install	Write the LaunchAgent plist, load it, and report status.
uninstall	Unload the agent and delete the plist.
start / stop	Load or unload the agent.
restart	Unload, load, and report status.
status	The default. launchd state, the last log line, and how many faults have been reported.
logs [n]	Tail the watch log. Default 60 lines.
problems	Print only the FLEET_PROBLEM, FLEET_UNREACHABLE and FLEET_RECOVERED transitions. Says so explicitly when there are none.
run	Run fleet-watch.mjs in the foreground, for debugging.

Its node lookup deliberately tries /opt/homebrew/bin/node and /usr/local/bin/node before command -v node, which is the opposite of what fleetctl.sh does. Whoever runs the installer probably has nvm active, and nvm's node path embeds its version — baking that into a plist means the watchdog stops surviving reboots the day that version is uninstalled, silently, months later, which is the failure mode a watchdog least wants.

Its ThrottleInterval is 30 rather than 10: if it cannot start, retrying twice a minute only fills the log faster than anyone will read it.

Set WATCH_NOTIFY=1 in fleet.env before install to show local macOS notifications for sustained FLEET_PROBLEM/FLEET_UNREACHABLE transitions and their recovery. Logging remains enabled regardless.

See also

• Configuration — every variable these scripts read
• Operations — when to run which of them
• Admission control and autoscaling — what the hooks are for
• HTTP API — the endpoints and actions behind the dashboard