willet

A daemon that manages a GitHub Actions runner scale set and runs every job in a fresh, ephemeral microsandbox microVM.

It works like actions-runner-controller, but runs on a single Linux host with KVM instead of Kubernetes. Each job gets hardware isolation, with its own guest kernel, instead of a container. willet is new (v0.x), so settings may still change between minor versions.

GitHub ──long poll──▶ willet ──runner config──▶ microVM (ghcr.io/gerritlansing/willet-runner)
                         │                           └─ run.sh: one job, then exits
                         └─ destroys the VM when the runner exits

Requirements

• Linux on amd64 or arm64 with KVM (/dev/kvm) and glibc 2.34 or later (Ubuntu 22.04, Debian 12, RHEL 9 or newer).
• To build from source instead: Go 1.27.3 or later, with CGO and a C toolchain.
• The microsandbox runtime, which the daemon installs into ~/.microsandbox on first start, next to its image cache. On every start it updates an older runtime there to the version it was built for. It never downgrades a newer one, and it leaves alone a runtime selected with MSB_PATH.
• A GitHub App (recommended) or classic personal access token that can manage self-hosted runners for the target repository or organization. Enterprise-level runners need a classic token, because GitHub Apps can't register them. See GitHub's docs for the permissions.

Quick start

Download a release and, optionally, check that it was built by this repository's release workflow:

v=0.1.1 arch=amd64   # or arm64
curl -LO https://github.com/gerritlansing/willet/releases/download/v$v/willet_${v}_linux_$arch.tar.gz
gh attestation verify willet_${v}_linux_$arch.tar.gz --repo gerritlansing/willet
tar -xzf willet_${v}_linux_$arch.tar.gz && cd willet_${v}_linux_$arch

To build from source instead, run make build; the binary is bin/willet.

Create the configuration:

cp willet.env.example willet.env
chmod 600 willet.env

Edit willet.env: set WILLET_URL, WILLET_NAME and either the three WILLET_APP_* settings or WILLET_TOKEN (comment out the WILLET_APP_* lines). Start the daemon:

./willet --env-file willet.env

The first start installs the microsandbox runtime and pulls the runner image (about 1 GB). It's ready when it logs Listening for jobs. To try it, add this workflow to the repository and run it from the Actions tab:

# .github/workflows/willet-test.yml
on: workflow_dispatch
jobs:
  hello:
    runs-on: willet   # WILLET_NAME, or one of WILLET_LABELS if set
    steps:
      - run: echo "Hello from $RUNNER_NAME"

Configuration

willet.env.example documents every setting. Each is an environment variable WILLET_<NAME> or a flag --<name> (WILLET_MAX_RUNNERS, --max-runners); flags override the process environment, which overrides the env file given with --env-file or WILLET_ENV_FILE. Unknown WILLET_* keys in the env file stop startup, so a typo can't silently fall back to a default.

Sizing: WILLET_MAX_RUNNERS caps VMs in any state, booting and teardown included. Size the host for WILLET_MAX_RUNNERS × WILLET_MEMORY plus one VM for image refreshes.

Runner image and build environment

VMs boot the willet runner image, ghcr.io/gerritlansing/willet-runner:latest, unless you set WILLET_RUNNER_IMAGE. It is GitHub's runner image, ghcr.io/actions/actions-runner, plus iptables for Docker and the tools actions and scripts commonly expect: Docker Compose, gh, git-lfs, yq, build-essential, Python's pip, venv and pipx, zstd and everyday utilities; see images/runner. Language toolchains are left to setup-* actions.

The image is rebuilt when GitHub releases a new runner version, for amd64 and arm64, and tagged three ways:

• 2.337.0-1: one build, never changed. The number after the dash counts changes to the image's recipe.
• 2.337.0: the newest build of that runner version.
• latest: the newest build.

Jobs can install tools at run time with setup-* actions or sudo apt-get install, but every job starts in a fresh VM and repeats the download. For tools most jobs use, build them into an image once:

FROM ghcr.io/gerritlansing/willet-runner:latest
USER root
RUN apt-get update \
 && apt-get install -y --no-install-recommends python3-venv \
 && rm -rf /var/lib/apt/lists/*
USER runner

Push it to a registry and point WILLET_RUNNER_IMAGE at it.

A custom image needs a working actions runner installation in WILLET_RUNNER_DIR (default /home/runner), with run.sh executable by WILLET_RUNNER_USER (default runner), and, with Docker on, Docker's requirements. Building FROM the willet image covers all of this. At startup the daemon boots the image once and checks the runner, Docker and the network, so a broken image or unreachable server fails at start, not on the first job.

For a private registry, set WILLET_REGISTRY_USERNAME and WILLET_REGISTRY_PASSWORD_FILE (for GHCR, a token with read:packages). The credentials are used on the host for the pull only and never reach a VM.

Docker in jobs

container: jobs, service containers and Docker container actions work out of the box: each VM runs its own Docker daemon, as GitHub-hosted runners do, with its data on a separate sparse disk (WILLET_DOCKER_DISK, default 20 GiB). Set WILLET_DOCKER=false to turn it off.

The image must include dockerd and iptables. The willet image has both; GitHub's stock image lacks iptables, so add it or turn Docker off.

Every VM starts with an empty Docker cache, so each job downloads its images again, and pulls from Docker Hub count against its anonymous rate limit for your host's IP address. A pull-through cache on the host fixes both. Run one bound to loopback:

docker run -d --name docker-hub-mirror --restart always -p 127.0.0.1:5000:5000 \
  -v docker-hub-mirror:/var/lib/registry \
  -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:3

and point willet at it with WILLET_DOCKER_REGISTRY_MIRROR=http://host.microsandbox.internal:5000. VMs reach the host by that name, and willet allows only that port. Docker mirrors cover Docker Hub images only. If the cache is down, Docker falls back to Docker Hub, and the daemon warns at startup.

Keeping the runner up to date

Ephemeral runners don't update themselves, and GitHub stops sending jobs to a runner more than 30 days behind the latest release. New runner versions come out every 3–7 weeks.

• Tagged image (the default, :latest): every WILLET_IMAGE_REFRESH_INTERVAL (default 24h) the daemon re-pulls the tag and starts new runners from the digest it now resolves to. Running jobs keep their image, old images are removed once unused, and a failed refresh keeps the last good image. The willet image's latest follows each runner release; a tag of your own only moves when you rebuild and push it.
• Digest-pinned image (repo@sha256:…): used exactly as given and never refreshed; updating it within the 30 days is up to you.

Private networks and GitHub Enterprise Server

Runner VMs can reach the public internet and DNS. Private ranges, loopback, link-local and cloud metadata addresses, and the host itself are blocked (microsandbox network security). To let jobs reach a GitHub Enterprise Server (GHES), package mirror or other private service, allow its addresses:

WILLET_NETWORK_ALLOW=10.0.5.10,10.20.0.0/16:443

Hosts reached by name need their address allowed without a port, because microsandbox's DNS rebinding protection drops private DNS answers unless an entry covers the address on all ports. For ghes.example.com at 10.0.5.10:

WILLET_NETWORK_ALLOW	WILLET_DNS_REBIND_PROTECTION	Jobs can reach
10.0.5.10	true (default)	every port on 10.0.5.10
10.0.5.10:443	false	only port 443 on 10.0.5.10

Either way, connections stay limited to the public internet and your allow list. Turning rebinding protection off gives up a defence against a malicious public domain resolving to one of your allowed private addresses.

If a VM can't reach the server in WILLET_URL, the startup check stops the daemon and says what to allow.

Running as a service

Create a service user in the kvm group, install the binary and configuration, and add a systemd unit:

sudo useradd --system --create-home --groups kvm willet
sudo install -m 755 willet /usr/local/bin/   # bin/willet if built from source
sudo install -m 600 willet.env /etc/willet.env

# /etc/systemd/system/willet.service
[Unit]
Description=GitHub Actions scale set on microsandbox
After=network-online.target
Wants=network-online.target

[Service]
User=willet
EnvironmentFile=/etc/willet.env
StateDirectory=willet
ExecStart=/usr/local/bin/willet
Restart=always
# Restart slowly after a failure instead of hitting systemd's start rate limit.
RestartSec=30
# The daemon's own shutdown takes at most 60s; this leaves headroom.
TimeoutStopSec=90

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now willet
sudo journalctl -u willet -f   # wait for "Listening for jobs"

Stopping or restarting willet interrupts running jobs: it destroys every VM rather than waiting for jobs to finish.

• EnvironmentFile is read by systemd as root, so /etc/willet.env can stay root-owned with mode 600. The daemon itself reads the GitHub App private key file and any registry password file, so the service user must be able to read those.
• StateDirectory= creates /var/lib/willet for the single-instance lock.
• The service user's ~/.microsandbox holds willet's runtime, images and VMs. When willet updates the runtime, anything else running as that user gets the new version too. To give willet its own runtime under a shared user, set MSB_HOME to a directory of its own in /etc/willet.env.

How it works

• Startup: the daemon takes a per-scale-set lock, so a second daemon for the same scale set on the host exits. It pins the runner image, checks it and the network from a test VM, gets or creates the scale set, opens GitHub's message session, and then removes its own VMs left over from a crash.
• Scaling: the target is min(WILLET_MAX_RUNNERS, WILLET_MIN_RUNNERS + assigned jobs). For each missing runner, the daemon requests a single-use runner config from GitHub, boots a VM (about 300 ms once the image is cached, plus about a second to start Docker), and starts run.sh.
• After a job: the ephemeral runner exits and its VM is destroyed, disk and all. A runner still alive two minutes after its job completed is destroyed anyway. Surplus idle runners are removed.
• Failures: a VM keeps its capacity slot until it is confirmed destroyed, and failed teardowns are retried, so the host is never oversubscribed.
• Shutdown (SIGINT/SIGTERM): the daemon destroys all VMs, deregisters idle runners and closes the GitHub session (with WILLET_DELETE_ON_EXIT, it also deletes the scale set), giving up on any step still running after a minute. VMs it couldn't destroy are removed at the next start; GitHub expires runners left behind.

Limitations

• One host per scale set. GitHub allows one message session per scale set, so a second host using the same scale set waits, with a warning, and does nothing. Give each host its own scale set. The same wait happens briefly after a crash, until GitHub expires the old session.

Contributing

Report bugs and ask questions in issues. See CONTRIBUTING.md for development setup and tests.

Security

Report vulnerabilities privately as described in SECURITY.md.

License

MIT

willet is an independent project, not affiliated with or endorsed by GitHub or microsandbox.