[addisdev]

Why ultra keeps falling under its 100 GB floor

Measured on ultra on 2026-10-22, read-only:

• Trend (hmap disk_usable_gb): 225 GB usable right after the 10-17 reclaim, down to about 112 GB on 10-22. That is roughly 20 GB a day net. On top of it, free space moves 15–25 GB within an hour while jobs run (cleanup log: 59–84 GB plain and 97–128 GB usable across 10-22 evening ticks).

• What's new since 10-17 (files created since 10-17 16:00 that still exist, by allocated size): about 100 GB in total.

  • ~/Library/Developer (simulators): 22.4 GB
  • ~/.android/avd: 20.5 GB
  • offsite-outbox: 10.0 GB
  • Colima: 9.7 GB
  • ~/.gradle/caches: 6.1 GB
  • /private/var/folders: 5.3 GB
  • runner _work: about 5 GB

  So CI devices (simulators and AVDs) are the largest thing growing, and cleanup.sh reclaimed almost none of them.

• The host-wide "skip while any job runs" steps are not the gap: DerivedData is 3.5 GB and no unavailable simulators were listed. The gap is in the per-runner steps, which do run on a busy fleet but missed these devices.

What this changes (all per-runner, so they run while other runners build)

1. CI simulators with an app word. Jerv's ci/sim.py names its devices ci-<runner> jerv iOS 27.0. The kit-shaped regex never matched that name, so 9 devices holding 19 GB had never been erased (one belonged to ultra-jerv-6, a runner that no longer exists). The match now allows one word before the platform.

2. Idle runners' Android AVDs (ci-<runner>, plus Jerv's shots-<runner>). Nothing removed them: 17 AVDs, 20 GB. android-emulator-runner recreates the AVD at the start of every job (force-avd-creation defaults to true), so an AVD sitting between jobs is pure waste. An AVD is removed only when all of these hold:

   • its runner is not building;
   • no emulator has it open (no live hardware-qemu.ini.lock pid and no -avd <name> process);
   • it was not written in the last 30 minutes.

   Night QA's fleet-explore-* and any other AVD are never touched.

3. Leftover _work/_update. A runner self-update leaves its 0.45 GB staging copy behind; 9 runners still had one. It is removed when it is older than 6 h, the runner is idle, and no process is running from it.

Dry run on ultra just now (not under pressure, busy runners skipped): 11 AVDs (~12 GB), ci-ultra-jerv-6 (2 GB) and one _update (0.4 GB). Under pressure, the 1 GB simulator limit also takes the Jerv devices (2–2.7 GB each).

Tests

scripts/test-cleanup.sh: two new cases (app-named sims; AVDs and _update with busy, open, fresh and foreign cases). 63 passed. shellcheck --severity=warning clean.

Not in this PR

• The floor itself. FLEET_ADMIT_MIN_FREE_DISK_GB=100 is ultra's untracked fleet.env value. It matches the Homelab Map's host floor, which exists for Dozehound's video nights (about 50 GB of renders a week). Lowering it is the owner's call.
• Runner binaries: about 91 copies of the same bin/externals (~0.45 GB each). Per-runner _tool caches: 18.8 GB. Both could be shared or cloned. That is a separate change.

🤖 Generated with Claude Code