Skip to content

ci: guest rows on demand on the guest runner; a run beside other work ends HOST NOT QUIET - #1173

Merged
Japabu merged 2 commits into
mainfrom
ci/guest-runner
Nov 4, 2026
Merged

Japabu merged 2 commits into
mainfrom
ci/guest-runner

Conversation

@Japabu

@Japabu Japabu commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The owner ruled on 2026-11-04 that ToyOS guest (QEMU) tests run only on a dedicated, otherwise idle x86 Linux host with KVM, registered as a GitHub self-hosted runner. Host checks stay on GitHub-hosted runners. Until now, agents had no shared, serialized place to run guest rows. The harness also could not tell a run on a busy host from a run on a quiet one, so host noise showed up as test reds.

The nightly is frozen by an owner ruling of the same day. This PR therefore adds only the on-demand path and leaves nightly.yml and guest.yml untouched.

Change

On-demand guest rows: .github/workflows/guest-rows.yml

Its only trigger is workflow_dispatch, and its one input is filter (optional; empty runs the whole suite). The run tests the head of the dispatched branch.

  • Runner. Runs on [self-hosted, toyos-guest]. run-name: guest-rows <branch> <filter> lets a dispatcher find its own run.
  • Refused on main (if: github.ref != 'refs/heads/main'). A job on main's ref would hold a token of main's cache scope, and main's caches are restored by every other ref.
    • Code from an earlier job can outlive that job on a persistent runner. Such code therefore must never see a main-scope token.
    • To test main's commit, dispatch from a branch that points at it.
  • No permissions and no cache action. permissions: {} and no actions/cache: the runner's own workspace is the only cache. No secret is read.
  • Checkout. clean: false keeps target/ warm. submodules: true lets a toolchain the store lacks be built from rust/. persist-credentials: false keeps the token out of .git/config.
  • One command. The suite step is cargo run -- --ci rows "$FILTER".
  • Serial logs. On failure, the run's kept serial logs are uploaded as the serial artifact. The driver first removes what earlier runs kept, so the artifact holds only this run's logs.
gh workflow run guest-rows.yml --ref <pushed branch> -f filter='virtsmp --jobs 4'
gh run list --workflow guest-rows.yml        # find it by title
gh run view <id> --log

The runner takes one job at a time, so this queue replaces the ad-hoc ssh runner, and no separate lock is needed.

cargo run -- --ci rows <args> (build/ci.rs)

This runs cargo test --test toyos-build -- <args>, with each argument split at whitespace in Rust, so the shell neither splits nor globs.

It shares guest() with the nightly's job:

  • the same instrument check per architecture: QEMU at build/ci/qemu-version, the firmware QEMU declares, and a /dev/kvm that opens;
  • the same $TMPDIR leak check.

It differs in two ways:

  • It installs no cache-restored sysroot. The toolchain comes from the host's own store, ~/.cache/toyos, which persists across jobs. A revision whose toolchain key is not in the store builds it on the runner first, which is why the timeout is 360 min.
  • It first removes target/red-run-serial.

Host-not-quiet outcome (tests/common/hostload.rs, Tally)

#1166's CPU budget is a set of admission locks and measures no load, so there was no existing measurement to reuse. The new reading follows the pattern of the suspend detector (clock.rs).

  • What it compares. Over the whole run, the host's busy CPU time against the run's own CPU time.
    • Busy time on Linux comes from /proc/stat's aggregate line. Idle and iowait are taken as one sum, because iowait alone can decrease (Documentation/filesystems/proc.rst). Steal counts as busy.
    • Busy time on macOS comes from Mach's HOST_CPU_LOAD_INFO, whose 32-bit counters wrap.
    • The run's own time is getrusage of itself plus every reaped child.
  • When it is read. After every worker has joined. By then every QEMU and build the harness started has been waited for.
  • The bound. If other work averaged more than one host CPU, the run ends test result: HOST NOT QUIET. and exits 3, whatever the rows said. The reasoning: a whole CPU that the guest budget admitted guests against was then not theirs for the run.
  • Reporting. Every run's summary prints host: other work used … of CPU beside this run, N CPUs on average.
  • Serial logs. Run::exit now keeps them on every status but green, HOST NOT QUIET included.

Security guard

only_a_dispatch_reaches_a_self_hosted_runner replaces workflows_run_against_main_on_hosted_runners, which forbade self-hosted runners. It enforces:

  • Dispatch only. A job whose runs-on is not a single GitHub-hosted image label is allowed only in a workflow whose triggers are exactly [workflow_dispatch]. Any runs-on it cannot read counts as self-hosted, so a list, a label, an expression or a multi-line value all fall under this rule.
  • No privileges. Such a workflow holds permissions: {} and no other permissions block. It uses no actions/cache, and every job's condition is exactly github.ref != 'refs/heads/main'. The reader test refuses each widened variant.
  • Local reusable workflows only. Every job-level uses: must be a local ./.github/workflows/ file.
  • One self-hosted workflow. The set of self-hosted workflows is exactly guest-rows.yml.

Instrument

build/ci/qemu-version moves from 11.1.1 to 11.1.2, the guest runner host's QEMU: the instrument moved. Consequences:

  • guest.yml's frozen container still installs 11.1.1. That is recorded in the freeze issue, whose exit now includes moving the container to 11.1.2.
  • The Mac's QEMU is 11.1.1. Upgrading it is the owner's call; aarch64 runs on the Mac need 11.1.2 to match the declaration.

Instructions and records

  • Root CLAUDE.md, tests/CLAUDE.md (a new "Where guests run" section), implementer.md and reviewer.md now say that guest tests run on the guest runner through the dispatch, and that aarch64 guests also run on a quiet Mac on demand.
    • The Mac clause is the owner's ruling of 2026-11-04: "only run arm guest tests on the mac and only if its silent". HOST NOT QUIET is what decides whether a given Mac run was silent.
    • The old "Guest tests run only on a quiet host" rule is replaced. The AArch64 rule is unchanged.
  • The freeze is recorded in issues/the-nightly-is-frozen-so-nothing-runs-the-guest-matrix.md, held by the owner, with the exit "the nightly resumes".
  • issues/the-build-runs-host-tools-outside-rust-and-qemu.md gains a row for the Actions runner agent on the guest host. Its firmware and Debian rows now say what each QEMU version is.

The owner must keep this repository setting

Settings → Actions → General → Fork pull request workflows from outside collaborators: "Require approval for all outside collaborators". The reviewer confirmed it is set (all_external_contributors).

The guard covers only this repository's own workflow files. A fork PR can add a workflow with on: pull_request and runs-on: [self-hosted, toyos-guest], and GitHub runs the PR's version of its workflows. On the free plan a runner cannot be restricted to selected workflows, so this approval is the only thing standing between such a PR and the host.

A branch in this repository is trusted already: pushing one needs write access, which is also what a dispatch needs. Its code can taint later verdicts on the persistent host. The owner accepted that by choosing a persistent runner; main's caches stay out of reach.

Runner host prerequisites

These are also stated in the workflow's header.

  • x86-64 Linux, dedicated and otherwise idle, with a /dev/kvm that the runner's user can open.
  • The Actions runner, registered for this repository with label toyos-guest, running one job at a time (the default for one runner process).
    • It runs as a dedicated user with no sudo.
    • Job code can read that user's files, the runner's .credentials among them. That is GitHub's standard self-hosted model.
  • QEMU 11.1.2 (build/ci/qemu-version), with qemu-system-x86_64 and qemu-system-aarch64 on PATH. A different version reds the instrument step.
  • QEMU's firmware descriptors for x86-64 OVMF and AArch64 edk2, in /usr/share/qemu/firmware/*.json or QEMU's data dirs. build/firmware.rs finds them through QEMU's interop spec.
  • rustup with stable on the runner user's PATH.
  • git, a C and C++ toolchain (cc, c++, ar), CMake and Python 3. These are needed only when the store lacks a revision's toolchain.
  • A persistent $HOME (the store) and a runner workspace kept between jobs.

The aarch64 rows run on this runner under TCG; that is expected.

Serial logs carry no machine identifiers

I checked a kept log from a local guest run (r1/uart-0.log, x86-64 under TCG). It contains no MAC, no IP, no host path or name, no serial number and no model string. The kernel reads no CPU brand string.

Under KVM on the runner, the microcode: and fpu: lines will carry the host CPU's CPUID signature and microcode revision. Those identify a CPU model and stepping, not a machine.

Separately, --ci's instrument line prints /proc/cpuinfo's model name into the job log. That is existing guest() behaviour, and the run logs of a public repository are public. Whether to keep it for the guest host is open (see Risks).

Checks and evidence

Evidence: r1 in pr-1173/r1 and r0 in pr-1173/r0. Logs are masked; each exits.txt lists exit codes and revisions.

r1, at 2d29d98db unless stated:

  • cargo run -- --ci host: exit 1, 108 of 109 steps green. The one red is two build-system tests this change does not touch: buildlock::…a_narrow_guest_does_not_overtake_a_wide_one_waiting and sysroot::…the_primary_s_fork_checkout_is_moved_to_its_pin.
    • The run was under host load average 22 to 28 from other agents.
    • The PR's build/ diff is only ci.rs and qemu-version, and neither test reads either.
    • Both passed alone, 3 of 3 at the same head (isolated-*.log).
    • The buildlock red is a test race: its probe takes any holder of the queue for the queued wide guest, and the log shows otherwise. It is recorded in issues/a-narrow-guest-test-reads-any-queue-holder-as-the-wide-waiter.md.
    • The sysroot red, "uncommitted work" in its own scratch fork, is a sighting with no established cause; its log is in r1-host-ci.log.
    • Every step this change can affect is green: the lint, the harness checks and the driver's tests.
  • cargo test --test toyos-checks -- other_work_detector run_exit_status iowait_going_back: green on macOS. iowait_going_back feeds two /proc/stat readings in which iowait falls by 20 ticks and idle rises by 100, and expects 15 ticks busy out of 95.
  • cargo test -p toyos-build --lib -- ci::tests firmware: green, including the guard and its refusal cases.
  • Negative controls, each a checked patch that was built, run and reverted (r1/controls/):
    • C1, iowait counted on its own: iowait_going_back red ("busy 42949672.91s of …, and it is 150ms of 950ms").
    • C2, the workflow's main-ref refusal removed: the guard red.
    • C3, an actions/cache/restore step added to the workflow: the guard red.
  • A local guest row at the r1 code, before the commit: a_drives_row_holds_one_process_per_function passed, the run exited 3 with 13.29 CPUs of other work, and its serial log was kept (this run's serial logs are kept at …). That is B3's harness half, end to end.
  • actionlint 1.7.12 (release binary, sha256 checked, toyos-guest declared as a self-hosted label, shellcheck unavailable): guest-rows.yml is clean. The only finding is the pre-existing toolchain.yml:20 concurrency.queue.

r0, still valid for what r1 did not change: the detector self-check and controls M1, M2 and M3, which are the getrusage attribution, the tick unit and the exit order.

Not run end to end. The runner is not registered, so the workflow, permissions: {} with checkout, the artifact upload and the instrument step on a Linux/KVM host have not run. On Linux, the /proc/stat path itself has not run; its arithmetic is now unit-tested on any host. ci.yml's Linux host job runs toyos-checks once the PR leaves draft, and must be green before landing.

First-run verification once the runner exists (orchestrator)

  1. First dispatch. Push a branch at main's head after landing, then gh workflow run guest-rows.yml --ref <that branch> -f filter='--list'. Expect:

    • the job lands on the runner, and checkout succeeds with permissions: {};
    • both instrument steps are green: QEMU 11.1.2, firmware …, /dev/kvm opens for x86-64 and another architecture's machine: emulated for aarch64;
    • the toolchain is built or found in the store, and the job is green.

    Note the toolchain time.

  2. Quiet baseline. -f filter='<one cheap x86-64 row> <one cheap aarch64 row>'. Expect host: other work used … N CPUs well under 1, test result: ok., and no toolchain rebuild.

  3. A wide run (S1). One full x86-64 dispatch, recording the N CPUs on average line. Kernel work the run causes is charged to no process: writeback, IRQ time, kswapd. This measures how close that comes to the one-CPU bound on this host. If it comes near, the bound needs revisiting before rows rely on it.

  4. Queueing. Dispatch the same request twice. The second waits until the first ends.

  5. Main refused. gh workflow run guest-rows.yml --ref main must show the job skipped.

  6. Logs on a red. On a deliberately red row, for example a filter that matches nothing: the run is red, and the serial artifact holds only that run's logs, or warns that it found none.

  7. Optional loaded-host check. With the owner running one CPU-bound process per core on the host, a row ends HOST NOT QUIET.

Risks

  • Dilution. The reading averages over the whole run, so a short burst of other work inside a long run can stay under one CPU on average. A per-test reading would need the CPU time of children that are still running, which getrusage cannot give. The dedicated host is the primary guarantee, and this check is a backstop.
  • Kernel work nobody is charged for. This lands on the "other work" side, as S1 notes; step 3 measures it.
  • A child left running. If something the harness starts is still running at the reading, it counts as other work. The failure mode is a false HOST NOT QUIET, never a false green.
  • Persistent workspace. clean: false leaves untracked files from earlier revisions in place, and a branch's code can taint later runs on the host. Both are inherent to a persistent runner.
  • Cold toolchain. A revision that changes the toolchain key builds LLVM and the sysroot on the runner, which takes hours, and later dispatches queue behind it.
  • Public CPU model. The instrument line publishes the guest host's CPU model name in public run logs. This is open for the orchestrator or owner to decide.

🤖 Generated with Claude Code

Japabu and others added 2 commits November 4, 2026 10:39
… is not a verdict

Owner ruling (2026-11-04): guest tests run on a dedicated, otherwise idle
self-hosted runner; host checks stay on GitHub's runners. The nightly is
frozen, so this adds the on-demand path only and leaves nightly.yml and
guest.yml as they are.

- `guest-rows.yml`: `workflow_dispatch` alone, inputs `rev` and `filter`,
  on `[self-hosted, toyos-guest]`. It checks out `rev` without cleaning the
  workspace, so `target/` stays warm, and the toolchain comes from the
  host's own store (`~/.cache/toyos`), as on a dev host. No secret, no
  persisted token.
- `cargo run -- --ci rows <args>`: the harness with those arguments, held
  to the same instrument check as the nightly's guest job (QEMU at
  build/ci/qemu-version, its firmware, a `/dev/kvm` that opens) and the
  same $TMPDIR leak check, without installing a restored sysroot.
- The harness reads the host's busy CPU time (/proc/stat, Mach's
  HOST_CPU_LOAD_INFO) against its own and its reaped children's
  (getrusage) over the run. Other work averaging more than one host CPU
  ends the run `HOST NOT QUIET`, exit 3, whatever the rows said.
- `only_a_dispatch_reaches_a_self_hosted_runner` replaces the test that
  forbade a self-hosted runner: a job off GitHub's hosted images is
  allowed only in a workflow `workflow_dispatch` alone starts, and a
  job-level reusable workflow must be this repository's own.
- Instructions say guest tests run on the guest runner through that
  dispatch, aarch64 also on a quiet Mac on demand; the frozen nightly is
  recorded as an open item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ext and a comment reflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu

Japabu commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review of #1173 at 4068d86ba (r0)

Architecture

The ownership is right. The workflow is a thin dispatch. --ci rows reuses guest()'s instrument check and $TMPDIR check instead of copying them, with one install flag. Arguments are split in Rust, so the shell neither splits nor globs. The other-work reading belongs to the harness and lives in tests/common, its only user. Exit 3 sits beside the existing suspend status 2.

Net growth is +473/−37:

  • production: about 56 lines of workflow, about 45 in build/ci.rs, 163 in hostload.rs and about 30 in tests/toyos.rs;
  • tests: about 155 lines (tests/checks/hostload.rs, the run_exit_status additions, the guard and its reader test).

The three host-side negative controls (M1, M2, M3) test the right defects. The guard fails closed on every runs-on shape it cannot read. Reusing the --ci host result from 626577bd4 is valid: I checked git diff 626577bd4 4068d86ba, and it touches only the saturating_sub, a usage string and a comment reflow.

The remaining problems are in the trust boundary of a persistent runner, one Linux arithmetic defect, red evidence that a dispatcher cannot reach, and a first-run plan that cannot pass on the intended host.

Security answers

  • Triggers. Among main's workflow files, only workflow_dispatch reaches the label. I checked the guard's readers. A PR branch from this repository can carry its own self-hosted workflow, as the PR says. Fork PRs are covered by the approval setting, which I confirmed live: actions/permissions/fork-pr-contributor-approval = all_external_contributors.
  • Token and secrets. persist-credentials: false keeps the token out of .git/config, and no secret is referenced. The token's real exposure is through the persistent runner, covered in B1.
  • Injection. None. rev goes to an action input and filter goes through env with a quoted "$FILTER". run-name is not shell. filter can pass any harness flag, but a dispatcher who chooses rev already chooses the code that runs.

Findings

BLOCKER B1 — .github/workflows/guest-rows.yml:21-44: unreviewed code runs with main's cache scope on a persistent runner.

  • gh workflow run guest-rows.yml -f rev=… without --ref runs on refs/heads/main. The job then checks out any rev GitHub serves, which includes unmerged branches and fork PR heads.
  • On a non-ephemeral self-hosted runner, the job's code runs as the user that owns the runner installation. That code can replace the runner's bundled node or worker binaries. The next job's actions/checkout would then run attacker code holding that job's ACTIONS_RUNTIME_TOKEN.
  • permissions: does not limit actions/cache writes. A main-ref run's runtime token writes main-scope cache entries.
  • ci.yml restores main's host-sealed-* cache (target/, ~/.cargo/registry) into PR and merge-queue host checks. nightly.yml restores main-scope sysroots into the release it publishes.
  • This breaks two reviewer.md rules: "No privileged triggers that execute untrusted code as main" and "Only main publishes caches other refs restore".
  • Fix, which is also simpler:
    • Drop the rev input. Dispatch with gh workflow run guest-rows.yml --ref <branch> -f filter=… and check out the run's own commit (checkout's default).
    • Add a job-level if: github.ref != 'refs/heads/main'. A main-ref dispatch then never reaches the machine, so a runner backdoored by an earlier branch cannot obtain a main-scope token. To test main's commit, dispatch from a branch that points at it.
    • Update tests/CLAUDE.md:7, run-name and the PR's command text.
    • Optionally, the guard could assert both properties on guest-rows.yml.
  • That one branch's code can taint later verdicts on this host is inherent to a persistent runner that writers dispatch to. The owner accepted that by choosing it. Main's caches are a separate trust boundary and must stay out of reach.

BLOCKER B2 — tests/common/hostload.rs:117: a - b per field panics when /proc/stat's iowait decreases.

  • The kernel documents that this happens. Documentation/filesystems/proc.rst, /proc/stat, under iowait: "The value of iowait field in /proc/stat will decrease in certain conditions". An in-progress idle period is reported as iowait while a task waits on that CPU, and is credited to idle if the task wakes elsewhere first.
  • [profile.dev] (Cargo.toml:412) keeps overflow checks on, so the subtraction panics.
  • Consequences:
    • other_work_detector can flake red in Linux host CI and the merge queue, since its window is about 1 s.
    • A guest run can panic after every row has finished, which loses the summary and replaces exit 0/1/3 with 101.
  • Fix: take the difference of idle plus iowait as one sum. That sum does not go backwards. Use wrapping arithmetic on the raw fields.
  • None of the Linux path has executed yet. The PR's own ci.yml host job (skipped while the PR is a draft) is the cheapest check and must be green before landing.

BLOCKER B3 — guest-rows.yml:53-56 and tests/common/lane.rs:94: a red's serial logs cannot be reached.

  • guest.yml uploads target/red-run-serial/**/uart-*.log on failure; guest-rows.yml uploads nothing. A red's 16550 logs stay on a host that dispatchers have no access to, and they pile up in the persistent target/.
  • Run::exit keeps serial logs only on exit 1. A run with reds that ends HOST NOT QUIET (exit 3) deletes them.
  • The PR's own Risks section calls this out. Root CLAUDE.md still asks for reds to be investigated, and says a noise-only red "may be recorded as a sighting". Both need the log.
  • Fix:
    • Keep the logs whenever failures is non-empty, whatever the exit code.
    • On failure(), upload only this run's kept directory, without continue-on-error. The harness already prints that path. Alternatively, clear target/red-run-serial at job start.

BLOCKER B4 — the PR's host prerequisites and first-run step 1: QEMU 11.1.1 against a host running 11.1.2.

  • instrument (build/ci.rs:1406) compares exact strings with build/ci/qemu-version (11.1.1).
  • On 11.1.2, both architectures' instrument steps return Err. guest() adds the suite step only when every earlier step passed, so every dispatch goes red before booting anything, and step 1's expected QEMU 11.1.1 … /dev/kvm opens cannot appear.
  • Pick one of two options before the first run, and say which in the PR:
    • (a) Install 11.1.1 on the host.
    • (b) Move the declaration to 11.1.2 in a commit that says the instrument moved. guest.yml's Debian snapshot carries 11.1.1, so the frozen nightly would red at its instrument when resumed. Record that in the freeze issue's exit, or move the snapshot.
  • Since the runner is now the only place guests run, (b) holds the declaration to the QEMU that actually measures. The choice is the orchestrator's.

BLOCKER B5 — CLAUDE.md:60: "aarch64 guests also run on a quiet Mac on demand" has no owner source shown.

  • The ruling as relayed says guest tests run only on the dedicated x86 Linux KVM host. The PR claims "as briefed" for the unchanged AArch64 rule, not for this clause.
  • The clause also leaves open who runs it and on which Mac. The PR's own evidence shows the development Mac reading 8 to 12 CPUs of other work.
  • Cite the owner's decision or delete the clause. If it stays, HOST NOT QUIET already decides whether a given Mac run counts as quiet, so no other wording is needed.

SUGGESTION S1 — first-run plan, step 2: also measure a wide run.

  • Step 2 measures the quiet baseline only on two cheap rows.
  • getrusage never charges the run for kernel work it causes: kworker writeback of 300 MB images, IRQ time under CONFIG_IRQ_TIME_ACCOUNTING, and kswapd. A wide, parallel run is where this could approach one CPU.
  • If it does, every run on the host exits 3 and masks its reds.
  • One full x86-64 dispatch, recording the N CPUs on average line, settles this and also justifies the one-CPU bound for this host. Today that bound is a reasoned inference, which is acceptable for a backstop.

SUGGESTION S2 — runner hygiene in the prerequisites.

  • Register the runner as a dedicated user with no sudo.
  • The runner's .credentials files are readable by job code. That is GitHub's standard self-hosted model, but worth one line in the workflow header's host list.

Answers to the review questions

  • Is exit 3 hiding a regression? Not on a quiet host. Exit 3 outranks a red, but FAIL lines and the failure count still print, and verdicts() carries them into the step summary. A red that reproduces quietly comes back on the quiet re-run, which matches the ruling. The real loss is B3's deleted logs.
  • Is the measurement correct?
    • /proc/stat fields and units: steal counted as busy, guest time excluded because user and nice already include it, USER_HZ taken from sysconf(_SC_CLK_TCK).
    • Mach CPU_STATE_* order and 32-bit wrap handling.
    • RUSAGE_SELF plus RUSAGE_CHILDREN semantics.
    • All of these are correct, apart from B2.
  • Is the dispatch minimal? Yes. It uses one driver path, and with B1's fix it loses its only input that needed trust.
  • Are the instruction edits one-home and consistent? Yes, apart from B5. tests/CLAUDE.md holds the procedure, root points to it, and reviewer.md replaces its "GitHub-hosted only" rule rather than appending an exception to it.

SEND BACK

@Japabu

Japabu commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Review r1 of #1173 at c32b69e8e

This round checks only the fixes for r0's B1–B5, S1 and S2, and the head commit's issue record. I read git diff 4068d86ba c32b69e8e in full, the callers of the changed code (dispatch, guest, step, Run::exit and the suite's exit paths), and the r1 evidence at 067de8d2.

Architecture

The shape is unchanged and still right, and the fixes made it smaller:

  • The workflow lost its only input that needed trust (rev).
  • Host is now two sums, not a per-field vector.
  • copy_serial creates the kept directory only when it copies a log, which makes the "kept at" line truthful.

The guard pins exactly the three properties the B1 ruling asked for.

Previous findings

B1: resolved. guest-rows.yml has these properties:

  • dispatch only, with no rev input;
  • permissions: {} at the top level only;
  • no actions/cache;
  • the job condition if: github.ref != 'refs/heads/main'.

refuses_main_and_writes_nothing asserts all three. It refuses a job-level permissions: because it compares untrimmed lines. Step-level if: lines at 8 spaces don't match its if: prefix, so if: failure() doesn't confuse it. The reader test refuses five widened variants, and controls C2 and C3 show the guard turning red.

Can a non-main ref still reach anything privileged? I don't see a path:

  • ci.yml runs only on pull_request and merge_group. Those refs restore caches from their own scope and main's, never from a head branch's scope. nightly.yml is disabled and dispatch-only off schedule.
  • The branch-scope runtime token that a backdoored runner could take from a later job therefore writes caches that no privileged run restores.
  • Its artifacts go only to its own run.
  • No workflow consumes guest-rows through workflow_run or download-artifact.
  • Cross-branch taint on the persistent host remains, and the owner accepted it.

B2: resolved. Linux now keeps busy = user+nice+system+irq+softirq+steal and idle = idle+iowait, and differences them with wrapping arithmetic.

  • macOS: the sum of the u32-wrapped fields, differenced with wrapping_sub and masked to u32::MAX, gives the true delta mod 2^32. That is correct for any interval under 2^32 ticks.
  • iowait_going_back uses iowait 40→20 and idle 1000→1100, with busy 150→165. That gives 15 busy out of 95, which I checked by hand.
  • The test would have panicked on the old per-field a - b. Control C1, which counts iowait alone, turns it red.
  • It runs on every host and is deterministic, because both sides go through the same ticks().

B3: resolved.

  • Run::exit keeps logs for every code != 0: red, suspend, metal-staged and HOST NOT QUIET. The panicking Drop path still keeps them as well.
  • forget_kept_serial runs as the first step of --ci rows. Steps run eagerly in order, so it runs before guest()'s instruments and suite.
  • Exit 3 makes the suite step's status non-success, so the step goes red, dispatch exits 1, and if: failure() uploads target/red-run-serial/**/uart-*.log. It does this without continue-on-error.
  • r1-smoke.log (exit 3, logs kept) shows the harness half end to end. The upload itself waits for first-run step 6.

B4: resolved as ruled.

  • build/ci/qemu-version is now 11.1.2.
  • The freeze issue says guest.yml's container reds at its instrument until it moves, and its exit now includes "its guest container on 11.1.2".
  • The host-tools row says which QEMU each source carries.
  • The remaining 11.1.1 strings are fixture provenance (build/licence.rs, lib/acpi/fixtures/qemu-11.1.1*), a source citation (build/arch.rs:179) and a parser input (build/firmware.rs:307). None of them is a declaration.

B5: settled by the owner's 2026-11-04 ruling. I did not reopen it.

S1: first-run step 3 now measures a wide run. S2: the workflow header and the prerequisites both state the dedicated no-sudo user and the readable .credentials. Both suggestions are taken.

The host-CI reds and the new record

The PR is independent of both reds:

  • The PR's build/ diff is ci.rs (one dispatch arm, one function and the guard test) and qemu-version. Neither buildlock.rs nor sysroot.rs, nor either test, reads them.
  • Both tests passed alone 3 of 3 at 2d29d98db (isolated-*.log).
  • c32b69e8e adds only the issue file, so the r1 evidence still applies at the head.

issues/a-narrow-guest-test-reads-any-queue-holder-as-the-wide-waiter.md is accurate:

  • The log (r1-host-ci.log:574-584) shows waiting for the guest CPU queue (wide), then narrow waiting and narrow acquiring, then wide acquiring.
  • So when the probe's try_lock failed, the wide guest was waiting for the queue, not holding it, which is exactly what the record says.
  • build/buildlock.rs:917-922 is the probe as the record describes it.
  • The frontmatter is valid (open/tooling). The exit is testable, and the evidence is cited by commit.

The sysroot red, a stat-dirty library/std/src/lib.rs with identical blob hashes in its own scratch fork under load, is reported as an unexplained sighting with its log. That is honest, and nothing in this PR can cause it.

Findings

No blockers.

SUGGESTION S3 — tests/common/hostload.rs:103: ticks(busy + idle) can still panic on Linux if aggregate idle+iowait ever reads lower.

  • since now wraps each difference. A small backwards step in idle would then become about 2^64, and the plain + would overflow in the dev profile.
  • The kernel documents only iowait going backwards, and taking the sum handles that, so this is not a defect against the specification.
  • Mark::reading discards the total anyway. busy.wrapping_add(idle), or computing the total only in host_since, closes the last panic path at no cost.

SUGGESTION S4 — tests/common/lane.rs:69-71: the edited doc comment was not reflowed. "…when it is not / green, so they are / copied to…" has a stray short line in the middle of the paragraph.

SUGGESTION S5 — the new issue record could name the likely holder.

  • The sibling test's comment at build/buildlock.rs:887-888 already names one mechanism that holds a lock: a sibling test's fork keeps the inherited lock description until its exec. Under load, that fits a probe that sees the queue held while neither guest holds it.
  • One clause would point the fixer at it. Without it, the record is still correct as written.

LAND

@Japabu
Japabu marked this pull request as ready for review November 4, 2026 10:36
@Japabu
Japabu enabled auto-merge November 4, 2026 10:36
@Japabu
Japabu added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 2c2f957 Nov 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant