[Japabu]

Problem

Power is one of the subsystems that has to reach Linux parity, and nothing on either bench reads it today. The last reading, from before deep idle (#1073) and AMD's 0x415 (#1089), put ToyOS's idle package at 3.16 W on the T14 against Linux's 1.18 to 1.57 W (#866). ToyOS had no counter for the package's energy (energy.idle_60s.package was unsupported), so no run could produce a figure.

What this changes

The kernel's counters carry the package's power (toyos_abi::counters, SYS_COUNTERS). They are all under Rights::TRACE, since package energy is a side channel (CVE-2020-8694):

counter	Intel	AMD	read on
package_energy	MSR_PKG_ENERGY_STATUS 611H	PKG_ENERGY_STAT C001_029BH	boot CPU
rapl_power_unit	MSR_RAPL_POWER_UNIT 606H	RAPL_PWR_UNIT C001_0299H	boot CPU
package_c2/c3/c6/c7	60DH, 3F8H, 3F9H, 3FAH	none documented	boot CPU
core_energy	not read (PP0 is a different, all-core plane)	CORE_ENERGY_STAT C001_029AH	each CPU

Sources: SDM 325462-093 Vol. 3B §17.10 and Vol. 4 Table 2-20; PPR 55922 (Family 17h Model 60h, the AMD bench), checked against 55772 and OSRR 56255. Each counter is admitted the way core_c6 already is, in kernel/pure/cpu.rs, which has host tests:

• Intel: the DisplayModels whose SDM sections name Table 2-20. 06_AFH is left out, because Table 2-57 restates its residency in other units. 06_66H is left out of package residency only, because Table 2-42 gives its 3F8H no rate. The one table with a different energy unit (Table 2-8, Silvermont) belongs to none of the listed models.
• AMD: Family 17h with CPUID_Fn80000007_EDX[RAPL]. PPR 55901 (Family 19h Model 11h) makes the core counter 64 bits wide and the package counter an L3 complex's, so that family is not read.
• Any guest: nothing is read under a hypervisor.
• A wrong admission still faults at boot, never inside a read, because bring-up reads every admitted register once.

osbench's energy group runs the same code on both systems and reads the same registers. ToyOS reads them through SYS_COUNTERS. Linux reads them as root through /dev/cpu/<n>/msr. One rule decodes both (osbench_rules::energy).

• Waiting for quiet. It first waits up to 120 s for three seconds in a row of at most 10 ms of machine CPU time each. The wait measures CPU time alone: whether the cores had reached their deepest idle state is each row's core C6/C7 note.

• The loads:

  • 60 s idle;
  • 30 s of one thread sleeping 10 ms at a time;
  • 30 s of one thread spinning;
  • 30 s of every CPU spinning.

• One round a second, alike on both systems (power::round): the package's counters and every CPU's. ToyOS's SYS_COUNTERS round kicks each CPU, and Linux's reads each CPU's MSRs, an IPI to each but its own. Before 55922d7a9, Linux read only the package each second, on whichever CPU the reader ran on, and each CPU only at a load's ends; the scoreboard below was measured that way.

• Each second it records:

  • package power (mW);
  • PC2/PC3/PC6/PC7 residency (permille);
  • under a spin, energy per compute chunk (nJ).

• Across each load, off its first and last rounds, it records:

  • each CPU's APERF/MPERF ratio and core C6/C7 residency;
  • on AMD, the cores' power (cores_mw);
  • on ToyOS, which processes ran.

• Units have no better direction. A spin that ran faster may draw more, and work is not slowed to save power.

• Linux's setup. Linux states its panel (DRM DPMS), backlight, rfkill switches and links as setup conditions. osbench-compare refuses to mix runs that differ on them.

• Its own boot. energy leaves all and gets its own metal row, osbench_energy: the benchcase image with that one job. Idle power belongs to the whole machine, so nothing else may run beside it. The T14 and the AMD bench. The README gives the run protocol, including how Linux matches ToyOS's devices: radios blocked, panel lit.

• Linux's registers come from crate::arch (AArch64 answers none). AMD's are read only on Family 17h with RAPL, as the kernel admits them; on Intel a register a model lacks reads as absent.

Rights::TRACE's contract names the package and core energy it now grants, the RAPL channel (CVE-2020-8694).

The counters metal judge now holds package_energy and core_energy presence to the bring-up line. It exempts the 32-bit energy counters from the "went backwards" check, and prints the package's power across its idle second and its spin as a reading, decoded by osbench_rules::energy.

Checks

The head is ed35ba937. Review round 2 adds two issue changes: the AMD issue now says its runs predate the matched read, with the cores' range across the runs, and the offline judge's selection defect is filed as tooling (issues/no-filter-word-takes-the-counters-metal-row-without-counters-silent.md). Host CI at ed35ba937: exit 0, 105 steps green (log). Below that: review round 1 (55922d7a9) and the T14 issue's phase-3 update (cf0fc945f), on the rebase onto 0c7e62ca7 (#1097). Evidence: r8/README.md.

• cargo run -- --ci host at cf0fc945f: exit 0, 105 steps green (log).
• QEMU osbench_capability_groups_exit_0 at 55922d7a9: exit 0 (log). The head adds only an issue file.
• osbench clippy, -D warnings, on x86_64- and aarch64-unknown-linux-musl: exit 0. The Linux binary builds; its sha256 is a6655896…7943a057c8cc.
• The changed counters judge over round 1's T14 readback: PASS counters, exit 0 (log). That boot ran at 890f5c709, and the kernel's counters code is unchanged since then.
  • No filter word takes that row without the shared member counters_silent, whose boot has no readback. So a checked patch dropped it from the selection for this run alone, and the same command reverted it.
  • Negative control: the same readback with its bring-up line saying package_energy=false gives FAIL counters, exit 1 (log).
• Since 55922d7a9, Linux reads every CPU each second. No metal run has used that method yet; the rounds below predate it.

Round 1 on metal (r2/README.md): one ToyOS boot (890f5c709) and one Linux run per bench.

• osbench_energy passes on both readbacks.
• Each kernel admitted what the rule predicts.
• Cross-check of the decode. One CPU spinning draws the same on both systems at equal chunk rates: T14 8351 against 7996 mW, AMD 6848 against 6795 mW. That confirms the units decode alike on both systems.
• First readings (one run a side, unresolved):

bench	row	Linux	ToyOS
T14	energy.idle.package	1473 mW	7015 mW
T14	energy.idle.pc2 / pc6	99 / 21 ‰	994 / 0 ‰
T14	energy.wake_10ms.package	1183 mW	2889 mW
AMD	energy.idle.package	1357 mW	1555 mW
AMD	energy.wake_10ms.package	1372 mW	1709 mW

• T14 detail. On the T14 every ToyOS core is in C7 99.7 to 99.9 % of the idle minute, while the package stays in PC2 and never reaches PC3 or deeper.
• Not yet explained. ToyOS's idle minute (5.1 to 18.1 W) drew more than its later 10 ms sleeper (2.9 W) at the same residency. 5e5c395b9 adds the per-second series, so the order of the readings shows when the power fell.
• Two Linux defects found in this round, fixed in 5e5c395b9.
  • The group pinned itself to CPU 0 before counting the CPUs, so Linux's spin.all ran one thread and its per-CPU notes read CPU 0 alone.
  • The quiet bar was strict against /proc/stat's 10 ms tick, so the AMD bench's Linux never read quiet.

Round 2, the scoreboard (r4/README.md, table): four ToyOS boots and four Linux runs per bench at 5e5c395b9, interleaved. Every judge passes, and every Linux run had its radios blocked. A resolved row is one where every run of one system lies beyond every run of the other (p = 0.029). Each figure is the median, over a system's four runs, of each run's per-second p50. Energy is a mean quantity and the idle series is right-skewed, so by the medians of the runs' means the T14's idle is 8272 against 2167 mW, 3.8×, and the AMD's 1688 against 1501 mW, 1.12×.

bench	row	Linux	ToyOS	ToyOS/Linux (p50s)
T14	energy.idle.package	1648 mW	7137 mW	4.33
T14	energy.idle.pc2 / pc6	102 / 21 ‰	994 / 0 ‰
T14	energy.wake_10ms.package	1219 mW	2863 mW	2.35
AMD	energy.idle.package	1472 mW	1688 mW	1.15
AMD	energy.spin.all.package	31676 mW	31254 mW	0.99

• Not resolved: the runs overlap. That holds for the one-CPU spin on both benches and for the T14's all-CPU spin. An unresolved row is not a finding of equal power.
• AMD's 10 ms sleeper is unresolved. The runs overlap.
• The T14's per-second idle series scatters 4.6 to 20.6 W on ToyOS and 1.0 to 9.8 W on Linux. Under the 10 ms sleeper, both are flat: ToyOS 2.86 W, Linux 1.22 W.
• Two separate gaps on the T14, which phase 2 takes up:
  • a steady floor of about 1.6 W, with the package in PC2;
  • a further surcharge of about 4.3 W that appears only when no CPU wakes (Linux's equivalent is about 0.4 W).

Unresolved

• The T14's idle gap is a recorded defect, issues/the-t14s-idle-package-stays-in-pc2.md. This PR adds that file. Phase 2's scouts ruled out the driverless functions, the deepest core state, and the cores and graphics as consumers (r5/README.md). Phase 3 (r7/README.md) named the two causes by putting ToyOS's state onto Linux. NVMe APST off costs Linux about 3.5 W at idle and caps it at PC3. No graphics driver holds Linux in PC2 at 2.8 W. The I219 is not a factor. "Panel lit on both" is the fair state, and the deficit is ToyOS's to fix, not by slowing work.
• PC8 to PC10 on Tiger Lake. SDM Vol. 4 does not give 630H to 632H for 06_8CH, so ToyOS reads only PC2 to PC7. On the T14 the deepest package states are therefore invisible on both systems through these rows.
• Machines with more than one package count only the boot processor's package. This is recorded in issues/the-kernel-owns-cpu-performance-state.md.
• The scoreboard predates the matched read. Rounds 1 and 2 were measured with Linux reading only the package each second, while ToyOS kicked every CPU. That asymmetry works against ToyOS. That it cannot explain the T14's gap is an inference: there the 10 ms sleeper, at 100 wakes a second, draws less than idle. Its size on the AMD bench is unmeasured. Linux-only runs of the head's binary would measure what the matched read costs Linux.
• The AMD bench's idle deficit (1688 against 1472 mW) is issues/the-amd-benchs-idle-package-draws-15-percent-over-linuxs.md, added here.

🤖 Generated with Claude Code

[Japabu]

Review of #1100 at 31be41287, round 1

Architecture

The structure fits. Only the kernel can read these MSRs. They join the existing SYS_COUNTERS round and its per-CPU admission in kernel/pure/cpu.rs, with host tests, in the same way as core_c6. The ABI addition belongs at the kernel/program boundary, and osbench decodes both systems' data with one host-tested rule (osbench_rules::energy). The energy group has its own boot, and the PR gives the reason: idle power belongs to the whole machine. Net production growth is about +150 lines in the kernel and ABI. The rest is osbench, at about +500, plus tests and records.

Registers, checked against the documents rather than against the PR's account.

• SDM 093 Vol. 3B §17.10.1: ESU is bits 12:8 and a step is 1/2^ESU J.
• Fig. 17-37: MSR_PKG_ENERGY_STATUS uses bits 31:0, and bits 63:32 are reserved.
• Table 2-20 defines 60DH, 3F8H, 3F9H and 3FAH as counting at the TSC's rate.
• Only three tables restate those package-residency addresses:
  • 2-42, for 06_66H: 3F8H with no rate. The PR excludes it.
  • 2-57, for 06_AFH. The PR excludes it.
  • 2-61, Xeon Phi. That model is not in the list.
• 606H is restated in 2-30, 2-32, 2-36 and 2-50, all with the same formula. Table 2-8 (Atom) has no listed model.
• PPR 55922:
  • C001_0299: ESU is bits 12:8, reset value 000A_1003h.
  • C001_029A: per core (_core[3:0]), bits 31:0.
  • C001_029B: bits 31:0.
  • CPUID_Fn80000007_EDX[14] is RAPL, and the test value 0x6799 matches the PPR's fixed bits.

All addresses, units, widths and masks are correct.

Fault safety. The admission is exactly what the SDM lists. Bring-up reads every admitted register on its own CPU, so a wrong admission fails loudly at boot and never inside a read. One residual risk comes with the established core_c6 design, because ToyOS has no rdmsr_safe: server models in the list (0x55, 0x6a, 0x8f, 0xcf, 0xad and others) are admitted to PC3 and PC7 because the SDM says they support them, and no bench can check that. I accept this given the vendor document.

Side channel. Every new counter needs COUNTERS|TRACE, the same right as APERF/MPERF (Hertzbleed). Rounds are at least 10 ms apart, which is far coarser than PLATYPUS-style sampling. Both the gating and its reason are sound. The right's own contract does not yet name the new channel (finding 3).

Findings

BLOCKER 1. tests/guest/osbench/README.md:168-169 and tests/guest/osbench/src/energy.rs:15-17: the read-cost description is false.

• The README says Linux "reads the package on CPU 0, where the reader is pinned". c523c94 removed that pin. sys/linux.rs power::package() opens /dev/cpu/{current_cpu()}/msr and pins nothing, and its own doc comment says so.
• The README (:158) says a quiet second is "under 10 ms". quiet tests <= QUIET_NS. That is the strict bar that round 1 found and fixed, so the README now describes the old defect.
• In the energy.rs module header, "read at the load's two ends, so Linux reads the package on the reader's own CPU" states a cause that does not exist. The line is also over-long.
• Fix: state the actual asymmetry in one sentence. Once a second, ToyOS's read kicks every CPU, while Linux reads the package on whichever CPU the reader runs on.

BLOCKER 2. tests/guest/osbench/src/sys/linux.rs:306-309: #[cfg(target_arch = ...)] outside the architecture modules.

• This is the only target_arch in osbench outside src/arch/, which the repository rule forbids.
• Fix: arch/aarch64.rs, which already exists for Linux AArch64, exports power_registers() -> None, and registers() calls crate::arch::power_registers() directly.

BLOCKER 3. sdk/abi/src/handle.rs:226-240: Rights::TRACE does not list what it now grants.

• This doc comment is where an image author decides whether to grant TRACE. It lists APERF/MPERF as "the Hertzbleed power channel". It does not list the package and core energy that now ride the right, which is a known side channel (CVE-2020-8694).
• The CVE note is only in Counter::needs. One clause in the right's contract fixes this.

BLOCKER 4. issues/the-t14s-idle-package-stays-in-pc2.md:16: the AMD idle deficit has no owner and no exit.

• energy.idle.package on the AMD bench is a resolved deficit: 1688 against 1472 mW, p = 0.029, every ToyOS run above every Linux run.
• This PR records it only as an aside in a T14 issue whose title and exit cover the T14 alone. That is silent debt under "No silent debt".
• Fix: widen the issue's scope and exit, or file a sibling issue.

BLOCKER 5. tests/toyos.rs:6749 and :6773-6853: the changed counters metal judge has never judged a readback.

• The PR body confirms it: "The counters row was not judged".
• The judge gains a presence check for package_energy and core_energy, an _energy exemption from the backwards check, and a power reading.
• Round 1 already booted the T14's counters row at 890f5c7. The PR says the kernel's counters code is unchanged since then, and I checked that by inspection. So no boot is needed. The cheapest sufficient check is to run that judge over the existing T14 readback, with a filter that selects only that row and not test_rs_counters_silent, and record the command and its exit code.

SUGGESTION A. The read-cost asymmetry is real, disclosed and unmeasured.

• ToyOS's per-second round kicks 7 or 15 other CPUs, and Linux's read is local. The asymmetry works against ToyOS.
• On the T14's 4.3× gap it cannot matter, since the 10 ms sleeper's 100 wakes/s draw less. My estimate, an inference, is single-digit mW on AMD against a 216 mW gap.
• The cheapest bound, or the cheapest fix: have Linux also read every CPU each second, by calling power::cpus() in sample(). The msr driver's IPIs then match ToyOS's kicks. One Linux-only AMD run would measure what it costs.

SUGGESTION B. The Linux half applies no model admission (arch/x86_64.rs:73-98).

• On AMD outside Family 17h, PPR 55901 makes the core counter 64-bit and gives the package counter per L3 complex. Linux would still decode them with the 32-bit package rule. cores_mw would then quietly disappear, and .package would carry the wrong label.
• The doc comment's "the ones ToyOS's kernel reads" is true only on the benches. Refuse on Linux wherever the kernel's rule refuses.

SUGGESTION C. The quiet gate.

• It measures CPU time only. The comments at energy.rs:9-10 and README :159-160 suggest it also waits out the ACPI server's idle-state handover, which it does not detect.
• Round 2's core C7 residency of 997-998 ‰ shows that the handover had finished.
• Either say what the gate establishes, or, on ToyOS, also wait for DeepIdles to advance.
• A run with quiet=false is admitted silently by both the row judge and the compare. All 16 round-2 runs read quiet, so no result changes.

SUGGESTION D. Energy is a mean quantity.

• The idle row's value is the p50 of a right-skewed per-second series.
• On the T14 the means are Linux 1.99-2.33 W against ToyOS 7.41-8.57 W, about 3.8×, against 4.33× on p50.
• The mean= field is already on each line. Saying which figure the scoreboard means would be enough.

SUGGESTION E. tests/toyos.rs:6842-6847 repeats the ESU and mask decoding instead of using osbench_rules::energy.

• It feeds only a printed reading, but it is a second decoder beside the PR's "one rule decodes both".

PR description (for the orchestrator)

• "Linux's reads only CPU 0, where its reader is pinned" is false, as in Blocker 1.
• "The extra wakes cost ToyOS a little idle power; they are not measured" states an estimate as fact. Label it as an inference.
• "Same work, same power … draw the same on both systems; those rows are not resolved" contradicts the README's rule that an unresolved row is "not equal". Say "not resolved: the runs overlap".
• "under 10 ms" should be "at most 10 ms".
• The measured figures match r4/table.md and r5:
  • T14 idle 7137 against 1648 mW, and the 10 ms sleeper 2863 against 1219 mW.
  • The 1.6 W floor and the 4.3 W surcharge are correct.
  • AMD idle 1688 against 1472 mW. AMD spin.all 31254 against 31676 mW is resolved, ToyOS lower.
  • The park and unbind ladders support the issue's "ruled out" list.

SEND BACK

[Japabu]

Review of #1100 at cf0fc945f, round 2

This round covers 31be41287..cf0fc945f: 55922d7a9, the review fixes, and cf0fc945f, the T14 issue's phase-3 update. Points settled in round 1 are not reopened.

The round-1 blockers

1. Read-cost description: resolved.
   • Both systems now read one round per second through power::round: the package and every CPU, on ToyOS one SYS_COUNTERS round, on Linux package() followed by cpus().
   • The README (:166-171), the energy.rs header and the sys/linux.rs doc now describe that accurately.
   • The README and PR body now say "at most 10 ms", which matches quiet's <=.
   • Each row's per-CPU notes and cores_mw come from its first and last rounds, which is correct.
2. target_arch outside arch/: resolved. arch/aarch64.rs exports power_registers() -> None, and sys/linux.rs now calls crate::arch::power_registers() without a cfg.
3. Rights::TRACE contract: resolved. It now names the package's energy and residency and AMD's core energy, citing CVE-2020-8694.
4. AMD deficit: filed as issues/the-amd-benchs-idle-package-draws-15-percent-over-linuxs.md. The front matter is valid and the exit is concrete. Two problems in its evidence remain; see the new blocker below.
5. counters judge: resolved.
   • r8 runs the changed judge at 55922d7a9 over round 1's T14 readback: PASS counters, exit 0.
   • The negative control sets package_energy=false in the bring-up line: FAIL, exit 1, with the expected message. This exercises the new presence check.
   • The selection patch only drops counters_silent, and it was reverted.
   • The kernel's counters code has not changed since 890f5c709 (by inspection), so that readback still applies.
   • The judge now decodes with osbench_rules::energy. It reaches that crate through a dev-dependency with a stated reason, following the toyos-parity precedent.

Suggestions

• A, C, D, E: taken, and correct as written.
• B, AMD half: taken. amd_family_17h_rapl applies the kernel's rule. Its family arithmetic adds the extended family unconditionally, but the result can only be 17h when the base family is 0FH, so the check is right.
• B, Intel half: I accept the decline.
  • My concern was registers that exist but are decoded wrongly. On AMD that is a likely future bench (Zen 3/4), and it is now fixed.
  • On Intel, the documented exceptions are 06_AFH, 06_66H's PC3 and Table 2-8's Atom unit. None is a bench, and none is plausibly one soon.
  • On those models ToyOS reports unsupported, so the table shows a Linux figure beside a reason, not a false verdict.
  • A second copy of the 36-model list would be a second rule to keep in step.

Re-measure before landing, or state it?

Stating it is enough. No scoreboard re-run is needed to land.

• What lands is the instrument and two records. Neither record's exit can be met without new rounds, and those rounds will use the matched read.
• The PR body says the rounds predate the matched read, which way the asymmetry biases, and that its size on AMD is unmeasured. It labels the T14 argument as an inference.
• The new Linux path executes only code that already ran on both benches in r4 (package() every second, cpus() at row ends), now called once per second. It builds and passes clippy on both Linux targets. Inspection is enough here.

The durable record still has to carry the caveat, though:

BLOCKER. issues/the-amd-benchs-idle-package-draws-15-percent-over-linuxs.md:11-20: the evidence is stated without its method, and one figure is from a single run.

• The issue records the AMD deficit as resolved against ToyOS. It does not say that those runs predate the matched read, and the asymmetry inflates exactly this row. A later agent will read the issue, not the PR body.
• Fix: add one clause saying that rounds 1 and 2 had Linux reading only the package each second while ToyOS kicked every CPU. The other option is a Linux-only AMD run of the head's binary, which would measure that cost directly.
• "Their own energy counters give ToyOS 0 mW over the idle minute and Linux 11 mW" uses one run each. r4's cores_mw is 0, 1, 1, 0 for ToyOS and 11, 17, 20, 17 for Linux. Give the ranges or medians. The conclusion, that about 200 mW lies outside the cores, stands either way.

cf0fc945f, the T14 issue

I checked its figures against r7:

• A1 APST off: 1622 → 5127 mW, deepest state PC3.
• B0 nomodeset: 2796 mW at 976 ‰ PC2, sleeper 2864 against ToyOS's 2831.
• ToyOS with the planes off: PC2 995 → 338 ‰, with PC3 at 656 ‰.
• The I219 changed nothing on either system.

The issue states these accurately, and it correctly leaves the display driver as the owner's decision.

New suggestion

• r8 found that no filter word selects the counters row without counters_silent, so an offline judge of that row needs a patch. That is a harness defect; record it as a tooling issue rather than leaving it only in evidence.

PR body

Accurate at this head. It labels the statistic, the method change and the inference, and it fixes the four wordings from round 1.

SEND BACK. One small blocker remains: the AMD issue's evidence clause.

[Japabu]

Orchestrator: r2's record blocker checked at ed35ba9. The AMD issue states that its runs predate the matched read and gives the per-run core figures, and the tooling issue is filed. Landing.