Skip to content

nvme: the controller's own idle power states, by APST in Linux's tiers - #1131

Draft
Japabu wants to merge 9 commits into
mainfrom
nvme-apst
Draft

nvme: the controller's own idle power states, by APST in Linux's tiers#1131
Japabu wants to merge 9 commits into
mainfrom
nvme-apst

Conversation

@Japabu

@Japabu Japabu commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Problem

ToyOS's NVMe driver never set Autonomous Power State Transitions, so a drive stayed in its operational state whenever it was idle. issues/diskserver-drives-nvme-apst.md records the track.

Behavior

  • Where Identify Controller's APSTA says the controller supports APST, the driver sets Feature 0Ch (NVMe Base 2.1 §5.1.25.1.6, §8.1.17.2). It does so at bring-up and after every reset. The driver logs every power state and the table it set.
  • The table is built by system/nvme/apst from the drive's own descriptors. Each state goes to the deepest admitted state of the first tier that holds one deeper than itself, reporting no higher idle power:
    • a round trip within 15 ms, after 100 ms idle;
    • else one within 100 ms (EXIT_BOUND_US), after 2 s.
  • The tiers are Linux's (nvme_core's apst_* parameters, read on the bench). On the T14 the rule reproduces the table Linux set there, 0,1,2,3->4 after 100 ms, and a test holds that drive. A state that leaves a latency unreported is never a target.
  • osbench measures the cost and reads Linux's side:
    • persist.write_4k_fsync_after_idle times the first durable write after 10 s idle, on both systems.
    • The osbench_disk metal row runs files, nvme and persist.
    • On Linux, the nvme group reads the controller's APST table through the UAPI admin passthrough.

Status: not ready (r6, Linux's tiers, both benches)

Evidence: https://github.com/ToyOSOrg/evidence/blob/c2f9be162f22bb54e90a1d14b5091cf6ab0ad785/r6/README.md

gate (fixed before the run) T14 AMD
wake cost: after-idle minus back-to-back, against Linux's 13.52 against 8.29 ms: fail 7.76 against 7.67 ms: fail by 1%
worst after-idle sample, against Linux's 19.93 against 17.09 ms: fail 19.34 against 11.74 ms: fail
back-to-back disk rows no regression no regression
  • Both systems set the same table: 0..3->4 after 100 ms, as Linux does on both benches.
  • The wait is the drive's wake. The first write after idle completes at once. The next command waits 12.0 ms (T14) or 7.3 ms (AMD), against 0.1-0.9 ms without APST. Linux pays the same wake.
  • T14: ToyOS's after-idle write is 14.5-15.0 ms against Linux's steady 13.0 ms. The like-for-like gate also swings with Linux's own back-to-back mode (4.7-5.0 ms in r6, 1.0 ms in r3).
  • AMD: the usual sample matches Linux. One sample per boot, the same one each boot, pays two device waits (17.5-19.3 ms). It is unexplained.
  • Idle is reported, not gated. These boots ran the GOP display. benchcase: the shipped image's display driver runs in the benchmark boot #1149 brings tgl-display into the benchmark image.

Checks

  • cargo test -p nvme-apst: 9 pass.
    • Mutation: taking a tier's shallowest state fails exactly the BC501 test.
    • The earlier mutation (a state past the bound) still fails the bound test.
  • osbench clippy, on both Linux musl targets with the shape's lints: exit 0.
  • QEMU nvme_disk_boots: PASS (r2 and r4, with the diagnostic probes).
  • cargo run -- --ci host at a5d9ffc99 (r6's revision): exit 0, 107 steps. The branch is rebased onto main; cargo run -- --ci host at the rebased head 8500478e8: exit 0, 109 steps.

Evidence

Risks

  • Not yet measured:
    • The T14's extra wake cost has no named cause.
    • The tiers have no metal run yet.
  • The tiers rest on one inference. That they compare the round trip (ENLAT+EXLAT) is read from Linux's parameter names and one drive's table, not from documentation.
  • AMD oracle: its Linux side has one energy run and one persist run of the planned three and two. The others were void (ssh).
  • Not established: that the slow write mode tracks APST off (A 4/4 slow, B 2/5).

🤖 Generated with Claude Code

Japabu added a commit that referenced this pull request Nov 1, 2026
@Japabu @claude
issues: APST did not lower the T14's idle package under ToyOS
4c965bb 
Under ToyOS, r1-r3 of PR #1131 measured APST at 50x and 500x on the T14.
The drive rests in PS4 between I/Os, but the package stays in PC2 at
6-7 W either way. "APST is the larger share and comes first" no longer
holds.

What remains, with or without APST:
- a 6.3 W baseline, against Linux nomodeset's 2.8 W in the same PC2;
- 2-20 seconds a minute 4 W above it with every core in C7, against
  Linux's 1-2.

r4 times them against the kernel's diary and the drive's I/O.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Japabu added a commit that referenced this pull request Nov 1, 2026
@Japabu @claude
issues: r4's readings on the T14's idle excursions and its slow writes
a5d9ffc 
Under PR #1131's r4, each idle second was set against the kernel's diary
and the drive's I/O. The excursions follow none of: task wakes, timer
fires, idle entries, CPU C0, or NVMe commands. The PC2 issue says so,
with the drive-state rates measured over r1-r4 and the one link setting
that differs from Linux's.

The slow-write issue gets the arm tally (4/4 boots without APST slow,
2/5 with) and the drive's HMB fields. HMPRE and HMMIN are 0 and Feature
0Dh is refused, so a missing Host Memory Buffer is not its cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu Japabu changed the title nvme: the controller's own idle power states, by APST under a latency bound nvme: the controller's own idle power states, by APST in Linux's tiers Nov 1, 2026
Japabu and others added 8 commits November 2, 2026 00:00
@Japabu @claude
nvme: the controller's own idle power states, by APST under a latency…
af1599f 
… bound

The NVMe driver now sets Autonomous Power State Transitions (NVM Express
Base 2.1 §5.1.25.1.6, §8.1.17.2) where Identify Controller's APSTA says the
controller supports them, at bring-up and again after every reset. Until
now a ToyOS drive stayed in its operational state whenever idle; on the
T14, Linux with APST off draws 5127 against 1622 mW idle and its package
goes no deeper than PC3 (evidence pr-1100 r7).

The table is built by `nvme-apst`, a pure crate nested in the driver, from
the drive's own power state descriptors (Figure 313), with no table of
drives:

- a target is a non-operational state whose stated entry plus exit latency,
  the most the first I/O after idle waits (§8.1.17), is within 100 ms; a
  state that reports either as 0h ("not reported") is never one;
- each state steps to the shallowest admitted state deeper than itself, by
  that round trip, reporting no higher idle power;
- each step waits 50 times its round trip, so waking costs at most 1/50 of
  the idle time it follows.

The driver logs every state and the table it set. A controller that
refuses the table is reported and served from its operational state.

osbench's `persist.write_4k_fsync_after_idle` times the same durable write
as `persist.write_4k_fsync`, each after 10 s idle: the first I/O after idle,
on both systems. The `osbench_disk` metal row runs `files`, `nvme` and
`persist` alone for a change like this one. `nvme_disk_boots` holds that the driver
states what it made of the controller's power states; QEMU's supports none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
nvme-apst: each step waits 500 times its round trip, not 50
bb783e7 
On the T14, APST with a 50x idle time drew 502 mW more package power at
idle than no APST across r1's three pairs, and 604 mW more in r2 with B
run first (evidence pr-1131 r1, r2). The excess is in seconds above
10 W; the baseline is the same in every arm.

The same table with 500x (PS3 after 1 s, PS4 3 s later) drew no more
than the control, while the drive still rested in PS4 through the idle
gaps (Get Features 02h at each arrival). Excluding PS4 did the same. What
the two share is that neither enters PS4 within some 400 ms of activity.
The mechanism behind the 10 W seconds is not established. This is the
candidate under test, not a settled cause.

Waking then costs at most 1/500 of the idle time it follows. The tests'
idle times follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
issues: APST did not lower the T14's idle package under ToyOS
e48de6e 
Under ToyOS, r1-r3 of PR #1131 measured APST at 50x and 500x on the T14.
The drive rests in PS4 between I/Os, but the package stays in PC2 at
6-7 W either way. "APST is the larger share and comes first" no longer
holds.

What remains, with or without APST:
- a 6.3 W baseline, against Linux nomodeset's 2.8 W in the same PC2;
- 2-20 seconds a minute 4 W above it with every core in C7, against
  Linux's 1-2.

r4 times them against the kernel's diary and the drive's I/O.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
osbench: Linux's nvme group says the controller's APST table
e11261a 
Linux documents no APST policy. Its kernel-parameters.txt names none,
and the module's parameters name tiers without what they compare. The
table it actually sets is the drive's to answer: Get Features 0Ch (NVM
Express Base 2.1 §5.1.25.1.6, Figure 393) through the admin passthrough
of Linux's UAPI linux/nvme_ioctl.h (struct nvme_passthru_cmd,
NVME_IOCTL_ADMIN_CMD). It reads nothing but the feature, and like the
group's reads it takes root.

The note gives each transition as `from->to after <ITPT> ms`, the form
ToyOS's driver logs its own table in, so the two systems' tables for one
drive sit side by side. ToyOS's half is unchanged: its driver already
says the table it set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
nvme-apst: Linux's two tiers, in place of a multiple of the round trip
1cc23f9 
Each state now goes to the deepest admitted state, deeper than itself and
reporting no higher idle power, of the first tier that holds one:
- a round trip within 15 ms, after 100 ms idle;
- else one within 100 ms (EXIT_BOUND_US), after 2 s.

Linux's nvme_core names these tiers: apst_primary_latency_tol_us=15000,
apst_primary_timeout_ms=100, apst_secondary_latency_tol_us=100000 and
apst_secondary_timeout_ms=2000. That is their values and modinfo
descriptions on the T14 (evidence pr-1131 r3 L0). The descriptions do not
say what a tier's latency compares. On the T14's BC501 (PS3 1+1 ms,
PS4 1+5 ms) Linux set 0,1,2,3->4 after 100 ms (pr-1131 r5, read through
osbench's nvme group), which the round trip's reading reproduces. The test
of that drive holds it.

The multiple (50x in r1-r2, 500x in r3) was ToyOS's own choice. r3 found
it no better on the gates that judge APST, wake cost and worst sample,
and the idle excursions it was raised for follow no drive event (r4).
Matching Linux's policy makes the A/B a comparison of the two systems'
paths, not of their policies.

Mutation: taking a tier's shallowest state, the old stepping, fails
exactly the BC501 test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
issues: r4's readings on the T14's idle excursions and its slow writes
d69af91 
Under PR #1131's r4, each idle second was set against the kernel's diary
and the drive's I/O. The excursions follow none of: task wakes, timer
fires, idle entries, CPU C0, or NVMe commands. The PC2 issue says so,
with the drive-state rates measured over r1-r4 and the one link setting
that differs from Linux's.

The slow-write issue gets the arm tally (4/4 boots without APST slow,
2/5 with) and the drive's HMB fields. HMPRE and HMMIN are 0 and Feature
0Dh is refused, so a missing Host Memory Buffer is not its cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
issues: the PC2 figures' display and lid condition
7ab2c3b 
The owner reported both bench lids closed from 2026-11-01, with no record
of when they were closed. Every Linux figure in the issue had its display
on as software holds it: DPMS On, the backlight at its setting, as
osbench energy recorded. The display-off step is the exception. Each
ToyOS-against-Linux figure is therefore for one lid state whose physical
state is unknown. Evidence pr-1131 lid/ restates the sessions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
issues: the APST figures' display, and Linux's slow durable writes
8500478 
The PC2 issue's APST figures came from the benchmark image. That image
started no display driver until #1149, so they read the GOP's display.

In PR #1131's r6, Linux's own back-to-back write_4k_fsync on the T14 ran
4.73-5.02 ms in all three runs, against 1.0-1.08 ms in r3. The slow
durable-write mode is not ToyOS's alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Japabu @claude
issues: a DATA commit flushes the drive twice
58ba186 
PR #1131's r7 split the T14's first durable write after idle, command by
command, with no command of the driver's own. The drive's wake (the
first write and first Flush, about 13 ms) matches Linux's 12 ms. ToyOS's
excess is the commit's second Flush, 2.25-2.5 ms after a wake against
0.36-0.43 ms warm, plus about 1 ms of host time between commands.
Fs::sync flushes, writes the superblock and flushes again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Japabu