[Japabu]

Draft. The metal A/B has run and been judged. Ready for a focused check. Companion to #1133, which fixes the same stall in userland libc.

Problem

A load whose address bits 11:0 match an older in-flight store waits on it. AMD Family 17h judges store-to-load forwarding by those bits alone (56305 rev. 3.02, §2.12); Intel documents it as 4K aliasing.

The kernel's memcpy was compiler_builtins' rep movsb; rep movsq; rep movsb, because the sysroot is built without ermsb.

Resulting behaviour

• Owner: the kernel. The kernel now defines memcpy in kernel/src/arch/x86_64/copy.rs, which overrides compiler_builtins' weak symbol. memmove and memset stay compiler_builtins'. The fix doesn't belong in the rust/ fork: whether a copy goes down depends on the CPU, and compiler_builtins chooses its copy at compile time only.
• Every copy is one rep movsb, except on AMD: there, a copy of at least 64 bytes whose destination is 1–31 bytes above its source modulo 4 KiB goes highest first.
• The descending copy uses general registers, 8 bytes per store, with every store but the top and bottom 8 bytes aligned. The kernel touches no vector register (build/fpgate.rs).
• The rule is chosen at boot from CPUID's vendor and logged as memcpy: highest first Rule { window }. Intel and other vendors never go down.
• The routing is not shared with libc: a forward copy a short way above its source modulo 4 KiB copies down #1133. The libc loops are AVX/SSE2, which the kernel may not use, and only a two-line predicate would be common.

Distances the kernel's copies hit

A measurement-only probe ran under QEMU; the distances depend only on software placement, so QEMU stands in for metal. Results: analysis.

workload	bytes in copies ≥ 64 B at d 1..511	at d 1..63
boot + osbench system (pipes)	0.39%	0.06%
osbench ipc	0.61%	0.00%
osbench files (kernel tmpfs)	12.3% of 1.67 GB	1.26%

Metal A/B

Base b49e6ae = main 190f3b1 + measurement-only osbench kcopyat; head dc53533 = this change at its first review. Branch kernel-memcpy-ab, four boots per bench, all passed. Material: judgement, rule, readbacks. Each figure is the mean of its arm's two boots' p50s.

• The forward path gains on both benches. Replacing the qword sequence with one rep movsb made a kernel tmpfs read of 4 or 8 KiB take 0.92–0.96× as long, at d = 0 and at every distance outside the window. The file metadata rows took 0.71–0.93× as long, the T14's pipe throughput rose 7–17%, and no file, IPC or pipe row was slower in both pairs (K3, K6). The AMD's ipc rows were slower in one head boot only (2-head; 3-head was faster than base).
• Zen 2 (Ryzen 7 4800U):
  • In-process, rep movsb waits at every distance under 32 bytes, at every length from 4 to 64 KiB: 1946 ns at 8 KiB and 20463 ns at 64 KiB, against 88 and 962 at d = 0. From 32 bytes up it never waits.
  • The descending copy takes 260 ns at 8 KiB and 2009 ns at 64 KiB, the same at every distance. The window of 32 at every length holds (K2, K4, K5).
• Tiger Lake (i5-1135G7): the descending copy lost to rep movsb at every distance and length: at 8 KiB, 241–256 ns against 58–197 (K4, K5 failed). Following the rule, the follow-up commit sends no Intel copy down; with one window left, the length cap and the distance-under-length test go too.
• The issue's exit holds (K1). A kernel read 16 bytes above its source takes 1.17× (AMD) and 1.10× (T14) the same read at d = 0, for 8 KiB.
• The base kernel did not stall at d < 32 on the AMD: its read rows are flat across distances. So rep movsq doesn't wait there and rep movsb does. The kernel's gain comes from the faster forward path; the AMD window is what keeps that path's rep movsb from its stall.
• Unexplained: this run's T14 rep movsb at d = 16 takes 136 ns at 8 KiB. libc: a forward copy a short way above its source modulo 4 KiB copies down #1133's copyat on the same bench and microcode measured 750. Here the T14's slow distances follow 64-byte misalignment, not a page-alias window. The AMD's stall reproduces (1946 against copyat's 2605).

Checks at the head

• After rebasing onto main c81fc83 (head c020b27, the same change set; one Cargo.toml member conflict, where main had dropped kernel/dma): host CI passed, 109 steps, exit 0; x86-64 clippy with -D warnings and the x86-64 tests both exit 0 (evidence). Everything below was run at fe91241.
• cargo clippy -p kernel-copies --all-targets --target x86_64-apple-darwin -- -D warnings: exit 0.
• cargo test -p kernel-copies for x86-64 (Rosetta): exit 0. It checks the descending copy at every length to 300 and beyond, at every store offset, and each rule at every distance below 1 KiB and beyond, all against copy_within.
• Mutations, each red with exit 101: dropping the bottom store, an 8-byte step loop that stops at 16, and a rule without its window (logs).
• cargo run -- --build-only: exit 0. The disassembly shows the kernel's memcpy with no call or panic path.
• Host CI cargo run -- --ci host at fe91241: 108 steps green, exit 0 (log). This host is aarch64; the x86-64 lines above cover kernel-copies.
• QEMU kcopyat_runs at dc53533: PASS. TCG reports AMD, so short-distance reads went down; every read and in-process copy was checked against its source (log). The follow-up only removes paths, so this check still applies.

Risks and open decisions

• Zen 2's descending copy runs 3× a rep movsb at d = 0 (260 against 88 ns at 8 KiB), against 22× stalled. Parity would mean letting the kernel copy save and use vector registers. The issue's exit holds without that, so the question is not raised.
• Each rule is measured on one microarchitecture: AMD's on Zen 2, Intel's on Tiger Lake. Other AMD and Intel cores take their vendor's rule unmeasured; the track issue records this.
• issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md exists only on libc: a forward copy a short way above its source modulo 4 KiB copies down #1133, and its premise (the kernel's copy was one rep movsb) is false. Its exit is met here (K1). Noted on #1133: whichever lands second deletes it.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1139 at 42fb869

Architecture

The kernel is the right owner. compiler_builtins makes its mem functions weak (#[linkage = "weak"], compiler-builtins/src/macros.rs:414) so that a platform can supply its own. Choosing a window per CPU needs CPUID at runtime, which a no_std builtins crate should not hold as policy, and the vendor choice is made in arch boot code, where it belongs. memmove and memset are unchanged. The descending copy uses only GPRs, which follows the owner's decision to measure the 8-byte version first. I confirmed by inspection that the disassembly evidence shows no FP/vector instruction.

I checked the following and found no defect:

• Linkage. The kernel target is x86_64-unknown-none (build/arch.rs:83), so target_os = "none" holds. The toyos profile uses no LTO, so the strong #[no_mangle] definition takes precedence over the weak one. The evidence disassembly names memcpy as this code.
• Recursion. The body is inline asm, branches and RIP-relative loads, with no call and no panic path.
• Order and SMP. set_rule runs in after_console, before interrupts() and AP startup. Copies before it use NEVER, which is plain rep movsb. Any mix of WINDOW and MOST values still copies correctly, so Relaxed cannot cause a correctness fault.
• DF. Every Ring 0 entry runs cld (entry.rs:20, nmi, spurious, unclaimed), and FMASK masks DF.
• copy_down for every n ≥ 8 and every alignment. aligned_end ∈ (n−8, n]. Loop accesses stay in [0, n). The top store covers [aligned_end, n) and the bottom store covers [0, p<8). Inner stores are 8-aligned on the destination. Wrapping arithmetic stays correct even for a range that ends at 2^64.
• The predicate. LEAST..=most is empty for NEVER. d < min(window, n) is the true forward-alias condition. d = 0 includes src == dest and same-offset-in-another-page; both take rep movsb.
• User copies. They go through pinned direct-map frames with no fault fixups (user_ptr.rs:150-205), so changing the instruction sequence breaks no recovery path. Re-reading a racing user source within a single copy creates no new TOCTOU, because the result lands in the kernel buffer.

Net: about 168 production lines (copy.rs 158, boot +9, mod +1) and 87 test lines in a new host package. The package follows the toyos-libc-copies #[path] precedent, which is justified because the kernel is a freestanding binary and kernel/pure may not contain asm.

Findings

BLOCKER 1 — kernel/src/arch/x86_64/copy.rs:51-62: x86-64 host CI fails on -D warnings.
WINDOW, MOST, set_rule and rule() are used only by the cfg(target_os = "none") memcpy and by boot. In kernel-copies' test build they are dead code, and the PR's own Rosetta log (host/copies-x86-42fb869.log) shows four dead_code warnings. CI's host job runs on ubuntu-24.04 (x86-64, .github/workflows/ci.yml:43) with clippy --workspace --all-targets … -D warnings (build/clippy.rs:71-73), so the merge queue goes red there. The "108 steps green" run was on an aarch64 host, where the module compiles out, so that result does not cover this.

• Fix: put the two statics, set_rule and rule() under cfg(target_os = "none") next to memcpy.
• Evidence needed: cargo clippy -p kernel-copies --all-targets --target x86_64-apple-darwin -- -D warnings with exit 0.

BLOCKER 2 — commit message and PR body: the base is described wrongly, and the change is larger than stated.
The commit says "The kernel's memcpy was compiler_builtins' one rep movsb", and both the commit and the PR say every other copy "stays one rep movsb". The sysroot is built without +ermsb: std_config (build/sysroot.rs:1113-1144) sets no rustflags, and the track issue this PR edits says the same at :11-19.

• So the base memcpy is rep_param, then rep movsb; rep movsq; rep movsb. I checked this in an existing main kernel ELF (target/x86_64-unknown-none/toyos/kernel): compiler_builtins::mem::memcpy disassembles to exactly those three string operations.
• This PR therefore also changes every copy outside the window on every CPU to a single rep movsb. On Zen 2, which has no ERMS or FSRM, that is the very change whose cost the track issue says is owed.
• Consequences:
  • The A/B's K3 premise ("untouched" outside the window) is false. K3 and K6 on AMD measure two changes, so a regression would be attributed to the wrong one.
  • The pr-1133 figures in the commit (2605 ns against 90 ns) timed userland's rep movsb, not the kernel's old memcpy.
• Fix:
  • Say this in the commit message. The PR body is the orchestrator's to correct.
  • Read AMD's K3 d0 rows and its K6 rows as the cost of the forward-path change.
  • Record that figure in the track issue. This also makes issues/kernel-forward-copies-…md:24-25's "no proving machine is family 0x17" false: the 4800U is family 0x17. That line is directly affected by this change, so correct it in the same edit.

BLOCKER 3 — kernel/src/arch/x86_64/copy.rs:9-10: the hardware explanation gets the direction wrong.
The comment says "copied highest first, that store comes 4 KiB minus d later." In a descending copy, the store with the same low 12 bits as the load at offset x is at x−d, and it comes d bytes after the load. The nearest older store with those bits is 4 KiB − d bytes earlier. That older store is the one that matters, and the comment says the opposite.

• Fix: "copied highest first, the nearest older store with those bits is 4 KiB − d back."

BLOCKER 4 — kernel/src/arch/x86_64/copy.rs:64-67, 83: the safety contract is narrower than how memcpy is actually called.
LLVM's LangRef allows llvm.memcpy operands that are "equal or non-overlapping", and its lowering calls this symbol with dest == src. So the claim at :83 that "memcpy's contract is copy_by's" ("do not overlap") is false for real callers. The code is correct for that case (d = 0 takes rep movsb, and copy_down would also be correct). The fix is wording only.

• Fix: state "disjoint or identical" in copy_by's # Safety.

BLOCKER 5 — copy.rs:33-41 together with the issue edit: vendor-wide rules are each extrapolated from one microarchitecture, and the issue record does not say so.

• Rule::AMD cites Zen 2 but applies to every AMD core, Zen 3-5 included.
• Rule::INTEL cites Tiger Lake but applies to every Intel core, including E-cores.
• The PR body says these are unmeasured. The only issue that tracks the kernel stall exists on libc: a forward copy a short way above its source modulo 4 KiB copies down #1133, so if this PR lands first, main records nothing (root: "No silent debt").
• Fix: one sentence in the track issue this PR already edits, naming the measured microarchitecture and the exit for the rest.

HELD FOR THE A/B (not judged here) — copy.rs:33-41.
"beats rep movsb under 192", "past its L1 cache … outruns" and "at every length" are estimates written as facts. After K4/K5, these comments must give the measured reason or say nothing.

• One point is already inconsistent: Tiger Lake's L1D is 48 KiB, and a 16 KiB copy touches 32 KiB of source plus destination. "Past its L1 cache" therefore does not explain a 16 KiB cap. Either the cap or the stated reason has to change.

Tests and evidence

• The host tests are hermetic and deterministic, and each has a single claim:
  • Lengths 8..300 and beyond, at all 8 destination alignments, compared over the whole buffer, so writes outside the range are caught.
  • Every rule at d −5..1018 and further distances.
  • The predicate's edges, including the wrap below the source.
• The three mutations each go red with exit 101.
• If copy_by swapped its branches, no test would catch it. That only costs performance; the A/B head rows and inspection of the 6-line function cover it.
• The QEMU run was on dc53533. Its fix diff is byte-identical to 42fb869's (I checked), so that evidence still applies. The disassembly establishes the override.
• Main has moved two commits since 190f3b1 and touches none of these files.

Suggestions (optional)

• copy.rs:33-34: "only a load overlapping the store before it in part waits" is hard to parse. Try: "a load waits only on a store whose 32-byte block it partly overlaps, so distances of 32 and more never wait."
• memmove's forward branch has the same stall for non-overlapping callers at a short mod-4 KiB distance. If the kernel owned memset/memmove too, the track issue's sysroot +ermsb key move would disappear. This is noted only; it is not in scope here.

SEND BACK