[Japabu]

Round 11 (the last bench round) found a regression in 5 of 5 pairs: the head's 256 KiB placement rows read 0.93 of base (0.80 in one pair). By the rule the drop is the PR's. Its cause is not yet found, and what follows is the orchestrator's decision.

Problem

A userland memcpy of 8 KiB took 2615 ns on the AMD bench (Ryzen 7 4800U, Zen 2) against 90 ns, depending only on where the copy's two ends sit. #1121's IPC copy split first showed it.

• Vendor fact. AMD's Family 17h "uses address bits 11:0 to determine STLF eligibility" and says to avoid in-flight stores with the same bits 11:0 to different addresses where a load may need forwarding from one of them (Software Optimization Guide for AMD Family 17h Models 30h and Greater Processors, 56305 rev. 3.02, §2.12, p. 39). A forward copy whose destination sits a short way above its source modulo 4 KiB does exactly that with every load.
• Metal, AMD. A forward rep movsb from 4 to 64 KiB stalls at d = 8, 16, 24 and 31 and at no distance tried from 32 to 2048 (kernel: memcpy goes highest first where a forward copy would stall #1139's in-process rows). osbench copyat (libc's memcpy) reads 2615 ns at 8 KiB and d = 16, 90 at every other distance, and stalls at d = 16 up to 64 MiB.
• Metal, T14 (i5-1135G7). copyat, with the ends 1 MiB + d apart, reads 769 ns at d = 16, 714 at 64 and 206 at 256, against 74 at d = 0. Intel documents 4K aliasing on bits 11:0. But the same bench read 124 ns for a heap pair 64 KiB + 16 apart and 136 ns for kernel: memcpy goes highest first where a forward copy would stall #1139's in-process copy at d = 16. So the window's extent on Tiger Lake is measured, but its mechanism is not established (recorded).
• dlmalloc puts two back-to-back 64 KiB Vecs exactly 16 apart. On the AMD that pair's copy took 2570 ns.

Change

sdk/libc's x86-64 copy_forward, inlined into memcpy and memmove's forward branch, returns dest:

• Under 256 B: main's rep movsb behind one compare. In the disassembly that is main's instructions plus cmp rdx,0xff; jbe and one register move.
• 256 B or more: six more instructions inline, with no call, ask whether the destination is 1..511 bytes above the source modulo 4096. Every other copy is one rep movsb.
• At such a distance: a tail jump to copy_short_alias. It copies highest first where the ranges are disjoint and the copy is at most 1 MiB, so any store a load could match is 4 KiB minus the distance behind it. Overlapping ranges (which keeps an overlapping memmove lowest first) and longer copies get one rep movsb.
• Disassembly.

The loop:

• uses 32-byte AVX loads and stores where CPUID and XCR0 allow, asked once, and SSE2 otherwise;
• stores its top 64 bytes unaligned once and every other store aligned to its width, since a store split across two lines cost both benches about as much as the stall.

The kernel's copy is #1139's, which also took the kernel issue this PR once carried. #1150's round 7 found that the paged read's user copies run at distance 0, so the kernel has no part in this.

Since the last review

• Round 7 failed D3, and its small copies paid 5–7% because memcpy called copy_forward. Round 8 moved the route out of line from 4 KiB: small copies recovered (0.985–0.99), but D3 still failed on mem.memcpy.262144 (0.84). It also measured copies under 4 KiB stalling 3–13× at d16.
• Round 9, by the orchestrator's ruling: the distance check is inline from 256 B, and the 256 KiB row is tested against buffer placement.
• Review r8's record findings are taken: the issue's exit C6 covers short copies, the T14 window is recorded as a choice within a taper, and ROUTED_LEAST's comment is corrected.

Checks

• Host CI: cargo run -- --ci host at 97ae8d6: 108 steps green, exit 0 (log).
• libc tests toyos-libc-copies at 97ae8d6, x86-64 under Rosetta and arm64 natively: exit 0 each.
  • Every page distance below 1 KiB, at lengths 0–8192 including 255/256/257, with both ends in one 64-aligned buffer.
  • Both loops called directly, at every destination offset in a 32-byte store.
  • The overlap guard, including overlaps under 4 KiB (n from 3601). Negative control at 7c67bbf: guard removed first fails at gap 3600 n 3800, exit 101; restored. The tests pass at 7c67bbf, exit 0.
  • Round 5's head- and tail-store mutations still apply: the loops are unchanged.
• Review of 97ae8d6: LAND, no blockers. Its suggestions are taken at 7c67bbf, which changes only tests and records, so round 9's images still apply.
• Guest copyat_runs at fb4ca4a (guest-rows run 42024430030, x86-64 QEMU with AVX): PASS, 5 steps green. That covers every routed copy, now from 256 B, checked against its source, and the placement rows running to exit 0. log
• AVX loop: Rosetta has no AVX, so the host tests cover only SSE2. QEMU's x86-64 CPU declares +avx, and the guest's copyat_runs checked every routed copy against its source at round 5: PASS.
• Metal, round 5 (base 80421e7 against head 0f861df, four boots per bench):
  • AMD: routed 8 KiB copies 80–86 ns against 2615; the heap pair 2570 → 81 ns; 64 KiB at d16 20.4 → 1.0 µs; 1 MiB at d16 330 → 20 µs. It lost at 4 MiB with d256 (597 against 385 µs), which the 1 MiB cap on every CPU answers.
  • T14: routed copies 82–103 ns against 714–773 at d 16–64; the heap pair 121 → 92 ns. Since round 5 the copy code changed only by dropping the cap's ERMS condition, which the T14 already met, so these rows stand.
  • IPC: ipc.reqrep.* 0.99–1.00 on both benches.
• Metal, round 7 (head before the restructure) (AMD only; base4 6cdf182 = main + measurement, head4 993461c, branch amd-memcpy-ab4; four boots base, head, head, base, all passed; rule, judgement):
  • D1 holds: main still stalls. 8 KiB at d16: base 1953 ns against 90 at d0.
  • D2 holds: routed 8 KiB copies at d 16..384 take 80 ns.
  • D3 FAILS: mem.memcpy.262144 read 36905 MB/s against base's 40756, 0.906 against the 0.909 bound. Both pairs agree. That copy is at distance 0 and never routed, and its only new code is two compares, so the cause is unexplained. The row is on record as unstable on this bench (round 5's base 19390/29365, the scoreboard's 45% range). Small rows lose 5–7% (64 B 0.93, 256 B 0.93, 1 KiB 0.95), as round 5 measured: the hot path's cost, within the bound.
  • D4 holds: the heap pair 1953 → 80 ns.
  • D5 holds: 64 KiB at d16 20.4 → 1.0 µs, 1 MiB at d16 330 → 21 µs. Round 5's 4 MiB d256 loss is gone (383.7 against 383.6 µs), since above 1 MiB both arms run rep movsb.
  • D6 holds: ipc.reqrep.* 1.00–1.01 in pair 1; base boot 4 was the slow one in pair 2.
• Metal, round 8 (code with the 4 KiB threshold) (AMD; base5 a049276 against head5 d7e0c70, which carries this PR's code at 643b89b; two pairs, all passed; judgement):
  • D1, D2, D4, D5 and D6 hold.
  • Small rows recovered: mem.memcpy.64 0.985, .256 0.987, .1024 0.99. .4096 reads 0.97, because those copies take copy_long's call.
  • D3 FAILS: mem.memcpy.262144 0.84. Copyat's unrouted 256 KiB d0 copy is 1.13× slower too, while 64 KiB and 1 MiB are equal. The 256 KiB copy runs about 15 instructions more before the same rep movsb, which cannot explain 1–1.5 µs. Unexplained.
  • Below 4 KiB (unjudged, base): d16 against d0 reads 90 against 30 ns at 256 B, 270 against 31 at 1 KiB and 511 against 40 at 2 KiB. These copies stall and are not routed. The issue's new exit C6 records this.
• Metal, round 9 (AMD, base6 85c7607 against head6 fb4ca4a; boot 4 was rerun alone about an hour after boot 3; judgement):
  • C6 holds: at d16, 256 B 90 → 20 ns, 1 KiB 270 → 20, 2 KiB 511 → 30. mem.memcpy.256, .1024 and .4096 read 0.98–0.99.
  • D1, D2, D4 and D5 hold. D6 holds by its pair clause: pair 1 reads 0.98–0.99, while pair 2 is skewed by the rerun base boot, which read every ipc row faster, the 8-byte row included.
  • D3 FAILS: mem.memcpy.262144 0.858. P fails: the placement spread (base 1.12, head 1.09) is under the gap, and head's median is 0.90 of base's. So the gap does not follow buffer placement. Reported, not judged: head's two boots differ (placement rows 0.97 and 0.81 of base), while base's agree.
• Metal, round 10 (AMD, nine boots: base B, a shifted base S whose memcpy is byte-identical to B's but sits at the head's address, and head H; rule, judgement):
  • Q1 holds: H reads 0.863 of B on mem.memcpy.262144.
  • Q2 fails as written, so by the rule the gap is the PR's.
  • But on mem.memcpy.262144 S reads 0.80 of B, lower than H; the clause failed because S fell below H's range.
  • On the placement rows S reads 0.98 of B. H reads about 0.82 in two of its three boots and B's level in the third.
• Metal, round 11 (AMD, five B/H pairs, B b76abc3, H 3e491cf; judgement): Regression, 5 of 5 pairs (placement-row Q_H/Q_B 0.795, 0.931, 0.926, 0.927, 0.927). Buffer virtual addresses and clocks are identical in both arms. mem.memcpy.262144, in a separate process that never runs the AVX loop, reads 0.84.

Why metal: the stall is in the CPU's load/store unit, and QEMU's timing cannot show it. Supported: the T14 and the AMD bench.

Evidence: rounds 1–6 at https://github.com/ToyOSOrg/evidence/tree/782346f21f39f6b2e87329546febaf433a8f7c2f/amd-memcpy, round 7 at https://github.com/ToyOSOrg/evidence/tree/459eb1230c2458f815dd74886eb14a5d6d1304d1/amd-memcpy/round-7, round 8 at https://github.com/ToyOSOrg/evidence/tree/675b23c48eeeeea932715455eca793b4284edb78/amd-memcpy/round-8, round 9 at https://github.com/ToyOSOrg/evidence/tree/dfd53b1e0d6f481c0812ad62ece2febc1c1d7e0b/amd-memcpy/round-9, round 10 at https://github.com/ToyOSOrg/evidence/tree/6d717e2ad8683be5d02947ddc57dd3576e40ada6/amd-memcpy/round-10.

Recorded limitations and risks

Recorded in issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md (exit: round 5's C1, C3 and C4 as written; the 64 B outlier explained):

• mem.memcpy.262144 on the AMD: 0.906, 0.84 and 0.858 in rounds 7–9 (D3). Round 9's placement test rules out buffer placement. About 6 extra instructions cannot explain 1 µs, so the cause is unexplained; code placement is untested.
• AMD above 1 MiB: the stall stays as on base (4 MiB at d16 3.4×, 64 MiB at d16 2.0× of d0).
• T14 routed copies: 1.1–1.4× of rep movsb's best (10× before). Where aligned stores leave the loads 16 bytes off a 32-byte boundary, the T14 pays for split loads.
• T14 mechanism: what decides its stall is not established; the window is set by copyat's rows.
• Other CPUs are unmeasured. The route is correct on any x86-64.
• The T14's mem.memcpy.64 outlier boot is unexplained.
• The Intel optimization manual's 4K-aliasing citation is not quoted in the evidence; only AMD's §2.12 is.

🤖 Generated with Claude Code

[Japabu]

Review r1 of #1133 at 4a3aa62

Architecture

The owner is right. sdk/libc's memcpy/memmove are what std links into every userland binary (build/libc.rs, CRATE), and C programs get them through the staticlib, so Rust's copy_nonoverlapping/copy_from_slice and C's memcpy both take the new route. Recording the kernel's compiler_builtins copy as an issue instead of fixing it here matches the accepted scope. The literal 4096 is right as written: it is the width of the alias compare (bits 11:0), not PAGE_SIZE. The change is small: about 35 lines of production code, 44 of tests and a 29-line issue.

I checked copy_forward/copy_down_64 (x86_64.rs:33–81) by hand and found them correct:

• Bounds. With p = n ≥ 64, every load and store in the loop is [p, p+64) with p ≥ 1 and p + 64 ≤ n. The head is [0, 64). Nothing reads or writes outside either buffer. movups takes any alignment. The last loop step ends at p ∈ [1, 64], so together with the head it covers everything from 0 up.
• Order. The head is loaded before any store and stored last. The loop loads each 64-byte block and then stores it, from the top down. Between disjoint ranges this is exact.
• memmove. The forward branch is taken when dest ≤ src or dest ≥ src + n. An overlapping dest < src fails disjoint and stays lowest-first with rep movsb. A disjoint dest > src is routed safely. dest == src has distance == 0 and is not routed. A memcpy with UB overlap and dest < src also stays lowest-first, which is benign.
• The alias argument. At distance k ∈ (0, 512), the next step's loads [p-64, p) never meet the stores just made at ≡ [p+k, p+k+64) mod 4 KiB. The one store a load could match lies 4096 − k bytes, or ≥ 56 steps, earlier. That is 224 stores, well beyond Zen 2's store queue.
• asm operands. The clobbers (xmm0–7, flags by default, memory by default) are complete.

---

BLOCKER

sdk/libc/arch/src/lib.rs:224–237 — the "page distance" this test sweeps is not the copy's distance.
src and got are separate heap allocations, and got is reallocated on every iteration. The copy's real distance is (got_base − src_base + d) mod 4096, not d, and the allocator decides got_base − src_base.

The implementer's own negative control shows it. In r3/mutation-head.log the head-store mutation breaks every routed copy, but the first failure is reported as n 1000 at 1 distance 109. If the label were true, the test would have failed first at distance 1, at 0, n 64. So on the Rosetta/macOS heap, labelled distances 1–108 were not routed at all. Under glibc on CI's x86-64 runners, got normally lands right after src's chunk, so the sweep is shifted by a constant. Distances 1..~79 (16, the case that matters, among them) and the 511/512 edge are then probably not exercised.

Consequence: a defect specific to a distance, or to the 512 boundary, can land green. The test's claim, the module header's added line (lib.rs:5, "a forward copy at every page distance below 1 KiB") and the PR's Checks bullet are all false as stated.

Cheapest fix: take both ends from one buffer, aligned to 64 by construction. For example, allocate pattern(...) once, pick src at a 64-aligned offset, put dest at src + 4096 + d inside the same allocation, and compare the whole buffer. The distance is then exact, and the "line offset" at really is an offset within a line. Rerun the head and guard mutations afterwards: the head mutation should report distance 1.

Evidence gap — routed copies above cache size are unmeasured.
The route applies at every n ≥ 64. The only routed rows in the A/B rule are 8 KiB (A1/A3) and possibly the 64 KiB IPC rows (A5). mem.memcpy.* uses two huge-page mappings at distance 0 (tests/guest/osbench/src/mem.rs:237), so A2 never measures the SSE2 path.

Failure scenario: a 1 MiB or 64 MiB copy at distance 16 or 256 moves from rep movsb's fast-string path to 16-byte temporal stores with RFO, and could get slower than base on the T14. Nothing in the planned run would show it.

Cheapest check: add copyat rows for a routed large copy (for example 1 MiB and 64 MiB at d = 16 and d = 256) to the base/head pair on both benches. If a regression shows, limit the route by size and record why. If the A/B already ran, one base/head pair per bench with those rows is enough.

---

SUGGESTION

• sdk/libc/src/arch/x86_64.rs:30–31. "the T14's Tiger Lake measures the same stall" puts a bench measurement in the source. Intel documents the same mechanism as 4K aliasing in the Intel 64 and IA-32 Architectures Optimization Reference Manual. Cite that, and leave the T14 numbers in the commit and evidence.
• sdk/libc/src/arch/x86_64.rs:51–52. The SHORT_ALIAS comment says the window is "as far as either bench was measured slower forward than down". 512 is not that: it is a choice between 256 (measured slow) and 1024 (measured fast), and A4 checks it. Say only what a reader needs, for example "a forward copy at a shorter distance stalls (measured to at least 256 on Tiger Lake)", or let the commit carry it.
• issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md:11. The issue cites the ToyOS branch amd-memcpy-ab, whose commit "never lands". Cite the evidence repository path so the reference survives the branch's deletion.

Net: about +35 production lines, +44 test lines and +29 issue lines. The edit to the six-tenths issue is accurate: mem.memcpy's buffers are two huge mappings at distance 0.

SEND BACK

[Japabu]

Review r1, answered at 15a8464:

• BLOCKER, the page-distance test. Fixed. Both ends now come from one buffer, aligned to 64 by
  construction, with the destination at src + 8192 + d. I reran both controls: the head-store
  mutation first fails at copy_forward: n 64 at 0 distance 1, and the guard mutation at
  gap 3600 n 4100 (3584 is distance 512, which is not routed).

• Evidence gap, routed copies past the caches. Rows at 1 MiB and 64 MiB, distances 0, 16 and 256,
  are added and staged as one base2/head2 pair per bench (rule B4).

• The round-3 A/B came back meanwhile and failed on the T14:

  • the SSE2 loop read 136–151 ns against 62 at distance 0;
  • mem.memcpy.64 was 16% slower;
  • the AMD regressed from 90 to 140 ns at distances 32–384.

  So this revision makes the loop 32-byte AVX where CPUID and XCR0 allow, SSE2 otherwise. The overlap
  check and the AVX choice are out of line, so the hot path pays two compares. The round-4 pair also
  measures the AVX loop at every distance to set the window (B5). AVX correctness is checked in the
  QEMU guest, because Rosetta has no AVX.

• Suggestions. All three taken: Intel's 4K aliasing replaces the bench reading, SHORT_ALIAS's
  comment is cut, and the issue cites the evidence repository.

[Japabu]

A note from #1139 (the kernel's memcpy) on this PR's issues/a-kernel-copy-a-short-way-above-its-source-modulo-4-kib-stalls.md.

The issue's premise is wrong. It says the kernel's memcpy is compiler_builtins' single rep movsb. The sysroot is built without ermsb, so it was actually rep movsb; rep movsq; rep movsb.

Its exit is met by #1139. #1139's metal A/B ran four boots per bench. Base kernel reads were flat across distances, so the old sequence did not stall at short distances. #1139 replaces it with one rep movsb, and on AMD it sends copies under 32 bytes of distance highest first. With that:

• a kernel tmpfs read 16 bytes above its source takes 1.17× (AMD) and 1.10× (T14) the same read at distance 0, for 8 KiB;
• reads at every other distance are 0.92–0.96× base.

Evidence: https://github.com/ToyOSOrg/evidence/tree/96718d0140daaf3ee44722bf72f5b711e815fc84/kmemcpy/round-3/judgement.md

Whichever of #1133 and #1139 lands second deletes the issue file. #1139 never added it. If #1133 lands second, it should drop the file from its branch.

[Japabu]

Round 7: does #1133 still claim something? Yes, on the AMD

Checked after #1139, #1150's round 7 and main 15c26bf:

• Vendor fact, read from the document. 56305 rev. 3.02 §2.12, p. 39: "The processor uses address bits 11:0 to determine STLF eligibility. Avoid having multiple stores with the same 11:0 address bits, but to different addresses … in-flight simultaneously where a load may need STLF from one of them." The document is the Software Optimization Guide for AMD Family 17h Models 30h and Greater Processors. The comment had called it "for AMD EPYC 7002", and that is now fixed.
• Metal. Two independent instruments measured libc's forward rep movsb stalling on Zen 2 at d < 32:
  • this PR's copyat read 2615 ns against 90 at 8 KiB and d = 16;
  • kernel: memcpy goes highest first where a forward copy would stall #1139's in-process rows stall at d = 8, 16, 24 and 31 and at nothing tried from 32 up, at 4–64 KiB.
• What changed since. kernel: memcpy goes highest first where a forward copy would stall #1139 owns the kernel's copy and took the kernel issue. kernel: user copies go through the process's own mapping, and none runs under a lock (S0-a's lock half) #1150 r7 found the paged read's user copies at d = 0. Neither touches libc, so they reduce the scope only to userland, which is what this PR already was.

Correction to the record: the T14's 512 window rests on copyat's rows alone. A heap pair 64 KiB + 16 apart and #1139's in-process copy at d = 16 did not stall there, so the issue now says the Tiger Lake mechanism is not established.

Rebased (no conflicts). Host CI exit 0, and Rosetta x86-64 tests exit 0. The AMD A/B on today's main is staged (rule D1–D6, stated before the run; D1 closes the PR if main no longer stalls): https://github.com/ToyOSOrg/evidence/tree/eeac5483dfdaaebade0685885018f9b098d40186/amd-memcpy/round-7

[Japabu]

Review, round 8: head abfe08b against main 15c26bf

Architecture. The fix belongs in libc, and that is where it is. libc owns userland's memcpy, and the kernel's copy (#1139) is a separate contract because the kernel cannot use vector registers. The restructure is the right shape. copy_forward is inlined and returns dest. Copies under 4 KiB run main's rep movsb behind one cmp/jbe. Everything else tail-jumps to an out-of-line copy_long, so short copies no longer pay for the route.

I checked the disassembly in memcpy-disasm.txt (head5 d7e0c7037 has the same sdk/ as abfe08b). It shows main's instructions plus cmp rdx,0xfff; jbe and one redundant mov rdi,rax. That supports the claim that round 7's 5–7% came from the call and the rbx save.

Net change: about +150 lines of production code in x86_64.rs, +75 lines of tests and a new 74-line issue. Most of that is the two descending loops and CPU detection, and both earn their place.

The libc copy goes down on every x86-64 CPU over a 512-byte window. The kernel's copy uses a per-vendor rule: AMD below 32 bytes, NEVER on Intel. Both choices rest on different loops and different recorded measurements, so they do not contradict each other.

Correctness (checked by reading the code)

• Overlap. memmove sends dest <= src or dest >= src + n to copy_forward. In copy_long, abs_diff(dest, src) < n catches every overlapping case and sends it to rep movsb, which copies lowest first. The guard's mutation turns the overlap test red. A disjoint copy downward has no ordering constraint, so it is correct whichever side dest is on. wrapping_sub % 4096 gives the right distance when dest < src, because 2^64 is a multiple of 4096. A copy with dest == src has distance 0 and uses rep movsb.
• Loops. I traced both loops for n ≥ 64. p starts at aligned_end, which lies in [n-w+1, n]. The loop exits with 0 < p <= 64, so the 64-byte head store and 64-byte tail store cover everything else. Every movaps/vmovdqa address is dest + aligned_end - 64k, which is aligned to w, and no store falls outside [0, n). The clobbers are complete: xmm0–11, and ymm0–15 for the AVX loop because of vzeroupper. The loop bodies are byte-identical to round 5's head 0f861df, so round 5's QEMU AVX check and mutations still apply.
• Direction flag. movsb relies on the ABI's DF = 0. copy_backward (unchanged) sets DF only across its single rep and clears it afterwards.
• AVX detection. It checks CPUID.1:ECX bits 28 and 27, then XCR0 bits 1 and 2. That is correct, and the kernel saves YMM state through XSAVE (kernel/src/arch/x86_64/fpu.rs).
• Threshold boundary. n >= 4096 takes the long path, and the tests cover 4095, 4096 and 4097. AArch64 only changes its return type, and memory.rs handles both architectures the same way.
• Vendor citations. I checked both against the vendor's own text of 56305. "uses address bits 11:0 to determine STLF eligibility" is in §2.12, and the vzeroupper advice is in §2.11.6 "Mixing AVX and SSE". The title is now correct.

Evidence

• Host CI. The log starts with rev abfe08bf4…, reports 108 steps all green, exits 0, and includes toyos_libc_copies (108 tests passed, arm64 debug).
• libc tests. Both the Rosetta x86-64 and native arm64 runs show 108 passed and exit 0. They were run at 643b89b; the only later change touches the issue file, so they still apply.
• Negative control. The mutation was made at 19f36d8, before the return-type change, but the guard line it removes is unchanged at head. With it, an_overlapping_forward_copy_a_page_apart_agrees fails with exit 101. Valid.
• AVX loop. No host covers it (Rosetta has no AVX). The round 5 guest check covers it, and the loop has not changed since.
• Metal. I checked the round 5 table against the issue's numbers: T14 base 74 / head 59 at d0, 82–103 routed, 714–773 at d 16–64, and heap 121 → 92. They match.

Findings

BLOCKER

1. issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md:52-56 and its Exit — the "Below 4 KiB, unmeasured" bullet has no exit, so the issue can close with it unresolved.
   • Why it matters. The vendor mechanism the code cites does not depend on length. Any forward copy longer than d loads bytes that match recent stores on bits 11:0. On the T14, distances 32–511 are routed only from 4 KiB. So ROUTED_LEAST = 4096 is the shortest length anyone has tried, not a measured lower bound.
   • No cost reason for 4 KiB. After this restructure the threshold only decides which copies pay the out-of-line call. Copies below the threshold pay nothing for it, so the ruling that small copies must not pay does not set the value at 4 KiB.
   • Round 8 already measures this. Its 256/1024/2048 B rows at d0 and d16 measure exactly this question, but the rule leaves them unjudged.
   • Fix. Add an exit criterion: judge those rows, and lower ROUTED_LEAST to the shortest length that stalls, or record that none does. "No silent debt" requires an exit as well as the observation.

SUGGESTION

• sdk/libc/src/arch/x86_64.rs:85-86 — "the shortest length a stall has been measured at" suggests shorter lengths were measured and did not stall. "the shortest length measured; a stall there is" would say what happened. (The next sentence does already state that shorter copies are unmeasured.)
• Issue, T14 bullet — "The 512 window is set by these rows" overstates how sharp the edge is. Base reads 122 at 512, 108 at 640, 99 at 768 and 88 at 1024, against 74 at d0. The slowdown tapers off; it does not end at 512, and 512 itself is not routed. One clause saying the edge is a choice within a taper would keep the record honest.
• sdk/libc/src/arch/x86_64.rs:40-44 — "Intel calls it 4K aliasing (… Optimization Reference Manual)" gives no section, and per the handoff the implementer has not read it. The PR records this. Adding the section number when someone reads it is enough; it does not block.

Round 8's D3 / 256 KiB gap is excluded from this review, as instructed.

SEND BACK (one blocker: the issue's exit for copies under 4 KiB)

[Japabu]

Review, round 9: head 97ae8d6 (changes since abfe08b)

Architecture. The shape is unchanged and still right: libc owns userland's memcpy, and the route stays in the x86-64 architecture module. The only structural change moves the distance compare from the out-of-line copy_long into the inlined copy_forward, and renames the out-of-line half copy_short_alias. Now only a copy that will actually be routed pays for a jump. This is the smallest way to route 256 B–4 KiB copies without giving every such copy a call, so it answers round 8's C6 blocker directly. Net change since abfe08b: +44/−38 lines across the code, tests and the issue. Production code shrank by a few lines. Nothing from the measurement branch leaks in: git diff 97ae8d604 fb4ca4a2b touches only commit-measure.txt, tests/guest/osbench/*, tests/images/benchcase.toml and tests/toyos.rs, and sdk/ is identical. So the copyat.place.* rows and copyat_runs stay on amd-memcpy-ab6.

Correctness (checked by reading the code and the disassembly)

• Distance arithmetic. mov ecx,eax; sub ecx,esi; and ecx,0xfff computes (dest − src) mod 4096. Doing it in 32 bits is exact, because 4096 divides 2^32. dec maps 0 to 0xffffffff, so ja skips it. 1..511 map to 0..510, and 510 = 0x1fe, which is the inclusive bound for jbe/not-ja. So the routed set is exactly 1..511, the same as copy_long's old distance.wrapping_sub(1) < SHORT_ALIAS − 1. When dest is below src by k, the result is 4096 − (k mod 4096). It is routed only when k mod 4096 is in 3585..4095, the case the overlap guard exists for. Wraparound across the address space reduces mod 2^64 the same way.
• Overlap, both directions. memmove decides the overlap before copy_forward, so dest > src with overlap still goes to copy_backward. dest == src has distance 0 and is not routed. Every routed forward overlap (dest < src and src − dest < n, which needs n ≥ 3586) hits abs_diff < n and runs rep movsb. A disjoint routed copy goes down, which is correct whichever side dest is on.
• Lengths. cmp rdx,0xff; jbe sends 255 to rep movsb and routes from 256, which matches ROUTED_LEAST = 256. At 4095 and 4096 the behaviour is now the same, since nothing changes there. In the loops, aligned_end ≥ n − 31 ≥ 225 at n = 256, so both loops run at least three iterations and exit with 0 < p ≤ 64. The preloaded head and tail cover the rest. Round 8's analysis holds for every n ≥ 64, and 256 is well above that.
• Tail jump. memcpy and memmove restore rdi and pop rbp before jmp copy_short_alias, with rsi and rdx untouched. copy_short_alias returns dest in rax. This is correct.
• Direction flag. Unchanged. The forward path relies on the ABI's DF = 0, and copy_backward sets DF only across its one rep.
• Under 256 B. head6's memcpy and memmove forward branch are base6's instructions plus cmp rdx,0xff; jbe and one redundant mov rdi,rax. The README and PR body say so. head6 carries the same sdk/ as 97ae8d6.

Evidence

• Host CI. The log begins rev 97ae8d604… and ends "108 step(s), all green", exit 0.
• libc tests. Rosetta x86-64 and native arm64 each show 108 passed, exit 0.
• Negative control. The patch removes only the abs_diff guard. an_overlapping_forward_copy_a_page_apart_agrees fails at gap 3600, n 4100, exit 101.
• AVX loop at 256 B–4 KiB. No host run covers it. The loop is byte-identical to the one round 5's guest check passed, and its arithmetic does not depend on length above 64. Guest run 42024430030 at fb4ca4a is still queued. Its copyat checks every 256/1024/2048 B d16 copy against its source under QEMU's +avx. Record its result before landing. It confirms rather than establishes, so it does not block.

Findings

BLOCKER: none.

SUGGESTION

• sdk/libc/arch/src/lib.rs:282 — the new over values 1 and 64 give the only overlap cases below 4 KiB (n 3601–4159). They cannot detect a missing guard. The descending loops load the source's first 64 bytes before storing anything, so an overlap of 64 bytes or less is never read after it is clobbered. That is why the mutation's first failure is gap 3600, n 4100. The guard does not depend on length, so this is not a gap in what the code needs. If the intent was to cover a routed overlap under 4 KiB, use over = 200 instead of 64 (n 3800–4295): those cases would catch the mutation.
• issues/a-user-copy-a-short-way-above-its-source-modulo-4-kib-still-costs.md:52-54 — "round 9 measures 64 and 128 B" describes a pending run. It goes stale once that run lands, so replace it with the result when round 9 is judged. Also, "routed from 256 B since round 9" names a measurement round, not a revision.
• Same file, C6 (:73-76) — "every mem.memcpy.* row within 1.1 times main's" includes mem.memcpy.262144. That row failed D3 at 0.84 for a reason no one has tied to short-copy routing. As written, C6 can fail because of that unrelated question. Excluding the row while D3/P is open, or pointing at their outcome, would keep C6 about short copies. I am not reopening D3 itself.

LAND (no blocker. Round 9's metal judgement and the queued guest run are outside this review and remain the orchestrator's landing conditions.)

[Japabu]

Closed by round 12's binding outcome

Round 12 ran twenty AMD boots in five rotated blocks of base B, head H, head with AVX off N, and base with memcpy moved to H's address S. The rule was written before the run; judgement.

• Premise holds: H drops on the 256 KiB placement rows in 5 of 5 blocks (0.80–0.94 of B).
• A fails: N, the AVX-off head, drops too, in 5 of 5 blocks (0.89–0.94). Its 8 KiB routed copies also read 140 ns against the 127 ns bound.
• P fails: S, with base's memcpy moved to H's address, clears in all blocks (1.03 of B). The cost follows the routed code, not memcpy's address, and its mechanism is unknown.

So by the rule this closes. The stall it measured stays real: AMD forward copies 1–31 bytes above their source modulo 4 KiB take 3× at 256 B, 22× at 8 KiB and 3.5× at 1 MiB (56305 rev. 3.02 §2.12). #1198 records the stall, this PR's fix and its unexplained 256 KiB cost as a finding, with an exit for a future approach.

Branches kept for reference: amd-memcpy (e936567), amd-memcpy-sse2 (7e7019f, the SSE2-only candidate).