[Japabu]

Reviewed in two rounds; head abae9a3 on main 24a538c. The parity boot ran: the 300 ms mark, p50 and loss pass; the 100 ms mark misses the pre-set rule by 30 KB, so the ramp issue stays open with the reading.

Problem

On the T14's LAN path a new ToyOS connection delivers 2–3 MB in its first 100 ms and about 24 MB by 300 ms, against Linux's 6 MB and 28–29 MB (issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md). In #1112's session a diagnostic arm, L = ∞ without pacing, reached Linux's marks (5.2–6.4 MB and 28.3–29.7 MB) and resent nothing. So RFC 9406 §4.3's L = 8 is what holds the ramp back. That section gives L = ∞ only to a paced sender, so the unpaced arm could not land.

Change

• Pacing, per connection in [tcp] (src/pace.rs, RFC 9002 §7.7). A token bucket fills at N · cwnd / srtt, and every data segment sent spends from it.
  • Capacity: the bucket holds 1 ms at the rate, at most half of cwnd, and never less than the initial window.
  • The millisecond departs from §7.7, which bounds a burst at the initial window unless the path is known to absorb more. The millisecond sizes a fast sender's bursts, and so its wakes, by time rather than by the initial window.
  • The half-window bound keeps the bucket below cwnd at any srtt, so a window past the initial one always leaves in more than one burst. That is what lets L = ∞ hold whatever the round trip (review r1: before it, at srtt ≤ ~2 ms the bucket was cwnd or more and never held).
  • Release: new data the bucket can't cover is held, and offered again when half the bucket has come back. The sender wakes once per half the bucket's refill time, and a wake up to that much late loses no rate. That is half a millisecond while the millisecond sets the bucket, less where half the window sets it (125 µs at cwnd 146,000 and srtt 1 ms), and more where the initial window does. In a small window that wait is half the initial window's time at the rate, even when one segment would do.
  • Wakes: the release instant is a connection deadline. Netstack already waits on Tcp::next_deadline, so no thread or timer is added.
• Spent but never held, each with its reason in the module header:
  • retransmissions: recovery's pipe and a timeout's one-segment window bound them;
  • new data in SACK recovery: RFC 6675's pipe and the ACK clock bound it, as they do retransmissions;
  • a loss probe of new data: RFC 8985 §7.3's single segment at the PTO, which a hold would only delay;
  • a persist probe of new data: one segment into a closed window (RFC 9293 §3.8.6.1);
  • ACK-only segments aren't paced at all (§7.7).
  • Without a smoothed RTT nothing is paced; cwnd is at most the initial window until the first sample.
• L = ∞ (RFC 9406 §4.3, for a paced sender): HyStart++ grows the initial slow start by all an ACK acknowledged, or a quarter of it in CSS.
• N is 1.25 in avoidance, the value §7.7 gives, and 2 in slow start. Slow start doubles cwnd across the round trip, so a round's data is twice the window it began with, and 1.25 · cwnd / srtt would hold the doubling back. The Q arm below measured 1.25 throughout and lost to 2 at 100 ms in every pair.
• limited: a pacing hold leaves Cc::limited as the last send set it. Clearing it would stop slow start's growth and CUBIC's clock while the pacer spreads a window that is in use.
• Counter: the time a pacing hold keeps data waiting is counted as tcp.send-limited.pacing.ns, and osbench reads it.
• The departure record (tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112): a send joins a short run only if it is short too, as before tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112's review r2. HTTP's shape is a short header, then a window of body. Without this rule the body joins the header's run, and all but its first partway ACK go untimed. The bound returns to two runs per SMSS in flight plus two, about 16 KB at 250 segments. This was review r3's optional suggestion on tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112.
• Tests changed:
  • s_cc_017 and s_cc_018 follow L = ∞. s_cc_017's boundary is now 10 of the window's 20 segments.
  • Two net.rs downloads on a 15 ms gigabit path asserted that the receive window reaches the whole 4 MiB buffer. The window grows to twice what is read per round trip, 3.75 MB on that path. Unpaced, it reached the whole buffer; paced, it stops 3 KB short. They now assert the rule's bound, which still separates them from the 65,535 they guard against.
• The ramp issue stays open. It gains the A/B as relative evidence and the parity boot's reading, which misses the pre-set rule at 100 ms by 30 KB (Metal below).

Tests

• pace.rs (unit tests):
  • no smoothed RTT, no pacing
  • a slow bucket holds the initial window and releases at half (1.25 ms at 2 · 29,200 / 10 ms)
  • avoidance's 1.25 (2 ms)
  • a fast bucket holds a millisecond (250 segments at srtt 8 ms)
  • new: a bucket holds at most half the window (50 of 100 segments at srtt 1 ms)
  • an idle bucket fills only to its capacity
• tests/pacing.rs, through the connection:
  • a window past the bucket leaves 5 + 5 at 11.25 and 12.5 ms, and the 2.5 ms hold is counted
  • an ACK during a pacing hold grows cwnd
  • an acknowledgment leaves through a hold
• s_rt_025: a 200 B header, then four segments of body 1 ms later; four ACKs give four samples. Its control (any send joining a short run) fails it with exit 101.
• At 3a0744d: [tcp] cargo test --no-fail-fast 442 passed, exit 0; clippy on the crate clean. f7c9eb2 and ff0dc5a change only comments (pace.rs's header, one issue bullet), so these checks and the parity boot of 3a0744d still apply; abae9a3 changes only the issue; the pacer's unit tests and clippy were rerun on it, clean.
• Eight controls at 3a0744d, each applied, run and reverted, all exit 101:
  • N = 1.25 in slow start
  • release at one segment
  • no initial-window floor
  • no millisecond term
  • no half-window bound (fails the new test)
  • a hold clearing limited
  • no pacing deadline
  • L = 8
• Not covered by a test: data that spends but isn't held spending from the bucket. No harness scenario made the spend observable.
• cargo run -- --ci host at 3a0744d: 107 steps, all green, exit 0.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/bdee450e90704beb00206486e052500b4d099c47/r6 (this round and the staged boot; earlier rounds in r1–r5)

Metal

Parity boot, 2026-11-01 (3a0744d on current main, between two Linux runs; image and binary hashes checked first; every run rc 0, the judge exit 0). The Linux binaries were rebuilt at this head, and the peer's are byte-identical to the A/B's.

	p50 MB/s	100 ms	300 ms	1 s	2 s	resent	pacing hold, ms	css / entry rounds	RTT samples, departure / echo
Linux L1, L2	113, 113	6.10, 6.38	28.63, 29.56	109.1, 110.5	224.6, 226.1	0
ToyOS p1	114	6.07	29.19	110.7	227.0	0	73.3	420/134	33,319/0

• The rule set before the run (at or above the lower Linux run at 100 and 300 ms, p50 within 1 MB/s, nothing resent) is not met. 300 ms, p50 and loss pass. The 100 ms mark is 30 KB short (0.5 %).
• That gap is under the marks' own spread (280 KB between the two Linux runs, 260 KB across the A/B's P runs), so one boot can't separate it from noise.
• The ramp issue stays open with this reading (abae9a3). Closing it needs another boot, or a judgement on the rule, which is yours to make.
• The pacer held 73 ms in the row, against 33–41 ms with the A/B's bucket: the half-window bound holds more on this short round trip, and the ramp held.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/8f08a8e9399817fc669e395dc717ee236a18ed35/r7/t14

A/B, 2026-10-31 (relative evidence; every boot rc 0, every judge exit 0). Peer: the AMD bench's Linux over its Wi-Fi. Order: Linux L1, c1 p1 q1 c2 p2 q2 c3 p3 q3, Linux L2.

• C = 15fdb0e (tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112 after its review r1)
• P = b3aee3b (this PR on it, with the earlier bucket)
• Q = P with N = 1.25 throughout, a diagnostic arm that does not land

Marks are each row's mean over five connections, received by the peer, in MB:

	p50 MB/s	100 ms	300 ms	1 s	resent	pacing hold, ms	css / entry rounds	RTT samples, departure / echo
Linux L1, L2	112, 112	5.68, 5.71	26.07, 26.23	106.4, 105.7	0
C	112, 112, 112	2.84, 3.07, 2.03	24.99, 25.35, 23.83	105.4–106.8	1,448 B in c1	0	678/166, 564/165, 462/137	32,805/2, 33,038/0, 32,647/0
P	114, 114, 114	6.34, 6.39, 6.60	29.29, 29.87, 29.80	110.9–111.1	0	32.7, 40.6, 40.7	374/104, 488/130, 619/136	33,398/0, 33,002/0, 33,378/0
Q	114, 114, 114	5.89, 6.25, 6.27	29.12, 29.45, 29.53	110.2–111.0	0	95.4, 96.4, 79.7	439/114, 763/205, 599/190	32,656/0, 33,628/0, 33,177/0

• P vs C: P's 300 ms mark beat C's in all three pairs (one-sided p = 1/8), with p50 114 against 112, and P resent nothing.
• N: P's 100 ms mark beat Q's in every pair, by 0.1–0.5 MB. Q held data two to three times as long under the pacer. Neither arm lost anything.
• The pacer: it held 33–41 ms per row in P, so it acted on this path but rarely.
• tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112 on metal: in C every sample but two came from a departure. The two echo samples were in c1, the row with a resent loss probe.
• CSS: 3.2–4.6 entries per entering round, against 2.0–2.7 in tcp: an ACK is timed from its newest segment's departure, at the clock's resolution #1112's session, but fewer rounds entered (104–205 against 229–327). Nothing exited slow start through CSS.
• Evidence: https://github.com/ToyOSOrg/evidence/tree/58dcdecab592116996c0e3cc84e4e3c8b7aa50c7/r4/t14

This needs metal because the claim is the ramp on a real gigabit path against Linux on the same machine. The host harness establishes the pacer's arithmetic and the connection's behaviour, but not a real path's queue.

Risks

• The half-window bound costs wakes on short round trips. A held connection wakes once per half bucket, which in slow start is every min(0.5 ms, srtt / 8), and never more than once per half initial window sent. At the T14's early ~2 ms srtt that is about 4,000 a second per connection while it ramps. The parity boot read the result, not the cost.
• On this path the line itself (1 Gb/s) spread most bursts. The pacer's burst bound for a slower bottleneck downstream is reasoned, not measured.
• 1 ms is chosen, not measured. A smaller span bounds bursts tighter at more wakes.
• N = 2 in slow start can overshoot more than 1.25 before HyStart++'s delay exit. On this path neither arm lost anything; a path with a shallow buffer is unmeasured.

🤖 Generated with Claude Code