[Japabu]

Status: draft. Review round 1's findings are addressed (below); the T14 A/B is done.

Problem

The T14's wired upload (I219 to the AMD bench's Wi-Fi peer) reads 85-89 MB/s under ToyOS against Linux's 110-113. The driver keeps line rate with frames in hand. The receiver-side ramp marks added here read where the rest goes (session at d1ea418, means of five connections a row):

received by	100 ms	300 ms	1 s	2 s
Linux	6.1 MB	28.5 MB	109 MB	224 MB
ToyOS	0.32 MB	2.0 MB	39 MB	151 MB

From 1 s on ToyOS runs at line rate. The whole gap is each new connection's first second, with nothing lost. The peer acknowledges about 50 KB (34 segments) per ACK, and each growing ACK added one SMSS (RFC 5681 §3.1's min(N, SMSS)), so slow start grew a few percent per round trip. Also, 75-76 % of ACKs grew nothing, because the card's room had ended the opportunity before them.

Change

• HyStart++ in the initial slow start (RFC 9406, which RFC 9438 §4.10 says CUBIC SHOULD use).
  • Growth per ACK is min(N, L·SMSS), with L = 8 because [tcp] does not pace (§4.3).
  • A round whose minimum RTT rises by RttThresh enters CSS, which grows by a quarter. CSS resumes slow start when the RTT falls back under its baseline. After 5 rounds it sets ssthresh = cwnd, and CUBIC starts with K = 0 and W_max = cwnd_prior = cwnd (RFC 9438 §4.10).
  • The constants are §4.3's. Slow start after a loss stays RFC 5681's.
• A card-held stop with at least half of cwnd in flight leaves the window in use. The old rule required less than one segment of room. This is a deliberate departure from RFC 9438 §5.8, under whose definition a flow sending less than cwnd allows is application limited, card-held or not. Half of cwnd in flight is RFC 7661 §4.3's validated window, which RFC 9438 §4.6 admits; cwnd stays within twice the flight, and a loss sets ssthresh from the flight (§4.6). The rule has not been measured on its own: the earlier card-full arm (d1ea418's B, +4–5 MB/s) was limited = true at every card stop, not this rule, and the A/B below measures it only together with HyStart++.
• Instrument (commit 1). osbench-lan protocol /3: the sink reports what it had received by 100 ms/300 ms/1 s/2 s, so both OSes' ramps are read the same way at the receiver. [tcp] counts ACKs by growth gate and their growth. Commit 2 adds tcp.hystart.{css,css.rounds,resumed,exits,exit.cwnd.bytes}; css.rounds (review round 1) counts rounds that entered CSS, beside css's entries.
• The diagnostic branch's per-connection cwnd-at-mark counters and the I219 EEE/PBA/full-ring diagnostics are dropped. They answered one-off questions, and the peer-side marks measure the ramp better.
• the_receive_window_grows_by_what_is_read_per_round_trip quadrupled the path RTT after its writer had started. HyStart++ reads that, as specified, as a queue and exits. The writer now starts after the change, and the test still checks the receiver's window.

Checks

• [tcp] host tests: 423 passed. New: s_cc_018 (stretch ACK grows by eight segments), s_cc_019-022 (RttThresh clamp, CSS entry and exit, resume, L only in the initial slow start); s_cc_017 now holds both sides of the half-window boundary.
• Twelve negative controls, each a checked patch applied, run and reverted (exit 101, failing exactly its test).
• cargo run -- --ci host at 94e3ee70d: 105 steps, all green. Rebased onto a5b330f (main) after the A/B: [tcp] 423 passed, and cargo run -- --ci host at 1250226: 105 steps, all green.
• Review round 1 (ae8e78b): cargo run -- --ci host 105 steps, all green; [tcp] 424 passed.
• Evidence: diagnostic session at d1ea41856, host CI and controls

Metal: T14 A/B, 2026-10-28

osbench_lan, run as Linux, A B A B A B, Linux. A = f895223 (the fix reverted), B = 94e3ee70d (the fix, same code as the head). Every boot rc = 0 and every judge exit 0. The peer and the T14's Linux ran this branch's osbench (lan/3). Evidence r2.

	upload p50, MB/s	received by 100 ms	300 ms	1 s	2 s
Linux (2)	113, 113	6.4–6.5 MB	29.7 MB	110–111 MB	226–227 MB
A (3)	87, 87, 87	0.33–0.34 MB	2.0–2.1 MB	34–38 MB	144–149 MB
B (3)	112, 112, 112	2.1–3.3 MB	23.9–25.4 MB	106–107 MB	222–223 MB

• B confirms (rule set before the run): faster in every pair (one-sided p = 1/20), with delivered-by-1 s up from 34–38 MB to 106–107 MB. No loss: 0 retransmitted bytes, 0 RTOs, 0 recoveries, 0 loss probes.
• Parity by the preset rule (inside this session's Linux runs) is missed by 1 MB/s. B's rounds ran 111–115, Linux's 113–114. B is inside Linux's 109–114 across the record's earlier sessions. The issue stays open on the residual, renamed to it: issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md.
• The residual is each connection's first 300 ms. In every B row HyStart++ entered CSS 590–702 times (about 120–140 per connection) and resumed every time, never exiting. Whether these are jittery rounds or flaps within one round (s_cc_023) is not yet read: tcp.hystart.css.rounds will separate them on the next session. Whether either accounts for the residual is not separated. The leads are RFC 9406 §4.3's tunable thresholds on such a path, or pacing with L = ∞, as a follow-up.
• The download row did not move: A 103–107, B 107–109, Linux 107–108.
• osbench_lan needs the T14's I219 and the AMD peer. The AMD bench has no NIC ToyOS drives, and ToyOS has no Wi-Fi driver, so no other bench or row runs this path today.

Review round 1

• B1: the half-window rule is stated as a departure from §5.8, justified by RFC 9438 §4.6 and RFC 7661 §4.3 with the cwnd ≤ 2·flight bound, in cc.rs's header, Cc::limited, hand_off, s_cc_017 and above.
• B2: the "4-5 MB/s for this rule" claim is corrected above.
• B3: the record is renamed to its residual, and the one citation moved.
• B4: the timestamps-off inertness is recorded in the RTT issue.
• S1: s_cc_017 pins the boundary exactly (9 of 18 segments grow, 8 do not); controls > for >=, half less a segment, and a quarter each red it.
• S2: net.rs's doc rewrapped. S3: tcp.acks.bytes dropped. S4: osbench's counter doc shortened.
• S5: s_cc_023 holds the within-round flap; tcp.hystart.css.rounds counts rounds entered. Its two controls (the round's first never reset, always first) red it.
• Controls and CI: evidence r3

Risks

• HyStart++'s delay exit on a jittery path: observed as CSS entries that always resume (see above). None ended slow start on this path. A path where it does would continue under CUBIC from K = 0.
• ToyOS offers timestamps in every SYN and keeps PAWS; the T14 session negotiated them (SMSS 1448). Only against a peer that declines them does the delay exit never fire, since there is one Karn sample per round trip, under RFC 9406's 8.
• RTT samples come from TSecr at 1 ms, the finest tick RFC 7323 §5.4 allows. That is coarse against the 4 ms threshold on a 2 ms path. Without timestamps there is one Karn sample a round, so the delay exit is inert and slow start runs with L = 8 until a loss. Both are recorded with their exit (per-segment send times, RFC 8985 §6.1) in issues/tcp-samples-rtt-at-a-millisecond.md, which the stacked step-1 PR closes.
• RFC 9406 §4.2 checks for CSS on every ACK of a round, and a round's running minimum only falls, so one round can enter and resume repeatedly (s_cc_023: four entries in one round). That may account for much of the 590–702 entries; tcp.hystart.css.rounds separates the two on the next metal session.

🤖 Generated with Claude Code

[Japabu]

Review of #1102 at 1250226

Architecture

The ownership is right. Congestion control stays in [tcp]'s Cc. HyStart++ is a small HyStart state with no variant that cannot occur: None stands for the RFC's infinity, and Option<Css> holds the CSS phase. It runs only while ssthresh == SSTHRESH_INITIAL, so after any loss or RTO the code returns to RFC 5681 §3.1 by construction. The ramp instrument belongs to osbench's own protocol (osbench-lan/3, no stale /2 left in the tree). It measures both OSes the same way, at the receiver. Production grows by about 135 lines in [tcp] (about 110 of them HyStart) and about 85 lines of osbench instrument. Tests grow by about 140 lines. The dropped I219 and per-connection diagnostics were never on main, so nothing on main is deleted.

RFC 9406 / 9438 conformance, clause by clause

• Growth min(N, L·SMSS), with L = 8 because [tcp] does not pace (§4.3): correct. CSS growth min(N, L·SMSS)/4: correct. Growth is applied before the ACK's RTT is observed, which is the §4.2 order. So the ACK that enters CSS grows at the full rate, and the ACK that resumes grows at a quarter.
• RttThresh = clamp(last/8, 4 ms, 16 ms), with the check current >= last + thresh. It needs at least 8 samples in the round and a last-round minimum. All of these match §4.2 and §4.3, and s_cc_019, s_cc_020 and the strict-comparison control pin them.
• Rounds: windowEnd is set from SND.NXT at the first ACK, not at initialisation. This only moves which round the first sample falls in, which is harmless. A round ends when una >= windowEnd, which is correct.
• CSS round count: the round CSS starts in counts as round 1, and slow start ends when the fifth round ends. That matches "that partial round counts towards the limit". The mutant that adds a round is caught.
• Resume when the current minimum falls below the baseline, with at least 8 samples in the round: correct, and controlled.
• Exit: ssthresh = prior = cwnd. The next ACK creates the epoch with w_max == None, which gives K = 0 and W_max = cwnd. That is RFC 9438 §4.10. Because prior = cwnd, the Reno-friendly estimate starts with α = 1, per §4.3.
• Loss during slow start or CSS: on_loss and on_timeout set ssthresh from flight, which ends HyStart. Later slow starts are RFC 5681's min(N, SMSS), capped at ssthresh (s_cc_022 and its control).
• Receive window: a peer-window stop sets limited = false, so nothing grows. RTT is still observed, and the CSS/slow-start flapping that follows is the behaviour §4.3 accepts for limited senders.
• Without timestamps: there is one Karn sample per round, so N_RTT_SAMPLE is never reached and the delay exit is inert (see blocker 4).

Half-window rule

Mechanism: every data hand-off provisionally sets limited = in_flight >= cwnd/2. A stop seen by next_segment (User, Window or Congestion) overwrites it. So the provisional value survives only when the caller's credit ended the opportunity.

• Application-limited senders still grow nothing in the normal case.
• The remaining edge case is the application's data and the card's credit running out on the same segment. Then the ACKs before the next opportunity can grow cwnd, but never beyond 2× flight. This is bounded and transient, and I accept it.
• The cwnd ≤ 2·flight bound is the actual safety argument. Losses are safe because ssthresh comes from flight (reduced(flight)), which is what RFC 9438 §4.6 requires before admitting RFC 7661. The textual justification, however, is wrong (blocker 1).

Tests and controls

• I read all twelve control patches and logs (mut-summary-2.txt, at 94e3ee70d). Each one fails the test it targets, and the failure is the intended defect:
  • L = 1
  • a strict threshold comparison
  • one sample instead of 8
  • CSS rounds + 1
  • CSS growth divisor 1
  • no resume
  • prior not set at exit
  • MAX_RTT_THRESH = 32 ms
  • L after a loss
  • the old card rule
  • the card stop always limited
  • the writer not held
• 94e3ee70d and 1250226 have the same code diff (it differs only in hunk offsets after the rebase). ci-host-4.log at 1250226 ends 105 step(s), all green, exit 0.
• the_receive_window_grows_by_what_is_read_per_round_trip: by inspection the change is legitimate. The test's claims are the receiver window's upper bound and the reader never waiting under the receiver's own SRTT estimate. Both are receiver-side. The 4× RTT step mid-transfer made the sender's HyStart++ exit, as specified, and the reader then waited on cwnd, which is the control's received mismatch. Holding the writer until after the step removes the sender confound and keeps both receiver claims able to fail.
• The writer-not-held control only shows that the change is needed. It is not a control for the test's own claim, but inspection is enough here.

Evidence

I checked r2/ab against the PR's table. Upload p50 is A 87/87/87, B 112/112/112, Linux 113/113. Received-by figures and the HyStart counters match (CSS 702/590/679, resumed equal, 0 exits). In every B row: 0 retransmitted bytes, 0 RTOs, 0 recoveries, 0 loss probes. A1 sent 2 loss probes. The images are hashed. The A/B is the whole change reverted onto the measured base, which is the right control. It does not separate HyStart++ from the half-window rule.

Counters

• The tcp.hystart.* counters earn their place. They produced the only reading of the residual (CSS entered 590–702 times, 0 exits), and the issue's next lead depends on them. exit.cwnd.bytes is the planned way to tell an early exit apart.
• acks.cwnd-limited / acks.not-limited and cwnd.grown.bytes diagnosed both defects this PR fixes, and they show the gate's effect after the fix (94 % of B's ACKs not limited).
• tcp.acks.bytes is the weak one: see S3.

Findings

BLOCKER 1 — the half-window rule's justification misstates RFC 9438 §5.8, and the module header is now false.

• system/netstack/shard/tcp/src/cc.rs:3-4 still says "Growth happens only when the window, not the application or the peer, stopped the sender last (RFC 9438 §5.8)". After this change, growth also follows a stop on the card's credit, with cwnd room left.
• conn.rs:1292-1294 ("The caller is neither the application nor the peer (RFC 9438 §5.8)"), tests/congestion.rs:109-110, the Cc::limited doc ("The window … ended …, or the caller's credit ended it") and the PR body all read §5.8 as covering only the application and the receiver window.
• §5.8's definition is "A flow is application limited if it is currently sending less than what is allowed by the congestion window"; the two causes it lists follow "This can happen if". A card-held sender with room left is application limited by that definition. The rule is a deliberate departure, justified by RFC 9438 §4.6, which admits RFC 7661's mechanisms ("safe to use…"), and by RFC 7661 §4.3's validated phase, with cwnd ≤ 2·flight and a flight-based ssthresh as the safety bound.
• Fix: say that in the comments, and correct the header to describe the growth gate as it now is.

BLOCKER 2 — false evidence claim in the PR description. "The previous A/B measured this rule alone as worth about 4-5 MB/s." Arm B of that session is r1/d1ea41856/patches/arm-b-card-full-is-window-limited.patch, which is self.cc.limited = true, not the half-window rule. The issue record describes it correctly ("a card-held stop counted as cwnd's"). The half-window rule has not been measured on its own; r2 measures it only together with HyStart++. This correction belongs to the orchestrator. It does not need a new run, only a true sentence.

BLOCKER 3 — issues/netstack-sends-tcp-at-a-third-of-what-the-t14s-wired-link-carries.md: the tree now refutes the slug and title. Upload is at 112 against 113. issues/README.md § Slugs: "one the tree has refuted is renamed by the work that takes it up, with every citation moved". This PR takes it up and changes its exit to the first 300 ms. Rename it to the remaining defect (the initial ramp trailing Linux's on the T14), or close it and file the residual as a new defect. Move issues/no-published-run-compares-toyos-with-linux-on-one-machine.md:50 with it. The opening paragraph's "41 MB/s" claim should then describe the residual.

BLOCKER 4 — the timestamps-off weakness is only in the PR body. Without timestamps, sample() gives one Karn sample per round (self.timing), so samples >= N_RTT_SAMPLE never holds. HyStart++'s delay exit is then inert, and the initial slow start runs with L = 8 until a loss. RFC 9406 §4.3 RECOMMENDS 8 samples, and L = 8 is justified only by that delay exit. Root's "No silent debt" requires recording it. The exit of issues/tcp-samples-rtt-at-a-millisecond.md (per-segment send times, RFC 8985 §6.1) would fix this case too, so a paragraph there is enough. Retitle the issue if it no longer fits.

SUGGESTION 1 — tests/congestion.rs s_cc_017. The test exercises 13,140/17,520 (75 %) and 4,380/29,200 (15 %), so any threshold between those survives, for example cwnd/4 or 2·cwnd/3. The PR says it "holds both sides of the half-window boundary". One ACK exactly at cwnd/2 and one a segment under it would actually pin the rule.

SUGGESTION 2 — system/netstack/shard/tcp/tests/net.rs:430. The inserted sentence makes one 174-column doc line in a paragraph wrapped at 100. Rewrap it.

SUGGESTION 3 — tcp.acks.bytes. It equals tcp.sent.bytes exactly in every r2 row (for example 1,708,654,637 in b1), because it differs only under loss. Bytes per ACK can be derived from the existing counters, so consider dropping it.

SUGGESTION 4 — tests/guest/osbench/src/sys/toyos.rs:401-402. The doc's inventory ("bytes, losses and the time … waited …, and the card's full-ring waits and missed frames") no longer covers the ACK, growth, HyStart and wire.received entries. Shorten it rather than extending the list.

SUGGESTION 5 (a lead for the residual, an inference I have not measured). observe checks entry on every ACK once there are 8 samples. Take a round whose minimum keeps falling while it stays at or above last + thresh, for example samples 7,…,7 then 6 then 5 with last = 2 ms. It enters CSS, resumes at the next lower sample, then re-enters on the following ACK with a new baseline. This is literal §4.2 pseudocode, so it is not a conformance defect. It does mean that 590–702 entries may count flaps within rounds rather than jittery rounds. A host test of that sequence, or a per-round count, would separate the two before anyone tunes MIN_RTT_THRESH or N_RTT_SAMPLE.

SEND BACK

[Japabu]

Review r2 of #1102 at ae8e78b

The one new commit since 1250226 answers r1. Merge base is unchanged (a5b330f). Its code changes are comments, the Conservative { first } flag and its counter, the test rewrites and the dropped tcp.acks.bytes.

r1 blockers

• B1, resolved. The text no longer misreads §5.8.
  • cc.rs:3-5 now defers to Cc::limited.
  • Cc::limited's doc is accurate.
  • conn.rs:1296-1300 states the departure from §5.8 as one, cites RFC 7661 §4.3's validated window that RFC 9438 §4.6 admits, and gives the actual safety bound: cwnd ≤ 2·flight, with ssthresh from flight.
  • The s_cc_017 doc and the PR body say the same.
• B2, resolved. The PR body now says the half-window rule was never measured alone, and that d1ea418's arm B was limited = true.
• B3, resolved.
  • The record is now issues/tcp-opens-a-connections-window-slower-than-linux-in-its-first-300-ms.md, with a title and opening paragraph that state the residual. I checked the opening's numbers against r2/ab.
  • git grep of the old slug at ae8e78b finds nothing.
  • The one citation, issues/no-published-run-compares-toyos-with-linux-on-one-machine.md:50, has moved.
• B4, resolved. issues/tcp-samples-rtt-at-a-millisecond.md now records the timestamps-off case: one Karn sample per round, the delay exit inert, L = 8 until a loss. It has a retitle and an exit that covers both cases.

New tests and counter

• s_cc_017. I traced it by hand.
  • The ACK of the whole initial window grows cwnd to 26,280, so half is 13,140, which is 9 segments. Credit 9 leaves exactly 13,140 in flight and the next ACK grows 1,460. Credit 8 leaves 11,680 and the next ACK grows 0.
  • The controls > for >=, half less a segment, and a quarter each fail it (r3 controls/summary.txt, exit 101, only s_cc_017).
  • The r1 card controls no longer apply textually, but by inspection the rewritten test still fails both. "Always limited" grows 1,460 at credit 8. The old room < SMSS rule grows 0 at credit 9.
• s_cc_023. I traced it by hand against observe. On the 9,…,9, 8, 8, 7, 7, 6, 6, 6, 6 round with last = 2 ms, it enters at index 7, then resumes and re-enters at 8/9, 10/11 and 12/13. Each lower sample resumes, and the next re-enters while the minimum is still ≥ last + 4 ms. The 5 ms round resumes, and the 12 ms round's entry is that round's own first.
• css_this_round. The flag is set on entry and reset at round end, after that ACK's entry is reported. So an entry on the round-ending ACK counts toward the round it ends, which is correct. Both controls (never reset, always first) fail s_cc_023.
• Counter wiring (conn.rs:777-782): correct by inspection.
• CI: ci-host-ae8e78b4e.log reports rev ae8e78b4e… and 105 step(s), all green, exit 0.

Findings

SUGGESTION 1 — tcp.hystart.css.rounds (counters.rs). It counts rounds that entered CSS at least once. A reader beside RFC 9406's CSS_ROUNDS will take it for rounds spent in CSS, which it is not, because CSS rounds 2–5 without a new entry do not count. A name like tcp.hystart.css.entry-rounds would remove the ambiguity before the next metal session reads it.

SUGGESTION 2 — tests/send_limit.rs:101. each_ack_is_counted_with_what_it_acknowledged_and_grew still names the acknowledged-bytes count that S3 removed. Its doc was updated; the name was not.

SUGGESTION 3 — PR body, for the orchestrator. "resumed every time, never exiting: the Wi-Fi-jitter risk below, now observed" reads as an established cause. The new last risk bullet correctly says that within-round flapping may account for much of the 590–702 entries, so the attribution is an inference until css.rounds is read.

No blocker remains.

LAND

[Japabu]

Orchestrator: r2 LAND. I corrected the body's CSS attribution to an inference (suggestion 3). Suggestions 1–2 are left to the follow-up, since the queued session's images already carry the counter's current name. Landing.