[Japabu]

Problem

With timestamps, [tcp] took every RTT sample from the echoed TSval, which ticks at 1 ms, the finest RFC 7323 §5.4 allows. On the T14's 2 ms LAN path that quantum is half the round trip, and a quarter of HyStart++'s 4 ms threshold (#1102). Without timestamps there was one Karn sample a round trip, so a round never reached HyStart++'s 8 samples and its delay exit was inert.

Change

• Every send of new data records its departure, at the stack's own clock. An ACK is timed from the newest segment it newly acknowledges (RFC 8985 §6.1), with and without timestamps.
• The record holds runs, not segments, each with its first and last send instants. A send at the same instant as the last joins its run. Any send also joins a run shorter than SMSS, until an ACK lands in that run (review r2: before that bound, a Nagle-off sender of small periodic writes had its ACKs timed from a run's first send up to 140 ms back).
  • Only the oldest run can be both short and closed, so the record holds at most a run per SMSS in flight plus two. No cap drops a send.
  • An ACK of a whole run is timed from its last send, and an ACK partway into a run that left at one instant (a burst) from that instant.
  • In a run of several instants, the first partway ACK is timed from the first send. That overstates by less than the joins took before that ACK ended them. A later partway ACK is not timed.
• Review r1 found the first version's 64-segment cap: at the T14's 230–255 segments in flight, about three ACKs in four still took the 1 ms echo. Counters tcp.rtt.departure and tcp.rtt.echo now name each sample's source, and osbench reads them.
• That is also the segment the receiver did not hold waiting for a second. A delayed ACK inflates the sample less than an echo of the earliest segment does.
• Karn (RFC 6298 §3): any retransmission clears the record, because the ACK that repairs a hole covers what was held behind it. An expiry also clears it. RFC 6298 §5: after a back-off, a new measurement "can only happen when new data has been sent and acknowledged", so the backed-off RTO stands until then (s_pl_002).
• The echoed timestamp times only data that a resend or an expiry took out of the record, which includes a retransmission (RFC 7323 §4.1). Every ACK of new data is now a sample, so each is weighted by RFC 7323 Appendix G's ExpectedSamples.
• Closes issues/tcp-samples-rtt-at-a-millisecond.md, including its timestamps-off case.
• A second commit takes tcp: HyStart++ in the initial slow start, and a card-held window half in flight is in use #1102's review r2 suggestions: tcp.hystart.css.rounds becomes tcp.hystart.css.entry-rounds, since it counts rounds that entered CSS, not RFC 9406's CSS_ROUNDS. The send_limit test that still named the dropped acknowledged-bytes count is renamed.

Tests

• s_rt_008 now reads the newest segment's 15 ms, where the old single timer read the first one's 20 ms.
• s_rt_011: an invalid echo is counted, and the ACK is timed by its own departure.
• s_rt_017: Karn after a fast retransmission. s_rt_018: a 1.5 ms sample that an echo would read as 1 ms. s_rt_019: one sample per ACK without timestamps.
• s_rt_020 (the review's case): more than 64 segments in flight. An ACK of the first 64 and then, 0.2 ms later, an ACK of only the rest are both timed from their departure, and no sample is an echo. It fails under the first version's rule.
• s_rt_021: an ACK of small sends and a full segment after them is timed from the segment (15 ms, not 20).
• s_rt_022 (review r2's case): 100 B every 10 ms with Nagle off, each ACKed 20 ms later. srtt stays 20 ms. It fails at 15fdb0e (21.25 ms at the ACK at 30 ms).
• s_rt_023: three stretch ACKs into one burst are each timed. s_rt_024: a later partway ACK into a run of writes is not timed, and the rest's ACK is timed from the last write.
• Eleven controls at 417b74b, each applied, run and reverted, all exit 101, each failing its tests:
  • joins continuing past an ACK: s_rt_022
  • the first version's rule (one entry a segment, at most 64): s_rt_020, s_rt_024
  • a whole run timed from its first send: s_rt_021, s_rt_022, s_rt_024
  • a partway ACK timed from the run's last send: s_rt_022, s_rt_024
  • a later partway ACK timed: s_rt_024
  • a burst's later partway ACKs left untimed: s_rt_023
  • only wholly acknowledged runs time: s_rt_019, s_rt_020, s_rt_022, s_rt_023, s_rt_024
  • the echo preferred: s_rt_018, s_rt_020
  • no echo fallback: s_rt_010
  • the record kept across a resend: s_rt_010, s_rt_017, RFC 8985 §7.3's probe test
  • the record kept across an expiry: s_pl_002
• At 417b74b: [tcp] cargo test --no-fail-fast 432 passed, exit 0; cargo run -- --ci host 105 steps, all green, exit 0. Evidence: https://github.com/ToyOSOrg/evidence/tree/96c693b58d242d8ec236e0805b34b714306ec4c2/r3 (earlier rounds in r1, r2)

Metal: T14, osbench_lan, one session (all boots rc 0, all judges exit 0)

This session ran the first version, with the 64-segment cap, so its C rows don't measure the fix. The pacing A/B (#1114) is restaged with C = 15fdb0e, and that session reads tcp.rtt.departure and tcp.rtt.echo. Review r2's fix changes only how sub-SMSS sends join a run. The bulk upload sends full segments, and its runs are bursts left at one instant, which both versions time the same, so the staged images still measure this PR.

The peer is the AMD bench's Linux over its Wi-Fi. Order: Linux L1, b1 c1 d1 b2 c2 d2 b3 c3 d3, Linux L2.

• B = ae8e78b (tcp: HyStart++ in the initial slow start, and a card-held window half in flight is in use #1102's head), C = 1480e1a (this PR before the rebase; the rebase touched nothing under system/netstack), D = C with L = ∞ unpaced, a diagnostic arm that does not land.
• Marks are each row's mean of five connections, received by the peer, in MB.

	p50 MB/s	100 ms	300 ms	1 s	2 s	css / entry rounds	retransmitted
Linux L1, L2	113, 114	5.86, 6.04	27.2, 29.0	108.0, 110.1	223.6, 225.7		0
B	112, 112, 112	2.21, 2.06, 2.35	24.00, 23.99, 24.66	105.5–105.8	222.1–222.4	644/327, 598/295, 614/229	0
C	112, 112, 112	3.22, 2.41, 2.23	25.57, 24.26, 24.31	105.2–106.6	221.6–222.7	779/319, 740/287, 599/250	1,448 in c1 (one loss probe)
D	114, 113, 114	6.39, 5.24, 5.71	29.65, 28.30, 29.08	109.9–111.2	226.5–227.6	582/251, 601/297, 628/311	0

Under the rule set before the run:

• C vs B: not confirmed. C's 300 ms mark beats B's in two triples of three (c3 24.31 < b3 24.66), with the same p50. On this path finer samples neither help nor hurt the ramp. This PR lands on its own grounds: it removes the 1 ms quantum and gives timestamps-off peers a working delay exit.
• Flaps: css is 1.97–2.68 times the rounds that entered it (8 of 9 rows at 2× or more), so most entries are within-round flaps (s_cc_023), not one jittery round each. Every entry resumed except one each in b3 and c1, still in CSS when the connection ended; none exited.
• D vs C: the L = 8 cap holds the ramp. D reaches Linux's marks at 100 ms (5.2–6.4 against 5.9–6.0) and 300 ms (28.3–29.7 against 27.2–29.0), with p50 113–114 and nothing retransmitted. CSS counts are unchanged, so CSS isn't what holds it. Next is step 3, pacing with L = ∞ (RFC 9406 §4.3), in its own PR.

Evidence (readbacks' test-runner lines, verdicts, judge logs, the Linux runs, image hashes, D's patch): https://github.com/ToyOSOrg/evidence/tree/110096df4b2c1ce75af25caa415677aa0985579e/r1

Risks

• A stretch ACK that first lands partway into a run of small writes is timed from the run's first write: at most the joins' span high, which ends at that ACK. Later partway ACKs into such a run give no sample.
• The record grows with the flight, 32 bytes a run: about 8 KB for the T14's 250 segments.
• Without timestamps the delay exit can now fire. That is the case RFC 9406 recommends, and it changes behaviour for peers that decline timestamps. No such peer has been measured.

🤖 Generated with Claude Code

[Japabu]

Review r1 — #1112 at f1a230e

Architecture. The design fits. RTT sampling stays inside [tcp]'s Sync. A per-connection departure record replaces Karn's single timer, and Karn's rule is enforced at the two places that can create ambiguity: hand_off for a resend and expire for an expiry. The echo remains for retransmissions, which RFC 7323 §4.1 allows. The clock is the shard's monotonic Instant in nanoseconds (netstack/src/main.rs:344), and rtt.rs works in µs, so the resolution claim holds wherever a departure is used. Net change: production +33/−22 in conn.rs, tests +55, and the issue deleted. The rename commit is clean: no reference to css.rounds or HyStartCssRounds remains, and osbench's READ list matches.

Checked and correct:

• Partial ACKs. A NewReno or RFC 6675 partial ACK only covers data sent before the retransmission, which cleared the record. It therefore finds no departure, and only a valid echo can time it.
• Resends and probes. These never produce a departure sample. A loss-probe resend clears the record, and so does a persist probe of una. A probe of new data is timed, and that is valid because it was sent once. A segment that resends and extends past SND.NXT clears the record and records nothing.
• s_pl_002. Clearing at expiry keeps the backed-off RTO until data sent after the expiry is timed. The second half of s_pl_002 (ACK at 250 after the 200 ms expiry, next expiry at 2401) depends on it.
• SACK. Samples come only from cumulative ACKs, as on main. With reordering and no loss, the newest cumulatively covered segment can be one already SACKed. That gives a sample no worse than RFC 8985 §6.1's choice of the newly delivered segment, so it is acceptable.
• The four controls (rtt-mut-summary-4.txt, rev 1480e1a, all exit 101). Each one fails the test it targets. The tcp source at f1a230e differs from that revision only by the counter rename, so reusing the controls is valid.
• c1's loss probe. It is not a concern. It was one tail resend of 1,448 B, D-SACKed (dsack_rcvd=1) and loss_probe_recovery=0, so it cost no cwnd. d1 also probed once (new data). c1's srtt at congestion stops (5.6 ms, against 6.9–7.9 ms elsewhere) may have shortened the PTO. That is an inference from one event in 15 connections and does not block.

BLOCKER

1. system/netstack/shard/tcp/src/conn.rs:859 with :1324 and :410. Past 64 segments in flight, most ACKs are still timed by the 1 ms echo. The PR's central claim and the issue's closure are false on the path that motivated them.

   • What happens. A segment sent while departures is full is not recorded. An ACK that covers only such segments pops nothing, and departed.map(..).or(echoed) then takes the echoed TSval, a difference of two millisecond floors.
   • Rough share on a steady flight of F segments. The record holds the first 64. The ACKs for the remaining F − 64 find nothing and use the echo. The sends those ACKs release are also untimed, so the pattern repeats every round.
   • The T14 runs are past the cap. cwnd at congestion stops averaged 331–372 KB, about 230–255 segments, in every B and C run (congestion_stops_cwnd_bytes / congestion_stops). On that path roughly three ACKs in four are still echo samples. The Linux peer negotiates timestamps by default.
   • The quantum reaches HyStart++. An echo can read up to 1 ms below the true round trip, and HyStart++ takes each round's minimum, so the 1 ms quantum reaches its delay comparison. That is the defect the issue describes.
   • Possible link to the result. This may explain why srtt at stops barely moved (B 7.1–7.9 ms, C 5.6–7.3 ms) and why C matched B. That link is an inference.
   • Metal evidence. No counter records which source timed a sample. "No 1 ms quantum" on metal is therefore unmeasured.
   • Second effect. An ACK that crosses from the last recorded entry into untimed data is timed from that older entry, which overstates the round trip by the gap. The comment at :410 ("one stretch ACK's worth after another") does not describe either effect.
   • Fix. Do not let the cap fall back to the echo. Use the echo only for data the record cleared through a resend or expiry. Alternatively, time every segment in flight, as the issue's exit proposed: a record sized by the flight or kept on the tx scoreboard.
   • Cheapest check. A host test with fixture_ef: send more than TIMED segments, then ACK an untimed one at a sub-millisecond offset. Assert the sample is not the echo's whole millisecond. It fails at f1a230e.
   • Also update. The PR's "Risks" paragraph and the ExpectedSamples sentence should match the result.

2. system/netstack/shard/tcp/src/conn.rs:1160. The expiry comment is false.

   • What it says. "An expiry puts every departure in doubt, whether or not its retransmission has left."
   • Why it is wrong. Before the retransmission leaves, an ACK can only be for the original, so its timing is not ambiguous. The real reason is RFC 6298 §5: after a back-off, a new RTT measurement "can only happen when new data has been sent and acknowledged", so the backed-off RTO stands until then. s_pl_002 holds that behaviour.
   • Consequence. A reader who believes the stated reason could remove the clear for the not-yet-left case and break the back-off.
   • Fix. State the real reason in one line, or delete the comment.

SUGGESTION

• tests/rtt.rs: s_rt_018 comes before s_rt_017, and s_rt_019 discards set_options's outputs with let (_, _) where s_rt_008 asserts nothing(&outs). Both are minor.

SEND BACK

[Japabu]

Review r2 — #1112 at 15fdb0e

Both r1 blockers are resolved:

• Every send is recorded. The echo now times only data that a resend or an expiry took out of the record. In sample, departed is None only when the acknowledged data has no run left, and that happens only after clear().
• The expiry comment now gives RFC 6298 §5's real reason.

The rest of the run-record design is correct:

• Karn. hand_off and expire still clear the whole record. A send that resends and also extends past SND.NXT clears the record and records nothing.
• Partial ACKs across runs. A NewReno or RFC 6675 partial ACK covers only cleared data, so only a valid echo can time it. An ACK that spans several runs takes the newest run it touches, and an ACK partway into a run takes that run's time.
• SACK. Samples come only from cumulative ACKs, as settled in r1.
• Memory bound. It holds. A send at a different instant opens a new run only when the previous run, or the send itself, is at least SMSS. Any two adjacent runs therefore hold at least SMSS between them, which gives at most 2·flight/SMSS + 1 runs of 16 B each, about 92 KB at a 4 MiB flight. Every ACK that moves SND.UNA passes through sample, so acknowledged runs never accumulate.
• Counters. tcp.rtt.departure and tcp.rtt.echo are registered and read by osbench.
• Tests and controls. s_rt_020 is the r1 case, and its control (the old cap) fails it. All eight controls at 15fdb0e exit 101, each on the tests it targets. [tcp] tests and --ci host pass (105 steps, exit 0).

BLOCKER

1. system/netstack/shard/tcp/src/conn.rs:1348 with :862–866. The joining rule's cost is unbounded in time: a sub-SMSS run keeps growing while ACKs land inside it, and each of those ACKs is timed from its first send.
   • What the code does. The join test (run.end.since(run.start) < smss && len < smss) never looks at whether the run is already partly acknowledged. A partway ACK neither trims run.start nor moves run.at.
   • Trace. Take a nodelay sender writing 100 B every 10 ms on a 20 ms path, with the peer ACKing each segment:
     • Every write joins run A at at = 0 until A reaches 1,460 B. That takes 15 writes, 140 ms.
     • The ACK at 20 ms is timed 20 ms, which is correct.
     • The ACK at 30 ms (for the write at 10 ms) is timed 30 ms, the one at 40 ms is timed 40 ms, and so on up to about 160 ms.
     • The next run repeats the pattern.
   • Size of the inflation. It approaches (SMSS ÷ write size) × write interval. That is RTO-scale for small writes, and can reach seconds for tiny periodic ones.
   • Consequences.
     • srtt and RTO are inflated.
     • The loss probe, set at 2·srtt, is delayed.
     • HyStart++'s round minimum rises for an app-limited flow still in slow start, which risks a spurious delay exit.
   • Regression. On main, this traffic was sampled correctly: by the echo with timestamps, and by Karn's one timer without.
   • Where the doc falls short. The field doc says such a send is "timed from its run's start, never later". It does not say that "earlier" is bounded only by how long the run takes to collect SMSS bytes. s_rt_021 covers a full segment after small sends, but not ACKs that land inside a growing small run.
   • Fix (direction only, the choice is the implementer's). Bound what one run can cost while keeping the flight bound. For example:
     • stop joins into a run once an ACK has landed inside it;
     • keep the run's last send instant, and time an ACK that covers the whole run from it;
     • take no departure sample from a later partway ACK into an already-sampled run.
   • Cheapest check. A host test: nodelay, 100 B every 10 ms, each ACKed 20 ms later. Assert that srtt stays at 20 ms. It fails at 15fdb0e.

SUGGESTION

• None new.

SEND BACK

[Japabu]

Review r3 — #1112 at 417b74b

The r2 blocker is fixed.

• Fix. Small writes join a run only until an ACK lands in it. An ACK that covers a whole run is timed from that run's last send.
• Test. s_rt_022 is the r2 case: srtt holds at 20 ms. At 15fdb0e it fails at 21.25 ms (s_rt_022-at-15fdb0e9b.log).

The bound holds, at flight/SMSS + 2 runs of 32 B each.

• A send at a different instant opens a new run only when the previous run is at least SMSS or has been sampled.
• Only the front run can be sampled, because a partway ACK lands only in the front run.
• So every run except the front and the back is at least SMSS.
• An ACK that pops a run and then lands partway in the next one cannot meet an already-sampled next run: a run is sampled only after every run before it has been popped.

The no-sample rule's effect on RTO.

• A run that left over several instants stops growing at its first ACK, so the ACK covering all of it comes about one round later. That ACK is timed from the run's last send.
• In between, sample returns None. RTO, back-off and expiries are untouched, so nothing collapses or inflates.
• sampled stays false for at most that round, so the loss-probe precondition of RFC 8985 §7.3 is delayed by at most that much.
• A loss inside the run triggers a resend, which clears the record. Timing then falls back to the echo or Karn, as settled.
• Returning None without trying the echo is right: that data is still in the record.
• In s_rt_022's pattern, every ACK is a correct 20 ms sample. With writes faster than the round trip, a run of 15 writes gives two exact samples, the first partway ACK and the whole-run ACK.

Karn is unchanged from r2: hand_off and expire each clear the whole record.

Tests and evidence.

• Each of the 11 controls at 417b74b exits 101 on the tests it targets. Two were regenerated at the same revision; the note in summary.txt explains why, and the reruns are valid.
• [tcp] tests and --ci host pass (105 steps, exit 0).
• s_rt_023 and s_rt_024 each hold one claim: every ACK into a one-instant burst is timed, and a later partway ACK into a run of writes is not.

Comments. The field doc is long, but each sentence carries part of the contract: the join rule, the bound, or which instant times which ACK. It is accurate.

SUGGESTION

• conn.rs:1358: dropping len < smss lets a full burst join a short run that has not been sampled yet.
  • Case. A small header goes out at T1, and a window of body follows at T2.
  • Cost. That window becomes one run spanning T1 to T2. The first partway ACK is overstated by T2 − T1, the later ones take no sample, and the ACK of the whole run is exact. That round loses most of its per-ACK samples, so HyStart++ may skip its delay decision for it.
  • Option. Restoring len < smss in the join test avoids this. The price is going back to r2's bound of two runs per SMSS.
  • Optional; the bound and the timing are correct either way.

LAND

[Japabu]

Orchestrator: r3 LAND. The header-then-body join suggestion (restore len < smss, at the cost of r2's bound) goes to #1114 as a follow-up, since #1114 stacks on this. Landing.