Repository navigation
bench: corpus scale run keeps up at 1.4 req/s per restore path and misses at 1.87 with the host idle #1159
Description
The open-loop corpus scale run (make bench-chromium-corpus MEASURE=scale, 4 guest vCPUs, screenshot requests on the 14-page corpus, FILE and UFFD restores interleaved) keeps up at 1.40 requests per second per restore path (2.8 in all) and misses its departure gate at 1.87: in 3 of 10 burst-and-path cells it completed 1.78 to 1.79 per second inside the 60 s window, every request still succeeding. Two earlier attempts at 2.8 per path failed outright, requests waiting until they timed out. Record: results/reqbench-20261022-corpus-s13/scale-20261023-000033-1468478/ on branch report/close-not-measured.
The host was not short of anything it samples:
| 0.93 / 1.40 / 1.87 req/s per path | |
|---|---|
machine CPU busy per burst (bursts.jsonl machine_proc_stat_delta) |
3.9-4.1% / 5.8-6.2% / 8.2-8.9% of 192 vCPUs |
PSI some avg10, max over the run (host-samples.jsonl) |
CPU 0, memory 0, I/O 0.14 |
| MemAvailable, min | 362 of 377 GiB |
burst median latency (artifact_latency_ms.median) |
715-797 / 780-837 / 816-1,542 ms |
So the limit is something that serializes requests, not the host's capacity. Candidates the records cannot separate: the replay server (one process serving every clone), fcvm's per-clone setup under shared locks (state directory, loopback IP allocation, namespace creation), the memory server for UFFD (though FILE saturates at the same rate), and the single-process Python driver.
What to measure next:
- per-request phase timings at 1.87 against 1.40 (spawn to port, wait for target, navigate) from
requests.jsonl, which the run already holds, to see which stage grows; - the replay server's per-request service time from its access log;
- a run with the replay replaced by a static local page, to take the corpus server out.
Separately, the harness refuses a whole run on any single failed request (one teardown ENOENT in 2,015 requests refused an otherwise complete run on 2026-10-22). That is the right default for latency records, but it means a rate above the ceiling invalidates every rate below it in the same run; a scale run could instead record a failing rate as failed and keep the others.
Activity
ejc3 commentedon Oct 23, 2026
Stage medians from
requests.jsonl(successful requests launched in the scoring windows, both paths) at 0.93 / 1.40 / 1.87 req/s per path: navigate 309 / 355 / 531 ms; spawn to port 107 / 111 / 117; wait for target 226 / 226 / 227; screenshot 70 / 72 / 70. Only navigation grows, which is when the clone fetches from the shared replay server. The replay's own service time is not recorded; timing it is the next measurement.ejc3 commentedon Oct 23, 2026
Harness follow-up from the same campaign:
reqscale.pyrequires each rate to give a whole number of requests per 15 s ramp and whole cycles of the corpus per 60 s window (for the 14-page corpus, multiples of 7/15 req/s), butmake bench-chromium-corpus MEASURE=scaleonly learns that after it has built the golden and run the diag, about 8 minutes in; two attempts (0.7 and 0.8) were refused that way. The Makefile's exampleSCALE_RATES(2,4,8) fails the corpus rule. Checking the schedule before the golden would save the box time.ejc3 commentedon Oct 23, 2026
Correction to the first line of this issue, from a review of the record: at 1.87 per path the host did complete as many requests as were offered. Counting every request that finished inside each 60 s window (including those launched in the ramp) gives 1.89 per second on each path; the departure gate counts only the window's own launches that also finished inside it (1.78 to 1.79), and that fell short because latency grew (burst medians up to 1,542 ms, 3 to 6 requests still running at the window's end against 0 or 1 at the lower rates). So 1.40 is the highest rate passing the gates, and the point where the host stops keeping up lies between 1.87 and 2.8. Machine CPU busy is 2.7 to 6.2% once guest time is counted once (it is already inside user). The replay's own per-answer time (
duration_msincorpus-access.log) stays flat: median 0.60 / 0.61 / 0.51 ms, p95 8.6 / 8.6 / 7.5 ms, so the added navigation time is outside its request handler.Record the corpus open-loop ceiling: the replay server's accept threa…
ejc3 commentedon Oct 26, 2026
Measured; record and numbers in #1340 (REVIEW.md section 'The corpus open-loop ceiling is the replay server's listener'). The s13 ceiling is the replay server's single accept thread (it runs the TLS handshake in accept()) behind a listen backlog of 5: at 1.40 per path the thread is 43% busy and 13,869 connections were dropped in the scoring windows, at 1.87 it is 62% busy, 26,105 dropped, median navigation 1,072 ms, and the departure gate fails as in s13. --backlog 1024 alone passes 2.33; handshake in the handler thread passes 2.8, and 3.27 to 4.2 with 0 to 3 failed requests per rate; at 2 guest vCPUs it passes 4.2 and collapses at 5.6. With the replay out of the path (guest-local page) 5.6 per path completes 3,357 of 3,360 and 8.4 collapses. Open: the limit above about 4.2 per path with the fix (the accept thread is idle there).
ejc3 commentedon Oct 26, 2026
Merged in #1340 (replay listener findings) and the harness fixes in #1347.