Skip to content

WO20: /benchmarks page — reproducible runtime benchmarks (scripts + script-generated data) - #239

Merged
ronaldtse merged 1 commit into
mainfrom
wo20-benchmarks
Nov 16, 2026
Merged

ronaldtse merged 1 commit into
mainfrom
wo20-benchmarks

Conversation

@ronaldtse

@ronaldtse ronaldtse commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

WO20 deliverable A: /benchmarks page + checked-in reproducible scripts (issue #238)

What's here

  • scripts/benchmarks/ — checked-in, runnable benchmark drivers per runtime plus one orchestrator:
    • corpus parse — every corpus .isc, median-of-3, exactly the parity-README methodology (in-process timing for ruby/ts/py; iscrs parse wall clock for rs);
    • golden replay — all 12,839 rows of the RS golden capture (isc-wf0.jsonl) through each runtime (in-process; rs via its native single-process iscrs capture);
    • transliteration throughput — a committed fixed input set (fixtures/input-set.jsonl, every 25th golden row, sampled once and frozen) so the series stays comparable as the golden evolves. rs has no batch transliteration CLI, so its throughput is derived from the capture leg and marked as such.
    • generate.mjs runs every leg strictly serially (one runtime at a time — the parity methodology) and writes src/data/benchmarks.json. Numbers are never hand-edited; the page renders only what the script produced.
  • /benchmarks page — current-HEAD numbers, machine profile, per-tree commit provenance, the parity-README audit-day table (2026-11-13) kept frozen as the baseline, and exact reproduce commands.
  • Nav: "Benchmarks" under Developers (header + drawer).
  • One CI-green regeneration is included: the committed src/data/benchmarks.json is the output of the scripts on this machine, and CI (astro check + build) passes.

Sibling-checkout requirement

The scripts read the four runtime engines and the map corpus from sibling checkouts of the monorepo layout (documented in scripts/benchmarks/README.md and on the page). All locations are env-overridable (INTERSCRIPT_MAPS/RS/TS/RUBY/PY, RUBY_BIN). First run used fresh origin/main clones — the local interscript-ts/interscript-ruby/interscript-rs checkouts sit on diverged branches, and the page records every tree's commit so provenance is explicit:

Tree Commit Version
interscript-rs 74b4b8d 0.3.2
interscript-ts b74cfdaf 5.10.0
interscript-ruby 4c47eb52 3.7.2
py ISC engine (unmerged wo17/py-override-placement branch of interscript-ruby) 5781a996 0.1.0
maps c55bc5c 428 maps

Machine: Apple M1 Max, macOS, ruby 3.4.8 (CI-covered; see finding below), node 24, python 3.10.5.

Results (current HEAD, median of 3, serial)

leg rs ts ruby py
corpus parse (428 maps) 26.4 s (wall) 15.3 s 167.4 s 4.7 s
golden replay (12,839 rows) 66.6 s (capture incl. parse) 7.1 s — 12,839/12,839 exact 7.6 s — 12,839/12,839 exact 61.8 s — 12,836/12,839 (3 diverge)
fixed-set throughput 193 rows/s (derived) 2,236 rows/s 2,191 rows/s 74 rows/s

vs the frozen audit-day baseline: rs 79 s → 26 s; py 81 s → 4.7 s (the generated-parser branch); ts 16 s → 15 s; ruby 161 s → 167 s.

Findings surfaced by the first run (not fixed here)

  1. py diverges on 3 of 12,839 golden rows — the Ў rows of the three thin ALA-LC Belarusian maps (gki-bel, mvd-bel, odni-bel), the newest corpus additions (maps v3.2.3 exhaustive per-row tests); ts and ruby replay them byte-exact. The run's guard reports divergence rather than hiding it.
  2. The bare ruby parse sweep is toolchain-sensitive. On the default local toolchain (ruby 4.0.7, with the interscript 3.5.0 and interscript-maps 3.2.0 gems installed) the sweep fails 120/428 corpus maps with MapNotFoundError from parse-time dependency locate (31 still fail even with an explicit Interscript.load_path). On CI-covered ruby 3.4.8 — no interscript gems installed, ruby -I <tree>/lib — the identical sweep parses 428/428 and replays the golden byte-exact. The benchmark scripts therefore expose RUBY_BIN and record the binary in the machine profile; the root cause of the 4.0.7 dependency-locate behavior is flagged for the owner, not fixed here.
  3. The py numbers come from the unmerged py/ISC branch of the interscript-ruby repo; interscript-py main still carries the legacy .imp engine and cannot run the ISC corpus at all.

Verification

  • npx astro check — 0 errors; npm run build — complete (benchmarks page prerendered); npm test — 239 passed (the one failing file, test/api-endpoints.test.ts, fails identically on a clean main checkout — pre-existing local-env server-start flake; CI does not run vitest).
  • npm run format:check / lint — my files clean; the repo-wide drift (25 files / 1 error) pre-exists on main and is untouched.

…rated

Deliverable A of WO20 (issue #238): published, reproducible evidence for
SOTA 2027.

scripts/benchmarks/ — checked-in drivers per runtime plus one
orchestrator:

- corpus parse, median-of-3, the parity-README methodology (in-process
  for ruby/ts/py, iscrs parse wall for rs)
- golden replay: the RS golden capture (isc-wf0.jsonl, 12,839 rows)
  through each runtime; rs via native iscrs capture
- transliteration throughput on a committed fixed input set
  (fixtures/input-set.jsonl, every 25th golden row, frozen); rs has no
  batch CLI, so its throughput derives from the capture leg and is
  marked as such

generate.mjs runs every leg strictly serially and writes
src/data/benchmarks.json — numbers are never hand-edited; the page
renders only what the script produced. The scripts read the four
runtime trees and the map corpus from sibling checkouts (env-overridable
INTERSCRIPT_* and RUBY_BIN); the requirement is documented in
scripts/benchmarks/README.md and on the page itself.

The page keeps the audit-day parity-README table (2026-11-13) frozen as
the baseline and tracks current-HEAD above it, with per-tree commit
provenance and a machine profile. Generated from fresh origin/main
clones: rs 0.3.2 (74b4b8d), ts 5.10.0 (b74cfdaf), ruby 3.7.2
(4c47eb52), py 0.1.0 (5781a996, the unmerged py/ISC branch of
interscript-ruby), maps c55bc5c, on an Apple M1 Max / ruby 3.4.8.

Findings surfaced by the first run (recorded on the page): py diverges
on 3 of 12,839 golden rows (the new Ў rows of gki-bel/mvd-bel/odni-bel);
ruby origin/main fails to parse 120/428 corpus maps under ruby 4.0.7
(CI covers 3.3/3.4 — the benchmarks run on 3.4.8, which parses 428/428).
@ronaldtse
ronaldtse merged commit 130a3db into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant