You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
WO20: published benchmarks page + API plane completion status #238
SOTA 2027 needs published, reproducible evidence. Two deliverables:
A. Benchmarks page on the site (interscript.github.io)
Reproducible benchmark scripts (checked in): per-runtime corpus parse (median-of-3, the parity-README methodology), per-runtime transliteration throughput on a fixed input set, golden replay time.
A /benchmarks page rendering current numbers + machine profile + exact reproduce commands, regenerated by script (never hand-edited numbers).
Keep the parity-README table as the audit-day baseline; the page tracks current-HEAD.
B. API plane completion (interscript/interscript-api)
Audit the repo against the agreed architecture (code-only npm package, maps as deployment artifacts, no-regex engine directive, bundled maps surface, ml surface). Produce a status table: done / partial / missing.
Complete the missing pieces that are code-only (worker code, wrangler config with generated types, package guard tests, package.json exports/files hygiene). Deployment itself (wrangler deploy, secrets) stays owner-held — record the exact deploy commands, do not run them.
Rules: PRs only; the benchmarks PR includes the scripts + page + one CI-green regeneration; the API PR includes the status table in the body. Branches: wo20-benchmarks, wo20-api-status.
WO20 milestone: deliverable B (API plane) — PR open
Audit + code-only completion is up as interscript/api#30 (branch wo20-api-status).
Key findings from the audit:
CI on the API plane's main has been red since the WO15 merge — src/ml.ts imports createPlaneModel from interscript/ml, but the pinned engine ^4.0.0 predates the plane module. The PR bumps the engine to ^5.14.0 (parity gate stays byte-identical, 48/48 green) — this was the one worker-code gap that actually broke the plane.
The software repo had no wrangler config and no generated types: the PR adds the canonical self-deploy wrangler.jsonc + committed worker-configuration.d.ts (npm run gen:types) with a CI freshness gate; @cloudflare/workers-types is superseded.
Package hygiene: CI now builds before vitest, the wrangler dry-run smoke is strict (no more || true), and test/package.test.ts grows phantom-entry / shipped-.d.ts-closure / manifest entry-point guards.
Owner-held (recorded in the PR, not run): wrangler deploy paths, ML_TOKEN secret, and the src/generated/models.json snapshot refresh (index v5 → current models.yaml is a release-round action).
Note: the issue names interscript/interscript-api, which is the legacy Ruby Lambda (rollback plane). The agreed Cloudflare architecture lives in interscript/api (software) + interscript/api.interscript.org (deployment); the PR body's status table audits the whole plane and leaves the legacy repo unchanged.
Deliverable A (benchmarks page, branch wo20-benchmarks) is in flight: scripts checked in, clean serial regeneration running on fresh origin/main clones of all four runtimes (the local interscript-ts/interscript-ruby/interscript-rs checkouts sit on diverged branches, so the run uses fresh clones and records every tree's SHA in the generated data). PR to follow.
Checked-in reproducible scripts under scripts/benchmarks/: per-runtime corpus parse (median-of-3, the parity-README methodology), golden replay (all 12,839 rows of the RS golden capture), and transliteration throughput on a committed fixed input set (every 25th golden row, sampled once and frozen). generate.mjs runs every leg strictly serially and writes src/data/benchmarks.json — numbers are script-generated, never hand-edited.
/benchmarks page renders the current-HEAD numbers, machine profile, per-tree commit provenance, exact reproduce commands, and keeps the parity-README audit-day table (2026-11-13) frozen as the baseline. Nav link added.
The scripts require sibling checkouts of the four runtimes + maps (env-overridable INTERSCRIPT_*/RUBY_BIN); the requirement is stated in scripts/benchmarks/README.md and on the page. The first run used fresh origin/main clones — the long-lived local interscript-ts/interscript-ruby/interscript-rs checkouts sit on diverged branches, so every tree's SHA is recorded in the generated data.
One CI-green regeneration included (astro check 0 errors, build green).
Current-HEAD headline numbers (median of 3, serial, Apple M1 Max, ruby 3.4.8):
leg
rs 0.3.2
ts 5.10.0
ruby 3.7.2
py 0.1.0 (ISC)
corpus parse, 428 maps
26.4 s
15.3 s
167.4 s
4.7 s
golden replay, 12,839 rows
66.6 s (capture)
7.1 s, byte-exact
7.6 s, byte-exact
61.8 s, 12,836 exact
fixed-set throughput
193 rows/s (derived)
2,236 rows/s
2,191 rows/s
74 rows/s
Deliverable B: API plane — interscript/api#30 (wo20-api-status), CI green
Status audit table in the PR body; code-only completion landed. The headline: CI on the API plane's main had been red since the WO15 merge — src/ml.ts imports createPlaneModel from interscript/ml while the pinned engine ^4.0.0 has no plane module, and vitest.config.ts aliased that import to a sibling ../interscript-ts checkout that doesn't exist on CI runners. Fixed by bumping the engine to ^5.14.0 and dropping the alias (48/48 tests against the published package, parity gate byte-identical). Also: canonical self-deploy wrangler.jsonc + generated worker-configuration.d.ts with a CI freshness gate, package-guard tests (phantom files, shipped-.d.ts closure, manifest entry points), strict wrangler dry-run smoke, onnxruntime-node devDep for the plane decode tests. Deployment stays owner-held; the exact wrangler deploy/e2e commands are recorded in the PR body — nothing was deployed.
Findings for the owner (flagged, not fixed)
py runtime divergence: the py ISC engine misses 3 of 12,839 golden rows — the Ў rows of gki-bel/mvd-bel/odni-bel (maps v3.2.3's newest exhaustive per-row tests); ts and ruby replay them byte-exact. Recorded on the benchmarks page.
Ruby bare-parse sweep is toolchain-sensitive: on ruby 4.0.7 with the interscript/interscript-maps gems installed, 120/428 corpus maps fail with MapNotFoundError at parse-time dependency locate (31 remain with an explicit load path); on CI-covered ruby 3.4.8 without those gems, 428/428. Benchmarks pin RUBY_BIN and record the binary; root cause flagged for investigation.
interscript-py main cannot run the ISC corpus at all (legacy .imp engine); all ISC py numbers come from the unmerged wo17/py-override-placement branch of interscript-ruby.
interscript/api's src/generated/models.json snapshot (index v5) is behind the current models registry — refresh is a release-round action alongside the deployment repo pin bump.
Activity
ronaldtse commentedon Nov 16, 2026
WO20 milestone: deliverable B (API plane) — PR open
Audit + code-only completion is up as interscript/api#30 (branch
wo20-api-status).Key findings from the audit:
mainhas been red since the WO15 merge —src/ml.tsimportscreatePlaneModelfrominterscript/ml, but the pinned engine^4.0.0predates the plane module. The PR bumps the engine to^5.14.0(parity gate stays byte-identical, 48/48 green) — this was the one worker-code gap that actually broke the plane.wrangler.jsonc+ committedworker-configuration.d.ts(npm run gen:types) with a CI freshness gate;@cloudflare/workers-typesis superseded.|| true), andtest/package.test.tsgrows phantom-entry / shipped-.d.ts-closure / manifest entry-point guards.wrangler deploypaths,ML_TOKENsecret, and thesrc/generated/models.jsonsnapshot refresh (index v5 → currentmodels.yamlis a release-round action).interscript/interscript-api, which is the legacy Ruby Lambda (rollback plane). The agreed Cloudflare architecture lives ininterscript/api(software) +interscript/api.interscript.org(deployment); the PR body's status table audits the whole plane and leaves the legacy repo unchanged.Deliverable A (benchmarks page, branch
wo20-benchmarks) is in flight: scripts checked in, clean serial regeneration running on fresh origin/main clones of all four runtimes (the localinterscript-ts/interscript-ruby/interscript-rscheckouts sit on diverged branches, so the run uses fresh clones and records every tree's SHA in the generated data). PR to follow.ronaldtse commentedon Nov 16, 2026
WO20 complete — both deliverables landed as PRs
Deliverable A: benchmarks page — #239 (
wo20-benchmarks)scripts/benchmarks/: per-runtime corpus parse (median-of-3, the parity-README methodology), golden replay (all 12,839 rows of the RS golden capture), and transliteration throughput on a committed fixed input set (every 25th golden row, sampled once and frozen).generate.mjsruns every leg strictly serially and writessrc/data/benchmarks.json— numbers are script-generated, never hand-edited./benchmarkspage renders the current-HEAD numbers, machine profile, per-tree commit provenance, exact reproduce commands, and keeps the parity-README audit-day table (2026-11-13) frozen as the baseline. Nav link added.INTERSCRIPT_*/RUBY_BIN); the requirement is stated inscripts/benchmarks/README.mdand on the page. The first run used fresh origin/main clones — the long-lived localinterscript-ts/interscript-ruby/interscript-rscheckouts sit on diverged branches, so every tree's SHA is recorded in the generated data.Current-HEAD headline numbers (median of 3, serial, Apple M1 Max, ruby 3.4.8):
Deliverable B: API plane — interscript/api#30 (
wo20-api-status), CI greenStatus audit table in the PR body; code-only completion landed. The headline: CI on the API plane's
mainhad been red since the WO15 merge —src/ml.tsimportscreatePlaneModelfrominterscript/mlwhile the pinned engine^4.0.0has no plane module, andvitest.config.tsaliased that import to a sibling../interscript-tscheckout that doesn't exist on CI runners. Fixed by bumping the engine to^5.14.0and dropping the alias (48/48 tests against the published package, parity gate byte-identical). Also: canonical self-deploywrangler.jsonc+ generatedworker-configuration.d.tswith a CI freshness gate, package-guard tests (phantom files, shipped-.d.tsclosure, manifest entry points), strict wrangler dry-run smoke,onnxruntime-nodedevDep for the plane decode tests. Deployment stays owner-held; the exactwrangler deploy/e2e commands are recorded in the PR body — nothing was deployed.Findings for the owner (flagged, not fixed)
Ўrows ofgki-bel/mvd-bel/odni-bel(maps v3.2.3's newest exhaustive per-row tests); ts and ruby replay them byte-exact. Recorded on the benchmarks page.interscript/interscript-mapsgems installed, 120/428 corpus maps fail withMapNotFoundErrorat parse-time dependency locate (31 remain with an explicit load path); on CI-covered ruby 3.4.8 without those gems, 428/428. Benchmarks pinRUBY_BINand record the binary; root cause flagged for investigation.interscript-pymain cannot run the ISC corpus at all (legacy.impengine); all ISC py numbers come from the unmergedwo17/py-override-placementbranch of interscript-ruby.interscript/api'ssrc/generated/models.jsonsnapshot (index v5) is behind the current models registry — refresh is a release-round action alongside the deployment repo pin bump.feat: /benchmarks page — reproducible runtime benchmarks, script-gene…