From 2c14924bb4fd882240630a573cecc10a805bf26e Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sun, 18 Oct 2026 20:28:36 +0300 Subject: [PATCH 01/17] perf(closures): fast-mode template, defaults in it, one call per nested value MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The template a decoded object starts from was parsed from JSON, which V8 hands over in dictionary mode from 128 keys on, so a record of 128 fields decoded seven times slower on closures than one of 127 (26 µs against 7 µs), and so did flags of 128 names on either backend. The template now comes from Object.fromEntries, keeps fast properties up to the engine's limit and holds every field at its default, so a missing field costs no store (a sparse record of 70 fields with three set decodes about four times faster); a nested value is one call instead of two, each field is read by a kind decided when the schema is built, and flags.unpack stores only the set bits (a record of 70 flags decodes about four and a half times faster on both backends). The wire, the decoded objects and the errors did not change; what remains is one keyed store per present field, about 60 ns against 30 in generated code, which the docs now state with the measured numbers. pnpm bench measures a record of 128 fields, and the new pnpm bench:sweep prints the cost per field from 4 to 256 fields, each (backend, width) point in a fresh process; the baseline, the comparison and the reports are regenerated. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 24 ++ CLAUDE.md | 17 +- README.md | 6 +- bench/baseline.json | 370 ++++++++++-------- bench/bench.js | 14 + bench/report/model.mjs | 1 + bench/results/compare.json | 208 +++++----- bench/results/sweep.json | 270 +++++++++++++ bench/sweep.js | 233 +++++++++++ docs/guide/architecture.md | 3 +- docs/guide/benchmarks.md | 36 +- docs/guide/runtimes.md | 35 +- docs/guide/standalone.md | 4 +- docs/public/benchmarks/coldstart.svg | 6 +- .../benchmarks/compare-numeric-bytes.svg | 6 +- docs/public/benchmarks/compare-numeric.svg | 42 +- .../benchmarks/compare-packets-bytes.svg | 6 +- docs/public/benchmarks/compare-packets.svg | 26 +- .../benchmarks/compare-records-bytes.svg | 6 +- docs/public/benchmarks/compare-records.svg | 38 +- .../benchmarks/compare-single-bytes.svg | 6 +- docs/public/benchmarks/compare-single.svg | 24 +- .../benchmarks/compare-sparse-bytes.svg | 6 +- docs/public/benchmarks/compare-sparse.svg | 44 +-- .../benchmarks/compare-strings-bytes.svg | 6 +- docs/public/benchmarks/compare-strings.svg | 40 +- .../public/benchmarks/compare-typed-bytes.svg | 6 +- docs/public/benchmarks/compare-typed.svg | 26 +- docs/public/benchmarks/engine-parse.svg | 218 ++++++----- docs/public/benchmarks/engine-stringify.svg | 218 ++++++----- docs/public/benchmarks/layouts.svg | 62 +-- package.json | 1 + src/closures.js | 169 +++++--- src/types.js | 31 +- tests/unit/codec.test.js | 59 +++ 35 files changed, 1513 insertions(+), 754 deletions(-) create mode 100644 bench/results/sweep.json create mode 100644 bench/sweep.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c2be09..29aaae0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **The closures backend decodes faster.** A decoded object starts as a copy of a template that + holds every field at its default, so a field that is missing costs no store: a sparse record of + 70 fields with three set decodes about four times faster. A nested value is one call instead of + two, and the loop reads each field by a kind decided when the schema is built. The same template + serves `flags`, where only the set bits are stored now: a record of 70 flags decodes about four + and a half times faster on both backends. The wire, the decoded objects and the errors did not + change. What remains is structural: the closures store each present field through one keyed + store, about 60 ns a field on decode against 30 in generated code (see + [Runtimes and CSP](https://protoarray.vercel.app/guide/runtimes)). +- `pnpm bench` measures a record of 128 fields, and the new `pnpm bench:sweep` prints the cost per + field from 4 to 256 fields for JSON and both backends, each point in a fresh process + (`bench/results/sweep.json`). + +### Fixed + +- **A record of 128 fields or more decoded seven times slower on closures than one of 127** (26 µs + against 7 µs): the template was parsed from JSON, which V8 hands over in dictionary mode from 128 + keys on, and a copy of such an object costs a transition per key. The template is built with + `Object.fromEntries` now and keeps fast properties up to the engine's limit; both backends stay + linear in the number of fields. The same applied to `flags` of 128 names or more on either + backend. + ## [2.0.0] - 2026-10-16 2.0 hands the definition of a schema to `@alexify/metaschema` and keeps the wire to protoarray. diff --git a/CLAUDE.md b/CLAUDE.md index c4027f8..0dc153a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,6 +36,7 @@ pnpm size [--max-gzip ] # esbuild bundle sizes of index.js and browse node bin/protoarray.js compile [--out f] [--format cjs|esm] [--runtime import|inline] [--only A,B] # the CLI pnpm bench [filter] [--json] [--save] [--compare] # engine ops/sec (median of 5 rounds ± spread), manual only; --save writes bench/baseline.json pnpm bench:compare [--scenarios a,b] [--adapters x,y] [--no-cold-start] [--save] # against JSON and 11 codecs, plus the cold start; --save merges into bench/results/compare.json +pnpm bench:sweep [--widths 4,8,…] [--standalone] [--json] [--save] # ns per field by record width, each (backend, width) point in a fresh process; --save writes bench/results/sweep.json pnpm bench:releases [--ref v1.0.0] # a release (a tag, via git archive) or the current tree on the compare scenarios, both backends; merges into bench/results/releases.json pnpm size:compare [--save] # browser bundle of one record's round trip per library, bench/results/size.json pnpm bench:report # the SVG charts (docs/public/benchmarks/) and the tables of README.md and docs/guide/benchmarks.md @@ -121,10 +122,10 @@ tests/ types/*.test-d.ts tsd assertions scripts/size.js bundle-size report and budget gate (CI smoke) scripts/size-compare.js cross-library bundle sizes; scripts/comparison.js renders the feature matrix -bench/ engine harness (bench.js), comparison (compare.js + compare/: adapters, scenarios, - measure, releases-worker), releases.js, report.js + report/ (model.mjs: the - chart models, svg.mjs: README images); baseline.json and results/*.json are - snapshots +bench/ engine harness (bench.js), the width sweep (sweep.js), comparison (compare.js + + compare/: adapters, scenarios, measure, releases-worker), releases.js, report.js + + report/ (model.mjs: the chart models, svg.mjs: README images); baseline.json + and results/*.json are snapshots docs/ VitePress site (protoarray.vercel.app), not published; comparison.json is the data of the feature matrix, .vitepress/theme/components/Bench*.vue the charts .claude/skills/ metaskills v1.0.5 skills (js-conventions, npm-publish adapted) @@ -181,8 +182,12 @@ docs/ VitePress site (protoarray.vercel.app), not published; compa 2. **A backend builds `{ encode, decode }` from the plan.** `codegen.js` emits one encode and one decode function per struct through `new Function` (array and object literals, so every decoded object has the hidden class an object literal would). `closures.js` composes closures over the - same plan and objects start as a spread of a literal-shaped template, so their hidden class is - still stable. The probe `new Function('return 1')` runs once, when the first schema that may + same plan and objects start as a spread of a template that holds every key at its default, so + their hidden class is still stable and a missing field costs no store; the template comes from + `Object.fromEntries` (`literal` in `types.js`), which keeps fast properties: V8 hands an object + parsed from JSON over in dictionary mode from 128 keys, one filled by keyed assignment from + about 18, and a spread of either costs a transition per key (`pnpm bench:sweep` tracks the cost + per field, `wide object (128 fields)` in `pnpm bench` the step). The probe `new Function('return 1')` runs once, when the first schema that may use generated code is built (`codegen.available()`), never for `{ codegen: false }` or a `compiled` codec, so such a program triggers no CSP report; `{ codegen }` forces a backend, `{ codegen: true }` throws `ERR_CODEGEN_UNAVAILABLE` where it cannot be honoured, and diff --git a/README.md b/README.md index a76c7e2..175118b 100644 --- a/README.md +++ b/README.md @@ -180,8 +180,8 @@ text or bytes and back. | Library | One record
round trips/s | 1,000 records
round trips/s | Sparse record
round trips/s | Typed record
round trips/s | Long strings
round trips/s | 100 wrpc packets
round trips/s | 32 doubles
round trips/s | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 572,000 | 554 | 2,730,000 | 640,000 | 719,000 | 16,400 | 194,000 | -| **protoarray**
codegen · v2.0.0 | 940,000 | 842 | 3,750,000 | 2,390,000 | 876,000 | 28,300 | 261,000 | -| **protoarray**
closures, no eval · v2.0.0 | 623,000 | 574 | 1,330,000 | 1,650,000 | 807,000 | 17,200 | 220,000 | +| **protoarray**
codegen · v2.0.0 | 938,000 | 847 | 3,740,000 | 2,380,000 | 888,000 | 28,500 | 257,000 | +| **protoarray**
closures, no eval · v2.0.0 | 630,000 | 575 | 1,800,000 | 1,620,000 | 826,000 | 16,600 | 222,000 | | **protoarray**
standalone module, no eval · v2.0.0 | 899,000 | 846 | 3,470,000 | 2,340,000 | 858,000 | 27,500 | 260,000 | | **protoarray**
codegen + metaschema check · v2.0.0 | 689,000 | 624 | 1,630,000 | 1,780,000 | 809,000 | 20,200 | 231,000 | | **protoarray**
columns layout · v2.0.0 | — | 879 | — | — | — | — | — | @@ -199,7 +199,7 @@ text or bytes and back. | devalue
stringify · parse · v6.0.3 | 184,000 | 171 | 1,010,000 | 383,000 | 344,000 | 5,960 | 123,000 | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 615,000 | 577 | 3,560,000 | 1,820,000 | 664,000 | — | 395,000 | -_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-16. A round trip is one encode and one decode of the payload; medians, rounded to three significant figures; — where the scenario does not apply._ +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. A round trip is one encode and one decode of the payload; medians, rounded to three significant figures; — where the scenario does not apply._ diff --git a/bench/baseline.json b/bench/baseline.json index 9816429..007381c 100644 --- a/bench/baseline.json +++ b/bench/baseline.json @@ -1,518 +1,558 @@ { "node": "v24.14.1", - "date": "2026-10-16T18:30:33.111Z", + "date": "2026-10-18T17:15:58.532Z", "results": [ { "name": "flat object (4 fields): JSON.stringify (keyed, reference)", - "opsPerSec": 5430865, + "opsPerSec": 5378497, "spread": 3 }, { "name": "flat object (4 fields): stringify (codegen)", - "opsPerSec": 6754475, - "spread": 3 + "opsPerSec": 6908081, + "spread": 1 }, { "name": "flat object (4 fields): check (metaschema) + stringify (codegen)", - "opsPerSec": 4506580, - "spread": 1 + "opsPerSec": 4552530, + "spread": 2 }, { "name": "flat object (4 fields): stringify (closures)", - "opsPerSec": 5641119, - "spread": 2 + "opsPerSec": 5905135, + "spread": 1 }, { "name": "flat object (4 fields): check (metaschema) + stringify (closures)", - "opsPerSec": 3810128, - "spread": 7 + "opsPerSec": 4080857, + "spread": 4 }, { "name": "flat object (4 fields): JSON.parse (keyed, reference)", - "opsPerSec": 3724096, - "spread": 2 + "opsPerSec": 3818097, + "spread": 1 }, { "name": "flat object (4 fields): parse (codegen)", - "opsPerSec": 6827288, - "spread": 2 + "opsPerSec": 7047625, + "spread": 1 }, { "name": "flat object (4 fields): parse (closures)", - "opsPerSec": 4310870, - "spread": 2 + "opsPerSec": 4446616, + "spread": 1 }, { "name": "nested object: JSON.stringify (keyed, reference)", - "opsPerSec": 1434024, - "spread": 1 + "opsPerSec": 1463062, + "spread": 2 }, { "name": "nested object: stringify (codegen)", - "opsPerSec": 1966355, + "opsPerSec": 1973269, "spread": 1 }, { "name": "nested object: check (metaschema) + stringify (codegen)", - "opsPerSec": 1055223, - "spread": 2 + "opsPerSec": 1074692, + "spread": 1 }, { "name": "nested object: stringify (closures)", - "opsPerSec": 1513275, - "spread": 2 + "opsPerSec": 1566011, + "spread": 1 }, { "name": "nested object: check (metaschema) + stringify (closures)", - "opsPerSec": 903182, + "opsPerSec": 926735, "spread": 1 }, { "name": "nested object: JSON.parse (keyed, reference)", - "opsPerSec": 924816, + "opsPerSec": 943705, "spread": 1 }, { "name": "nested object: parse (codegen)", - "opsPerSec": 1644326, + "opsPerSec": 1657169, "spread": 2 }, { "name": "nested object: parse (closures)", - "opsPerSec": 867708, + "opsPerSec": 860017, "spread": 1 }, { "name": "sparse object (3 of 20): JSON.stringify (keyed, reference)", - "opsPerSec": 6697156, + "opsPerSec": 6623864, "spread": 1 }, { "name": "sparse object (3 of 20): stringify (codegen)", - "opsPerSec": 6770678, - "spread": 2 + "opsPerSec": 6894832, + "spread": 1 }, { "name": "sparse object (3 of 20): check (metaschema) + stringify (codegen)", - "opsPerSec": 2065974, + "opsPerSec": 2127174, "spread": 1 }, { "name": "sparse object (3 of 20): stringify (closures)", - "opsPerSec": 2653609, - "spread": 2 + "opsPerSec": 2747115, + "spread": 1 }, { "name": "sparse object (3 of 20): check (metaschema) + stringify (closures)", - "opsPerSec": 1412770, - "spread": 1 + "opsPerSec": 1468460, + "spread": 2 }, { "name": "sparse object (3 of 20): JSON.parse (keyed, reference)", - "opsPerSec": 4663739, + "opsPerSec": 4554826, "spread": 1 }, { "name": "sparse object (3 of 20): parse (codegen)", - "opsPerSec": 7113587, + "opsPerSec": 7240902, "spread": 1 }, { "name": "sparse object (3 of 20): parse (closures)", - "opsPerSec": 1388388, + "opsPerSec": 1699421, "spread": 1 }, { "name": "typed object (enum, flags, date): JSON.stringify (keyed, reference)", - "opsPerSec": 2984920, - "spread": 2 + "opsPerSec": 2989570, + "spread": 1 }, { "name": "typed object (enum, flags, date): stringify (codegen)", - "opsPerSec": 4631975, + "opsPerSec": 4670819, "spread": 1 }, { "name": "typed object (enum, flags, date): check (metaschema) + stringify (codegen)", - "opsPerSec": 2247121, + "opsPerSec": 2271656, "spread": 1 }, { "name": "typed object (enum, flags, date): stringify (closures)", - "opsPerSec": 3127871, - "spread": 2 + "opsPerSec": 3125463, + "spread": 1 }, { "name": "typed object (enum, flags, date): check (metaschema) + stringify (closures)", - "opsPerSec": 1771924, + "opsPerSec": 1797984, "spread": 1 }, { "name": "typed object (enum, flags, date): JSON.parse (keyed, reference)", - "opsPerSec": 2118376, - "spread": 1 + "opsPerSec": 2122792, + "spread": 2 }, { "name": "typed object (enum, flags, date): parse (codegen)", - "opsPerSec": 3939181, - "spread": 2 + "opsPerSec": 3996320, + "spread": 1 }, { "name": "typed object (enum, flags, date): parse (closures)", - "opsPerSec": 2070272, - "spread": 3 + "opsPerSec": 2058430, + "spread": 1 }, { "name": "packet (union): JSON.stringify (keyed, reference)", - "opsPerSec": 3617670, + "opsPerSec": 3643315, "spread": 1 }, { "name": "packet (union): stringify (codegen)", - "opsPerSec": 5089820, - "spread": 2 + "opsPerSec": 5039237, + "spread": 1 }, { "name": "packet (union): check (metaschema) + stringify (codegen)", - "opsPerSec": 2839513, - "spread": 3 + "opsPerSec": 2830639, + "spread": 2 }, { "name": "packet (union): stringify (closures)", - "opsPerSec": 3955165, - "spread": 1 + "opsPerSec": 4046015, + "spread": 2 }, { "name": "packet (union): check (metaschema) + stringify (closures)", - "opsPerSec": 2326119, - "spread": 4 + "opsPerSec": 2348300, + "spread": 3 }, { "name": "packet (union): JSON.parse (keyed, reference)", - "opsPerSec": 2195014, + "opsPerSec": 2265030, "spread": 1 }, { "name": "packet (union): parse (codegen)", - "opsPerSec": 4242250, - "spread": 2 + "opsPerSec": 4227193, + "spread": 1 }, { "name": "packet (union): parse (closures)", - "opsPerSec": 2283962, - "spread": 2 + "opsPerSec": 2294634, + "spread": 1 }, { "name": "money (decimals): JSON.stringify (keyed, reference)", - "opsPerSec": 4872723, + "opsPerSec": 4886993, "spread": 1 }, { "name": "money (decimals): stringify (codegen)", - "opsPerSec": 9889055, + "opsPerSec": 10050521, "spread": 1 }, { "name": "money (decimals): check (metaschema) + stringify (codegen)", - "opsPerSec": 5171320, - "spread": 1 + "opsPerSec": 5208550, + "spread": 2 }, { "name": "money (decimals): stringify (closures)", - "opsPerSec": 6088127, + "opsPerSec": 6119243, "spread": 1 }, { "name": "money (decimals): check (metaschema) + stringify (closures)", - "opsPerSec": 3807913, + "opsPerSec": 3803160, "spread": 2 }, { "name": "money (decimals): JSON.parse (keyed, reference)", - "opsPerSec": 4119858, + "opsPerSec": 4114398, "spread": 1 }, { "name": "money (decimals): parse (codegen)", - "opsPerSec": 8957765, - "spread": 0 + "opsPerSec": 8991095, + "spread": 2 }, { "name": "money (decimals): parse (closures)", - "opsPerSec": 4705833, - "spread": 2 + "opsPerSec": 4754161, + "spread": 1 }, { "name": "sparse object (3 of 70): JSON.stringify (keyed, reference)", - "opsPerSec": 6642989, + "opsPerSec": 6625070, "spread": 1 }, { "name": "sparse object (3 of 70): stringify (codegen)", - "opsPerSec": 6194428, + "opsPerSec": 6256542, "spread": 1 }, { "name": "sparse object (3 of 70): check (metaschema) + stringify (codegen)", - "opsPerSec": 764247, - "spread": 0 + "opsPerSec": 802355, + "spread": 1 }, { "name": "sparse object (3 of 70): stringify (closures)", - "opsPerSec": 969898, + "opsPerSec": 1113692, "spread": 1 }, { "name": "sparse object (3 of 70): check (metaschema) + stringify (closures)", - "opsPerSec": 464778, - "spread": 2 + "opsPerSec": 501186, + "spread": 0 }, { "name": "sparse object (3 of 70): JSON.parse (keyed, reference)", - "opsPerSec": 4628758, - "spread": 2 + "opsPerSec": 4566282, + "spread": 1 }, { "name": "sparse object (3 of 70): parse (codegen)", - "opsPerSec": 5048858, - "spread": 2 + "opsPerSec": 5086982, + "spread": 1 }, { "name": "sparse object (3 of 70): parse (closures)", - "opsPerSec": 324960, + "opsPerSec": 465381, "spread": 1 }, { "name": "wide flags (70 names): JSON.stringify (keyed, reference)", - "opsPerSec": 5180605, - "spread": 2 + "opsPerSec": 5138144, + "spread": 1 }, { "name": "wide flags (70 names): stringify (codegen)", - "opsPerSec": 1328518, - "spread": 1 + "opsPerSec": 1313851, + "spread": 2 }, { "name": "wide flags (70 names): check (metaschema) + stringify (codegen)", - "opsPerSec": 1108638, + "opsPerSec": 1109344, "spread": 1 }, { "name": "wide flags (70 names): stringify (closures)", - "opsPerSec": 1131079, + "opsPerSec": 1104338, "spread": 1 }, { "name": "wide flags (70 names): check (metaschema) + stringify (closures)", - "opsPerSec": 955889, + "opsPerSec": 946163, "spread": 1 }, { "name": "wide flags (70 names): JSON.parse (keyed, reference)", - "opsPerSec": 3234115, + "opsPerSec": 3293230, "spread": 1 }, { "name": "wide flags (70 names): parse (codegen)", - "opsPerSec": 615841, - "spread": 1 + "opsPerSec": 2885873, + "spread": 2 }, { "name": "wide flags (70 names): parse (closures)", - "opsPerSec": 559861, + "opsPerSec": 2178617, + "spread": 1 + }, + { + "name": "wide object (128 fields): JSON.stringify (keyed, reference)", + "opsPerSec": 210927, + "spread": 2 + }, + { + "name": "wide object (128 fields): stringify (codegen)", + "opsPerSec": 367373, + "spread": 2 + }, + { + "name": "wide object (128 fields): check (metaschema) + stringify (codegen)", + "opsPerSec": 152565, + "spread": 1 + }, + { + "name": "wide object (128 fields): stringify (closures)", + "opsPerSec": 184771, + "spread": 1 + }, + { + "name": "wide object (128 fields): check (metaschema) + stringify (closures)", + "opsPerSec": 107704, + "spread": 1 + }, + { + "name": "wide object (128 fields): JSON.parse (keyed, reference)", + "opsPerSec": 76468, + "spread": 2 + }, + { + "name": "wide object (128 fields): parse (codegen)", + "opsPerSec": 319416, + "spread": 1 + }, + { + "name": "wide object (128 fields): parse (closures)", + "opsPerSec": 98991, "spread": 2 }, { "name": "bytes (1 KB): JSON.stringify + base64 (reference)", - "opsPerSec": 990286, + "opsPerSec": 1001593, "spread": 1 }, { "name": "bytes (1 KB): stringify (codegen)", - "opsPerSec": 1090504, - "spread": 1 + "opsPerSec": 1089536, + "spread": 3 }, { "name": "bytes (1 KB): stringify (closures)", - "opsPerSec": 1044250, - "spread": 1 + "opsPerSec": 1030871, + "spread": 2 }, { "name": "bytes (1 KB): JSON.parse + base64 (reference)", - "opsPerSec": 975737, + "opsPerSec": 975818, "spread": 1 }, { "name": "bytes (1 KB): parse (codegen)", - "opsPerSec": 722131, - "spread": 1 + "opsPerSec": 712790, + "spread": 2 }, { "name": "bytes (1 KB): parse (closures)", - "opsPerSec": 678094, - "spread": 1 + "opsPerSec": 679373, + "spread": 2 }, { "name": "bytes (1 KB): parse (codegen, validate: false)", - "opsPerSec": 720426, - "spread": 2 + "opsPerSec": 717336, + "spread": 1 }, { "name": "flat object (4 fields): stringify (codegen, validate: false)", - "opsPerSec": 7020952, - "spread": 1 + "opsPerSec": 7204373, + "spread": 2 }, { "name": "1000 records: JSON.stringify (keyed, reference)", - "opsPerSec": 1713, - "spread": 2 + "opsPerSec": 1665, + "spread": 3 }, { "name": "1000 records: stringify (rows)", - "opsPerSec": 2247, + "opsPerSec": 2229, "spread": 1 }, { "name": "1000 records: stringify (flat)", - "opsPerSec": 2113, - "spread": 1 + "opsPerSec": 2073, + "spread": 2 }, { "name": "1000 records: stringify (columns)", - "opsPerSec": 2217, - "spread": 0 + "opsPerSec": 2187, + "spread": 1 }, { "name": "1000 records: JSON.parse (keyed, reference)", - "opsPerSec": 1247, + "opsPerSec": 1255, "spread": 1 }, { "name": "1000 records: parse (rows)", - "opsPerSec": 2002, + "opsPerSec": 1939, "spread": 3 }, { "name": "1000 records: parse (flat)", - "opsPerSec": 1979, - "spread": 2 + "opsPerSec": 1978, + "spread": 1 }, { "name": "1000 records: parse (columns)", - "opsPerSec": 1980, - "spread": 2 + "opsPerSec": 1964, + "spread": 1 }, { "name": "converted record: stringify + read (codegen, stringify: auto, the builder)", - "opsPerSec": 6333762, - "spread": 4 + "opsPerSec": 6455159, + "spread": 2 }, { "name": "converted record: stringify + read (codegen, stringify: native)", - "opsPerSec": 5296090, - "spread": 4 + "opsPerSec": 5219837, + "spread": 2 }, { "name": "nested object: stringify + read (codegen, stringify: builder)", - "opsPerSec": 1611449, - "spread": 5 + "opsPerSec": 1550892, + "spread": 2 }, { "name": "nested object: stringify + read (codegen, stringify: native)", - "opsPerSec": 2147727, + "opsPerSec": 2146294, "spread": 2 }, { "name": "typed object (enum, flags, date): stringify + read (codegen, stringify: builder)", - "opsPerSec": 4954673, - "spread": 2 + "opsPerSec": 4994942, + "spread": 1 }, { "name": "resolved (renamed, defaulted): parse (codegen)", - "opsPerSec": 7080495, + "opsPerSec": 6980321, "spread": 1 }, { "name": "resolved (renamed, defaulted): stringify (codegen)", - "opsPerSec": 3711297, + "opsPerSec": 3604992, "spread": 3 }, { "name": "1000 records: stringify (columns, delta + intern)", - "opsPerSec": 2347, + "opsPerSec": 2350, "spread": 1 }, { "name": "1000 records: parse (columns, delta + intern)", - "opsPerSec": 2126, + "opsPerSec": 2075, "spread": 1 }, { "name": "nested object: stringify (standalone, inline)", - "opsPerSec": 2152445, + "opsPerSec": 2148499, "spread": 1 }, { "name": "nested object: parse (standalone, inline)", - "opsPerSec": 1732828, + "opsPerSec": 1707463, "spread": 1 }, { "name": "Schema.from (nested definition, codegen)", - "opsPerSec": 15786, - "spread": 5 + "opsPerSec": 15544, + "spread": 3 }, { "name": "Schema.from (nested definition, closures)", - "opsPerSec": 26081, - "spread": 6 + "opsPerSec": 23804, + "spread": 7 }, { "name": "Schema.from (nested snapshot, codegen)", - "opsPerSec": 27978, - "spread": 1 + "opsPerSec": 27583, + "spread": 3 }, { "name": "1000 records: parseItems (stream, 64 KB chunks)", - "opsPerSec": 885, - "spread": 2 + "opsPerSec": 859, + "spread": 1 }, { "name": "1000 records: parse (rows, whole text)", - "opsPerSec": 2011, + "opsPerSec": 1945, "spread": 1 } ], "coldStart": [ { "name": "nested object (codegen)", - "requireMs": 12.618166, - "buildMs": 4.004040999999997, - "firstCallMs": 0.23387500000000117 + "requireMs": 12.701958999999999, + "buildMs": 4.068999999999999, + "firstCallMs": 0.24329099999999926 }, { "name": "nested object (closures)", - "requireMs": 12.773834, - "buildMs": 3.543458000000001, - "firstCallMs": 0.22595900000000313 + "requireMs": 13.003459000000001, + "buildMs": 3.667375, + "firstCallMs": 0.24054200000000492 }, { "name": "packet (union, codegen)", - "requireMs": 12.534333, - "buildMs": 4.189916, - "firstCallMs": 0.15812500000000185 + "requireMs": 12.572125, + "buildMs": 4.212750000000003, + "firstCallMs": 0.1600830000000002 }, { "name": "sparse object (70 fields, codegen)", - "requireMs": 12.399542, - "buildMs": 4.313458999999998, - "firstCallMs": 0.4432919999999996 + "requireMs": 12.679124999999999, + "buildMs": 4.345458000000001, + "firstCallMs": 0.4432079999999985 }, { "name": "nested object (snapshot, codegen)", - "requireMs": 4.472791000000001, - "buildMs": 2.6734159999999996, - "firstCallMs": 0.23874999999999957 + "requireMs": 4.5177499999999995, + "buildMs": 2.7060829999999996, + "firstCallMs": 0.23887500000000017 } ] } diff --git a/bench/bench.js b/bench/bench.js index 804ea73..764eae4 100644 --- a/bench/bench.js +++ b/bench/bench.js @@ -133,6 +133,19 @@ const WIDE_FLAGS = { perms: { flags: Array.from({ length: 70 }, (_, i) => `n${i}`) }, }; const WIDE_FLAGS_OBJECT = { id: 1, perms: { n1: true, n35: true, n69: true } }; +// A record of 128 fields of the types that travel as they are: past 127 keys +// V8 keeps an object parsed from JSON in dictionary mode, which is where the +// closures backend's template used to fall off a cliff. +const WIDE_TYPES = ['string', 'integer', 'boolean', 'number']; +const WIDE128 = Object.fromEntries( + Array.from({ length: 128 }, (_, i) => [`f${String(i).padStart(5, '0')}`, WIDE_TYPES[i % 4]]), +); +const WIDE128_OBJECT = Object.fromEntries( + Object.keys(WIDE128).map((key, i) => { + const type = i % 4; + return [key, type === 0 ? `v${i}` : type === 1 ? i : type === 2 ? (i & 1) === 0 : i / 8]; + }), +); // A record of converted types only: where `stringify: 'auto'` writes the text itself. const CONVERTED = { id: 'integer', @@ -314,6 +327,7 @@ const main = async () => { ...pairs('money (decimals)', MONEY, {}, MONEY_OBJECT), ...pairs('sparse object (3 of 70)', SPARSE70, { sparse: true }, SPARSE70_OBJECT), ...pairs('wide flags (70 names)', WIDE_FLAGS, {}, WIDE_FLAGS_OBJECT), + ...pairs('wide object (128 fields)', WIDE128, {}, WIDE128_OBJECT), // JSON has no bytes: the reference converts with Buffer as an application would. [ 'bytes (1 KB): JSON.stringify + base64 (reference)', diff --git a/bench/report/model.mjs b/bench/report/model.mjs index eeb81ba..8bce888 100644 --- a/bench/report/model.mjs +++ b/bench/report/model.mjs @@ -481,6 +481,7 @@ const ENGINE = [ { prefix: 'packet (union)', label: 'Tagged union', sublabel: 'a wrpc packet' }, { prefix: 'money (decimals)', label: 'Money', sublabel: 'decimals' }, { prefix: 'wide flags (70 names)', label: 'Wide flags', sublabel: '70 names' }, + { prefix: 'wide object (128 fields)', label: 'Wide object', sublabel: '128 fields' }, { prefix: 'bytes (1 KB)', label: 'Bytes', sublabel: '1 KB, base64' }, ]; diff --git a/bench/results/compare.json b/bench/results/compare.json index 5d445a4..bf11fbc 100644 --- a/bench/results/compare.json +++ b/bench/results/compare.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-16T20:34:06.440Z", + "generatedAt": "2026-10-18T17:17:42.838Z", "machine": { "cpu": "Apple M3 Max", "cores": 16, @@ -32,7 +32,7 @@ "package": "@alexify/protoarray", "version": "2.0.0", "family": "text", - "commit": "72941bf" + "commit": "5a482c5" }, { "id": "protoarray-snapshot", @@ -50,7 +50,7 @@ "package": "@alexify/protoarray", "version": "2.0.0", "family": "text", - "commit": "72941bf" + "commit": "5a482c5" }, { "id": "protoarray-standalone", @@ -212,16 +212,16 @@ }, "protoarray": { "encode": { - "median": 2066250, - "min": 2040820, - "max": 2150642, - "spread": 5 + "median": 2060585, + "min": 2058311, + "max": 2095963, + "spread": 2 }, "decode": { - "median": 1723648, - "min": 1705544, - "max": 1726956, - "spread": 1 + "median": 1721172, + "min": 1705829, + "max": 1743201, + "spread": 2 }, "bytes": { "raw": 135, @@ -231,15 +231,15 @@ }, "protoarray-closures": { "encode": { - "median": 1500853, - "min": 1500230, - "max": 1507164, - "spread": 0 + "median": 1565372, + "min": 1555673, + "max": 1584881, + "spread": 2 }, "decode": { - "median": 1065375, - "min": 1060196, - "max": 1077570, + "median": 1052962, + "min": 1039818, + "max": 1063413, "spread": 2 }, "bytes": { @@ -542,15 +542,15 @@ }, "protoarray": { "encode": { - "median": 1958, - "min": 1955, - "max": 1974, - "spread": 1 + "median": 1978, + "min": 1970, + "max": 2000, + "spread": 2 }, "decode": { - "median": 1476, - "min": 1457, - "max": 1483, + "median": 1482, + "min": 1461, + "max": 1484, "spread": 2 }, "bytes": { @@ -561,15 +561,15 @@ }, "protoarray-closures": { "encode": { - "median": 1390, - "min": 1359, - "max": 1402, - "spread": 3 + "median": 1424, + "min": 1413, + "max": 1432, + "spread": 1 }, "decode": { - "median": 978, - "min": 974, - "max": 984, + "median": 965, + "min": 959, + "max": 968, "spread": 1 }, "bytes": { @@ -910,16 +910,16 @@ }, "protoarray": { "encode": { - "median": 7466131, - "min": 7284012, - "max": 7518319, - "spread": 3 + "median": 7410265, + "min": 7147069, + "max": 7506210, + "spread": 5 }, "decode": { - "median": 7541369, - "min": 7491594, - "max": 7555459, - "spread": 1 + "median": 7549897, + "min": 7389689, + "max": 7651469, + "spread": 3 }, "bytes": { "raw": 23, @@ -929,16 +929,16 @@ }, "protoarray-closures": { "encode": { - "median": 3029115, - "min": 2977120, - "max": 3038991, - "spread": 2 + "median": 3233774, + "min": 3109659, + "max": 3314224, + "spread": 6 }, "decode": { - "median": 2382353, - "min": 2379411, - "max": 2389168, - "spread": 0 + "median": 4067092, + "min": 4049652, + "max": 4142724, + "spread": 2 }, "bytes": { "raw": 23, @@ -1240,16 +1240,16 @@ }, "protoarray": { "encode": { - "median": 4717208, - "min": 4700002, - "max": 4802629, - "spread": 2 + "median": 4677198, + "min": 4556154, + "max": 4747663, + "spread": 4 }, "decode": { - "median": 4844478, - "min": 4839329, - "max": 4887903, - "spread": 1 + "median": 4832085, + "min": 4795753, + "max": 4890116, + "spread": 2 }, "bytes": { "raw": 36, @@ -1259,16 +1259,16 @@ }, "protoarray-closures": { "encode": { - "median": 3514161, - "min": 3504240, - "max": 3542441, - "spread": 1 + "median": 3518242, + "min": 3459464, + "max": 3524019, + "spread": 2 }, "decode": { - "median": 3118202, - "min": 3098851, - "max": 3127262, - "spread": 1 + "median": 3011498, + "min": 2953768, + "max": 3084242, + "spread": 4 }, "bytes": { "raw": 36, @@ -1570,15 +1570,15 @@ }, "protoarray": { "encode": { - "median": 1811907, - "min": 1805132, - "max": 1817117, + "median": 1824978, + "min": 1814318, + "max": 1836761, "spread": 1 }, "decode": { - "median": 1695038, - "min": 1667854, - "max": 1703789, + "median": 1730970, + "min": 1711097, + "max": 1743955, "spread": 2 }, "bytes": { @@ -1589,15 +1589,15 @@ }, "protoarray-closures": { "encode": { - "median": 1673013, - "min": 1668292, - "max": 1676170, - "spread": 0 + "median": 1719394, + "min": 1707707, + "max": 1729188, + "spread": 1 }, "decode": { - "median": 1557716, - "min": 1545152, - "max": 1578421, + "median": 1589045, + "min": 1565012, + "max": 1597617, "spread": 2 }, "bytes": { @@ -1900,15 +1900,15 @@ }, "protoarray": { "encode": { - "median": 66308, - "min": 65849, - "max": 66834, - "spread": 1 + "median": 66967, + "min": 65753, + "max": 67295, + "spread": 2 }, "decode": { - "median": 49455, - "min": 48725, - "max": 49587, + "median": 49676, + "min": 49477, + "max": 50345, "spread": 2 }, "bytes": { @@ -1919,15 +1919,15 @@ }, "protoarray-closures": { "encode": { - "median": 48093, - "min": 47942, - "max": 48209, + "median": 47938, + "min": 47806, + "max": 48186, "spread": 1 }, "decode": { - "median": 26769, - "min": 26446, - "max": 26819, + "median": 25505, + "min": 25225, + "max": 25579, "spread": 1 }, "bytes": { @@ -2211,16 +2211,16 @@ }, "protoarray": { "encode": { - "median": 406134, - "min": 405033, - "max": 411158, + "median": 396687, + "min": 390975, + "max": 400574, "spread": 2 }, "decode": { - "median": 733235, - "min": 729226, - "max": 738020, - "spread": 1 + "median": 729668, + "min": 727311, + "max": 742910, + "spread": 2 }, "bytes": { "raw": 582, @@ -2230,15 +2230,15 @@ }, "protoarray-closures": { "encode": { - "median": 360928, - "min": 354789, - "max": 365179, - "spread": 3 + "median": 369242, + "min": 367796, + "max": 374299, + "spread": 2 }, "decode": { - "median": 566586, - "min": 563770, - "max": 569043, + "median": 558993, + "min": 553339, + "max": 560697, "spread": 1 }, "bytes": { diff --git a/bench/results/sweep.json b/bench/results/sweep.json new file mode 100644 index 0000000..ea334c5 --- /dev/null +++ b/bench/results/sweep.json @@ -0,0 +1,270 @@ +{ + "node": "v24.14.1", + "date": "2026-10-18T17:17:42.800Z", + "records": 128, + "widths": [ + 4, + 8, + 16, + 32, + 48, + 64, + 96, + 127, + 128, + 256 + ], + "backends": [ + "json", + "codegen", + "closures" + ], + "points": [ + { + "backend": "json", + "n": 4, + "bytes": 57, + "encode": 4779292.018514145, + "decode": 3474444.1851938297 + }, + { + "backend": "codegen", + "n": 4, + "bytes": 21, + "encode": 6625982.475720087, + "decode": 6349315.659235001 + }, + { + "backend": "closures", + "n": 4, + "bytes": 21, + "encode": 5502404.594041766, + "decode": 4373059.258206903 + }, + { + "backend": "json", + "n": 8, + "bytes": 113, + "encode": 2690512.539358805, + "decode": 2177976.0326443342 + }, + { + "backend": "codegen", + "n": 8, + "bytes": 41, + "encode": 3975791.7289021416, + "decode": 3913844.0206123753 + }, + { + "backend": "closures", + "n": 8, + "bytes": 41, + "encode": 3084514.2383661373, + "decode": 2470115.9966480043 + }, + { + "backend": "json", + "n": 16, + "bytes": 227, + "encode": 1433197.7616470298, + "decode": 1159149.1721887207 + }, + { + "backend": "codegen", + "n": 16, + "bytes": 83, + "encode": 2249568.4460532637, + "decode": 2133532.4431170463 + }, + { + "backend": "closures", + "n": 16, + "bytes": 83, + "encode": 1676397.144381504, + "decode": 1316030.671285174 + }, + { + "backend": "json", + "n": 32, + "bytes": 459, + "encode": 744506.0447946758, + "decode": 602240.3751142889 + }, + { + "backend": "codegen", + "n": 32, + "bytes": 171, + "encode": 1196623.1670171223, + "decode": 1025700.5398154422 + }, + { + "backend": "closures", + "n": 32, + "bytes": 171, + "encode": 835815.3908088433, + "decode": 630611.056827418 + }, + { + "backend": "json", + "n": 48, + "bytes": 691, + "encode": 514251.6723835716, + "decode": 439339.0137872838 + }, + { + "backend": "codegen", + "n": 48, + "bytes": 259, + "encode": 855707.6904022597, + "decode": 692933.677063574 + }, + { + "backend": "closures", + "n": 48, + "bytes": 259, + "encode": 505048.3338707003, + "decode": 371178.50257839245 + }, + { + "backend": "json", + "n": 64, + "bytes": 923, + "encode": 393484.5227854831, + "decode": 334559.55081542145 + }, + { + "backend": "codegen", + "n": 64, + "bytes": 347, + "encode": 633396.0823539266, + "decode": 535412.4561554812 + }, + { + "backend": "closures", + "n": 64, + "bytes": 347, + "encode": 371193.0920986881, + "decode": 285851.8176175806 + }, + { + "backend": "json", + "n": 96, + "bytes": 1391, + "encode": 268760.3578472175, + "decode": 219829.57737333977 + }, + { + "backend": "codegen", + "n": 96, + "bytes": 527, + "encode": 446951.6214485828, + "decode": 350942.2428102296 + }, + { + "backend": "closures", + "n": 96, + "bytes": 527, + "encode": 252463.52085201052, + "decode": 187785.35403993685 + }, + { + "backend": "json", + "n": 127, + "bytes": 1861, + "encode": 199217.34018890007, + "decode": 168234.27061959688 + }, + { + "backend": "codegen", + "n": 127, + "bytes": 718, + "encode": 329819.8882865047, + "decode": 266389.25162292726 + }, + { + "backend": "closures", + "n": 127, + "bytes": 718, + "encode": 183251.06689976394, + "decode": 137917.57754063408 + }, + { + "backend": "json", + "n": 128, + "bytes": 1877, + "encode": 195403.41826130316, + "decode": 69143.89841197612 + }, + { + "backend": "codegen", + "n": 128, + "bytes": 725, + "encode": 331551.76232621307, + "decode": 239750.87106244764 + }, + { + "backend": "closures", + "n": 128, + "bytes": 725, + "encode": 182449.00273289852, + "decode": 137002.19475838103 + }, + { + "backend": "json", + "n": 256, + "bytes": 3829, + "encode": 99184.81910220427, + "decode": 33782.36501453661 + }, + { + "backend": "codegen", + "n": 256, + "bytes": 1525, + "encode": 164545.8591071086, + "decode": 116894.62028639753 + }, + { + "backend": "closures", + "n": 256, + "bytes": 1525, + "encode": 87830.26651769275, + "decode": 65902.0873733047 + } + ], + "fits": { + "decode": { + "json": { + "a": -2404.0899996710905, + "b": 111.85248114299786, + "r2": 0.8930535028981359 + }, + "codegen": { + "a": -125.0990502520126, + "b": 33.0521003317058, + "r2": 0.9958000734583876 + }, + "closures": { + "a": -203.08174219237844, + "b": 59.37292303230343, + "r2": 0.9991768155112374 + } + }, + "encode": { + "json": { + "a": 56.964465791606926, + "b": 39.12501385152096, + "r2": 0.9998404124776871 + }, + "codegen": { + "a": 56.75097987890585, + "b": 23.395830968891424, + "r2": 0.9997176480149081 + }, + "closures": { + "a": -139.7316572699885, + "b": 44.486234015111464, + "r2": 0.9991009801844328 + } + } + } +} diff --git a/bench/sweep.js b/bench/sweep.js new file mode 100644 index 0000000..6dbb7b7 --- /dev/null +++ b/bench/sweep.js @@ -0,0 +1,233 @@ +/** + * How the cost of a record grows with its width, per backend: the protocol of + * the article's e5 stand. Each (backend, width) point runs in a fresh process: + * 128 records of `n` fields (string, integer, boolean and number in turn) are + * encoded and decoded in rotation, and the number is the median of five rounds + * after a warm-up, so one object staying hot in the caches is not what is + * measured. + * + * Run: pnpm bench:sweep [--widths 4,8,16] [--standalone] [--json] [--save] + * + * Prints ns per operation and per field for JSON (the keyed object), generated + * code and closures (and the inline standalone module with `--standalone`), + * the closures/codegen ratio, the step from one width to the next where the + * widths are adjacent (127 → 128 is where a cliff would show), and the + * least-squares fit `a + b·n` of each column. `--save` writes + * bench/results/sweep.json. Manual only, not part of CI; the numbers are + * machine-specific. + */ + +const { execFileSync } = require('node:child_process'); +const fs = require('node:fs'); +const path = require('node:path'); +const vm = require('node:vm'); + +const RESULTS = path.join(__dirname, 'results', 'sweep.json'); +const WIDTHS = [4, 8, 16, 32, 48, 64, 96, 127, 128, 256]; +const BACKENDS = ['json', 'codegen', 'closures']; +const RECORDS = 128; +const TYPES = ['string', 'integer', 'boolean', 'number']; +const WARMUP = 2_000; +const ROUNDS = 5; +const ROUND_MS = 300; +const BATCH = 64; + +const args = process.argv.slice(2); +const option = (name) => { + const at = args.indexOf(name); + return at === -1 ? undefined : args[at + 1]; +}; + +const keyOf = (i) => `f${String(i).padStart(5, '0')}`; +const definitionOf = (n) => + Object.fromEntries(Array.from({ length: n }, (_, i) => [keyOf(i), TYPES[i % 4]])); +const recordOf = (n, r) => { + const out = {}; + for (let i = 0; i < n; i++) { + const type = i % 4; + let value; + if (type === 0) value = `v${r}-${i}`; + else if (type === 1) value = r * 1000 + i; + else if (type === 2) value = ((r + i) & 1) === 0; + else value = r + i / 8; + out[keyOf(i)] = value; + } + return out; +}; + +// Ops/sec of `fn` over the inputs in rotation: the clock is read once per +// batch, so it costs a fraction of a nanosecond per call. +const measure = (fn, inputs) => { + const count = inputs.length; + let k = 0; + const next = () => { + const input = inputs[k]; + k = k + 1 === count ? 0 : k + 1; + return input; + }; + for (let i = 0; i < WARMUP; i++) fn(next()); + const samples = []; + for (let round = 0; round < ROUNDS; round++) { + let iterations = 0; + const start = performance.now(); + while (performance.now() - start < ROUND_MS) { + for (let j = 0; j < BATCH; j++) fn(next()); + iterations += BATCH; + } + samples.push((iterations / (performance.now() - start)) * 1000); + } + samples.sort((a, b) => a - b); + return samples[Math.floor(ROUNDS / 2)]; +}; + +// One point, in this process: the backend's encode and decode of `n` fields. +const point = (backend, n) => { + const definition = definitionOf(n); + const values = Array.from({ length: RECORDS }, (_, r) => recordOf(n, r)); + let encode; + let decode; + let label = backend; + if (backend === 'json') { + encode = (value) => JSON.stringify(value); + decode = (text) => JSON.parse(text); + } else { + const metaschema = require('@alexify/metaschema'); + const { Schema } = require('../index.js'); + let schema = Schema.from(definition, { metaschema, codegen: backend === 'codegen' }); + if (backend === 'standalone') { + const { toSource } = require('../standalone.js'); + const module = { exports: {} }; + const source = `(function (module) {${toSource(schema, { runtime: 'inline' })}\n})`; + vm.runInThisContext(source)(module); + schema = module.exports; + } + if (schema.backend !== backend) throw new Error(`Expected ${backend}, got ${schema.backend}`); + label = schema.backend; + encode = (value) => schema.stringify(value); + decode = (text) => schema.parse(text); + } + const texts = values.map(encode); + const check = JSON.stringify(decode(texts[1])); + if (check !== JSON.stringify(values[1])) throw new Error(`${label} does not round-trip`); + return { + backend: label, + n, + bytes: texts[0].length, + encode: measure(encode, values), + decode: measure(decode, texts), + }; +}; + +const child = (backend, n) => { + const output = execFileSync(process.execPath, [__filename, '--child', backend, String(n)], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'inherit'], + }); + return JSON.parse(output); +}; + +// The least-squares line through (n, ns) and how well it fits. +const fit = (points) => { + const m = points.length; + let sx = 0; + let sy = 0; + let sxx = 0; + let sxy = 0; + for (const [x, y] of points) { + sx += x; + sy += y; + sxx += x * x; + sxy += x * y; + } + const b = (m * sxy - sx * sy) / (m * sxx - sx * sx); + const a = (sy - b * sx) / m; + const mean = sy / m; + let total = 0; + let residual = 0; + for (const [x, y] of points) { + total += (y - mean) ** 2; + residual += (y - (a + b * x)) ** 2; + } + return { a, b, r2: total === 0 ? 1 : 1 - residual / total }; +}; + +const ns = (ops) => 1e9 / ops; +const cell = (text, width = 26) => String(text).padStart(width); + +const report = (widths, backends, points) => { + const at = (backend, n) => points.find((p) => p.backend === backend && p.n === n); + const fits = {}; + for (const op of ['decode', 'encode']) { + console.log(`\n${op}: ns per operation (ns per field)`); + console.log( + cell('fields', 8) + backends.map((b) => cell(b)).join('') + cell('closures/codegen', 18), + ); + fits[op] = {}; + for (const n of widths) { + const row = backends.map((backend) => { + const p = at(backend, n); + return cell(`${ns(p[op]).toFixed(0)} (${(ns(p[op]) / n).toFixed(1)})`); + }); + const ratio = ns(at('closures', n)[op]) / ns(at('codegen', n)[op]); + console.log(cell(n, 8) + row.join('') + cell(`×${ratio.toFixed(2)}`, 18)); + } + const line = backends.map((backend) => { + const f = fit(widths.map((n) => [n, ns(at(backend, n)[op])])); + fits[op][backend] = f; + return cell(`${f.a.toFixed(0)} + ${f.b.toFixed(1)}·n R²=${f.r2.toFixed(3)}`, 26); + }); + console.log(`\n${cell('fit', 8)}${line.join('')}`); + for (let i = 1; i < widths.length; i++) { + if (widths[i] !== widths[i - 1] + 1) continue; + const steps = backends.map((backend) => { + const ratio = ns(at(backend, widths[i])[op]) / ns(at(backend, widths[i - 1])[op]); + return cell(`×${ratio.toFixed(2)}`); + }); + console.log(cell(`${widths[i - 1]}→${widths[i]}`, 8) + steps.join('')); + } + } + return fits; +}; + +const main = () => { + if (args[0] === '--child') { + process.stdout.write(JSON.stringify(point(args[1], Number(args[2])))); + return; + } + const widths = (option('--widths') ?? WIDTHS.join(',')) + .split(',') + .map(Number) + .filter((n) => Number.isInteger(n) && n > 0); + const backends = args.includes('--standalone') ? [...BACKENDS, 'standalone'] : BACKENDS; + const json = args.includes('--json'); + const points = []; + for (const n of widths) { + for (const backend of backends) { + const p = child(backend, n); + points.push(p); + if (!json) { + process.stderr.write( + `${backend.padEnd(11)} n=${String(n).padEnd(4)} ` + + `encode ${ns(p.encode).toFixed(0).padStart(7)} ns decode ${ns(p.decode).toFixed(0).padStart(7)} ns\n`, + ); + } + } + } + const fits = json ? null : report(widths, backends, points); + const snapshot = { + node: process.version, + date: new Date().toISOString(), + records: RECORDS, + widths, + backends, + points, + fits: fits ?? undefined, + }; + if (json) console.log(JSON.stringify(snapshot, null, 2)); + if (args.includes('--save')) { + fs.writeFileSync(RESULTS, `${JSON.stringify(snapshot, null, 2)}\n`); + if (!json) console.log(`\nSaved ${points.length} points to ${RESULTS}`); + } +}; + +main(); diff --git a/docs/guide/architecture.md b/docs/guide/architecture.md index d7119b2..449cb73 100644 --- a/docs/guide/architecture.md +++ b/docs/guide/architecture.md @@ -121,7 +121,8 @@ flowchart TB The generated code writes array and object literals, so every decoded object has the hidden class a literal of the schema's keys would, and the engine keeps the call sites monomorphic. The closures compose the same steps without compiling anything; a decoded object starts as a copy of a -template with the schema's keys, which keeps its hidden class stable too. The two backends are +template with the schema's keys at their defaults, which keeps its hidden class stable too and +makes a missing field free. The two backends are indistinguishable on the outside: the same wire, the same objects in the same key order and the same errors, which a property-based suite checks on random schemas, values and corrupted wires. Schema keys reach generated source only as `JSON.stringify(key)`, and generated code gets every diff --git a/docs/guide/benchmarks.md b/docs/guide/benchmarks.md index 23db221..087c792 100644 --- a/docs/guide/benchmarks.md +++ b/docs/guide/benchmarks.md @@ -49,8 +49,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 1,393,246 (×1.00) | 970,015 (×1.00) | 266 | 212 | 178 | -| **protoarray**
codegen · v2.0.0 | 2,066,250 (×1.48) | 1,723,648 (×1.78) | 135 | 134 | 110 | -| **protoarray**
closures, no eval · v2.0.0 | 1,500,853 (×1.08) | 1,065,375 (×1.10) | 135 | 134 | 110 | +| **protoarray**
codegen · v2.0.0 | 2,060,585 (×1.48) | 1,721,172 (×1.77) | 135 | 134 | 110 | +| **protoarray**
closures, no eval · v2.0.0 | 1,565,372 (×1.12) | 1,052,962 (×1.09) | 135 | 134 | 110 | | **protoarray**
standalone module, no eval · v2.0.0 | 2,038,604 (×1.46) | 1,606,745 (×1.66) | 135 | 134 | 110 | | **protoarray**
codegen + metaschema check · v2.0.0 | 1,148,641 (×0.82) | 1,723,478 (×1.78) | 135 | 134 | 110 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 1,398,695 (×1.00) | 1,472,331 (×1.52) | 115 | 127 | 117 | @@ -71,8 +71,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 1,466 (×1.00) | 891 (×1.00) | 271,215 | 33,793 | 29,133 | -| **protoarray**
codegen · v2.0.0 | 1,958 (×1.34) | 1,476 (×1.66) | 140,738 | 30,666 | 26,946 | -| **protoarray**
closures, no eval · v2.0.0 | 1,390 (×0.95) | 978 (×1.10) | 140,738 | 30,666 | 26,946 | +| **protoarray**
codegen · v2.0.0 | 1,978 (×1.35) | 1,482 (×1.66) | 140,738 | 30,666 | 26,946 | +| **protoarray**
closures, no eval · v2.0.0 | 1,424 (×0.97) | 965 (×1.08) | 140,738 | 30,666 | 26,946 | | **protoarray**
standalone module, no eval · v2.0.0 | 1,992 (×1.36) | 1,471 (×1.65) | 140,738 | 30,666 | 26,946 | | **protoarray**
codegen + metaschema check · v2.0.0 | 1,090 (×0.74) | 1,462 (×1.64) | 140,738 | 30,666 | 26,946 | | **protoarray**
columns layout · v2.0.0 | 2,059 (×1.40) | 1,535 (×1.72) | 138,758 | 27,412 | 21,053 | @@ -95,8 +95,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 6,800,381 (×1.00) | 4,562,955 (×1.00) | 33 | 53 | 36 | -| **protoarray**
codegen · v2.0.0 | 7,466,131 (×1.10) | 7,541,369 (×1.65) | 23 | 43 | 27 | -| **protoarray**
closures, no eval · v2.0.0 | 3,029,115 (×0.45) | 2,382,353 (×0.52) | 23 | 43 | 27 | +| **protoarray**
codegen · v2.0.0 | 7,410,265 (×1.09) | 7,549,897 (×1.65) | 23 | 43 | 27 | +| **protoarray**
closures, no eval · v2.0.0 | 3,233,774 (×0.48) | 4,067,092 (×0.89) | 23 | 43 | 27 | | **protoarray**
standalone module, no eval · v2.0.0 | 6,712,097 (×0.99) | 7,171,599 (×1.57) | 23 | 43 | 27 | | **protoarray**
codegen + metaschema check · v2.0.0 | 2,089,158 (×0.31) | 7,340,173 (×1.61) | 23 | 43 | 27 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 5,568,923 (×0.82) | 9,670,688 (×2.12) | 11 | 31 | 15 | @@ -117,8 +117,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 821,191 (×1.00) | 2,898,740 (×1.00) | 123 | 120 | 94 | -| **protoarray**
codegen · v2.0.0 | 4,717,208 (×5.74) | 4,844,478 (×1.67) | 36 | 46 | 36 | -| **protoarray**
closures, no eval · v2.0.0 | 3,514,161 (×4.28) | 3,118,202 (×1.08) | 36 | 46 | 36 | +| **protoarray**
codegen · v2.0.0 | 4,677,198 (×5.70) | 4,832,085 (×1.67) | 36 | 46 | 36 | +| **protoarray**
closures, no eval · v2.0.0 | 3,518,242 (×4.28) | 3,011,498 (×1.04) | 36 | 46 | 36 | | **protoarray**
standalone module, no eval · v2.0.0 | 4,666,737 (×5.68) | 4,696,261 (×1.62) | 36 | 46 | 36 | | **protoarray**
codegen + metaschema check · v2.0.0 | 2,803,310 (×3.41) | 4,851,985 (×1.67) | 36 | 46 | 36 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 3,837,319 (×4.67) | 12,143,747 (×4.19) | 13 | 33 | 17 | @@ -139,8 +139,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 1,557,669 (×1.00) | 1,335,778 (×1.00) | 816 | 266 | 229 | -| **protoarray**
codegen · v2.0.0 | 1,811,907 (×1.16) | 1,695,038 (×1.27) | 785 | 230 | 207 | -| **protoarray**
closures, no eval · v2.0.0 | 1,673,013 (×1.07) | 1,557,716 (×1.17) | 785 | 230 | 207 | +| **protoarray**
codegen · v2.0.0 | 1,824,978 (×1.17) | 1,730,970 (×1.30) | 785 | 230 | 207 | +| **protoarray**
closures, no eval · v2.0.0 | 1,719,394 (×1.10) | 1,589,045 (×1.19) | 785 | 230 | 207 | | **protoarray**
standalone module, no eval · v2.0.0 | 1,758,113 (×1.13) | 1,674,028 (×1.25) | 785 | 230 | 207 | | **protoarray**
codegen + metaschema check · v2.0.0 | 1,548,540 (×0.99) | 1,693,591 (×1.27) | 785 | 230 | 207 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 575,285 (×0.37) | 507,626 (×0.38) | 777 | 233 | 211 | @@ -161,8 +161,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 51,915 (×1.00) | 24,070 (×1.00) | 8,834 | 852 | 630 | -| **protoarray**
codegen · v2.0.0 | 66,308 (×1.28) | 49,455 (×2.05) | 4,332 | 685 | 501 | -| **protoarray**
closures, no eval · v2.0.0 | 48,093 (×0.93) | 26,769 (×1.11) | 4,332 | 685 | 501 | +| **protoarray**
codegen · v2.0.0 | 66,967 (×1.29) | 49,676 (×2.06) | 4,332 | 685 | 501 | +| **protoarray**
closures, no eval · v2.0.0 | 47,938 (×0.92) | 25,505 (×1.06) | 4,332 | 685 | 501 | | **protoarray**
standalone module, no eval · v2.0.0 | 65,648 (×1.26) | 47,442 (×1.97) | 4,332 | 685 | 501 | | **protoarray**
codegen + metaschema check · v2.0.0 | 33,902 (×0.65) | 49,890 (×2.07) | 4,332 | 685 | 501 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 24,172 (×0.47) | 37,015 (×1.54) | 3,691 | 778 | 609 | @@ -182,8 +182,8 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | Library | encode ops/s | decode ops/s | bytes | gzip | brotli | | --- | ---: | ---: | ---: | ---: | ---: | | JSON
stringify · parse · Node 24.14.1 | 323,242 (×1.00) | 483,381 (×1.00) | 764 | 406 | 392 | -| **protoarray**
codegen · v2.0.0 | 406,134 (×1.26) | 733,235 (×1.52) | 582 | 309 | 295 | -| **protoarray**
closures, no eval · v2.0.0 | 360,928 (×1.12) | 566,586 (×1.17) | 582 | 309 | 295 | +| **protoarray**
codegen · v2.0.0 | 396,687 (×1.23) | 729,668 (×1.51) | 582 | 309 | 295 | +| **protoarray**
closures, no eval · v2.0.0 | 369,242 (×1.14) | 558,993 (×1.16) | 582 | 309 | 295 | | **protoarray**
standalone module, no eval · v2.0.0 | 404,218 (×1.25) | 731,900 (×1.51) | 582 | 309 | 295 | | **protoarray**
codegen + metaschema check · v2.0.0 | 336,196 (×1.04) | 737,943 (×1.53) | 582 | 309 | 295 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 709,089 (×2.19) | 1,944,328 (×4.02) | 305 | 328 | 304 | @@ -199,7 +199,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | devalue
stringify · parse · v6.0.3 | 230,286 (×0.71) | 265,668 (×0.55) | 853 | 454 | 443 | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 2,243,379 (×6.94) | 479,670 (×0.99) | 764 | 406 | 392 | -_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-16. Medians; (×n) against JSON; bytes are the wire size of one payload (binary formats before any text encoding), gzip at level 6, brotli at quality 5. The spread of the rounds is in bench/results/compare.json._ +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. Medians; (×n) against JSON; bytes are the wire size of one payload (binary formats before any text encoding), gzip at level 6, brotli at quality 5. The spread of the rounds is in bench/results/compare.json._ @@ -352,8 +352,10 @@ built once; one built from a snapshot skips metaschema, and a standalone module change on one machine. The generated code is ahead of JSON on most shapes, ×1.2 to ×2.2. It is behind on a record of 70 flags, which travels as bitmask words JSON does not have to compute, on decoding bytes, where the base64 has to be canonical, and level with it on encoding a sparse -record of 70 fields. The closures stay near JSON on most shapes and fall behind on sparse -records, where they fill a decoded object one property at a time. +record of 70 fields. The closures stay near JSON on most shapes and behind generated code by the +cost of one keyed store per present field, ×1.6 to ×1.9 on decode: `pnpm bench:sweep` prints that +cost from 4 to 256 fields, each backend in a fresh process (about 60 ns a field against 30, with +no step at 128 fields; see [Runtimes and CSP](/guide/runtimes#what-the-fallback-costs)). diff --git a/docs/guide/runtimes.md b/docs/guide/runtimes.md index be52cbd..6957eca 100644 --- a/docs/guide/runtimes.md +++ b/docs/guide/runtimes.md @@ -51,18 +51,33 @@ reports nothing. `{ codegen: false }` is also the way to compare the two. ## What the fallback costs Closures cannot write object literals, so decoded objects start as a copy of a template with the -schema's keys (which keeps their hidden class stable) and are filled one property at a time. On a -typical record the closures backend stays within 10% of `JSON.stringify` and `JSON.parse` of the -keyed object, a little ahead on one record and a little behind on a list; the codegen backend is -×1.5 and ×1.8. The cost of closures grows with the number of fields, about 40 ns per field on -decode against a few in generated code, so a sparse record of 70 fields decodes about 15 times -slower than with codegen. The payload is identical either way; in a fresh process, loading the -package takes about 4 ms and building a first schema about 2 ms with either backend, a few -microseconds once warm (`pnpm bench` prints a cold-start table). +schema's keys at their defaults (which keeps their hidden class stable, and makes a missing field +free) and each field that is present is stored on its own, through a store whose key changes with +every field. That store is what the fallback costs. On decode it is about 60 ns a field against +30 in generated code and 47 for `JSON.parse` of the keyed object; on encode about 45 against 24 +and 39, as the table below measures (`pnpm bench:sweep`: records of string, integer, boolean and +number fields, 128 of them in rotation, each backend in a fresh process; Node.js 24, Apple M3 +Max). So on a typical record the closures backend stays near `JSON.stringify` and `JSON.parse` of +the keyed object, a little ahead on one record and a little behind on a list, where the codegen +backend is ×1.5 and ×1.8. The gap to generated code is structural: the closures are the same code +for every schema, so the engine's inline caches meet every schema's keys at the same sites, while +generated code has sites of its own per schema. Both backends grow linearly with the number of +fields, with no step at 128 fields, where V8 hands an object parsed from JSON over in dictionary +mode (which is why the template is not parsed from JSON). The payload is identical either way; in +a fresh process, loading the package takes about 4 ms and building a first schema about 2 ms with +either backend, a few microseconds once warm (`pnpm bench` prints a cold-start table). + +| Fields | `JSON.parse`, keyed | parse, codegen | parse, closures | `JSON.stringify`, keyed | stringify, codegen | stringify, closures | +| -----: | ------------------: | -------------: | --------------: | ----------------------: | -----------------: | ------------------: | +| 4 | 298 ns | 154 ns | 227 ns | 210 ns | 151 ns | 181 ns | +| 16 | 850 ns | 474 ns | 756 ns | 695 ns | 443 ns | 596 ns | +| 64 | 3.0 µs | 1.8 µs | 3.5 µs | 2.6 µs | 1.6 µs | 2.7 µs | +| 128 | 14.5 µs | 4.2 µs | 7.3 µs | 5.1 µs | 3.1 µs | 5.5 µs | +| 256 | 30.1 µs | 8.6 µs | 15.3 µs | 10.1 µs | 6.1 µs | 11.5 µs | A very wide record makes a large function: about 200 to 500 bytes of source a field, 4 ms to build -for 1,000 fields and 16 ms for 3,000. It still decodes two to ten times faster than closures do, -so the automatic choice stays codegen however wide a record is. +for 1,000 fields and 16 ms for 3,000. It still decodes about twice as fast as closures do, so the +automatic choice stays codegen however wide a record is. ## Without code generation, at its speed diff --git a/docs/guide/standalone.md b/docs/guide/standalone.md index edc87cf..7195fda 100644 --- a/docs/guide/standalone.md +++ b/docs/guide/standalone.md @@ -90,9 +90,9 @@ generation where it is allowed. module's own, so tell them by `name` and `code`, not by `instanceof`. Either way the module compiles no source text to load, and its schemas run at the speed of the codegen -backend: on the benchmark's nested record, ×1.5 faster to `stringify` and ×1.85 faster to `parse` +backend: on the benchmark's nested record, ×1.5 faster to `stringify` and ×1.8 faster to `parse` than `JSON` of the keyed object, where the closures backend a runtime without code generation -would otherwise use is about as fast as `JSON` (×1.5 and ×2 slower than the module). +would otherwise use is about as fast as `JSON` (×1.4 and ×2 slower than the module). ## What it does not write diff --git a/docs/public/benchmarks/coldstart.svg b/docs/public/benchmarks/coldstart.svg index 472cafa..082f5b4 100644 --- a/docs/public/benchmarks/coldstart.svg +++ b/docs/public/benchmarks/coldstart.svg @@ -156,7 +156,7 @@ avsc@5.7.9 15.7 ms in total load 11.6 ms · build 3.8 ms first encode 0.11 ms · first decode 0.17 ms -@alexify/protoarray@2.0.0 (72941bf) +@alexify/protoarray@2.0.0 (5a482c5) protoarray closures, no eval @@ -176,7 +176,7 @@ protobufjs@8.8.1 16.5 ms in total load 11.7 ms · build 4.5 ms first encode 0.14 ms · first decode 0.16 ms -@alexify/protoarray@2.0.0 (72941bf) +@alexify/protoarray@2.0.0 (5a482c5) protoarray codegen @@ -202,5 +202,5 @@ first encode 0.80 ms · first decode 0.06 ms 26 ms -Apple M3 Max · 16 cores · Node v24.14.1 · 2026-10-16 +Apple M3 Max · 16 cores · Node v24.14.1 · 2026-10-18 diff --git a/docs/public/benchmarks/compare-numeric-bytes.svg b/docs/public/benchmarks/compare-numeric-bytes.svg index 6a5ed5e..1947eb8 100644 --- a/docs/public/benchmarks/compare-numeric-bytes.svg +++ b/docs/public/benchmarks/compare-numeric-bytes.svg @@ -125,7 +125,7 @@ node:v8, Node 24.14.1, writes bytes protoarray · codegen 582 bytes 76% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -134,7 +134,7 @@ node:v8, Node 24.14.1, writes bytes protoarray · closures, no eval 582 bytes 76% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -194,5 +194,5 @@ devalue@6.0.3, writes JSON text 853 B Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-numeric.svg b/docs/public/benchmarks/compare-numeric.svg index 922898f..66c5937 100644 --- a/docs/public/benchmarks/compare-numeric.svg +++ b/docs/public/benchmarks/compare-numeric.svg @@ -123,25 +123,25 @@ cbor-x@1.6.6, writes bytes 269k -protoarray · codegen -261,365 ops/s -3.83 µs per payload -×1.35 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text -protoarray -codegen - -261k - protoarray · standalone module, no eval 260,402 ops/s 3.84 µs per payload ×1.34 the speed of JSON @alexify/protoarray@2.0.0 (72941bf), writes JSON text -protoarray -standalone module, no eval - -260k +protoarray +standalone module, no eval + +260k + +protoarray · codegen +256,979 ops/s +3.89 µs per payload +×1.33 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text +protoarray +codegen + +257k protoarray · codegen + metaschema check 230,970 ops/s @@ -154,14 +154,14 @@ cbor-x@1.6.6, writes bytes 231k protoarray · closures, no eval -220,478 ops/s -4.54 µs per payload -×1.14 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +222,361 ops/s +4.5 µs per payload +×1.15 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval - -220k + +222k JSON · stringify · parse 193,708 ops/s @@ -213,5 +213,5 @@ superjson@2.2.6, writes JSON text 73.5k Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-packets-bytes.svg b/docs/public/benchmarks/compare-packets-bytes.svg index c25a9a5..7590452 100644 --- a/docs/public/benchmarks/compare-packets-bytes.svg +++ b/docs/public/benchmarks/compare-packets-bytes.svg @@ -80,7 +80,7 @@ msgpackr@2.1.0, writes bytes protoarray · codegen 4,332 bytes 49% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -89,7 +89,7 @@ msgpackr@2.1.0, writes bytes protoarray · closures, no eval 4,332 bytes 49% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -185,5 +185,5 @@ bson@7.3.3, writes bytes 9.5 KB Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-packets.svg b/docs/public/benchmarks/compare-packets.svg index bbad193..82d8a9e 100644 --- a/docs/public/benchmarks/compare-packets.svg +++ b/docs/public/benchmarks/compare-packets.svg @@ -44,14 +44,14 @@ protoarray · codegen -28,327 ops/s -35.3 µs per payload -×1.72 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +28,520 ops/s +35.1 µs per payload +×1.73 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen - -28.3k + +28.5k protoarray · standalone module, no eval 27,540 ops/s @@ -84,14 +84,14 @@ msgpackr@2.1.0, writes bytes 20.2k protoarray · closures, no eval -17,197 ops/s -58.1 µs per payload -×1.05 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +16,648 ops/s +60.1 µs per payload +×1.01 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval - -17.2k + +16.6k JSON · stringify · parse 16,445 ops/s @@ -203,5 +203,5 @@ superjson@2.2.6, writes JSON text 3,615 Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-records-bytes.svg b/docs/public/benchmarks/compare-records-bytes.svg index d94a7d8..c41b193 100644 --- a/docs/public/benchmarks/compare-records-bytes.svg +++ b/docs/public/benchmarks/compare-records-bytes.svg @@ -111,7 +111,7 @@ cbor-x@1.6.6, writes bytes protoarray · codegen 140,738 bytes 52% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -120,7 +120,7 @@ cbor-x@1.6.6, writes bytes protoarray · closures, no eval 140,738 bytes 52% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -216,5 +216,5 @@ devalue@6.0.3, writes JSON text 304.2 KB Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-records.svg b/docs/public/benchmarks/compare-records.svg index 7e0c2fb..2d901e8 100644 --- a/docs/public/benchmarks/compare-records.svg +++ b/docs/public/benchmarks/compare-records.svg @@ -83,25 +83,25 @@ cbor-x@1.6.6, writes bytes 865 +protoarray · codegen +847 ops/s +1.18 ms per payload +×1.53 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text +protoarray +codegen + +847 + protoarray · standalone module, no eval 846 ops/s 1.18 ms per payload ×1.53 the speed of JSON @alexify/protoarray@2.0.0 (72941bf), writes JSON text -protoarray -standalone module, no eval - -846 - -protoarray · codegen -842 ops/s -1.19 ms per payload -×1.52 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text -protoarray -codegen - -842 +protoarray +standalone module, no eval + +846 avsc · Type.forSchema 786 ops/s @@ -144,14 +144,14 @@ fast-json-stringify@7.1.0, writes JSON text 577 protoarray · closures, no eval -574 ops/s +575 ops/s 1.74 ms per payload ×1.04 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval - -574 + +575 JSON · stringify · parse 554 ops/s @@ -233,5 +233,5 @@ superjson@2.2.6, writes JSON text 94 Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-single-bytes.svg b/docs/public/benchmarks/compare-single-bytes.svg index 38090db..0ab487c 100644 --- a/docs/public/benchmarks/compare-single-bytes.svg +++ b/docs/public/benchmarks/compare-single-bytes.svg @@ -84,7 +84,7 @@ protobufjs@8.8.1, writes bytes protoarray · codegen 135 bytes 51% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -93,7 +93,7 @@ protobufjs@8.8.1, writes bytes protoarray · closures, no eval 135 bytes 51% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -198,5 +198,5 @@ devalue@6.0.3, writes JSON text 311 B Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-single.svg b/docs/public/benchmarks/compare-single.svg index 07d394a..44ff39b 100644 --- a/docs/public/benchmarks/compare-single.svg +++ b/docs/public/benchmarks/compare-single.svg @@ -42,14 +42,14 @@ protoarray · codegen -939,732 ops/s -1.06 µs per payload +937,824 ops/s +1.07 µs per payload ×1.64 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen - -940k + +938k msgpackr · records, shared structures 916,227 ops/s @@ -102,14 +102,14 @@ protobufjs@8.8.1, writes bytes 689k protoarray · closures, no eval -623,082 ops/s -1.6 µs per payload -×1.09 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +629,514 ops/s +1.59 µs per payload +×1.10 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval - -623k + +630k fast-json-stringify · compiled stringify, JSON.parse 615,075 ops/s @@ -211,5 +211,5 @@ superjson@2.2.6, writes JSON text 109k Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-sparse-bytes.svg b/docs/public/benchmarks/compare-sparse-bytes.svg index d383dd0..c0fa705 100644 --- a/docs/public/benchmarks/compare-sparse-bytes.svg +++ b/docs/public/benchmarks/compare-sparse-bytes.svg @@ -91,7 +91,7 @@ msgpackr@2.1.0, writes bytes protoarray · codegen 23 bytes 70% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -100,7 +100,7 @@ msgpackr@2.1.0, writes bytes protoarray · closures, no eval 23 bytes 70% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -196,5 +196,5 @@ superjson@2.2.6, writes JSON text 42 B Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-sparse.svg b/docs/public/benchmarks/compare-sparse.svg index bc65766..8b3ee82 100644 --- a/docs/public/benchmarks/compare-sparse.svg +++ b/docs/public/benchmarks/compare-sparse.svg @@ -54,14 +54,14 @@ msgpackr@2.1.0, writes bytes 4M protoarray · codegen -3,751,781 ops/s +3,739,715 ops/s 267 ns per payload ×1.37 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen - -3.75M + +3.74M fast-json-stringify · compiled stringify, JSON.parse 3,556,803 ops/s @@ -132,35 +132,35 @@ bson@7.3.3, writes bytes 1.82M +protoarray · closures, no eval +1,801,438 ops/s +555 ns per payload +×0.66 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text +protoarray +closures, no eval + +1.8M + cbor-x · records 1,651,536 ops/s 605 ns per payload ×0.60 the speed of JSON cbor-x@1.6.6, writes bytes -cbor-x -records - -1.65M +cbor-x +records + +1.65M protoarray · codegen + metaschema check 1,626,285 ops/s 615 ns per payload ×0.60 the speed of JSON @alexify/protoarray@2.0.0 (72941bf), writes JSON text -protoarray -codegen + metaschema check - -1.63M - -protoarray · closures, no eval -1,333,542 ops/s -750 ns per payload -×0.49 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text protoarray -closures, no eval - -1.33M +codegen + metaschema check + +1.63M avsc · Type.forSchema 1,039,354 ops/s @@ -213,5 +213,5 @@ superjson@2.2.6, writes JSON text 639k Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-strings-bytes.svg b/docs/public/benchmarks/compare-strings-bytes.svg index 36e77de..d6555c9 100644 --- a/docs/public/benchmarks/compare-strings-bytes.svg +++ b/docs/public/benchmarks/compare-strings-bytes.svg @@ -80,7 +80,7 @@ protobufjs@8.8.1, writes bytes protoarray · codegen 785 bytes 96% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -89,7 +89,7 @@ protobufjs@8.8.1, writes bytes protoarray · closures, no eval 785 bytes 96% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -194,5 +194,5 @@ node:v8, Node 24.14.1, writes bytes 863 B Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-strings.svg b/docs/public/benchmarks/compare-strings.svg index 67476a1..5556596 100644 --- a/docs/public/benchmarks/compare-strings.svg +++ b/docs/public/benchmarks/compare-strings.svg @@ -42,14 +42,14 @@ protoarray · codegen -875,763 ops/s -1.14 µs per payload -×1.22 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +888,366 ops/s +1.13 µs per payload +×1.24 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen - -876k + +888k protoarray · standalone module, no eval 857,520 ops/s @@ -61,25 +61,25 @@ <path class="bar secondary" d="M217,141H606.6017921834435A4,4 0 0 1 610.6017921834435,145V151A4,4 0 0 1 606.6017921834435,155H217Z"/> <text class="value" x="616.6017921834435" y="154">858k</text> </g> -<g><title>protoarray · codegen + metaschema check -808,910 ops/s -1.24 µs per payload -×1.12 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +protoarray · closures, no eval +825,826 ops/s +1.21 µs per payload +×1.15 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray -codegen + metaschema check - -809k +closures, no eval + +826k -protoarray · closures, no eval -806,654 ops/s +<g><title>protoarray · codegen + metaschema check +808,910 ops/s 1.24 µs per payload ×1.12 the speed of JSON @alexify/protoarray@2.0.0 (72941bf), writes JSON text protoarray -closures, no eval - -807k +codegen + metaschema check + +809k JSON · stringify · parse 719,108 ops/s @@ -211,5 +211,5 @@ msgpackr@2.1.0, writes bytes 170k Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-typed-bytes.svg b/docs/public/benchmarks/compare-typed-bytes.svg index 600abff..6791814 100644 --- a/docs/public/benchmarks/compare-typed-bytes.svg +++ b/docs/public/benchmarks/compare-typed-bytes.svg @@ -84,7 +84,7 @@ msgpackr@2.1.0, writes bytes protoarray · codegen 36 bytes 29% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen @@ -93,7 +93,7 @@ msgpackr@2.1.0, writes bytes protoarray · closures, no eval 36 bytes 29% of the JSON bytes -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval @@ -198,5 +198,5 @@ superjson@2.2.6, writes JSON text 179 B Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/compare-typed.svg b/docs/public/benchmarks/compare-typed.svg index 4cbb2b7..ed8a57a 100644 --- a/docs/public/benchmarks/compare-typed.svg +++ b/docs/public/benchmarks/compare-typed.svg @@ -54,14 +54,14 @@ protobufjs@8.8.1, writes bytes 2.92M protoarray · codegen -2,389,998 ops/s -418 ns per payload -×3.73 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +2,376,690 ops/s +421 ns per payload +×3.71 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray codegen - -2.39M + +2.38M protoarray · standalone module, no eval 2,340,726 ops/s @@ -114,14 +114,14 @@ avsc@5.7.9, writes bytes 1.68M protoarray · closures, no eval -1,652,181 ops/s -605 ns per payload -×2.58 the speed of JSON -@alexify/protoarray@2.0.0 (72941bf), writes JSON text +1,622,603 ops/s +616 ns per payload +×2.54 the speed of JSON +@alexify/protoarray@2.0.0 (5a482c5), writes JSON text protoarray closures, no eval - -1.65M + +1.62M msgpackr · pack · unpack 1,386,649 ops/s @@ -213,5 +213,5 @@ superjson@2.2.6, writes JSON text 213k Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · -2026-10-16 +2026-10-18 diff --git a/docs/public/benchmarks/engine-parse.svg b/docs/public/benchmarks/engine-parse.svg index a18478d..eae014b 100644 --- a/docs/public/benchmarks/engine-parse.svg +++ b/docs/public/benchmarks/engine-parse.svg @@ -1,4 +1,4 @@ - + diff --git a/docs/public/benchmarks/engine-stringify.svg b/docs/public/benchmarks/engine-stringify.svg index 8e37992..9fbbbc6 100644 --- a/docs/public/benchmarks/engine-stringify.svg +++ b/docs/public/benchmarks/engine-stringify.svg @@ -1,4 +1,4 @@ - + diff --git a/docs/public/benchmarks/layouts.svg b/docs/public/benchmarks/layouts.svg index 9e12529..198d87e 100644 --- a/docs/public/benchmarks/layouts.svg +++ b/docs/public/benchmarks/layouts.svg @@ -46,54 +46,54 @@ 3,000 JSON · keyed objects -encode 1,713 lists/s · 584 µs -decode 1,247 lists/s · 802 µs +encode 1,665 lists/s · 601 µs +decode 1,255 lists/s · 797 µs JSON keyed objects - -1,713 - -1,247 + +1,665 + +1,255 rows · an array of arrays, the default -encode 2,247 lists/s · 445 µs -decode 2,002 lists/s · 500 µs +encode 2,229 lists/s · 449 µs +decode 1,939 lists/s · 516 µs rows an array of arrays, the default - -2,247 - -2,002 + +2,229 + +1,939 flat · one array, a fixed stride -encode 2,113 lists/s · 473 µs -decode 1,979 lists/s · 505 µs +encode 2,073 lists/s · 482 µs +decode 1,978 lists/s · 506 µs flat one array, a fixed stride - -2,113 - -1,979 + +2,073 + +1,978 columns · an array per field -encode 2,217 lists/s · 451 µs -decode 1,980 lists/s · 505 µs +encode 2,187 lists/s · 457 µs +decode 1,964 lists/s · 509 µs columns an array per field - -2,217 - -1,980 + +2,187 + +1,964 columns, delta + intern · ids as differences, cities and tags interned -encode 2,347 lists/s · 426 µs -decode 2,126 lists/s · 470 µs +encode 2,350 lists/s · 426 µs +decode 2,075 lists/s · 482 µs columns, delta + intern ids as differences, cities and tags interned - -2,347 - -2,126 + +2,350 + +2,075 -pnpm bench, bench/baseline.json · Node v24.14.1 · 2026-10-16 +pnpm bench, bench/baseline.json · Node v24.14.1 · 2026-10-18 diff --git a/package.json b/package.json index 0656017..4773075 100644 --- a/package.json +++ b/package.json @@ -87,6 +87,7 @@ "size:compare": "node scripts/size-compare.js", "bench": "node bench/bench.js", "bench:compare": "node bench/compare.js", + "bench:sweep": "node bench/sweep.js", "bench:releases": "node bench/releases.js", "bench:report": "node bench/report.js", "docs:dev": "vitepress dev docs", diff --git a/src/closures.js b/src/closures.js index fa989cd..000deba 100644 --- a/src/closures.js +++ b/src/closures.js @@ -149,28 +149,34 @@ const build = (plan, options) => { const lead = node.tag === undefined ? 0 : 1; const width = size + lead; const end = tail + lead; + // Identity fields have no encoder: their value is their wire form. const encoders = node.fields.map((field) => field.node.kind === 'scalar' ? scalarEncoder(field.node, '') : encoder(field.node), ); - // The wire form of one field, or null for one that is missing. - const wireOf = (object, i) => { - const key = keys[i]; - const value = key === null ? null : object[key]; - if (value == null) return null; - const encode = encoders[i]; - if (encode === null) return value; - try { - return encode(value); - } catch (error) { - throw nest(error, paths[i]); - } - }; + // The bit of each field in its sparse segment, computed once. + const bits = Int32Array.from(keys, (key, i) => 1 << (i % WORD)); if (!sparse) { return (object, t) => { if (typeof object !== 'object' || Array.isArray(object)) refuse('object', ''); const out = new Array(width); if (lead === 1) out[0] = t; - for (let i = 0; i < size; i++) out[i + lead] = wireOf(object, i); + for (let i = 0; i < size; i++) { + const key = keys[i]; + const value = key === null ? null : object[key]; + let wire = null; + if (value != null) { + const encode = encoders[i]; + if (encode === null) wire = value; + else { + try { + wire = encode(value); + } catch (error) { + throw nest(error, paths[i]); + } + } + } + out[i + lead] = wire; + } let length = width; while (length > end && out[length - 1] === null) length--; if (length < width) out.length = length; @@ -179,26 +185,39 @@ const build = (plan, options) => { } // One [mask, ...present values] segment per 31 fields: `start` is where // the current mask goes, `keep` where the last segment with something in - // it ends (the first one is always kept). + // it ends (the first one is always kept), `boundary` the field that opens + // the next segment. return (object) => { if (typeof object !== 'object' || Array.isArray(object)) refuse('object', ''); const out = [0]; let mask = 0; let start = 0; let keep = 0; + let boundary = WORD; for (let i = 0; i < size; i++) { - if (i > 0 && i % WORD === 0) { + if (i === boundary) { out[start] = mask; if (start === 0 || mask !== 0) keep = out.length; start = out.length; out.push(0); mask = 0; + boundary += WORD; } - const wire = wireOf(object, i); - if (wire !== null) { - mask |= 1 << (i % WORD); - out.push(wire); + const key = keys[i]; + const value = key === null ? null : object[key]; + if (value == null) continue; + const encode = encoders[i]; + let wire = value; + if (encode !== null) { + try { + wire = encode(value); + } catch (error) { + throw nest(error, paths[i]); + } + if (wire === null) continue; } + mask |= bits[i]; + out.push(wire); } out[start] = mask; if (start === 0 || mask !== 0) keep = out.length; @@ -302,73 +321,103 @@ const build = (plan, options) => { }; }; - // Objects start as a copy of a literal-shaped template so every decoded - // record has the same hidden class, as an object literal would. Scalars are - // handled in the loop; only nested values go through a call. + // What the loop of a record does with one field's wire value: store it as + // it is, check it first, check and convert it in one call, convert it + // unchecked, call the decoder of a nested value, or skip it (a field only + // the writer has). The kind is decided once per field, not once per value. + const AS_IS = 0; + const CHECKED = 1; + const CONVERTED = 2; + const UNCHECKED = 3; + const NESTED = 4; + const SKIPPED = 5; + const kindOf = (field) => { + if (field.key === null) return SKIPPED; + const { node } = field; + if (node.kind !== 'scalar') return NESTED; + // The order of codegen's `convert`: a converted type checks and converts + // in one call under validation, an identity type is checked on its own. + if (validate.decode && node.checkedUnpack !== undefined) return CONVERTED; + if (node.unpack !== null) return UNCHECKED; + return validate.decode && node.check !== null ? CHECKED : AS_IS; + }; + const readerOf = (field, kind) => { + const { node } = field; + if (kind === NESTED) return decoder(node); + if (kind === CHECKED) return node.check; + if (kind === CONVERTED) return node.checkedUnpack; + return kind === UNCHECKED ? node.unpack : null; + }; + + // Objects start as a copy of a template with every key of the schema in its + // order and every field at its default, so each decoded record has the same + // hidden class, as an object literal would, and a field that is missing + // costs no store. The loop reads each field by its kind, with one direct + // call for a nested value; codegen writes the same steps out per field. const structDecoder = (node) => { const { sparse, tag } = node; - // A field only the writer has (Schema.resolve) has no key: it is not read. const keys = node.fields.map((field) => field.key); const size = keys.length; const lead = tag === undefined ? 0 : 1; - const fields = node.fields.map((field) => { - const scalar = field.node.kind === 'scalar'; - return { - path: field.key === null ? null : step(field.key), - required: field.required && validate.decode, - def: field.def, - check: scalar && validate.decode ? field.node.check : null, - code: field.node.code, - expected: field.node.expected, - checked: scalar ? converter(field.node) : null, - convert: scalar ? field.node.unpack : null, - decode: scalar ? null : slotDecoder(field.node, field.required, field.def), - }; - }); + const kinds = Uint8Array.from(node.fields, kindOf); + const readers = node.fields.map((field, i) => readerOf(field, kinds[i])); + const paths = keys.map((key) => (key === null ? null : step(key))); + const required = node.fields.map((field) => field.required && validate.decode); + const codes = node.fields.map((field) => field.node.code); + const expecteds = node.fields.map((field) => field.node.expected); + const bits = Int32Array.from(keys, (key, i) => 1 << (i % WORD)); // A branch's template has every key of the branch, its discriminator too; // a resolved record's has the reader's keys in its order, with the // reader's fields the writer does not have at their defaults. const order = node.order === undefined ? keys : node.order; - const template = literal(lead === 0 ? order : tag.keys, 'null'); + const template = literal(lead === 0 ? order : tag.keys, null); + for (const field of node.fields) if (field.key !== null) put(template, field.key, field.def); if (node.extras !== undefined) for (const extra of node.extras) template[extra.key] = extra.def; return (array, strict, t) => { if (!Array.isArray(array)) fail('structure', '', 'array'); const { length } = array; const object = { ...template }; if (lead === 1) object[tag.key] = tag.values[t]; + if (!sparse && strict && length > size + lead) fail('extra', ''); let mask = 0; let next = 0; - if (!sparse && strict && length > size + lead) fail('extra', ''); + // The field that opens the next sparse segment: the first mask is + // required, a later segment may be missing. + let boundary = 0; for (let i = 0; i < size; i++) { - const key = keys[i]; - const field = fields[i]; - let value = null; + let value; if (!sparse) value = i + lead < length ? array[i + lead] : null; else { - // The first mask is required; a later segment may be missing. - if (i % WORD === 0) { + if (i === boundary) { mask = i === 0 || next < length ? array[next++] : 0; if (typeof mask !== 'number' || (mask | 0) !== mask || mask < 0) { fail('structure', '', 'mask'); } + boundary += WORD; } - if ((mask & (1 << (i % WORD))) !== 0) value = array[next++]; + value = (mask & bits[i]) !== 0 ? array[next++] : null; + } + const kind = kinds[i]; + if (kind === SKIPPED) continue; + if (value == null) { + if (required[i]) fail('required', paths[i]); + continue; } - if (key === null) continue; - if (field.decode !== null) { - object[key] = field.decode(value, field.path, strict); - } else if (value == null) { - if (field.required) fail('required', field.path); - object[key] = field.def; - } else if (field.checked !== null) { - const out = field.checked(value); - if (out === undefined) fail(field.code, field.path, field.expected); - object[key] = out; - } else { - if (field.check !== null && !field.check(value)) { - fail(field.code, field.path, field.expected); + if (kind === AS_IS) object[keys[i]] = value; + else if (kind === CONVERTED) { + const out = readers[i](value); + if (out === undefined) fail(codes[i], paths[i], expecteds[i]); + object[keys[i]] = out; + } else if (kind === CHECKED) { + if (!readers[i](value)) fail(codes[i], paths[i], expecteds[i]); + object[keys[i]] = value; + } else if (kind === UNCHECKED) object[keys[i]] = readers[i](value); + else { + try { + object[keys[i]] = readers[i](value, strict); + } catch (error) { + throw nest(error, paths[i]); } - object[key] = field.convert === null ? value : field.convert(value); } } if (sparse && strict && next !== length) fail('extra', ''); diff --git a/src/types.js b/src/types.js index 21ad92c..02d9739 100644 --- a/src/types.js +++ b/src/types.js @@ -48,10 +48,6 @@ const kernel = (base64) => { return Number.isFinite(at) ? at : undefined; }; - // An object parsed from a literal has the hidden class a literal would have. - const literal = (keys, value) => - JSON.parse(`{${keys.map((key) => `${JSON.stringify(key)}:${value}`).join(',')}}`); - // A view from another realm fails instanceof; its tag tells it apart from the // other views, and isView from an object that merely claims the tag. const isBytes = (v) => @@ -98,6 +94,16 @@ const kernel = (base64) => { } else object[key] = value; }; + // The template a decoded object starts from, every key at `value`: a copy + // of it has one hidden class per schema, as an object literal would, as + // long as the template keeps fast properties. `Object.fromEntries` does, up + // to V8's limit of 1020 descriptors (`__proto__` becomes an own key, as in + // JSON.parse); an object parsed from JSON is handed over in dictionary mode + // from 128 keys on, and one filled by keyed assignment from about 18, and a + // copy of a dictionary-mode object costs a transition per key every time + // (17 µs for 128 keys against 0.2 µs). + const literal = (keys, value) => Object.fromEntries(keys.map((key) => [key, value])); + // The path of a dictionary's value under `path`, on an error: its key comes // from the payload, so it is written as itself only when it is a plain name, // and otherwise as a quoted JSON string cut to 64 characters, never with a @@ -247,14 +253,21 @@ const kernel = (base64) => { // decoder reads either form and ignores the bits it has no name for. const flags = (names) => { const size = names.length; - const template = literal(names, 'false'); + const template = literal(names, false); + // The template holds every name at false: only the set bits are stored. const unpack = (v) => { const single = typeof v === 'number'; if (!single && !Array.isArray(v)) return v; const object = { ...template }; - for (let i = 0; i < size; i++) { - const word = single ? (i < WORD ? v : 0) : v[(i / WORD) | 0]; - object[names[i]] = typeof word === 'number' && (word & (1 << (i % WORD))) !== 0; + const words = single ? 1 : Math.min(v.length, Math.ceil(size / WORD)); + for (let w = 0; w < words; w++) { + const word = single ? v : v[w]; + if (typeof word !== 'number' || word === 0) continue; + const base = w * WORD; + const stop = Math.min(base + WORD, size); + for (let i = base; i < stop; i++) { + if ((word & (1 << (i - base))) !== 0) object[names[i]] = true; + } } return object; }; @@ -479,7 +492,7 @@ const kernel = (base64) => { // false, the writer's names the reader lacks are dropped. flags: (writer, reader) => { const { names } = reader; - const template = literal(names, 'false'); + const template = literal(names, false); const own = new Set(writer.names); const shared = names.filter((name) => own.has(name)); const project = (object) => { diff --git a/tests/unit/codec.test.js b/tests/unit/codec.test.js index 286a193..dee4795 100644 --- a/tests/unit/codec.test.js +++ b/tests/unit/codec.test.js @@ -35,6 +35,65 @@ for (const [backend, options] of BACKENDS) { assert.deepStrictEqual(schema.decode([null, 1], null), decoded); }); + // On V8 an object parsed from JSON with 128 keys or more is in dictionary + // mode, and so is one filled by keyed assignment from about 18: the template + // a decoded record starts from must be neither, on either side of the line. + const WIDE_TYPES = ['string', 'integer', 'boolean', 'number']; + const wideKeys = (width) => Array.from({ length: width }, (_, i) => `f${i}`); + const wideValue = (keys) => + Object.fromEntries( + keys.map((key, i) => { + const type = i % 4; + return [key, type === 0 ? `s${i}` : type === 1 ? i : type === 2 ? i % 3 === 0 : i / 4]; + }), + ); + for (const width of [127, 128, 129]) { + test(`Codec (${backend}): a record of ${width} fields decodes with every key in order`, () => { + const keys = wideKeys(width); + // Every eighth field is optional and left out: it decodes to null. + const definition = Object.fromEntries( + keys.map((key, i) => [key, `${i % 8 === 7 ? '?' : ''}${WIDE_TYPES[i % 4]}`]), + ); + const full = wideValue(keys); + const value = { ...full }; + const expected = { ...full }; + for (let i = 7; i < width; i += 8) { + delete value[keys[i]]; + expected[keys[i]] = null; + } + const dense = define(definition, options); + const decoded = dense.decode(dense.encode(value)); + assert.deepStrictEqual(Object.keys(decoded), keys); + assert.deepStrictEqual(decoded, expected); + assert.deepStrictEqual(dense.parse(dense.stringify(value)), expected); + // A sparse record of the same width, every field optional, three present. + const optional = Object.fromEntries(keys.map((key, i) => [key, `?${WIDE_TYPES[i % 4]}`])); + const sparse = define(optional, { ...options, sparse: true }); + const few = { f0: 's0', f64: 's64', [keys[width - 1]]: full[keys[width - 1]] }; + const back = sparse.decode(sparse.encode(few)); + assert.deepStrictEqual(Object.keys(back), keys); + assert.deepStrictEqual(back, { + ...Object.fromEntries(keys.map((key) => [key, null])), + ...few, + }); + }); + } + + test(`Codec (${backend}): flags past 127 names decode as an object of every name`, () => { + const names = Array.from({ length: 130 }, (_, i) => `n${i}`); + const schema = define({ perms: { flags: names } }, options); + const none = Object.fromEntries(names.map((name) => [name, false])); + const wire = schema.encode({ perms: { n0: true, n31: true, n129: true } }); + assert.deepStrictEqual(wire, [[1, 1, 0, 0, 32]]); + const decoded = schema.decode(wire).perms; + assert.deepStrictEqual(Object.keys(decoded), names); + assert.deepStrictEqual(decoded, { ...none, n0: true, n31: true, n129: true }); + // One integer is the first word; the words a payload stops short of are false. + assert.deepStrictEqual(schema.decode([3]).perms, { ...none, n0: true, n1: true }); + assert.deepStrictEqual(schema.decode([[0, 2]]).perms, { ...none, n32: true }); + assert.deepStrictEqual(schema.decode([[0]]).perms, none); + }); + test(`Codec (${backend}): extra trailing elements are ignored unless strict`, () => { const schema = define({ id: 'number' }, options); assert.deepStrictEqual(schema.decode([1, 'future']), { id: 1 }); From bc4fc4ee6c13cf841671327969fd565c7304c4c8 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Tue, 20 Oct 2026 22:51:35 +0300 Subject: [PATCH 02/17] feat(evolve): compatibility check of schema versions, baselines and the CLI commands snapshot and check The ./evolve subpath: compatible(previous, next, { strict }) walks two versions by position (fields, tuple elements, enum values, flag names, a union's values, the item or value of a collection, references once per pair of entities) and reports every place where the newer one reads an older payload differently, with the rule it breaks (inserted, moved, removed, renamed, required, retired, type, optional, appended, wire, flat, default), its severity and the canonical forms of the two nodes. baseline(schemas) writes schemas as JSON (snapshots, the options that shape the wire, the options of the entities reached, the canonical form and the id) and restore(baseline) builds them again, refusing one that does not rebuild to its canonical form; register() installs Schema.compatible. protoarray snapshot writes the baseline of a module, protoarray check compares a module with one by export name (--only, --strict, --json; exit code 1 when incompatible or unusable, 2 for the command line). The CLI is a table of commands now; its usage names every command. Docs: "Checking it in CI" in the evolution guide, the API page, the errors table, README and CHANGELOG. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 21 + README.md | 8 +- bin/protoarray.js | 223 ++++++-- docs/api/exports.md | 37 +- docs/guide/errors.md | 6 +- docs/guide/evolution.md | 130 +++++ docs/guide/standalone.md | 3 + evolve.d.ts | 126 +++++ evolve.js | 1 + package.json | 8 +- scripts/size.js | 1 + src/evolve.js | 564 +++++++++++++++++++++ tests/fixtures/evolution/v1.js | 11 + tests/fixtures/evolution/v2.js | 11 + tests/fixtures/evolution/v3.js | 12 + tests/fixtures/evolution/v4.js | 22 + tests/types/evolve.test-d.ts | 41 ++ tests/unit/cli.test.js | 167 +++++- tests/unit/evolve.test.js | 837 +++++++++++++++++++++++++++++++ tests/unit/export-parity.test.js | 2 + 20 files changed, 2176 insertions(+), 55 deletions(-) create mode 100644 evolve.d.ts create mode 100644 evolve.js create mode 100644 src/evolve.js create mode 100644 tests/fixtures/evolution/v1.js create mode 100644 tests/fixtures/evolution/v2.js create mode 100644 tests/fixtures/evolution/v3.js create mode 100644 tests/fixtures/evolution/v4.js create mode 100644 tests/types/evolve.test-d.ts create mode 100644 tests/unit/evolve.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c2be09..636bba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **`@alexify/protoarray/evolve`**: `compatible(previous, next, { strict? })` checks two versions + of a schema by position, the way the evolution rules read, and reports every place where the + newer one reads an older payload differently: `{ ok, problems }`, each problem with its `path`, + the `rule` it breaks (`inserted`, `moved`, `removed`, `renamed`, `required`, `retired`, `type`, + `optional`, `appended`, `wire`, `flat`, `default`), its `severity` (`error`, `warning`, `info`) + and the canonical forms of the two nodes; `ok` is false on an error, with `strict` on a warning + too. Fields, tuple elements, enum values, flag names and a union's values compare by position; + references by the entity they reach, once per pair; a key that changes where neither key is + anywhere else is a rename, any other shift is an error. `baseline(schemas)` writes schemas as + JSON (their snapshots, the options that shape their wire, their canonical forms and ids) and + `restore(baseline)` builds them again, refusing one that does not rebuild to its canonical form + (`ERR_INVALID_SNAPSHOT`); `register()` installs `Schema.compatible`. The core does not carry it. +- **`protoarray snapshot [--only A,B] [--out ]`** writes the baseline of the + schemas a module exports, and **`protoarray check --against [--only + A,B] [--strict] [--json]`** compares the module with it, by export name: one line per problem + and a summary (`--json`: `{ ok, schemas }`), exit code 1 when incompatible or when the module + or the baseline cannot be used, 2 for a wrong command line. The usage of the command names + every command now, and `--out` refuses to write over the module it reads (the message says so). + ## [2.0.0] - 2026-10-16 2.0 hands the definition of a schema to `@alexify/metaschema` and keeps the wire to protoarray. diff --git a/README.md b/README.md index a76c7e2..046a445 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,8 @@ const User = Schema.from(await (await fetch('/schemas/user')).json()); validates the values with it, protoarray carries them, and both read a reference, an optional field or a stored entity's id the same way. - **Evolves safely.** Append fields at the end and old and new versions keep understanding each - other; a fingerprint (`schema.id`) tells versions apart in a handshake. For changes that break + other; a fingerprint (`schema.id`) tells versions apart in a handshake, and `protoarray check` + holds a module to the rules in CI, against the baseline that shipped. For changes that break the rules, `resolve` reads and writes another version by name, Avro-style, and a registry wraps payloads as `[id, payload]`. - **Typed.** A schema takes its TypeScript types from the metaschema schema it is built from; @@ -86,7 +87,7 @@ const User = Schema.from(await (await fetch('/schemas/user')).json()); it recognises a metaschema schema by its brand and reads its snapshot, and takes the metaschema module as an option (`{ metaschema }`) only to read a raw definition. CommonJS with ESM named imports, no build step, one package for Node.js 18+ and browsers; the registry, resolve, - streaming and standalone extras are subpaths of 1.8–2.5 KB each. Generated code where the + streaming, standalone and evolve extras are subpaths of 1.8–3 KB each. Generated code where the runtime allows it, and under a strict CSP or on edge runtimes either a closures fallback or a standalone module generated at build time. @@ -122,7 +123,7 @@ protoarray alone. | Streaming | `encodeEach`/`decodeEach`; NDJSON, `TransformStream`s and the items of a large array read as they arrive (`@alexify/protoarray/stream`) | | Standalone modules | `protoarray compile` writes the generated code of your schemas into a module, for a strict CSP and edge runtimes, with no metaschema at runtime (`@alexify/protoarray/standalone`) | | Validation | the wire on decode by default: `DecodeError` with codes, paths like `orders[1].sku` and the schema name; the values before encode with metaschema's `check` and `parse` | -| Evolution | append-only positions; `{ strict: true }` rejects elements beyond the schema; `schema.id` or your own `fingerprint` tells versions apart | +| Evolution | append-only positions; `{ strict: true }` rejects elements beyond the schema; `schema.id` or your own `fingerprint` tells versions apart; `compatible(previous, next)` and `protoarray check` hold a module to the rules in CI (`@alexify/protoarray/evolve`) | See the [guide](https://protoarray.vercel.app/guide/schema) for every form, and [Migrating from 1.x](https://protoarray.vercel.app/guide/migrating-from-1) for the upgrade: the @@ -524,6 +525,7 @@ const { resolve } = require('@alexify/protoarray/resolve'); const { SchemaRegistry } = require('@alexify/protoarray/registry'); const { parseLines, parseItems /* … */ } = require('@alexify/protoarray/stream'); const { toSource } = require('@alexify/protoarray/standalone'); +const { compatible, baseline, restore } = require('@alexify/protoarray/evolve'); ``` ```ts diff --git a/bin/protoarray.js b/bin/protoarray.js index 0276b9a..4d0bd3e 100755 --- a/bin/protoarray.js +++ b/bin/protoarray.js @@ -2,20 +2,31 @@ /** * protoarray compile [--out ] [--format cjs|esm] [--runtime import|inline] * [--only ] + * protoarray snapshot [--only ] [--out ] + * protoarray check --against [--only ] [--strict] [--json] * - * Loads a module (CommonJS or ES), takes the schemas it exports and writes - * the module @alexify/protoarray/standalone's toSource makes of them: their - * generated code, for runtimes that forbid compiling source text. A schema is - * a protoarray Schema or a metaschema Schema (built with Schema.from); a - * metaschema Model stands for its entities, each under its name. A module - * whose export is one schema gives a module with that one as its default - * export; otherwise the schemas keep their export names. `--only` keeps the - * named ones. Without `--out` the source goes to stdout; `--out` makes the - * directory it names and never writes over the module it reads. The generated - * module needs no metaschema. + * Every command loads a module (CommonJS or ES) and takes the schemas it + * exports. A schema is a protoarray Schema or a metaschema Schema (built with + * Schema.from); a metaschema Model stands for its entities, each under its + * name. A module whose export is one schema stands for that one (as `default` + * where a name is needed); otherwise the schemas keep their export names. + * `--only` keeps the named ones. Without `--out` the output goes to stdout; + * `--out` makes the directory it names and never writes over the module it + * reads. * - * Exit codes: 0 done (`--help` included), 1 the module or its schemas could - * not be compiled, 2 the command line is wrong. + * `compile` writes the module @alexify/protoarray/standalone's toSource makes + * of the schemas: their generated code, for runtimes that forbid compiling + * source text. The generated module needs no metaschema. + * + * `snapshot` writes the baseline of @alexify/protoarray/evolve: the schemas as + * JSON, to keep with the code. `check` compares the schemas of the module with + * the ones of a baseline, by export name, and reports every place where the + * module reads an older payload differently (one line per problem, or + * `{ ok, schemas }` with `--json`); `--strict` fails on warnings too. + * + * Exit codes: 0 done (`--help` included; `check`: compatible), 1 the module, + * its schemas or the baseline could not be used (`check`: incompatible too), + * 2 the command line is wrong. */ const fs = require('node:fs'); @@ -23,10 +34,7 @@ const path = require('node:path'); const { pathToFileURL } = require('node:url'); const { Schema } = require('../index.js'); const { toSource } = require('../standalone.js'); - -const USAGE = - 'Usage: protoarray compile [--out ] [--format cjs|esm] ' + - '[--runtime import|inline] [--only ]'; +const { compatible, baseline, restore } = require('../evolve.js'); const BRAND = Symbol.for('alexify.protoarray.brand'); const FOREIGN = Symbol.for('alexify.metaschema.brand'); @@ -34,29 +42,11 @@ const isObject = (value) => value !== null && typeof value === 'object'; const isSchema = (value) => isObject(value) && value[BRAND] === true; const isForeign = (value, brand) => isObject(value) && value[FOREIGN] === brand; -const VALUED = new Set(['--out', '--format', '--runtime', '--only']); - // A mistake on the command line, and what the CLI itself refuses of a module // that loads: told from an error inside the module, whose stack is reported. class UsageError extends Error {} class Refusal extends Error {} -const parseArgs = (argv) => { - if (argv.includes('--help') || argv.includes('-h')) return { help: true }; - const [command, ...rest] = argv; - if (command !== 'compile') throw new UsageError(USAGE); - const options = {}; - const files = []; - for (let i = 0; i < rest.length; i++) { - const arg = rest[i]; - if (!arg.startsWith('--')) files.push(arg); - else if (VALUED.has(arg) && i + 1 < rest.length) options[arg.slice(2)] = rest[++i]; - else throw new UsageError(`Unknown or incomplete option ${arg}\n${USAGE}`); - } - if (files.length !== 1) throw new UsageError(USAGE); - return { file: files[0], ...options }; -}; - // CommonJS first; an ES module through import(), which require() cannot load // on older Node.js versions. const load = async (file) => { @@ -98,6 +88,8 @@ const exported = (loaded) => { return out; }; +const names = (only) => only.split(',').map((name) => name.trim()); + // The schemas a module exports: itself when it is one, the entities of a // Model it is, its schemas by name otherwise (an ES module's default export // when it has nothing else). @@ -105,9 +97,9 @@ const schemasOf = (loaded, only) => { const single = schemaOf(loaded); if (single !== null) return single; const all = isForeign(loaded, 'Model') ? { ...Schema.fromModel(loaded) } : exported(loaded); - const names = Object.keys(all); - if (names.length === 1 && names[0] === 'default') return all.default; - const chosen = only === undefined ? names : only.split(',').map((name) => name.trim()); + const keys = Object.keys(all); + if (keys.length === 1 && keys[0] === 'default') return all.default; + const chosen = only === undefined ? keys : names(only); const out = {}; for (const name of chosen) { if (!Object.hasOwn(all, name)) throw new Refusal(`The module exports no schema "${name}"`); @@ -117,6 +109,153 @@ const schemasOf = (loaded, only) => { return out; }; +// The schemas by name, a lone one as `default`. +const record = (schemas) => (isSchema(schemas) ? { default: schemas } : schemas); + +// What a command writes: to stdout, or to the file `--out` names, which is +// never the module it reads. +const write = (args, text) => { + if (args.out === undefined) { + process.stdout.write(text); + return; + } + fs.mkdirSync(path.dirname(path.resolve(args.out)), { recursive: true }); + fs.writeFileSync(args.out, text); +}; + +const compile = async (args) => { + const schemas = schemasOf(await load(args.file), args.only); + write(args, toSource(schemas, { runtime: args.runtime, format: args.format })); +}; + +const snapshot = async (args) => { + const schemas = record(schemasOf(await load(args.file), args.only)); + write(args, `${JSON.stringify(baseline(schemas), null, 2)}\n`); +}; + +// The baseline file as JSON; what is wrong with it is told without a stack. +const readBaseline = (file) => { + let text; + try { + text = fs.readFileSync(file, 'utf8'); + } catch (error) { + throw new Refusal(`Cannot read the baseline ${file}: ${error.message}`); + } + try { + return JSON.parse(text); + } catch (error) { + throw new Refusal(`The baseline ${file} is not JSON: ${error.message}`); + } +}; + +const SEVERITIES = ['error', 'warning', 'info']; + +// One line per problem, then a summary. +const formatted = (result) => { + const lines = []; + const counts = { error: 0, warning: 0, info: 0 }; + let total = 0; + for (const name of Object.keys(result.schemas)) { + total += 1; + for (const problem of result.schemas[name].problems) { + counts[problem.severity] += 1; + const { path: at } = problem; + const where = at === '' ? name : at[0] === '[' ? name + at : `${name}.${at}`; + lines.push(`${problem.severity.padEnd(7)} ${where}: ${problem.message} (${problem.rule})`); + } + } + const found = SEVERITIES.filter((severity) => counts[severity] > 0).map((severity) => { + const count = counts[severity]; + return `${count} ${severity}${count !== 1 && severity !== 'info' ? 's' : ''}`; + }); + const checked = `${total} schema${total === 1 ? '' : 's'} checked`; + const problems = found.length === 0 ? 'no problems' : found.join(', '); + lines.push(`${checked}: ${problems}; ${result.ok ? 'compatible' : 'incompatible'}`); + return `${lines.join('\n')}\n`; +}; + +// The schemas of the module against the ones of the baseline, paired by +// export name: one the module no longer exports is a warning, a new one is +// noted. `--only` names the pairs to check, on either side. +const check = async (args, usage) => { + if (args.against === undefined) { + throw new UsageError(`check needs --against \n${usage}`); + } + const strict = args.strict === true; + const current = record(schemasOf(await load(args.file))); + const previous = restore(readBaseline(args.against)); + const all = new Set([...Object.keys(previous), ...Object.keys(current)]); + const only = args.only === undefined ? [...all] : names(args.only); + const result = { ok: true, schemas: {} }; + for (const name of only) { + if (!all.has(name)) { + throw new Refusal(`Neither the module nor the baseline has a schema "${name}"`); + } + let entry; + if (!Object.hasOwn(current, name)) { + const message = `The module no longer exports "${name}"`; + const problem = { path: '', rule: 'removed', severity: 'warning', message }; + entry = { ok: !strict, problems: [problem] }; + } else if (!Object.hasOwn(previous, name)) { + const message = `"${name}" is not in the baseline: nothing to compare`; + entry = { ok: true, problems: [{ path: '', rule: 'appended', severity: 'info', message }] }; + } else entry = compatible(previous[name], current[name], { strict }); + result.schemas[name] = entry; + if (!entry.ok) result.ok = false; + } + process.stdout.write(args.json === true ? `${JSON.stringify(result)}\n` : formatted(result)); + if (!result.ok) process.exitCode = 1; +}; + +// The commands: their usage, the options that take a value, the flags, and +// what they do. +const COMMANDS = { + compile: { + usage: + 'protoarray compile [--out ] [--format cjs|esm] ' + + '[--runtime import|inline] [--only ]', + valued: ['out', 'format', 'runtime', 'only'], + flags: [], + run: compile, + }, + snapshot: { + usage: 'protoarray snapshot [--only ] [--out ]', + valued: ['only', 'out'], + flags: [], + run: snapshot, + }, + check: { + usage: 'protoarray check --against [--only ] [--strict] [--json]', + valued: ['against', 'only'], + flags: ['strict', 'json'], + run: check, + }, +}; + +const USAGE = `Usage: ${Object.values(COMMANDS) + .map((command) => command.usage) + .join('\n ')}`; + +const parseArgs = (argv) => { + if (argv.includes('--help') || argv.includes('-h')) return { help: true }; + const [name, ...rest] = argv; + if (!Object.hasOwn(COMMANDS, name)) throw new UsageError(USAGE); + const command = COMMANDS[name]; + const usage = `Usage: ${command.usage}`; + const options = {}; + const files = []; + for (let i = 0; i < rest.length; i++) { + const arg = rest[i]; + const option = arg.slice(2); + if (!arg.startsWith('--')) files.push(arg); + else if (command.flags.includes(option)) options[option] = true; + else if (command.valued.includes(option) && i + 1 < rest.length) options[option] = rest[++i]; + else throw new UsageError(`Unknown or incomplete option ${arg}\n${usage}`); + } + if (files.length !== 1) throw new UsageError(usage); + return { command, usage, file: files[0], ...options }; +}; + const main = async () => { const args = parseArgs(process.argv.slice(2)); if (args.help) { @@ -124,15 +263,9 @@ const main = async () => { return; } if (args.out !== undefined && path.resolve(args.out) === path.resolve(args.file)) { - throw new UsageError(`--out names the module it compiles: ${args.out}`); - } - const schemas = schemasOf(await load(args.file), args.only); - const source = toSource(schemas, { runtime: args.runtime, format: args.format }); - if (args.out === undefined) process.stdout.write(source); - else { - fs.mkdirSync(path.dirname(path.resolve(args.out)), { recursive: true }); - fs.writeFileSync(args.out, source); + throw new UsageError(`--out names the module it reads: ${args.out}`); } + await args.command.run(args, args.usage); }; // What went wrong, enough to act on: an error of protoarray or metaschema with diff --git a/docs/api/exports.md b/docs/api/exports.md index 9238806..91a284f 100644 --- a/docs/api/exports.md +++ b/docs/api/exports.md @@ -8,7 +8,7 @@ const { } = require('@alexify/protoarray'); ``` -Four subpaths carry what the core does not need, so a program that does not use them does not +Five subpaths carry what the core does not need, so a program that does not use them does not bundle them: | Subpath | Exports | Page | @@ -17,9 +17,12 @@ bundle them: | `@alexify/protoarray/registry` | `SchemaRegistry` | [Migrations and the Registry](/guide/resolve#the-registry) | | `@alexify/protoarray/stream` | `stringifyLines`, `parseLines`, `stringifyStream`, `parseStream`, `stringifyItems`, `parseItems`, `register` | [Streaming](/guide/streaming) | | `@alexify/protoarray/standalone` | `toSource`, `runtime`, `register` | [Standalone Modules](/guide/standalone) | +| `@alexify/protoarray/evolve` | `compatible`, `baseline`, `restore`, `register` | [Schema Evolution](/guide/evolution#checking-it-in-ci) | The package also installs a `protoarray` command: `protoarray compile ` (see -[Standalone Modules](/guide/standalone#protoarray-compile)). None of it requires +[Standalone Modules](/guide/standalone#protoarray-compile)), `protoarray snapshot ` and +`protoarray check --against ` (see +[Checking it in CI](/guide/evolution#checking-it-in-ci)). None of it requires `@alexify/metaschema`; schemas are built from what metaschema writes (see [Defining a Schema](/guide/schema)). @@ -142,6 +145,35 @@ interface ResolveOptions { A schema that reads what `writer` wrote as `reader` objects and writes them back as `writer` does (see [Migrations](/guide/resolve)). +## `compatible(previous, next, options?)` + +```ts +import { compatible, baseline, restore, register } from '@alexify/protoarray/evolve'; + +function compatible(previous: AnySchema, next: AnySchema, options?: CompatibleOptions): Compatibility; +function baseline(schemas: Readonly>): Baseline; +function restore(baseline: Baseline): Readonly>; +function register(): void; // installs Schema.compatible + +interface CompatibleOptions { strict?: boolean } // a warning fails the check too +interface Compatibility { ok: boolean; problems: readonly Problem[] } +interface Problem { + path: string; // as the errors spell it: 'address.city', 'orders[].sku', '' for the root + rule: Rule; // 'inserted' | 'moved' | 'removed' | 'renamed' | 'required' | 'retired' | 'type' + // | 'optional' | 'appended' | 'wire' | 'flat' | 'default' + severity: Severity; // 'error' | 'warning' | 'info' + message: string; + previous?: string; // the canonical form of the node in `previous`, where there is one + next?: string; // the same in `next` +} +interface Baseline { format: 1; snapshots: readonly Snapshot[]; schemas: Readonly> } +interface BaselineEntry { snapshot: number; entity: string; options: BaselineOptions; canonical: string; id: string } +``` + +Whether `next` reads what `previous` writes by position, with every rule a change breaks; +`baseline` writes schemas as JSON and `restore` builds them again (see +[Checking it in CI](/guide/evolution#checking-it-in-ci)). + ## Errors ```ts @@ -161,6 +193,7 @@ class DecodeError extends TypeError { code: DecodeErrorCode; path: string; schem | `AnySchema` | any `Schema` instance, what the functions and the registry accept | | `Collection` | the type of `schema.list` | | `SchemaOptions`, `ModelOptions`, `EntityOptions`, `DecodeOptions`, `Backend`, `Layout`, `WireType`, `CompiledCodec`, `Codec` | options and their values | +| `Compatibility`, `Problem`, `Rule`, `Severity`, `CompatibleOptions`, `Baseline`, `BaselineEntry`, `BaselineOptions` | what `@alexify/protoarray/evolve` gives and takes | | `SchemaErrorCode`, `EncodeErrorCode`, `DecodeErrorCode` | the `code` unions | See [TypeScript](/guide/typescript) for how a schema takes its types from metaschema. diff --git a/docs/guide/errors.md b/docs/guide/errors.md index e1e10ab..5472c76 100644 --- a/docs/guide/errors.md +++ b/docs/guide/errors.md @@ -125,7 +125,7 @@ Messages say where: `Unknown type "float" at "price"`. | Code | Cause | | --- | --- | -| `ERR_INVALID_SNAPSHOT` | not a snapshot of format 1, no entity of the name asked for, an entity that is not an object, a field without a type, a schema nested too deeply, a collection without its key or element, a tuple without elements, `decimals` outside 0–15, `aliases` that are not a list of keys, references that lead back to themselves with no record, a snapshot that breaks the reader unchecked (`cause`) | +| `ERR_INVALID_SNAPSHOT` | not a snapshot of format 1, no entity of the name asked for, an entity that is not an object, a field without a type, a schema nested too deeply, a collection without its key or element, a tuple without elements, `decimals` outside 0–15, `aliases` that are not a list of keys, references that lead back to themselves with no record, a snapshot that breaks the reader unchecked (`cause`), a baseline of `@alexify/protoarray/evolve` that is not one of format 1 or does not rebuild to the canonical form it was written with | | `ERR_UNKNOWN_TYPE` | an unknown type, a custom type with a check of its own and no wire type in `types` | | `ERR_INVALID_ENUM`, `ERR_INVALID_FLAGS` | an empty list, duplicates, wrong value types, `__proto__` (neither has a size limit) | | `ERR_INVALID_UNION` | a union without a `discriminator`, an empty union, a branch that is not a struct with the discriminator as a required enum without a default, the same value in two branches | @@ -134,9 +134,9 @@ Messages say where: `Unknown type "float" at "price"`. | `ERR_INTEGER_KEY` | a key or a flag name like `'1'`, which JavaScript would order first | | `ERR_INVALID_DEFINITION` | a struct without fields, an empty key | | `ERR_INVALID_OPTIONS` | an unknown option, a wrong type, a path that names no place of the schema or a place of another kind (`sparse` on a non-struct, a layout on anything but a collection of records, `delta` on a string, `intern` on anything but a string), `layout: 'rows'` on anything but a collection, `sparse` on a union's branch, an option of one schema given to `Schema.fromModel` instead of `entities`, an `entities` key that names no entity of the snapshot or an entity's own `entity`, `entities` or `metaschema`, a snapshot of several entities without `entity`, `entity` other than a metaschema Schema's own, a `fingerprint` that gives no non-empty string, an `aliases` path of `resolve` that names no field of the reader, a `maxLength` that is not a positive integer or `Infinity` | -| `ERR_SCHEMA_EXPECTED` | `Schema.from` got neither a metaschema `Schema` or `Model`, a snapshot, a protoarray schema nor a definition with `{ metaschema }`; `encode(schema, …)` and friends got something that is not a `Schema` | +| `ERR_SCHEMA_EXPECTED` | `Schema.from` got neither a metaschema `Schema` or `Model`, a snapshot, a protoarray schema nor a definition with `{ metaschema }`; `encode(schema, …)` and friends got something that is not a `Schema`; `compatible` got a resolved schema, `baseline` a schema not read from a snapshot (`schema.list`, a resolved one) | | `ERR_CODEGEN_UNAVAILABLE` | `{ codegen: true }` where `new Function` is blocked | | `ERR_COMPILED_MISMATCH` | a generated module's codec is of another schema than the snapshot given with it, or the module was generated by another version of protoarray | -| `ERR_INCOMPATIBLE` | `resolve` cannot read the writer as the reader: a narrowing type change, a required reader field the writer lacks, a union split between branches, a chain of references too long for the stack | +| `ERR_INCOMPATIBLE` | `resolve` cannot read the writer as the reader: a narrowing type change, a required reader field the writer lacks, a union split between branches, a chain of references too long for the stack (for `compatible` too) | | `ERR_ID_COLLISION` | a registry got two schemas of different signatures with one id: the fingerprint is too narrow for it | | `ERR_UNKNOWN_SCHEMA` | a registry's `encode` got `as` an id it does not know | diff --git a/docs/guide/evolution.md b/docs/guide/evolution.md index 90fbe62..e395034 100644 --- a/docs/guide/evolution.md +++ b/docs/guide/evolution.md @@ -89,6 +89,136 @@ By default, elements beyond the schema are ignored. If a payload must match the pass `{ strict: true }` to `decode` or `parse`: extra elements, values or columns throw `DecodeError` with code `extra`. +## Checking it in CI + +The rules above are kept by hand, and a definition that breaks one still builds: a version that +moves a position decodes the payloads of the other into the wrong keys without a word. +`compatible`, from the `@alexify/protoarray/evolve` subpath, reads two versions the way a +reviewer would, position by position, and names every rule a change breaks. It is a tool for +development and CI, kept out of the core and apart from [`resolve`](/guide/resolve), which reads +one version as another at runtime: + +```js +const { compatible } = require('@alexify/protoarray/evolve'); + +const V3 = Schema.from( + { + id: 'number', + name: 'string', + email: '?string', + score: { type: 'number', default: 0 }, + 'role?': { enum: ['admin', 'user'] }, + }, + { metaschema }, +); +compatible(V2, V3); // { ok: true, problems: [] } — an optional field appended + +const V4 = Schema.from( + { + id: 'integer', + fullName: 'string', + org: 'string', + email: '?string', + score: { type: 'number', default: 0 }, + 'role?': { enum: ['admin', 'user', 'guest'] }, + }, + { metaschema }, +); +compatible(V3, V4); +// { +// ok: false, +// problems: [ +// { path: 'id', rule: 'type', severity: 'error', message: 'Type changed: number → integer', previous: 'number', next: 'integer' }, +// { path: 'fullName', rule: 'renamed', severity: 'info', message: 'Field "name" is now "fullName"', previous: 'string', next: 'string' }, +// { path: 'org', rule: 'inserted', severity: 'error', message: 'Field "org" inserted at position 2: "email", "score" and "role" move further', next: 'string' }, +// { path: 'role', rule: 'appended', severity: 'warning', message: 'Enum value "guest" appended: deploy decoders before encoders', previous: 'enum(admin|user)', next: 'enum(admin|user|guest)' }, +// ], +// } +compatible(V3, V4, { strict: true }).ok; // false on a warning too +``` + +`ok` is false when a problem is an error, and with `{ strict: true }` when one is a warning. Each +problem names its place as the errors do (`address.city`, `orders[].sku`, `scores{}`, `pair[1]`, +`''` for the root), the rule it breaks, how bad it is, and the canonical forms of the two nodes +where there are two. What changes nothing on the wire is reported for information, so a review +sees it. `register()` from the same subpath installs it as `Schema.compatible(previous, next)`. + +| `rule` | Severity | When | Rule above | +| --- | --- | --- | --- | +| `inserted` | error | a field, a tuple element, an enum value, a flag name or a union value inserted before one that keeps its place, which moves | 1, Unions | +| `moved` | error | one that is elsewhere in the new version, not further (a swap, a reorder) | 1 | +| `removed` | error | one the new version does not have, or fewer positions | 1, 3 | +| `renamed` | info | a key, a value or an entity under another name at the same place, the old one nowhere else | 1 | +| `required` | error | a field or an element appended as required without a default; an optional one now required | 2 | +| `retired` | info | a slot that now takes anything (`?any`, `json`); **warning** when it takes only `null`, which a validating decoder holds older writers to | 3 | +| `type` | error | another type or form: `integer` for `number`, an enum for a string, `decimals` changed, a record for an id, a Set for an array, a map keyed otherwise, a union on another discriminator | 4, 7 | +| `optional` | info | a required slot now optional | 4 | +| `appended` | warning | an enum value, a flag name or a union value appended: deploy decoders before encoders | 4, 5 | +| `wire` | error | `sparse`, a `layout`, `delta` or `intern` switched at a place | 6 | +| `flat` | error | the record of a `flat` collection has another number of fields, optional or not | Nested records | +| `default` | warning | a default added, removed or changed: an empty slot in older payloads decodes differently | 2 | + +Fields, tuple elements, enum values, flag names and a union's values compare by position, and +references by the entity they reach, once per pair of entities (a recursion included). A stored +kind's id that moves because a field was appended to the definition (rule 8) is an `inserted` +problem whose message says where to declare it. The telling of a rename from a shift is +conservative: a key that changes at a position where neither the old nor the new key is anywhere +else is a rename; any other shift of a key is an error, since the wire cannot say which one it +was. + +### A baseline + +A CI job needs the version that shipped. `baseline` writes schemas as JSON: each one's snapshot +(once, however many schemas share it), the entity it is of, the options that shape its wire +(`sparse`, `layout`, `intern`, `delta`, `references`, `types`, the options of the entities it +reaches, and the name of a built-in fingerprint; a function of your own cannot travel, so it is +written as `null` and the id is recorded, not checked), and its canonical form and id as +checksums. `restore` builds them again and refuses a baseline that does not rebuild to the +canonical form it was written with (`ERR_INVALID_SNAPSHOT`: write it again): + +```js +const { baseline, restore } = require('@alexify/protoarray/evolve'); + +fs.writeFileSync('schemas.baseline.json', JSON.stringify(baseline({ User: V3 }), null, 2)); +// later, in another process +const shipped = restore(JSON.parse(fs.readFileSync('schemas.baseline.json', 'utf8'))); +compatible(shipped.User, V4); +``` + +The `protoarray` command does both over the schemas a module exports (a protoarray or a +metaschema `Schema`, the entities of a `Model`, as [`protoarray compile`](/guide/standalone#protoarray-compile) +takes them), paired by export name: + +```sh +protoarray snapshot src/schemas.js --out schemas.baseline.json # when a version ships +protoarray check src/schemas.js --against schemas.baseline.json # on every change +``` + +`check` prints one line per problem and a summary, and exits with 1 when any schema is +incompatible, or when the module or the baseline cannot be used (a baseline that does not +rebuild fails the job too), with 2 for a wrong command line: + +``` +error User.id: Type changed: number → integer (type) +info User.fullName: Field "name" is now "fullName" (renamed) +error User.org: Field "org" inserted at position 2: "email", "score" and "role" move further (inserted) +warning User.role: Enum value "guest" appended: deploy decoders before encoders (appended) +1 schema checked: 2 errors, 1 warning, 1 info; incompatible +``` + +A schema the baseline has and the module no longer exports is a `removed` warning; one the +module exports and the baseline lacks is noted (`appended`, info). `--only A,B` names the pairs +to check, `--strict` fails on warnings too and `--json` writes `{ ok, schemas: { Name: { ok, +problems } } }` instead of the lines. A workflow step holds every pull request to the rules: + +```yaml +- run: npx protoarray check src/schemas.js --against schemas.baseline.json --strict +``` + +Write the baseline again with `protoarray snapshot` when a version ships, and commit it with the +code: a change that has to break the rules then shows as a diff of the baseline in the same pull +request, alongside the [`resolve`](/guide/resolve) that bridges it. + ## Telling versions apart `schema.id` is a 32-bit FNV-1a fingerprint of the schema's signature: its canonical form diff --git a/docs/guide/standalone.md b/docs/guide/standalone.md index edc87cf..c5911ec 100644 --- a/docs/guide/standalone.md +++ b/docs/guide/standalone.md @@ -32,6 +32,9 @@ const Ping = new Definition('Ping', { at: 'date' }); module.exports = { model, Ping }; // compiles to the exports Address, User and Ping ``` +The same command writes and checks the baselines of the evolution guide, `protoarray snapshot` and +`protoarray check` (see [Checking it in CI](/guide/evolution#checking-it-in-ci)). + Options: | Option | | diff --git a/evolve.d.ts b/evolve.d.ts new file mode 100644 index 0000000..55027fc --- /dev/null +++ b/evolve.d.ts @@ -0,0 +1,126 @@ +/** + * Hand-written typings for @alexify/protoarray/evolve: whether a newer version of a schema keeps + * the wire of an older one, and a baseline of schemas a CI job checks the module against. + */ + +import type { AnySchema, Layout, Schema, Snapshot, WireType } from './index.js'; + +/** + * How bad a problem is: an `error` breaks the wire, a `warning` is survived by older peers but + * must be minded when deploying, `info` changes nothing on the wire. + */ +export type Severity = 'error' | 'warning' | 'info'; + +/** The rule a problem breaks (see the evolution guide, "Checking it in CI"). */ +export type Rule = + | 'inserted' + | 'moved' + | 'removed' + | 'renamed' + | 'required' + | 'retired' + | 'type' + | 'optional' + | 'appended' + | 'wire' + | 'flat' + | 'default'; + +export interface Problem { + /** The place, as the errors spell it (`address.city`, `orders[].sku`, `''` for the root). */ + readonly path: string; + readonly rule: Rule; + readonly severity: Severity; + readonly message: string; + /** The canonical form of the node in the previous version, where there is one. */ + readonly previous?: string; + /** The canonical form of the node in the next version, where there is one. */ + readonly next?: string; +} + +export interface Compatibility { + /** No problem is an error (with `strict`, none is a warning either). */ + readonly ok: boolean; + readonly problems: readonly Problem[]; +} + +export interface CompatibleOptions { + /** A warning fails the check too. */ + readonly strict?: boolean; +} + +/** The options a baseline carries: the ones that shape the wire and the canonical form. */ +export interface BaselineOptions { + readonly sparse: boolean | readonly string[]; + readonly layout: Layout | Readonly>; + readonly intern: boolean | readonly string[]; + readonly delta: readonly string[]; + readonly references: 'kind' | 'embed' | 'id'; + readonly types: Readonly>; + /** + * The name of a built-in fingerprint; `null` for a function of your own, which a baseline + * cannot carry: the id is then recorded, not checked. + */ + readonly fingerprint?: string | null; + /** The options of the entities the schema reaches, by name. */ + readonly entities?: Readonly>>; +} + +export interface BaselineEntry { + /** The index of the schema's snapshot in `snapshots`. */ + readonly snapshot: number; + /** The entity of that snapshot the schema is of. */ + readonly entity: string; + readonly options: BaselineOptions; + /** The canonical form the schema was written with: `restore` checks it. */ + readonly canonical: string; + readonly id: string; +} + +/** What `baseline` writes and `restore` reads: plain JSON. */ +export interface Baseline { + readonly format: 1; + /** Each snapshot once, however many schemas share it. */ + readonly snapshots: readonly Snapshot[]; + readonly schemas: Readonly>; +} + +/** + * Whether `next` reads what `previous` writes, and writes what it reads, by position: the two + * plans walked together, every place where the newer version reads an older payload differently + * reported with the rule it breaks. Throws `ERR_SCHEMA_EXPECTED` for anything but a schema of its + * own (a resolved schema is no version), `ERR_INCOMPATIBLE` for a chain of references too long + * for the stack. + */ +export function compatible( + previous: AnySchema, + next: AnySchema, + options?: CompatibleOptions, +): Compatibility; + +/** + * The schemas given, by name, as JSON: their snapshots, the options that shape their wire, and + * their canonical forms and ids as checksums. Each must have been read from a snapshot (not + * `schema.list`, not a resolved schema): `ERR_SCHEMA_EXPECTED` otherwise. + */ +export function baseline(schemas: Readonly>): Baseline; + +/** + * The schemas of a baseline, built again and checked against the canonical form each was written + * with (`ERR_INVALID_SNAPSHOT` when one does not rebuild to it: write the baseline again). + */ +export function restore(baseline: Baseline): Readonly>; + +/** Installs `Schema.compatible`. Once per process. */ +export function register(): void; + +declare module './index.js' { + namespace Schema { + /** `compatible` of `@alexify/protoarray/evolve`, once its `register()` has run. */ + function compatible( + previous: AnySchema, + next: AnySchema, + options?: CompatibleOptions, + ): Compatibility; + } +} diff --git a/evolve.js b/evolve.js new file mode 100644 index 0000000..8707e00 --- /dev/null +++ b/evolve.js @@ -0,0 +1 @@ +module.exports = require('./src/evolve.js'); diff --git a/package.json b/package.json index 0656017..0460dff 100644 --- a/package.json +++ b/package.json @@ -47,6 +47,10 @@ "types": "./standalone.d.ts", "default": "./standalone.js" }, + "./evolve": { + "types": "./evolve.d.ts", + "default": "./evolve.js" + }, "./package.json": "./package.json" }, "sideEffects": false, @@ -65,6 +69,8 @@ "stream.d.ts", "standalone.js", "standalone.d.ts", + "evolve.js", + "evolve.d.ts", "bin", "src", "README.md", @@ -79,7 +85,7 @@ "test": "node --test tests/unit/*.test.js", "test:coverage": "c8 --all --include 'src/**' --include 'bin/**' --100 --per-file --reporter text --reporter lcov node --test tests/unit/*.test.js", "test:types": "tsd", - "check:dts": "tsc --noEmit --strict --target es2022 --skipLibCheck false index.d.ts registry.d.ts resolve.d.ts stream.d.ts standalone.d.ts", + "check:dts": "tsc --noEmit --strict --target es2022 --skipLibCheck false index.d.ts registry.d.ts resolve.d.ts stream.d.ts standalone.d.ts evolve.d.ts", "lint": "oxlint src tests scripts bench bin", "format": "oxfmt src tests scripts bench bin", "format:check": "oxfmt --check src tests scripts bench bin", diff --git a/scripts/size.js b/scripts/size.js index c75aa6f..731c4e9 100644 --- a/scripts/size.js +++ b/scripts/size.js @@ -33,6 +33,7 @@ const SUBPATHS = [ { label: 'Stream (stream.js)', entry: 'stream.js' }, { label: 'Resolve (resolve.js)', entry: 'resolve.js' }, { label: 'Standalone (standalone.js)', entry: 'standalone.js' }, + { label: 'Evolve (evolve.js)', entry: 'evolve.js' }, ]; const bundle = async (entry, platform, minify) => { diff --git a/src/evolve.js b/src/evolve.js new file mode 100644 index 0000000..a46d30a --- /dev/null +++ b/src/evolve.js @@ -0,0 +1,564 @@ +// Schema.compatible: whether a newer version of a schema keeps the wire of an +// older one, by the rules of the evolution guide. A position is the contract, +// so the two plans are walked together position by position, and every place +// where the newer version reads an older payload differently is a problem, +// named after the rule it breaks: what breaks the wire is an error, what older +// peers survive but a deployment must mind is a warning, what changes nothing +// on the wire is for information. A baseline writes the schemas of a module +// as JSON (their snapshots and the options that shape their wire), and +// `restore` builds them again, so a CI job compares the schemas of today with +// the ones that shipped. +// +// The `@alexify/protoarray/evolve` subpath: a tool for development and CI, +// kept out of the core and apart from `./resolve`, which reads one version as +// another at runtime, by name. `register()` installs `Schema.compatible`. + +const { SchemaDefinitionError, overflowed, join, step } = require('./errors.js'); +const { PLAN, canonical, isSchema, referenced, fingerprint } = require('./plan.js'); +const { isObject } = require('./snapshot.js'); +const { Schema, SETTINGS, ENTITY, SAME } = require('./schema.js'); + +// The rules, by the problem they give, and how bad each one is. A retired +// slot is for information when it takes anything (`?any`) and a warning when +// it takes only null, which a validating decoder holds older writers to. +const RULES = Object.assign(Object.create(null), { + inserted: 'error', + moved: 'error', + removed: 'error', + renamed: 'info', + required: 'error', + retired: 'info', + type: 'error', + optional: 'info', + appended: 'warning', + wire: 'error', + flat: 'error', + default: 'warning', +}); + +const isJson = (node) => node.kind === 'scalar' && node.type === 'json'; +const isNull = (node) => node.kind === 'scalar' && node.type === 'null'; +// A list of differences is a scalar node over its item: a list all the same. +const isDeltas = (node) => node.kind === 'scalar' && node.type === 'deltas'; +const isList = (node) => node.kind === 'array' || isDeltas(node); + +// A key, a value or a name in a message: quoted, a number as it is. +const show = (value) => JSON.stringify(value); +const listed = (values) => { + const quoted = values.map(show); + if (quoted.length < 2) return quoted.join(''); + return `${quoted.slice(0, -1).join(', ')} and ${quoted[quoted.length - 1]}`; +}; +// `"a" moves up`, `"a" and "b" move further`. +const moves = (values, where) => `${listed(values)} move${values.length === 1 ? 's' : ''} ${where}`; + +// The type of a node as the rules compare it: its canonical form without the +// markers of its wire (`*` interned, `~` differences), which the `wire` rule +// reports apart. +const bare = (node) => canonical(node).replace(/[*~]$/, ''); + +// Two lists of names (keys, enum values, flag names, a union's values) +// walked together. The same name at both cursors is one place; a name the +// other side lacks at the same place is a rename when it is nowhere else, +// otherwise the side it is on tells an insertion from a removal, and a name +// both sides have elsewhere moved: of the two at the cursors, the one further +// from its own place (the older one when they are as far), which the walk +// then skips where it stands on the other side. After an insertion or a +// removal the walk realigns on the names that follow, so one change is one +// problem. The heuristic is conservative: a key that shifts for a reason it +// cannot name is an error, never a rename. +const aligned = (previous, next, hooks) => { + const inPrevious = new Map(); + const inNext = new Map(); + previous.forEach((name, i) => inPrevious.set(name, i)); + next.forEach((name, i) => inNext.set(name, i)); + const skipped = new Set(); + let i = 0; + let j = 0; + while (i < previous.length && j < next.length) { + if (skipped.has(previous[i])) { + i++; + continue; + } + if (skipped.has(next[j])) { + j++; + continue; + } + if (previous[i] === next[j]) { + hooks.same(i, j); + i++; + j++; + continue; + } + const to = inNext.get(previous[i]); + const from = inPrevious.get(next[j]); + if (to === undefined && from === undefined) { + hooks.renamed(i, j); + i++; + j++; + } else if (to === undefined) { + hooks.removed(i); + i++; + } else if (from === undefined) { + hooks.inserted(i, j); + j++; + } else if (Math.abs(to - i) >= Math.abs(from - j)) { + hooks.moved(i, to); + skipped.add(previous[i]); + i++; + } else { + hooks.moved(from, j); + skipped.add(next[j]); + j++; + } + } + for (; i < previous.length; i++) if (!skipped.has(previous[i])) hooks.removed(i); + for (; j < next.length; j++) if (!skipped.has(next[j])) hooks.appended(j); +}; + +// One slot of a struct, a tuple or a collection: its node, and whether it is +// required and what it defaults to where the field (or the root) says so. +const slotOf = (node, required, def) => ({ node, required, def }); +const rootOf = (plan) => { + const required = plan.kind === 'scalar' ? plan.required !== false : true; + return slotOf(plan, required, null); +}; + +const comparePlans = (previousPlan, nextPlan) => { + const problems = []; + const report = (path, rule, message, previous, next, severity = RULES[rule]) => { + const problem = { path, rule, severity, message }; + if (previous !== undefined) problem.previous = canonical(previous); + if (next !== undefined) problem.next = canonical(next); + problems.push(problem); + }; + + // A schema reached by reference compares once per pair of plans: a pair met + // again, on a recursion or from another field, has been compared. + const pairs = new Map(); + const pair = (p, n, path) => { + let byNext = pairs.get(p); + if (byNext === undefined) pairs.set(p, (byNext = new Set())); + if (byNext.has(n)) return; + byNext.add(n); + slot(rootOf(p), rootOf(n), path); + }; + + const slot = (p, n, path) => { + const from = p.node; + const to = n.node; + // A retired slot: what older writers sent is read as anything, or as null. + if (isNull(to) && !isNull(from)) { + const text = 'Retired as null: a validating decoder rejects the values older writers send'; + report(path, 'retired', text, from, to, 'warning'); + return; + } + if (isJson(to) && !isJson(from)) { + report( + path, + 'retired', + 'Retired as ?any: whatever older writers send is read as is', + from, + to, + ); + return; + } + if (p.required && !n.required) report(path, 'optional', 'Optional now', from, to); + if (!p.required && n.required) { + report(path, 'required', 'Required now, where older payloads may send null', from, to); + } + if (p.def !== n.def) { + const text = + p.def === null + ? `Default added (${show(n.def)}): an empty slot in older payloads decodes as it` + : n.def === null + ? `Default removed (${show(p.def)}): an empty slot in older payloads decodes as null` + : `Default changed from ${show(p.def)} to ${show(n.def)}: an empty slot in older ` + + 'payloads decodes differently'; + report(path, 'default', text, from, to); + } + node(from, to, path); + }; + + const node = (p, n, path) => { + if (p.kind === 'ref' || n.kind === 'ref') { + if (p.kind !== n.kind) { + report(path, 'type', `Type changed: ${bare(p)} → ${bare(n)}`, p, n); + return; + } + if (p.name !== n.name) { + report(path, 'renamed', `Entity ${show(p.name)} is now ${show(n.name)}`, p, n); + } + pair(p.slot.schema[PLAN], n.slot.schema[PLAN], path); + return; + } + if (isList(p) && isList(n)) { + list(p, n, path); + return; + } + if ( + p.kind !== n.kind || + (p.kind === 'scalar' && (p.type !== n.type || p.decimals !== n.decimals)) + ) { + report(path, 'type', `Type changed: ${bare(p)} → ${bare(n)}`, p, n); + return; + } + if (p.kind === 'scalar') scalar(p, n, path); + else if (p.kind === 'struct') struct(p, n, path); + else if (p.kind === 'union') union(p, n, path); + else if (p.kind === 'tuple') tuple(p, n, path); + else { + // A map or a dictionary: the value, under a key of the same type. + if (p.kind === 'map' && p.key.type !== n.key.type) { + report(path, 'type', `Type changed: ${bare(p)} → ${bare(n)}`, p, n); + return; + } + slot( + slotOf(p.value, p.valueRequired, null), + slotOf(n.value, n.valueRequired, null), + `${path}{}`, + ); + } + }; + + const wire = (path, what, p, n) => report(path, 'wire', `The wire changed: ${what}`, p, n); + const switched = (name, on) => `${name} switched ${on ? 'on' : 'off'}`; + + const scalar = (p, n, path) => { + if (p.type === 'enum') values(p.values, n.values, 'Enum value', path, p, n); + if (p.type === 'flags') values(p.names, n.names, 'Flag', path, p, n); + const interned = (p.intern === true) !== (n.intern === true); + const delta = (p.delta === true) !== (n.delta === true); + if (interned) wire(path, switched('intern', n.intern === true), p, n); + if (delta) wire(path, switched('delta', n.delta === true), p, n); + }; + + // The values of an enum, the names of flags: an index travels. + const values = (previous, next, word, path, p, n) => { + const rest = (i) => { + const after = previous.slice(i + 1).filter((value) => next.includes(value)); + return after.length === 0 ? '' : `: ${moves(after, 'up')}`; + }; + aligned(previous.map(show), next.map(show), { + same: () => {}, + renamed: (i, j) => { + report(path, 'renamed', `${word} ${show(previous[i])} is now ${show(next[j])}`, p, n); + }, + removed: (i) => { + report(path, 'removed', `${word} ${show(previous[i])} removed${rest(i)}`, p, n); + }, + inserted: (i, j) => { + const text = `${word} ${show(next[j])} inserted before ${show(previous[i])}, which moves`; + report(path, 'inserted', text, p, n); + }, + moved: (i, to) => { + const text = `${word} ${show(previous[i])} moved from position ${i} to ${to}`; + report(path, 'moved', text, p, n); + }, + appended: (j) => { + const text = `${word} ${show(next[j])} appended: deploy decoders before encoders`; + report(path, 'appended', text, p, n); + }, + }); + }; + + const struct = (p, n, path) => { + if (p.sparse !== n.sparse) wire(path, switched('sparse', n.sparse), p, n); + const at = (field) => join(path, step(field.key)); + const previous = p.fields.map((field) => field.key); + const next = n.fields.map((field) => field.key); + // The fields from `i` on that keep a place in the next version: what an + // insertion or a removal before them shifts. + const shifted = (i) => previous.slice(i).filter((key) => next.includes(key)); + // Rule 8: a stored kind's id, appended by metaschema after the fields of + // the definition, moves when a field is appended to the definition. + const hint = (keys) => { + const last = p.fields[p.fields.length - 1]; + const id = /Id$/.test(last.key) && !last.required && last.node.type === 'string'; + return id && keys.includes(last.key) + ? ` (rule 8: declare ${show(last.key)} in the definition and append after it)` + : ''; + }; + aligned(previous, next, { + same: (i, j) => slot(p.fields[i], n.fields[j], at(n.fields[j])), + renamed: (i, j) => { + const text = `Field ${show(previous[i])} is now ${show(next[j])}`; + report(at(n.fields[j]), 'renamed', text, p.fields[i].node, n.fields[j].node); + slot(p.fields[i], n.fields[j], at(n.fields[j])); + }, + removed: (i) => { + const keys = shifted(i + 1); + const text = + `Field ${show(previous[i])} removed from position ${i}` + + (keys.length === 0 ? '' : `: ${moves(keys, 'up')}`) + + '; keep the slot of a retired field, as ?any'; + report(at(p.fields[i]), 'removed', text, p.fields[i].node); + }, + inserted: (i, j) => { + const keys = shifted(i); + const text = + `Field ${show(next[j])} inserted at position ${j}: ${moves(keys, 'further')}` + + hint(keys); + report(at(n.fields[j]), 'inserted', text, undefined, n.fields[j].node); + }, + moved: (i, to) => { + const text = `Field ${show(previous[i])} moved from position ${i} to ${to}`; + report(at(p.fields[i]), 'moved', text, p.fields[i].node, n.fields[to].node); + }, + appended: (j) => { + if (!n.fields[j].required) return; + const text = `Field ${show(next[j])} appended as required without a default: older payloads do not have it`; + report(at(n.fields[j]), 'required', text, undefined, n.fields[j].node); + }, + }); + }; + + // The tags of a union are the indexes of the values of all its branches, + // in order: the values compare as an enum's, then each branch of the older + // version with every branch of the newer one that holds a value of it. + const union = (p, n, path) => { + if (p.key !== n.key) { + const text = `The union discriminates on ${show(n.key)} instead of ${show(p.key)}`; + report(path, 'type', text, p, n); + return; + } + values(p.values, n.values, 'Union value', path, p, n); + const owners = new Map(); + n.branches.forEach((branch, b) => { + for (let i = 0; i < branch.count; i++) owners.set(show(n.values[branch.first + i]), b); + }); + for (const branch of p.branches) { + const targets = new Set(); + for (let i = 0; i < branch.count; i++) { + const b = owners.get(show(p.values[branch.first + i])); + if (b !== undefined) targets.add(b); + } + for (const b of targets) struct(branch.node, n.branches[b].node, `${path}[${b}]`); + } + }; + + const tuple = (p, n, path) => { + const size = Math.min(p.items.length, n.items.length); + for (let i = 0; i < size; i++) slot(p.items[i], n.items[i], `${path}[${i}]`); + for (let i = size; i < p.items.length; i++) { + report(`${path}[${i}]`, 'removed', `Element ${i} removed`, p.items[i].node); + } + for (let i = size; i < n.items.length; i++) { + if (!n.items[i].required) continue; + const text = 'Element appended as required without a default: older payloads do not have it'; + report(`${path}[${i}]`, 'required', text, undefined, n.items[i].node); + } + }; + + // An array, a Set, a list of differences: the item, under the same layout. + const list = (p, n, path) => { + if (isDeltas(p) !== isDeltas(n)) wire(path, switched('delta', isDeltas(n)), p, n); + else if (p.container !== n.container) { + report(path, 'type', `Type changed: ${bare(p)} → ${bare(n)}`, p, n); + return; + } else { + if (p.layout !== n.layout) wire(path, `layout ${p.layout} → ${n.layout}`, p, n); + else if (p.layout === 'flat' && p.item.fields.length !== n.item.fields.length) { + const text = + `A flat record has ${n.item.fields.length} fields instead of ` + + `${p.item.fields.length}: the stride of the collection changed`; + report(path, 'flat', text, p, n); + } + } + slot(slotOf(p.item, p.itemRequired, null), slotOf(n.item, n.itemRequired, null), `${path}[]`); + }; + + pair(previousPlan, nextPlan, ''); + return problems; +}; + +const readOptions = (options) => { + const invalid = (text) => new SchemaDefinitionError('ERR_INVALID_OPTIONS', text); + if (options == null) return { strict: false }; + if (typeof options !== 'object') throw invalid('Invalid options: expected an object'); + for (const key of Object.keys(options)) { + if (key !== 'strict') throw invalid(`Invalid option "${key}"`); + } + if (options.strict !== undefined && typeof options.strict !== 'boolean') { + throw invalid('Invalid option "strict"'); + } + return { strict: options.strict === true }; +}; + +const version = (schema) => { + if (!isSchema(schema)) { + throw new SchemaDefinitionError('ERR_SCHEMA_EXPECTED', 'Schema instance expected'); + } + // A resolved schema reads one version as another: it is no version itself. + if (schema[PLAN].text !== undefined) { + const text = 'A resolved schema is no version to compare: give its writer or its reader'; + throw new SchemaDefinitionError('ERR_SCHEMA_EXPECTED', text); + } +}; + +// Whether `next` reads what `previous` writes and writes what it reads, by +// position: `ok` when no problem is an error (with `strict`, a warning +// neither), and every problem with the rule it breaks, its severity, a path +// as the errors spell it, and the canonical forms of the two nodes. +const compatible = (previous, next, options) => { + version(previous); + version(next); + const { strict } = readOptions(options); + if (previous[SAME](next)) return { ok: true, problems: [] }; + let problems; + try { + problems = comparePlans(previous[PLAN], next[PLAN]); + } catch (error) { + // Two versions are walked together, a chain of references included: one + // too long for the stack cannot be compared, which is no RangeError. + if (!overflowed(error)) throw error; + const message = 'The schemas are nested too deeply to compare'; + throw new SchemaDefinitionError('ERR_INCOMPATIBLE', message, next.name, '', error); + } + const worst = strict ? /^(error|warning)$/ : /^error$/; + const ok = !problems.some((problem) => worst.test(problem.severity)); + return { ok, problems }; +}; + +// The options that shape the wire and the canonical form, which a baseline +// carries: the ones that shape the codec (`codegen`, `validate`, +// `stringify`, `maxDepth`) change neither. +const WIRE = ['sparse', 'layout', 'intern', 'delta', 'references', 'types']; + +// The fingerprint as a baseline carries it: a built-in one by its name, the +// default one by nothing, and a function of your own, which cannot travel in +// JSON, as null, so the id is recorded and not checked. +const named = (settings) => { + const hash = settings.fingerprint; + const out = hash === fingerprint || typeof hash === 'string' ? hash : null; + return typeof out === 'function' ? undefined : out; +}; + +const wireOptions = (settings) => { + const out = {}; + for (const key of WIRE) out[key] = settings[key]; + const hash = named(settings); + if (hash !== undefined) out.fingerprint = hash; + return out; +}; + +const optionsOf = (schema) => { + const out = wireOptions(schema[SETTINGS]); + // The entities the schema reaches keep the options they were built with, + // which its id depends on. + const entities = {}; + for (const [name, entity] of referenced(schema[PLAN], [schema])) { + entities[name] = wireOptions(entity[SETTINGS]); + } + out.entities = entities; + return out; +}; + +const expected = (text) => new SchemaDefinitionError('ERR_SCHEMA_EXPECTED', text); + +// The schemas given, by name, as JSON: each one's snapshot (written once, +// however many schemas share it), the entity it is of, the options that +// shape its wire, and its canonical form and id as checksums. +const baseline = (schemas) => { + if (!isObject(schemas)) throw expected('A record of schemas by name expected'); + const snapshots = []; + const indexes = new Map(); + const out = { format: 1, snapshots, schemas: {} }; + for (const name of Object.keys(schemas)) { + const schema = schemas[name]; + if (!isSchema(schema)) throw expected(`${show(name)} is not a Schema`); + if (schema.snapshot === null) { + throw expected(`${show(name)} was not read from a snapshot: give a schema of its own`); + } + let index = indexes.get(schema.snapshot); + if (index === undefined) { + index = snapshots.length; + indexes.set(schema.snapshot, index); + snapshots.push(schema.snapshot); + } + out.schemas[name] = { + snapshot: index, + entity: schema[ENTITY], + options: optionsOf(schema), + canonical: schema.toString(), + id: schema.id, + }; + } + return out; +}; + +const broken = (text, name) => new SchemaDefinitionError('ERR_INVALID_SNAPSHOT', text, name); + +const isEntry = (entry) => + isObject(entry) && + Number.isInteger(entry.snapshot) && + typeof entry.entity === 'string' && + isObject(entry.options) && + (entry.options.fingerprint == null || typeof entry.options.fingerprint === 'string') && + typeof entry.canonical === 'string' && + typeof entry.id === 'string'; + +// The options as the schema takes them again: a fingerprint written as null +// (a function of the caller's own) is left out, so the default one applies. +const carried = (options) => { + const out = { ...options }; + if (out.fingerprint === null) delete out.fingerprint; + return out; +}; + +// The schemas of a baseline, built again from its snapshots and checked +// against the canonical form each one was written with, and against its id +// unless a fingerprint of the caller's own computed it. +const restore = (source) => { + if ( + !isObject(source) || + source.format !== 1 || + !Array.isArray(source.snapshots) || + !isObject(source.schemas) + ) { + throw broken('A baseline of format 1 with snapshots and schemas expected'); + } + const out = Object.create(null); + for (const name of Object.keys(source.schemas)) { + const entry = source.schemas[name]; + if (!isEntry(entry)) { + throw broken(`The entry ${show(name)} of the baseline is incomplete`, name); + } + const snapshot = source.snapshots[entry.snapshot]; + if (snapshot === undefined) { + throw broken(`The entry ${show(name)} names no snapshot of the baseline`, name); + } + const options = carried(entry.options); + if (isObject(options.entities)) { + const entities = {}; + for (const key of Object.keys(options.entities)) { + entities[key] = carried(options.entities[key]); + } + options.entities = entities; + } + const schema = new Schema(name, snapshot, { ...options, entity: entry.entity }); + // The id is checked where the baseline says how it was computed: by a + // built-in fingerprint, or by the default one when it names none. + const checked = entry.options.fingerprint === null || schema.id === entry.id; + if (schema.toString() !== entry.canonical || !checked) { + throw broken('The baseline does not rebuild to its canonical form: write it again', name); + } + out[name] = schema; + } + return out; +}; + +let registered = false; + +// Installs `Schema.compatible`, non-enumerable, once. +const register = () => { + if (registered) return; + registered = true; + Object.defineProperty(Schema, 'compatible', { + value: compatible, + writable: true, + configurable: true, + }); +}; + +module.exports = { compatible, baseline, restore, register }; diff --git a/tests/fixtures/evolution/v1.js b/tests/fixtures/evolution/v1.js new file mode 100644 index 0000000..22b62ff --- /dev/null +++ b/tests/fixtures/evolution/v1.js @@ -0,0 +1,11 @@ +// Four versions of the User record of the evolution guide, for the CLI tests: +// V1 is where it starts. +const { Schema } = require('@alexify/metaschema'); + +const User = new Schema('User', { + id: 'integer', + fullName: 'string', + role: { enum: ['admin', 'user'] }, +}); + +module.exports = { User }; diff --git a/tests/fixtures/evolution/v2.js b/tests/fixtures/evolution/v2.js new file mode 100644 index 0000000..d607999 --- /dev/null +++ b/tests/fixtures/evolution/v2.js @@ -0,0 +1,11 @@ +// V2: an optional field appended, which keeps every position. +const { Schema } = require('@alexify/metaschema'); + +const User = new Schema('User', { + id: 'integer', + fullName: 'string', + role: { enum: ['admin', 'user'] }, + email: '?string', +}); + +module.exports = { User }; diff --git a/tests/fixtures/evolution/v3.js b/tests/fixtures/evolution/v3.js new file mode 100644 index 0000000..59db74a --- /dev/null +++ b/tests/fixtures/evolution/v3.js @@ -0,0 +1,12 @@ +// V3: an enum value appended (decoders go first) and a field with a default. +const { Schema } = require('@alexify/metaschema'); + +const User = new Schema('User', { + id: 'integer', + fullName: 'string', + role: { enum: ['admin', 'user', 'guest'] }, + email: '?string', + score: { type: 'number', default: 0 }, +}); + +module.exports = { User }; diff --git a/tests/fixtures/evolution/v4.js b/tests/fixtures/evolution/v4.js new file mode 100644 index 0000000..a33ab8c --- /dev/null +++ b/tests/fixtures/evolution/v4.js @@ -0,0 +1,22 @@ +// V4: incompatible by position (a type widened, a field renamed, the enum +// values reordered, a default added), and a Model, whose entities the CLI +// exports under their own names. +const { Schema, Model } = require('@alexify/metaschema'); + +const User = new Schema('User', { + id: 'number', + name: { type: 'string', aliases: ['fullName'] }, + role: { enum: ['guest', 'user', 'admin'] }, + email: { type: 'string', default: '' }, + score: { type: 'number', default: 0 }, +}); + +const model = new Model( + {}, + new Map([ + ['Address', { Struct: {}, city: 'string', zip: '?string' }], + ['Contact', { Struct: {}, home: 'Address', phone: '?string' }], + ]), +); + +module.exports = { User, model }; diff --git a/tests/types/evolve.test-d.ts b/tests/types/evolve.test-d.ts new file mode 100644 index 0000000..0304d21 --- /dev/null +++ b/tests/types/evolve.test-d.ts @@ -0,0 +1,41 @@ +import { expectError, expectType } from 'tsd'; +import { Schema as Definition } from '@alexify/metaschema'; +import { Schema } from '../../index.js'; +import { + baseline, + compatible, + register, + restore, + type Baseline, + type Compatibility, + type Problem, + type Rule, + type Severity, +} from '../../evolve.js'; + +const V1 = Schema.from(new Definition('User', { id: 'number', fullName: 'string' })); +const V2 = Schema.from(new Definition('User', { id: 'number', name: 'string' })); + +expectType(compatible(V1, V2)); +expectType(compatible(V1, V2, { strict: true })); +expectType(compatible(V1, V2).ok); +expectType(compatible(V1, V2).problems); +expectError(compatible(V1, V2, { aliases: {} })); +expectError(compatible(V1, 1)); +declare const problem: Problem; +expectType(problem.rule); +expectType(problem.severity); +expectType(problem.previous); +expectType(problem.path); + +const written = baseline({ V1, V2 }); +expectType(written); +expectType<1>(written.format); +expectType(written.schemas.V1.snapshot); +expectType(written.schemas.V1.options.fingerprint); +expectError(baseline(V1)); +expectError(baseline({ V1: 1 })); +expectType>>(restore(written)); +expectType(restore(written).V1.decode([1])); +expectType(register()); +expectType(Schema.compatible(V1, V2)); diff --git a/tests/unit/cli.test.js b/tests/unit/cli.test.js index 6fac22c..4d4602f 100644 --- a/tests/unit/cli.test.js +++ b/tests/unit/cli.test.js @@ -122,7 +122,21 @@ test('CLI: what it refuses', () => { // The module it reads is never written over. const over = cli('compile', 'tests/fixtures/schemas.js', '--out', 'tests/fixtures/schemas.js'); assert.strictEqual(over.status, 2); - assert.match(over.stderr, /^--out names the module it compiles/); + assert.match(over.stderr, /^--out names the module it reads/); + // Without a command, the usage of every command; with one, its own. + const all = cli(); + assert.match(all.stderr, /^ {7}protoarray snapshot /m); + assert.match(all.stderr, /^ {7}protoarray check --against/m); + const one = cli('snapshot'); + assert.match( + one.stderr, + /^Usage: protoarray snapshot \[--only \] \[--out \]$/m, + ); + assert.doesNotMatch(one.stderr, /compile/); + // A flag of one command is no option of another. + const flag = cli('snapshot', 'tests/fixtures/schemas.js', '--strict'); + assert.strictEqual(flag.status, 2); + assert.match(flag.stderr, /^Unknown or incomplete option --strict/); }); test('CLI: a Model as the whole module, names with spaces, --out in a new directory', () => { @@ -203,3 +217,154 @@ test('CLI: a module of one schema, one that waits at its top, one that throws a assert.strictEqual(thrown.status, 1); assert.strictEqual(thrown.stderr, 'not an error\n'); }); + +test('CLI: snapshot writes a baseline and check compares a module with it', () => { + const dir = path.join(ROOT, 'tests', '.generated', 'evolution'); + fs.rmSync(dir, { recursive: true, force: true }); + const fixture = (version) => path.join('tests', 'fixtures', 'evolution', `${version}.js`); + const v1 = path.join(dir, 'v1.json'); + const written = cli('snapshot', fixture('v1'), '--out', v1); + assert.strictEqual(written.status, 0, written.stderr); + assert.strictEqual(written.stdout, ''); + const { baseline } = require('../../evolve.js'); + const { Schema: Definition } = require('@alexify/metaschema'); + const User = require('../fixtures/evolution/v1.js').User; + assert.deepStrictEqual( + JSON.parse(fs.readFileSync(v1, 'utf8')), + JSON.parse(JSON.stringify(baseline({ User: Schema.from(User) }))), + ); + assert.ok(User instanceof Definition); + // Without --out, the baseline goes to stdout; --only keeps the named schemas. + const printed = cli('snapshot', fixture('v4'), '--only', 'User,Address'); + assert.strictEqual(printed.status, 0, printed.stderr); + assert.deepStrictEqual(Object.keys(JSON.parse(printed.stdout).schemas), ['User', 'Address']); + // V2 keeps every position; V3 appends an enum value, a warning --strict fails on. + const two = cli('check', fixture('v2'), '--against', v1); + assert.strictEqual(two.status, 0, two.stderr); + assert.strictEqual(two.stdout, '1 schema checked: no problems; compatible\n'); + const three = cli('check', fixture('v3'), '--against', v1); + assert.strictEqual(three.status, 0, three.stderr); + assert.strictEqual( + three.stdout, + 'warning User.role: Enum value "guest" appended: deploy decoders before encoders (appended)\n' + + '1 schema checked: 1 warning; compatible\n', + ); + const strict = cli('check', fixture('v3'), '--against', v1, '--strict'); + assert.strictEqual(strict.status, 1); + assert.match(strict.stdout, /1 warning; incompatible\n$/); + // V4 breaks the rules, and brings two schemas the baseline does not have. + const four = cli('check', fixture('v4'), '--against', v1); + assert.strictEqual(four.status, 1); + assert.strictEqual( + four.stdout, + [ + 'error User.id: Type changed: integer → number (type)', + 'info User.name: Field "fullName" is now "name" (renamed)', + 'error User.role: Enum value "guest" inserted before "admin", which moves (inserted)', + 'error User.role: Enum value "admin" moved from position 0 to 2 (moved)', + 'info Address: "Address" is not in the baseline: nothing to compare (appended)', + 'info Contact: "Contact" is not in the baseline: nothing to compare (appended)', + '3 schemas checked: 3 errors, 3 info; incompatible', + '', + ].join('\n'), + ); + const json = cli('check', fixture('v4'), '--against', v1, '--json'); + assert.strictEqual(json.status, 1); + const result = JSON.parse(json.stdout); + assert.strictEqual(result.ok, false); + assert.deepStrictEqual(Object.keys(result.schemas), ['User', 'Address', 'Contact']); + assert.strictEqual(result.schemas.User.ok, false); + assert.strictEqual(result.schemas.User.problems.length, 4); + assert.deepStrictEqual(result.schemas.Address, { + ok: true, + problems: [ + { + path: '', + rule: 'appended', + severity: 'info', + message: '"Address" is not in the baseline: nothing to compare', + }, + ], + }); + // --only names the pairs to check. + const only = cli('check', fixture('v4'), '--against', v1, '--only', 'Address', '--json'); + assert.strictEqual(only.status, 0, only.stderr); + assert.deepStrictEqual(Object.keys(JSON.parse(only.stdout).schemas), ['Address']); + // A schema the module no longer exports is a warning, which --strict fails on. + const v4 = path.join(dir, 'v4.json'); + assert.strictEqual(cli('snapshot', fixture('v4'), '--out', v4).status, 0); + const gone = cli('check', fixture('v2'), '--against', v4, '--only', 'Address,Contact'); + assert.strictEqual(gone.status, 0, gone.stderr); + assert.strictEqual( + gone.stdout, + 'warning Address: The module no longer exports "Address" (removed)\n' + + 'warning Contact: The module no longer exports "Contact" (removed)\n' + + '2 schemas checked: 2 warnings; compatible\n', + ); + const goneStrict = cli('check', fixture('v2'), '--against', v4, '--strict', '--json'); + assert.strictEqual(goneStrict.status, 1); + assert.strictEqual(JSON.parse(goneStrict.stdout).schemas.Address.ok, false); + const nowhere = cli('check', fixture('v2'), '--against', v4, '--only', 'User, Nope'); + assert.strictEqual(nowhere.status, 1); + assert.strictEqual(nowhere.stderr, 'Neither the module nor the baseline has a schema "Nope"\n'); + // A path with brackets follows the name without a dot. + const list = path.join(dir, 'list.js'); + fs.writeFileSync( + list, + [ + "const ms = require('@alexify/metaschema');", + "module.exports = ms.Schema.from({ array: { v: 'string' } });", + '', + ].join('\n'), + ); + const listed = path.join(dir, 'list.json'); + assert.strictEqual(cli('snapshot', list, '--out', listed).status, 0); + fs.writeFileSync( + list, + [ + "const ms = require('@alexify/metaschema');", + "module.exports = ms.Schema.from({ array: { v: 'integer' } });", + '', + ].join('\n'), + ); + const items = cli('check', list, '--against', listed); + assert.strictEqual(items.status, 1); + assert.match(items.stdout, /^error {4}default\[\]\.v: Type changed: string → integer \(type\)$/m); +}); + +test('CLI: what check refuses', () => { + const dir = path.join(ROOT, 'tests', '.generated', 'evolution'); + fs.mkdirSync(dir, { recursive: true }); + const fixture = path.join('tests', 'fixtures', 'evolution', 'v1.js'); + const usage = /^Usage: protoarray check --against /m; + // No baseline named: the command line is wrong. + const missing = cli('check', fixture); + assert.strictEqual(missing.status, 2); + assert.match(missing.stderr, /^check needs --against $/m); + assert.match(missing.stderr, usage); + // A baseline that cannot be read, is not JSON, or does not rebuild: exit 1. + const absent = cli('check', fixture, '--against', path.join(dir, 'absent.json')); + assert.strictEqual(absent.status, 1); + assert.match(absent.stderr, /^Cannot read the baseline .*absent\.json: ENOENT/); + const text = path.join(dir, 'text.json'); + fs.writeFileSync(text, 'not json'); + const invalid = cli('check', fixture, '--against', text); + assert.strictEqual(invalid.status, 1); + assert.match(invalid.stderr, /^The baseline .*text\.json is not JSON: /); + const v1 = path.join(dir, 'forged.json'); + assert.strictEqual(cli('snapshot', fixture, '--out', v1).status, 0); + const forged = JSON.parse(fs.readFileSync(v1, 'utf8')); + forged.schemas.User.canonical = '{}'; + fs.writeFileSync(v1, JSON.stringify(forged)); + const stale = cli('check', fixture, '--against', v1); + assert.strictEqual(stale.status, 1); + assert.strictEqual( + stale.stderr, + 'SchemaDefinitionError ERR_INVALID_SNAPSHOT: The baseline does not rebuild to its canonical ' + + 'form: write it again (schema "User")\n', + ); + // The module is checked before the baseline is read. + const none = cli('check', 'tests/fixtures/empty.js', '--against', v1); + assert.strictEqual(none.status, 1); + assert.strictEqual(none.stderr, 'The module exports no schema\n'); +}); diff --git a/tests/unit/evolve.test.js b/tests/unit/evolve.test.js new file mode 100644 index 0000000..67ce0bb --- /dev/null +++ b/tests/unit/evolve.test.js @@ -0,0 +1,837 @@ +const { test } = require('node:test'); +const assert = require('node:assert'); +const { Schema } = require('../../index.js'); +const { compatible, baseline, restore, register } = require('../../evolve.js'); +const { resolve } = require('../../resolve.js'); +const { ms, define, named, model } = require('../support/schema.js'); + +// The four versions of the evolution guide: V1 to V3 follow the rules, V4 +// breaks them (a widened type, a rename, reordered enum values, a default). +const V1 = named('User', { id: 'integer', fullName: 'string', role: { enum: ['admin', 'user'] } }); +const V2 = named('User', { + id: 'integer', + fullName: 'string', + role: { enum: ['admin', 'user'] }, + email: '?string', +}); +const V3 = named('User', { + id: 'integer', + fullName: 'string', + role: { enum: ['admin', 'user', 'guest'] }, + email: '?string', + score: { type: 'number', default: 0 }, +}); +const V4 = named('User', { + id: 'number', + name: { type: 'string', aliases: ['fullName'] }, + role: { enum: ['guest', 'user', 'admin'] }, + email: { type: 'string', default: '' }, + score: { type: 'number', default: 0 }, +}); + +// The problems of a comparison as `[path, rule, severity]`, and their messages. +const found = (previous, next, options) => { + const { ok, problems } = compatible(previous, next, options); + const list = problems.map((problem) => [problem.path, problem.rule, problem.severity]); + return { ok, list, messages: problems.map((problem) => problem.message) }; +}; + +test('Evolve: the versions of the guide', () => { + assert.deepStrictEqual(compatible(V1, V2), { ok: true, problems: [] }); + assert.deepStrictEqual(compatible(V1, V1), { ok: true, problems: [] }); + // An enum value appended: decoders go first. A field with a default: nothing. + const grown = compatible(V2, V3); + assert.strictEqual(grown.ok, true); + assert.deepStrictEqual(grown.problems, [ + { + path: 'role', + rule: 'appended', + severity: 'warning', + message: 'Enum value "guest" appended: deploy decoders before encoders', + previous: 'enum(admin|user)', + next: 'enum(admin|user|guest)', + }, + ]); + assert.strictEqual(compatible(V2, V3, { strict: true }).ok, false); + assert.strictEqual(compatible(V2, V3, { strict: false }).ok, true); + const broken = found(V3, V4); + assert.strictEqual(broken.ok, false); + assert.deepStrictEqual(broken.list, [ + ['id', 'type', 'error'], + ['name', 'renamed', 'info'], + ['role', 'moved', 'error'], + ['role', 'moved', 'error'], + ['email', 'default', 'warning'], + ]); + assert.deepStrictEqual(broken.messages, [ + 'Type changed: integer → number', + 'Field "fullName" is now "name"', + 'Enum value "admin" moved from position 0 to 2', + 'Enum value "guest" moved from position 2 to 0', + 'Default added (""): an empty slot in older payloads decodes as it', + ]); + // Every problem carries the canonical forms of the two nodes. + const [widened] = compatible(V3, V4).problems; + assert.deepStrictEqual([widened.previous, widened.next], ['integer', 'number']); + const { list, messages } = found(V1, V4); + assert.deepStrictEqual(list.slice(2), [ + ['role', 'inserted', 'error'], + ['role', 'moved', 'error'], + ]); + assert.strictEqual(messages[2], 'Enum value "guest" inserted before "admin", which moves'); +}); + +test('Evolve: fields inserted, swapped, removed and renamed', () => { + const abc = define({ a: 'string', b: 'string', c: 'string' }); + const inserted = found(abc, define({ a: 'string', x: 'string', b: 'string', c: 'string' })); + assert.deepStrictEqual(inserted.list, [['x', 'inserted', 'error']]); + assert.deepStrictEqual(inserted.messages, [ + 'Field "x" inserted at position 1: "b" and "c" move further', + ]); + const [problem] = compatible( + abc, + define({ a: 'string', x: 'number', b: 'string', c: 'string' }), + ).problems; + assert.strictEqual(problem.previous, undefined); + assert.strictEqual(problem.next, 'number'); + const swapped = found(abc, define({ a: 'string', c: 'string', b: 'string' })); + assert.deepStrictEqual(swapped.list, [['b', 'moved', 'error']]); + assert.deepStrictEqual(swapped.messages, ['Field "b" moved from position 1 to 2']); + const reversed = found(abc, define({ c: 'string', b: 'string', a: 'string' })); + assert.deepStrictEqual(reversed.messages, [ + 'Field "a" moved from position 0 to 2', + 'Field "c" moved from position 2 to 0', + ]); + // A field that moved is skipped where it stands on the other side. + const abcd = define({ a: 'string', b: 'string', c: 'string', d: 'string' }); + const middle = found(abcd, define({ a: 'string', c: 'string', b: 'string', d: 'string' })); + assert.deepStrictEqual(middle.messages, ['Field "b" moved from position 1 to 2']); + const removed = found(abc, define({ a: 'string', c: 'string' })); + assert.deepStrictEqual(removed.list, [['b', 'removed', 'error']]); + assert.deepStrictEqual(removed.messages, [ + 'Field "b" removed from position 1: "c" moves up; keep the slot of a retired field, as ?any', + ]); + const shorter = found(abc, define({ a: 'string', b: 'string' })); + assert.deepStrictEqual(shorter.messages, [ + 'Field "c" removed from position 2; keep the slot of a retired field, as ?any', + ]); + // A key that changes where neither key is anywhere else is a rename. + const renamed = found(abc, define({ a: 'string', bee: 'string', c: 'string' })); + assert.deepStrictEqual(renamed.list, [['bee', 'renamed', 'info']]); + assert.strictEqual(renamed.ok, true); + // A rename with a type change is both. + const retyped = found(abc, define({ a: 'string', bee: 'number', c: 'string' })); + assert.deepStrictEqual(retyped.list, [ + ['bee', 'renamed', 'info'], + ['bee', 'type', 'error'], + ]); + // A shift the walk cannot explain is an error, never a rename: here "a" is + // gone and "b" moved up; "c" to "x" at the same place of the rest is a rename. + const shifted = found(abc, define({ b: 'string', x: 'string' })); + assert.deepStrictEqual(shifted.list, [ + ['a', 'removed', 'error'], + ['x', 'renamed', 'info'], + ]); + // After an insertion the walk realigns: one change, one problem. + const realigned = found(abc, define({ z: 'string', a: 'string', b: 'string', y: 'string' })); + assert.deepStrictEqual(realigned.list, [ + ['z', 'inserted', 'error'], + ['y', 'renamed', 'info'], + ]); + const mixed = found( + define({ a: 'string', b: 'string', c: 'string', d: 'string' }), + define({ + c: 'string', + b: 'string', + }), + ); + assert.deepStrictEqual(mixed.messages, [ + 'Field "a" removed from position 0: "b" and "c" move up; keep the slot of a retired field, as ?any', + 'Field "c" moved from position 2 to 0', + 'Field "d" removed from position 3; keep the slot of a retired field, as ?any', + ]); +}); + +test("Evolve: rule 8, a stored kind's id moves when a field is appended", () => { + const entity = (fields) => model({ User: { Entity: {}, ...fields } }).User; + const one = entity({ name: 'string' }); + assert.strictEqual(one.toString(), '{name:string,userId:?string}'); + const moved = found(one, entity({ name: 'string', email: '?string' })); + assert.deepStrictEqual(moved.list, [['email', 'inserted', 'error']]); + assert.deepStrictEqual(moved.messages, [ + 'Field "email" inserted at position 1: "userId" moves further ' + + '(rule 8: declare "userId" in the definition and append after it)', + ]); + // Declared in the definition, the id keeps its place. + const kept = entity({ name: 'string', userId: '?string' }); + assert.deepStrictEqual( + compatible(kept, entity({ name: 'string', userId: '?string', email: '?string' })), + { + ok: true, + problems: [], + }, + ); + // The hint is for an id only. + const plain = found( + define({ name: 'string', tail: '?string' }), + define({ name: 'string', x: '?string', tail: '?string' }), + ); + assert.deepStrictEqual(plain.messages, [ + 'Field "x" inserted at position 1: "tail" moves further', + ]); +}); + +test('Evolve: required, optional, retired and defaults', () => { + const base = define({ a: 'string', b: '?string', n: { type: 'number', default: 1 } }); + const required = found(base, define({ a: 'string', b: 'string', n: 'number', c: 'string' })); + assert.deepStrictEqual(required.list, [ + ['b', 'required', 'error'], + ['n', 'required', 'error'], + ['n', 'default', 'warning'], + ['c', 'required', 'error'], + ]); + assert.deepStrictEqual(required.messages, [ + 'Required now, where older payloads may send null', + 'Required now, where older payloads may send null', + 'Default removed (1): an empty slot in older payloads decodes as null', + 'Field "c" appended as required without a default: older payloads do not have it', + ]); + const relaxed = found( + base, + define({ + a: '?string', + b: '?string', + n: { type: 'number', default: 2 }, + c: '?string', + d: 'json', + }), + ); + assert.strictEqual(relaxed.ok, true); + assert.deepStrictEqual(relaxed.list, [ + ['a', 'optional', 'info'], + ['n', 'default', 'warning'], + ]); + assert.strictEqual( + relaxed.messages[1], + 'Default changed from 1 to 2: an empty slot in older payloads decodes differently', + ); + // A field with a default is optional: appended, it is no problem. + const defaulted = found( + base, + define({ + a: 'string', + b: '?string', + n: { type: 'number', default: 1 }, + s: { type: 'string', default: '' }, + }), + ); + assert.deepStrictEqual(defaulted.list, []); + // Retired as ?any (anything is read as is), or as null (a warning). + const any = found(base, define({ a: '?any', b: 'json', n: 'unknown' })); + assert.deepStrictEqual(any.list, [ + ['a', 'retired', 'info'], + ['b', 'retired', 'info'], + ['n', 'retired', 'info'], + ]); + assert.strictEqual(any.messages[0], 'Retired as ?any: whatever older writers send is read as is'); + const nulled = found( + base, + define({ a: 'null', b: '?string', n: { type: 'number', default: 1 } }), + ); + assert.deepStrictEqual(nulled.list, [['a', 'retired', 'warning']]); + assert.strictEqual(nulled.ok, true); + assert.strictEqual( + found(base, define({ a: 'null', b: '?string', n: { type: 'number', default: 1 } }), { + strict: true, + }).ok, + false, + ); + // A retired slot taken for a value again is a type change (and a required + // one where it was never required); null to any is a retirement. + const reused = found(define({ a: '?any', b: 'null' }), define({ a: 'string', b: '?any' })); + assert.deepStrictEqual(reused.list, [ + ['a', 'required', 'error'], + ['a', 'type', 'error'], + ['b', 'retired', 'info'], + ]); + assert.strictEqual(reused.messages[1], 'Type changed: json → string'); + assert.deepStrictEqual(found(define({ a: 'null' }), define({ a: 'null' })).list, []); +}); + +test('Evolve: types and the wire', () => { + const widened = found( + define({ a: 'integer', b: 'string', c: { type: 'number', decimals: 2 } }), + define({ a: 'number', b: { enum: ['x'] }, c: { type: 'number', decimals: 3 } }), + ); + assert.deepStrictEqual(widened.list, [ + ['a', 'type', 'error'], + ['b', 'type', 'error'], + ['c', 'type', 'error'], + ]); + assert.deepStrictEqual(widened.messages, [ + 'Type changed: integer → number', + 'Type changed: string → enum(x)', + 'Type changed: number(2) → number(3)', + ]); + const scalar = define({ a: 'string' }); + const record = define({ a: { b: 'string' } }); + assert.deepStrictEqual(found(scalar, record).messages, ['Type changed: string → {b:string}']); + assert.deepStrictEqual(found(record, scalar).list, [['a', 'type', 'error']]); + // The markers of the wire are not the type: switching them is the wire rule. + const plain = define({ a: 'string', ids: { array: 'integer' }, at: 'date' }); + const interned = define( + { a: 'string', ids: { array: 'integer' }, at: 'date' }, + { intern: ['a'] }, + ); + const deltas = define({ a: 'string', ids: { array: 'integer' }, at: 'date' }, { delta: ['ids'] }); + assert.deepStrictEqual(found(plain, interned).list, [['a', 'wire', 'error']]); + assert.deepStrictEqual(found(interned, plain).messages, [ + 'The wire changed: intern switched off', + ]); + assert.deepStrictEqual(found(plain, deltas).messages, ['The wire changed: delta switched on']); + assert.deepStrictEqual(found(deltas, plain).messages, ['The wire changed: delta switched off']); + assert.deepStrictEqual(found(deltas, deltas).list, []); + const [problem] = compatible(plain, deltas).problems; + assert.deepStrictEqual([problem.previous, problem.next], ['[integer]', '~[integer]']); + // A list of differences keeps its item under watch. + const dates = define({ a: 'string', ids: { array: 'date' }, at: 'date' }, { delta: ['ids'] }); + assert.deepStrictEqual(found(deltas, dates).list, [['ids[]', 'type', 'error']]); + // A column of differences, and a sparse record. + const columns = define( + { rows: { array: { id: 'integer', v: 'number' } } }, + { layout: { rows: 'columns' } }, + ); + const delta = define( + { rows: { array: { id: 'integer', v: 'number' } } }, + { layout: { rows: 'columns' }, delta: ['rows[].id'] }, + ); + assert.deepStrictEqual(found(columns, delta).list, [['rows[].id', 'wire', 'error']]); + assert.deepStrictEqual( + found(define({ a: 'string' }), define({ a: 'string' }, { sparse: true })).messages, + ['The wire changed: sparse switched on'], + ); + const sparse = found(define({ a: 'string' }, { sparse: true }), define({ a: 'string' })); + assert.deepStrictEqual(sparse.list, [['', 'wire', 'error']]); + assert.strictEqual(sparse.messages[0], 'The wire changed: sparse switched off'); +}); + +test('Evolve: collections, layouts and dictionaries', () => { + const rows = define({ items: { array: { sku: 'string', qty: '?number' } } }); + const flat = define( + { items: { array: { sku: 'string', qty: '?number' } } }, + { layout: { items: 'flat' } }, + ); + assert.deepStrictEqual(found(rows, flat).messages, ['The wire changed: layout rows → flat']); + // A flat record changes its stride with every field, optional too. + const wider = define( + { items: { array: { sku: 'string', qty: '?number', note: '?string' } } }, + { layout: { items: 'flat' } }, + ); + const widened = found(flat, wider); + assert.deepStrictEqual(widened.list, [['items', 'flat', 'error']]); + assert.strictEqual( + widened.messages[0], + 'A flat record has 3 fields instead of 2: the stride of the collection changed', + ); + // A column appended is fine; a row collection grows like a record. + const cols = (fields) => define({ array: fields }, { layout: 'columns' }); + assert.deepStrictEqual( + compatible(cols({ id: 'integer' }), cols({ id: 'integer', v: '?number' })), + { ok: true, problems: [] }, + ); + assert.deepStrictEqual( + found(rows, define({ items: { array: { sku: 'string', qty: '?number', note: '?string' } } })) + .list, + [], + ); + assert.deepStrictEqual( + found(rows, define({ items: { array: { sku: 'integer', qty: '?number' } } })).list, + [['items[].sku', 'type', 'error']], + ); + // A Set for an array is another type, with the same items. + const set = found(define({ tags: { array: 'string' } }), define({ tags: { set: 'string' } })); + assert.deepStrictEqual(set.messages, ['Type changed: [string] → set[string]']); + // Items and values that become required, under their paths. + const loose = define({ + list: { array: '?string' }, + dict: { object: { string: '?number' } }, + map: { map: { string: '?number' } }, + }); + const tight = define({ + list: { array: 'string' }, + dict: { object: { string: 'number' } }, + map: { map: { string: 'number' } }, + }); + assert.deepStrictEqual(found(loose, tight).list, [ + ['list[]', 'required', 'error'], + ['dict{}', 'required', 'error'], + ['map{}', 'required', 'error'], + ]); + const keyed = found( + tight, + define({ + list: { array: 'string' }, + dict: { object: { string: 'number' } }, + map: { map: { number: 'number' } }, + }), + ); + assert.deepStrictEqual(keyed.messages, ['Type changed: map → map']); + assert.deepStrictEqual( + found( + tight, + define({ + list: { array: 'string' }, + dict: { object: { string: 'string' } }, + map: { map: { string: 'number' } }, + }), + ).list, + [['dict{}', 'type', 'error']], + ); +}); + +test('Evolve: enums, flags and unions', () => { + const flags = (names) => define({ perms: { flags: names } }); + const grown = found(flags(['read', 'write']), flags(['read', 'write', 'admin'])); + assert.deepStrictEqual(grown.list, [['perms', 'appended', 'warning']]); + assert.strictEqual(grown.messages[0], 'Flag "admin" appended: deploy decoders before encoders'); + assert.deepStrictEqual(found(flags(['read', 'write']), flags(['read', 'all'])).messages, [ + 'Flag "write" is now "all"', + ]); + assert.deepStrictEqual(found(flags(['read', 'write']), flags(['write'])).messages, [ + 'Flag "read" removed: "write" moves up', + ]); + const numbers = found(define({ e: { enum: [1, 2] } }), define({ e: { enum: [1, '2', 3] } })); + assert.deepStrictEqual(numbers.messages, [ + 'Enum value 2 is now "2"', + 'Enum value 3 appended: deploy decoders before encoders', + ]); + const union = (branches, key = 'kind') => + define({ packet: { union: branches, discriminator: key } }); + const call = { kind: { enum: ['call'] }, args: { array: 'string' } }; + const event = { kind: { enum: ['event', 'signal'] }, name: 'string' }; + const two = union([call, event]); + // A branch appended is its values appended. + const added = found(two, union([call, event, { kind: { enum: ['ping'] } }])); + assert.deepStrictEqual(added.list, [['packet', 'appended', 'warning']]); + assert.strictEqual( + added.messages[0], + 'Union value "ping" appended: deploy decoders before encoders', + ); + // A value into any branch but the last moves every later tag. + const inserted = found( + two, + union([{ kind: { enum: ['call', 'cast'] }, args: { array: 'string' } }, event]), + ); + assert.deepStrictEqual(inserted.list, [['packet', 'inserted', 'error']]); + assert.strictEqual( + inserted.messages[0], + 'Union value "cast" inserted before "event", which moves', + ); + // The branches compare as records, under the branch's index. + const changed = found( + two, + union([ + { kind: { enum: ['call'] }, args: { array: 'integer' } }, + { ...event, at: '?date' }, + ]), + ); + assert.deepStrictEqual(changed.list, [['packet[0].args[]', 'type', 'error']]); + assert.deepStrictEqual( + found(two, union([call, { kind: { enum: ['event', 'signal'] }, name: 'string', at: 'date' }])) + .list, + [['packet[1].at', 'required', 'error']], + ); + // A branch whose values all went has nothing to compare with; one split in + // two compares with both. + assert.deepStrictEqual(found(two, union([call])).list, [ + ['packet', 'removed', 'error'], + ['packet', 'removed', 'error'], + ]); + const split = union([ + call, + { kind: { enum: ['event'] }, name: 'string' }, + { kind: { enum: ['signal'] }, name: 'string', x: 'number' }, + ]); + assert.deepStrictEqual(found(two, split).list, [['packet[2].x', 'required', 'error']]); + // The discriminator is the union's. + const { kind: callKind, ...callRest } = call; + const { kind: eventKind, ...eventRest } = event; + const renamed = found( + two, + union( + [ + { tag: callKind, ...callRest }, + { tag: eventKind, ...eventRest }, + ], + 'tag', + ), + ); + assert.deepStrictEqual(renamed.list, [['packet', 'type', 'error']]); + assert.strictEqual(renamed.messages[0], 'The union discriminates on "tag" instead of "kind"'); +}); + +test('Evolve: tuples and root scalars', () => { + const pair = define({ pair: { tuple: ['string', '?number'] } }); + assert.deepStrictEqual( + found(pair, define({ pair: { tuple: ['string', '?number', '?boolean'] } })).list, + [], + ); + const longer = found(pair, define({ pair: { tuple: ['string', 'number', 'boolean'] } })); + assert.deepStrictEqual(longer.list, [ + ['pair[1]', 'required', 'error'], + ['pair[2]', 'required', 'error'], + ]); + assert.strictEqual( + longer.messages[1], + 'Element appended as required without a default: older payloads do not have it', + ); + const shorter = found(pair, define({ pair: { tuple: ['integer'] } })); + assert.deepStrictEqual(shorter.list, [ + ['pair[0]', 'type', 'error'], + ['pair[1]', 'removed', 'error'], + ]); + assert.strictEqual(shorter.messages[1], 'Element 1 removed'); + assert.deepStrictEqual( + found( + define({ pair: { tuple: ['string', { tuple: ['number'] }] } }), + define({ pair: { tuple: ['string', { tuple: ['string'] }] } }), + ).list, + [['pair[1][0]', 'type', 'error']], + ); + // A root scalar keeps its own optionality. + assert.deepStrictEqual(found(define('string'), define('?string')).list, [ + ['', 'optional', 'info'], + ]); + assert.deepStrictEqual(found(define('?string'), define('string')).list, [ + ['', 'required', 'error'], + ]); + assert.deepStrictEqual(found(define('string'), define('integer')).messages, [ + 'Type changed: string → integer', + ]); + assert.deepStrictEqual(found(define('string'), define({ array: 'string' })).list, [ + ['', 'type', 'error'], + ]); + assert.deepStrictEqual(found(define({ array: 'string' }), define({ array: 'integer' })).list, [ + ['[]', 'type', 'error'], + ]); +}); + +test('Evolve: references, recursion and entities renamed', () => { + const entities = (address, rest = {}) => + model({ Address: { Struct: {}, ...address }, User: { Struct: {}, home: 'Address', ...rest } }); + const one = entities({ city: 'string' }); + const two = entities({ city: 'string', zip: '?string' }); + assert.deepStrictEqual(compatible(one.User, two.User), { ok: true, problems: [] }); + // What changed in the entity is reported where it is reached, once. + const changed = entities({ city: 'integer' }, { work: '?Address' }); + assert.deepStrictEqual(found(one.User, changed.User).list, [['home.city', 'type', 'error']]); + const renamed = model({ + Location: { Struct: {}, city: 'string' }, + User: { Struct: {}, home: 'Location' }, + }); + const moved = found(one.User, renamed.User); + assert.deepStrictEqual(moved.list, [['home', 'renamed', 'info']]); + assert.strictEqual(moved.messages[0], 'Entity "Address" is now "Location"'); + assert.deepStrictEqual(compatible(one.User, renamed.User).problems[0].previous, '@Address'); + // A record that becomes an id (a stored kind), or the reverse. + const stored = model({ + Address: { Entity: {}, city: 'string' }, + User: { Struct: {}, home: 'Address' }, + }); + assert.deepStrictEqual(found(one.User, stored.User).messages, [ + 'Type changed: @Address → string', + ]); + assert.deepStrictEqual(found(stored.User, one.User).messages, [ + 'Type changed: string → @Address', + ]); + // A recursion compares once per pair of plans. + const tree = (extra) => named('Tree', { v: 'number', kids: { array: 'Tree' }, ...extra }); + assert.deepStrictEqual(compatible(tree({}), tree({ x: '?string' })), { ok: true, problems: [] }); + assert.deepStrictEqual(found(tree({}), tree({ x: 'string' })).list, [['x', 'required', 'error']]); + // An alias (an entity whose root is a reference) stands for the entity. + const alias = (city) => + model({ + Address: { Struct: {}, city }, + Home: { Struct: {}, home: 'Address' }, + Place: { type: 'Home' }, + }); + assert.deepStrictEqual(found(alias('string').Place, alias('integer').Place).list, [ + ['home.city', 'type', 'error'], + ]); + // A reference to a scalar entity keeps the entity's own optionality. + const scalarEntity = (type) => model({ Name: { type }, User: { Struct: {}, name: 'Name' } }); + assert.deepStrictEqual(found(scalarEntity('string').User, scalarEntity('?string').User).list, [ + ['name', 'optional', 'info'], + ]); +}); + +test('Evolve: what cannot be compared', () => { + for (const [previous, next] of [ + [1, V1], + [V1, null], + [{}, V2], + ]) { + assert.throws(() => compatible(previous, next), { + code: 'ERR_SCHEMA_EXPECTED', + message: 'Schema instance expected', + }); + } + const resolved = resolve(V1, V2); + for (const [previous, next] of [ + [resolved, V2], + [V1, resolved], + ]) { + assert.throws(() => compatible(previous, next), { + code: 'ERR_SCHEMA_EXPECTED', + message: 'A resolved schema is no version to compare: give its writer or its reader', + }); + } + assert.throws(() => compatible(V1, V2, 1), { + code: 'ERR_INVALID_OPTIONS', + message: 'Invalid options: expected an object', + }); + assert.throws(() => compatible(V1, V2, { aliases: {} }), { + code: 'ERR_INVALID_OPTIONS', + message: 'Invalid option "aliases"', + }); + assert.throws(() => compatible(V1, V2, { strict: 1 }), { + code: 'ERR_INVALID_OPTIONS', + message: 'Invalid option "strict"', + }); + assert.deepStrictEqual(compatible(V1, V2, { strict: undefined }), { ok: true, problems: [] }); + // A chain of references too long for the stack is an error, not a RangeError. + const chain = (extra) => { + const entities = {}; + for (let i = 0; i < 10000; i++) { + entities[`E${i}`] = { + fields: { n: { type: `E${i + 1}`, one: `E${i + 1}`, required: false } }, + }; + } + entities.E10000 = { fields: { v: { type: 'number' }, ...extra } }; + return Schema.from({ format: 1, root: 'E0', entities }, { codegen: false }); + }; + assert.throws(() => compatible(chain({}), chain({ w: { type: 'string', required: false } })), { + code: 'ERR_INCOMPATIBLE', + message: 'The schemas are nested too deeply to compare', + }); + // Another error of the walk passes through as it is. + const PLAN = Symbol.for('alexify.protoarray.plan'); + const odd = define({ a: 'string' }); + const fake = Object.create(odd); + Object.defineProperty(fake, 'toString', { value: () => '{b:string}' }); + Object.defineProperty(fake, PLAN, { + value: { + kind: 'struct', + sparse: false, + get fields() { + throw new TypeError('not a plan'); + }, + }, + }); + assert.throws(() => compatible(odd, fake), { message: 'not a plan' }); +}); + +test('Evolve: a baseline writes the schemas as JSON and restore builds them again', () => { + const schemas = model( + { + Address: { Struct: {}, city: 'string', zip: '?string' }, + User: { Entity: {}, name: 'string', home: 'Address', items: { array: { sku: 'string' } } }, + }, + { + references: 'embed', + entities: { + Address: { intern: ['city'] }, + User: { sparse: true, layout: { items: 'columns' } }, + }, + }, + ); + const written = baseline({ User: schemas.User, Address: schemas.Address, Ping: V1 }); + const text = JSON.stringify(written); + const parsed = JSON.parse(text); + // One snapshot for the two entities of the model, another for V1. + assert.strictEqual(parsed.format, 1); + assert.strictEqual(parsed.snapshots.length, 2); + assert.deepStrictEqual(Object.keys(parsed.schemas), ['User', 'Address', 'Ping']); + assert.deepStrictEqual(parsed.schemas.User, { + snapshot: 0, + entity: 'User', + options: { + sparse: true, + layout: { items: 'columns' }, + intern: false, + delta: [], + references: 'embed', + types: {}, + entities: { + Address: { + sparse: false, + layout: 'rows', + intern: ['city'], + delta: [], + references: 'embed', + types: {}, + }, + }, + }, + canonical: schemas.User.toString(), + id: schemas.User.id, + }); + assert.strictEqual(parsed.schemas.Address.snapshot, 0); + assert.deepStrictEqual(parsed.schemas.Ping, { + snapshot: 1, + entity: 'User', + options: { + sparse: false, + layout: 'rows', + intern: false, + delta: [], + references: 'kind', + types: {}, + entities: {}, + }, + canonical: V1.toString(), + id: V1.id, + }); + const restored = restore(parsed); + assert.deepStrictEqual(Object.keys(restored), ['User', 'Address', 'Ping']); + for (const name of Object.keys(restored)) { + const original = { User: schemas.User, Address: schemas.Address, Ping: V1 }[name]; + assert.strictEqual(restored[name].toString(), original.toString()); + assert.strictEqual(restored[name].id, original.id); + assert.strictEqual(restored[name].name, name); + assert.deepStrictEqual(compatible(original, restored[name]), { ok: true, problems: [] }); + } + // The restored schemas work: the wire is the original's. + const user = { name: 'A', home: { city: 'K', zip: null }, items: [{ sku: 'x' }], userId: '1' }; + assert.deepStrictEqual(restored.User.encode(user), schemas.User.encode(user)); + // An anonymous schema has the entity '' of its snapshot. + const anonymous = define({ n: 'integer' }, { delta: [] }); + const entry = baseline({ one: anonymous }).schemas.one; + assert.strictEqual(entry.entity, ''); + assert.strictEqual(restore(baseline({ one: anonymous })).one.toString(), '{n:integer}'); + // The baseline writes the wire options of the snapshot's custom types too. + const typed = Schema.from( + { + format: 1, + root: 'T', + types: { money: { kind: 'scalar' } }, + entities: { T: { fields: { m: { type: 'money' } } } }, + }, + { types: { money: 'number' } }, + ); + const copy = restore(baseline({ T: typed })).T; + assert.strictEqual(copy.toString(), '{m:number}'); +}); + +test('Evolve: a baseline with a fingerprint of its own, and what restore refuses', () => { + const sha = (signature) => `x${signature.length}`; + const custom = define({ a: 'string' }, { fingerprint: sha }); + const written = JSON.parse(JSON.stringify(baseline({ A: custom }))); + // A function cannot travel: null says so, and the id is recorded, not checked. + assert.strictEqual(written.schemas.A.options.fingerprint, null); + assert.strictEqual(written.schemas.A.id, 'x10'); + const restored = restore(written); + assert.strictEqual(restored.A.toString(), '{a:string}'); + assert.notStrictEqual(restored.A.id, 'x10'); + // The entities a schema reaches carry theirs the same way. + const entities = model( + { Address: { Struct: {}, city: 'string' }, User: { Struct: {}, home: 'Address' } }, + { fingerprint: sha }, + ); + const reached = JSON.parse(JSON.stringify(baseline({ User: entities.User }))); + assert.strictEqual(reached.schemas.User.options.entities.Address.fingerprint, null); + assert.strictEqual(restore(reached).User.toString(), '{home:@Address}'); + // The default fingerprint names none, and the id is checked. + const plain = JSON.parse(JSON.stringify(baseline({ A: define({ a: 'string' }) }))); + assert.strictEqual(plain.schemas.A.options.fingerprint, undefined); + assert.strictEqual(restore(plain).A.id, define({ a: 'string' }).id); + const forged = (change) => { + const copy = JSON.parse(JSON.stringify(plain)); + change(copy); + return copy; + }; + const mismatch = { + code: 'ERR_INVALID_SNAPSHOT', + message: 'The baseline does not rebuild to its canonical form: write it again', + schema: 'A', + }; + assert.throws( + () => restore(forged((copy) => (copy.schemas.A.canonical = '{a:number}'))), + mismatch, + ); + assert.throws(() => restore(forged((copy) => (copy.schemas.A.id = '00000000'))), mismatch); + // A baseline written by hand may leave the entities out. + const bare = forged((copy) => delete copy.schemas.A.options.entities); + assert.strictEqual(restore(bare).A.id, plain.schemas.A.id); + // A fingerprint by name is given to the schema (none is built in yet). + assert.throws(() => restore(forged((copy) => (copy.schemas.A.options.fingerprint = 'sha1'))), { + code: 'ERR_INVALID_OPTIONS', + }); + for (const [source, message] of [ + [null, 'A baseline of format 1 with snapshots and schemas expected'], + [ + { format: 2, snapshots: [], schemas: {} }, + 'A baseline of format 1 with snapshots and schemas expected', + ], + [ + { format: 1, snapshots: {}, schemas: {} }, + 'A baseline of format 1 with snapshots and schemas expected', + ], + [ + { format: 1, snapshots: [], schemas: [] }, + 'A baseline of format 1 with snapshots and schemas expected', + ], + [ + { format: 1, snapshots: [], schemas: { A: 1 } }, + 'The entry "A" of the baseline is incomplete', + ], + [ + forged((copy) => (copy.schemas.A.options.fingerprint = 1)), + 'The entry "A" of the baseline is incomplete', + ], + [forged((copy) => delete copy.schemas.A.id), 'The entry "A" of the baseline is incomplete'], + [ + forged((copy) => (copy.schemas.A.snapshot = 3)), + 'The entry "A" names no snapshot of the baseline', + ], + ]) { + assert.throws(() => restore(source), { code: 'ERR_INVALID_SNAPSHOT', message }); + } + // A snapshot that does not read, and options that do not, are the schema's errors. + assert.throws(() => restore(forged((copy) => (copy.snapshots[0].entities = {}))), { + code: 'ERR_INVALID_SNAPSHOT', + message: 'The snapshot has no entity ""', + }); + assert.throws(() => restore(forged((copy) => (copy.schemas.A.options.sparse = 1))), { + code: 'ERR_INVALID_OPTIONS', + }); +}); + +test('Evolve: what baseline refuses', () => { + assert.throws(() => baseline([V1]), { + code: 'ERR_SCHEMA_EXPECTED', + message: 'A record of schemas by name expected', + }); + assert.throws(() => baseline({ A: 1 }), { + code: 'ERR_SCHEMA_EXPECTED', + message: '"A" is not a Schema', + }); + for (const schema of [V1.list, resolve(V1, V2)]) { + assert.throws(() => baseline({ A: schema }), { + code: 'ERR_SCHEMA_EXPECTED', + message: '"A" was not read from a snapshot: give a schema of its own', + }); + } + assert.deepStrictEqual(baseline({}), { format: 1, snapshots: [], schemas: {} }); +}); + +test('Evolve: register() installs Schema.compatible', () => { + assert.strictEqual(Schema.compatible, undefined); + register(); + register(); + assert.strictEqual(Schema.compatible, compatible); + assert.strictEqual(Object.getOwnPropertyDescriptor(Schema, 'compatible').enumerable, false); + assert.deepStrictEqual(Schema.compatible(V1, V2), { ok: true, problems: [] }); + // The metaschema module in the tests is the one injected by the support file. + assert.strictEqual(typeof ms.Schema, 'function'); +}); diff --git a/tests/unit/export-parity.test.js b/tests/unit/export-parity.test.js index 5e3d80f..0b25342 100644 --- a/tests/unit/export-parity.test.js +++ b/tests/unit/export-parity.test.js @@ -24,6 +24,7 @@ const entryPoints = [ ['stream.js', 'stream.d.ts'], ['resolve.js', 'resolve.d.ts'], ['standalone.js', 'standalone.d.ts'], + ['evolve.js', 'evolve.d.ts'], ]; // Every subpath of the exports map, with the typings it declares. @@ -33,6 +34,7 @@ const SUBPATHS = [ ['@alexify/protoarray/stream', 'stream.d.ts'], ['@alexify/protoarray/resolve', 'resolve.d.ts'], ['@alexify/protoarray/standalone', 'standalone.d.ts'], + ['@alexify/protoarray/evolve', 'evolve.d.ts'], ]; for (const [runtimeFile, dtsFile] of entryPoints) { From aacfedaa748932d920ced994dcf90a921ca0c5f1 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Tue, 20 Oct 2026 22:51:40 +0300 Subject: [PATCH 03/17] feat(schema): built-in 64-bit fingerprint, fingerprint: 'fnv1a32' | 'fnv1a64' The `fingerprint` option names the hash `id` is of the signature: 'fnv1a32' (the default, unchanged: eight hex digits, the same ids as before) or 'fnv1a64', the 64-bit FNV-1a over the same UTF-16 code units as sixteen hex digits, for a registry of many schemas, where 32 bits collide. A function of the signature is accepted as before; anything else is ERR_INVALID_OPTIONS. The name travels where a function could not: schema.list, the entities references reach and a resolved schema hash with it, a standalone module writes it into its options (the list of a module's schema hashed with the default before), and ERR_ID_COLLISION names it as the remedy. Docs: evolution.md gains the birthday bound and a table of collision probabilities for 32 and 64 bits ("A wider fingerprint"), with the other pages, the feature matrix, README and the CHANGELOG updated. Canonical forms, signatures and golden-1x are untouched. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 12 ++++++++ README.md | 10 +++--- docs/api/exports.md | 2 +- docs/comparison.json | 2 +- docs/guide/architecture.md | 3 +- docs/guide/comparison.md | 2 +- docs/guide/errors.md | 4 +-- docs/guide/evolution.md | 48 +++++++++++++++++++++-------- docs/guide/resolve.md | 5 +-- docs/guide/schema.md | 7 +++-- docs/guide/standalone.md | 8 +++-- docs/guide/wire-format.md | 5 +-- index.d.ts | 10 +++--- src/plan.js | 56 +++++++++++++++++++++++++-------- src/registry.js | 3 +- src/schema.js | 8 +++-- src/standalone.js | 12 ++++++-- tests/types/index.test-d.ts | 3 ++ tests/unit/registry.test.js | 7 +++-- tests/unit/resolve.test.js | 10 ++++++ tests/unit/schema.test.js | 58 +++++++++++++++++++++++++++++++++++ tests/unit/standalone.test.js | 27 ++++++++++++++++ 22 files changed, 244 insertions(+), 58 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c2be09..402f16a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **A built-in 64-bit fingerprint.** The `fingerprint` option names the hash `id` is of the + signature: `'fnv1a32'`, the default, unchanged (eight hex digits, the same ids as before), or + `'fnv1a64'`, the 64-bit FNV-1a over the same UTF-16 code units as sixteen hex digits, for a + registry of many schemas, where 32 bits collide (two of 65,536 schemas share an id with + probability 39%; one in 10¹⁰ with 64 bits). The name travels where a function cannot: + `schema.list`, the entities references reach and a resolved schema hash with it, a standalone + module writes it (the `list` of a module's schema hashed with the default before), and + `ERR_ID_COLLISION` names it as the remedy. A function of the signature is accepted as before; + anything else is `ERR_INVALID_OPTIONS` ("fnv1a32, fnv1a64 or a function"). + ## [2.0.0] - 2026-10-16 2.0 hands the definition of a schema to `@alexify/metaschema` and keeps the wire to protoarray. diff --git a/README.md b/README.md index a76c7e2..1fb2e44 100644 --- a/README.md +++ b/README.md @@ -77,9 +77,9 @@ const User = Schema.from(await (await fetch('/schemas/user')).json()); validates the values with it, protoarray carries them, and both read a reference, an optional field or a stored entity's id the same way. - **Evolves safely.** Append fields at the end and old and new versions keep understanding each - other; a fingerprint (`schema.id`) tells versions apart in a handshake. For changes that break - the rules, `resolve` reads and writes another version by name, Avro-style, and a registry - wraps payloads as `[id, payload]`. + other; a fingerprint (`schema.id`, 32 or 64 bits) tells versions apart in a handshake. For + changes that break the rules, `resolve` reads and writes another version by name, Avro-style, + and a registry wraps payloads as `[id, payload]`. - **Typed.** A schema takes its TypeScript types from the metaschema schema it is built from; hand-written typings for TypeScript 5.0 or newer. - **Zero dependencies, about 22 KB min+gzip.** protoarray requires nothing, metaschema included: @@ -122,7 +122,7 @@ protoarray alone. | Streaming | `encodeEach`/`decodeEach`; NDJSON, `TransformStream`s and the items of a large array read as they arrive (`@alexify/protoarray/stream`) | | Standalone modules | `protoarray compile` writes the generated code of your schemas into a module, for a strict CSP and edge runtimes, with no metaschema at runtime (`@alexify/protoarray/standalone`) | | Validation | the wire on decode by default: `DecodeError` with codes, paths like `orders[1].sku` and the schema name; the values before encode with metaschema's `check` and `parse` | -| Evolution | append-only positions; `{ strict: true }` rejects elements beyond the schema; `schema.id` or your own `fingerprint` tells versions apart | +| Evolution | append-only positions; `{ strict: true }` rejects elements beyond the schema; `schema.id` tells versions apart, 32 bits by default, `{ fingerprint: 'fnv1a64' }` for a registry of many schemas, or your own function | See the [guide](https://protoarray.vercel.app/guide/schema) for every form, and [Migrating from 1.x](https://protoarray.vercel.app/guide/migrating-from-1) for the upgrade: the @@ -412,7 +412,7 @@ table shows.
Add fields✅ append at the end; trailing nulls trimmed✅ no schema✅ keyed JSON: add properties✅ no schema✅ no schema; keys travel✅ no schema; keys in the header✅ no schema; keys travel✅ new tag numbers✅ new tag numbers✅ new tag numbers✅ with
default, via createResolver✅ new field ids; old readers skip them✅ append fields or explicit (id: n)✅ new @n ordinals; old data reads defaults✅ messages: new field indices✅ JSON: names travel; protobuf: new Sequence<N>❌ new key shifts sorted positions✅ no schema; keys travel✅ no schema; keys travel✅ no schema; keys travel✅ no schema; keys travel✅ no schema✅ no schema Rename a field✅ positions travel; aliases in resolve❌ names travel❌ names travel❌ names travel❌ names travel❌ names travel❌ names travel✅ tag numbers travel, not names✅ tag numbers travel, not names✅ tag numbers travel, not names✅ aliases in resolution; names not on wire✅ field ids travel, names do not✅ slots travel, names do not✅ names do not travel✅ indices and positions travel⚠️ protobuf with Sequence<N>; JSON names travel⚠️ positions travel, but sorted by key❌ names travel❌ names travel❌ names travel (keyMap maps names to numbers)❌ names travel❌ names travel❌ names travel Reader and writer schema resolution✅ resolve(writer, reader) (./resolve)— schemaless❌ none: decode is JSON.parse— schemaless— schemaless— schemaless— schemaless⚠️ tag matching only; no resolver step⚠️ tag matching only; no resolver step⚠️ tag matching only; no resolver step✅ reader.createResolver(writer): aliases, defaults, promotions⚠️ by field id; a mismatched type is skipped❌ none; append-only rules instead❌ none; layout rules instead❌ none❌ none❌ none— no schema— no schema— no schema— no schema— schemaless— schemaless -Schema id or fingerprint✅ schema.id, FNV-1a of the canonical form— schemaless❌ none ($id is a name)— schemaless— schemaless— schemaless— schemaless❌ none❌ none❌ none✅ type.fingerprint(): MD5 of canonical form❌ none in JS⚠️ 4-char file_identifier, set by hand⚠️ 64-bit type ids, stable across versions⚠️ @opcode 4-byte id, by hand❌ none❌ none❌ none for shared structures— no schema❌ none for shared structures— no schema— schemaless— schemaless +Schema id or fingerprint✅ schema.id, FNV-1a (32 or 64 bits) of the canonical form— schemaless❌ none ($id is a name)— schemaless— schemaless— schemaless— schemaless❌ none❌ none❌ none✅ type.fingerprint(): MD5 of canonical form❌ none in JS⚠️ 4-char file_identifier, set by hand⚠️ 64-bit type ids, stable across versions⚠️ @opcode 4-byte id, by hand❌ none❌ none❌ none for shared structures— no schema❌ none for shared structures— no schema— schemaless— schemaless Registry or envelope with the schema id✅ SchemaRegistry: [id, payload] (./registry)— schemaless❌ none— schemaless— schemaless— schemaless— schemaless⚠️ Any type URL; Confluent via third party⚠️ Any type URL; Confluent via @confluentinc/schemaregistry⚠️ google.protobuf.Any type URL only⚠️ container files embed schema; registry third-party❌ none (RPC envelope names the method)⚠️ file_identifier in the buffer header❌ segment framing carries no type id❌ none (opcode written by hand)❌ none❌ none (README: add a uint8 field by hand)❌ record ids only; structures kept out of band— no schema❌ record ids only; structures kept out of band— no schema— schemaless⚠️ shared symbol tables by name/version (Catalog) Unknown fields from a newer writer✅ trailing extras ignored, or rejected with strict— schemaless✅ JSON.parse keeps them; encode drops undeclared— schemaless— schemaless— schemaless— schemaless✅ dropped by default; discardUnknown = false keeps✅ kept in $unknown, re-encoded✅ dropped by default; unknownFields=true keeps⚠️ skipped by the resolver; not kept⚠️ skipped (input.skip), not kept⚠️ ignored; lost on unpack() → pack()✅ kept in the buffer; copyFrom copies them⚠️ ignored: skips to the message end✅ ignored; assertEquals rejects, assertPrune strips⚠️ trailing bytes ignored; no strict mode— no schema— no schema— no schema— no schema— schemaless— schemaless Decoding untrusted input diff --git a/docs/api/exports.md b/docs/api/exports.md index 9238806..c41ff5e 100644 --- a/docs/api/exports.md +++ b/docs/api/exports.md @@ -77,7 +77,7 @@ interface SchemaOptions { intern?: boolean | readonly string[]; // every string, or the strings at these paths: [table, body] delta?: readonly string[]; // integer, number, date or decimals lists and columns stringify?: 'auto' | 'builder' | 'native'; // how stringify writes the text - fingerprint?: (signature: string) => string; // computes `id`; default: FNV-1a 32-bit, 8 hex digits + fingerprint?: 'fnv1a32' | 'fnv1a64' | ((signature: string) => string); // what `id` is of the signature: FNV-1a, 8 or 16 hex digits, or a function maxDepth?: number; // levels of a recursion a payload may nest; default: the stack references?: 'kind' | 'embed' | 'id'; // how references travel; 'kind' as metaschema's check reads them types?: Readonly>; // the wire type of a custom type with a check of its own diff --git a/docs/comparison.json b/docs/comparison.json index c925941..3e58def 100644 --- a/docs/comparison.json +++ b/docs/comparison.json @@ -400,7 +400,7 @@ }, "fingerprint": { "mark": "✅", - "text": "`schema.id`, FNV-1a of the canonical form" + "text": "`schema.id`, FNV-1a (32 or 64 bits) of the canonical form" }, "envelope": { "mark": "✅", diff --git a/docs/guide/architecture.md b/docs/guide/architecture.md index d7119b2..7e89347 100644 --- a/docs/guide/architecture.md +++ b/docs/guide/architecture.md @@ -158,7 +158,8 @@ intermediate array (see [Runtimes](/guide/runtimes#how-stringify-writes-the-text The plan has a canonical form, a string that describes the wire and is the same for two schemas that write the same one: `schema.toString()`. A schema's **signature** is that form followed by the canonical form of every other entity it reaches by reference, and `schema.id` is its fingerprint -(FNV-1a, or your own `fingerprint` function of it), so a changed referenced entity changes the id. +(FNV-1a, 32 or 64 bits as the `fingerprint` option names, or your own function of it), so a +changed referenced entity changes the id. For a schema without references, these are the strings and ids 1.x computed from its own spelling of the definition. diff --git a/docs/guide/comparison.md b/docs/guide/comparison.md index a187fac..2e96678 100644 --- a/docs/guide/comparison.md +++ b/docs/guide/comparison.md @@ -152,7 +152,7 @@ what each one does. Add fields✅ append at the end; trailing nulls trimmed✅ no schema✅ keyed JSON: add properties✅ no schema✅ no schema; keys travel✅ no schema; keys in the header✅ no schema; keys travel✅ new tag numbers✅ new tag numbers✅ new tag numbers✅ with default, via createResolver✅ new field ids; old readers skip them✅ append fields or explicit (id: n)✅ new @n ordinals; old data reads defaults✅ messages: new field indices✅ JSON: names travel; protobuf: new Sequence<N>❌ new key shifts sorted positions✅ no schema; keys travel✅ no schema; keys travel✅ no schema; keys travel✅ no schema; keys travel✅ no schema✅ no schema Rename a field✅ positions travel; aliases in resolve❌ names travel❌ names travel❌ names travel❌ names travel❌ names travel❌ names travel✅ tag numbers travel, not names✅ tag numbers travel, not names✅ tag numbers travel, not names✅ aliases in resolution; names not on wire✅ field ids travel, names do not✅ slots travel, names do not✅ names do not travel✅ indices and positions travel⚠️ protobuf with Sequence<N>; JSON names travel⚠️ positions travel, but sorted by key❌ names travel❌ names travel❌ names travel (keyMap maps names to numbers)❌ names travel❌ names travel❌ names travel Reader and writer schema resolution✅ resolve(writer, reader) (./resolve)— schemaless❌ none: decode is JSON.parse— schemaless— schemaless— schemaless— schemaless⚠️ tag matching only; no resolver step⚠️ tag matching only; no resolver step⚠️ tag matching only; no resolver step✅ reader.createResolver(writer): aliases, defaults, promotions⚠️ by field id; a mismatched type is skipped❌ none; append-only rules instead❌ none; layout rules instead❌ none❌ none❌ none— no schema— no schema— no schema— no schema— schemaless— schemaless -Schema id or fingerprint✅ schema.id, FNV-1a of the canonical form— schemaless❌ none ($id is a name)— schemaless— schemaless— schemaless— schemaless❌ none❌ none❌ none✅ type.fingerprint(): MD5 of canonical form❌ none in JS⚠️ 4-char file_identifier, set by hand⚠️ 64-bit type ids, stable across versions⚠️ @opcode 4-byte id, by hand❌ none❌ none❌ none for shared structures— no schema❌ none for shared structures— no schema— schemaless— schemaless +Schema id or fingerprint✅ schema.id, FNV-1a (32 or 64 bits) of the canonical form— schemaless❌ none ($id is a name)— schemaless— schemaless— schemaless— schemaless❌ none❌ none❌ none✅ type.fingerprint(): MD5 of canonical form❌ none in JS⚠️ 4-char file_identifier, set by hand⚠️ 64-bit type ids, stable across versions⚠️ @opcode 4-byte id, by hand❌ none❌ none❌ none for shared structures— no schema❌ none for shared structures— no schema— schemaless— schemaless Registry or envelope with the schema id✅ SchemaRegistry: [id, payload] (./registry)— schemaless❌ none— schemaless— schemaless— schemaless— schemaless⚠️ Any type URL; Confluent via third party⚠️ Any type URL; Confluent via @confluentinc/schemaregistry⚠️ google.protobuf.Any type URL only⚠️ container files embed schema; registry third-party❌ none (RPC envelope names the method)⚠️ file_identifier in the buffer header❌ segment framing carries no type id❌ none (opcode written by hand)❌ none❌ none (README: add a uint8 field by hand)❌ record ids only; structures kept out of band— no schema❌ record ids only; structures kept out of band— no schema— schemaless⚠️ shared symbol tables by name/version (Catalog) Unknown fields from a newer writer✅ trailing extras ignored, or rejected with strict— schemaless✅ JSON.parse keeps them; encode drops undeclared— schemaless— schemaless— schemaless— schemaless✅ dropped by default; discardUnknown = false keeps✅ kept in $unknown, re-encoded✅ dropped by default; unknownFields=true keeps⚠️ skipped by the resolver; not kept⚠️ skipped (input.skip), not kept⚠️ ignored; lost on unpack() → pack()✅ kept in the buffer; copyFrom copies them⚠️ ignored: skips to the message end✅ ignored; assertEquals rejects, assertPrune strips⚠️ trailing bytes ignored; no strict mode— no schema— no schema— no schema— no schema— schemaless— schemaless Decoding untrusted input diff --git a/docs/guide/errors.md b/docs/guide/errors.md index e1e10ab..14f61a6 100644 --- a/docs/guide/errors.md +++ b/docs/guide/errors.md @@ -133,10 +133,10 @@ Messages say where: `Unknown type "float" at "price"`. | `ERR_RESERVED_KEY` | `__proto__` as a key | | `ERR_INTEGER_KEY` | a key or a flag name like `'1'`, which JavaScript would order first | | `ERR_INVALID_DEFINITION` | a struct without fields, an empty key | -| `ERR_INVALID_OPTIONS` | an unknown option, a wrong type, a path that names no place of the schema or a place of another kind (`sparse` on a non-struct, a layout on anything but a collection of records, `delta` on a string, `intern` on anything but a string), `layout: 'rows'` on anything but a collection, `sparse` on a union's branch, an option of one schema given to `Schema.fromModel` instead of `entities`, an `entities` key that names no entity of the snapshot or an entity's own `entity`, `entities` or `metaschema`, a snapshot of several entities without `entity`, `entity` other than a metaschema Schema's own, a `fingerprint` that gives no non-empty string, an `aliases` path of `resolve` that names no field of the reader, a `maxLength` that is not a positive integer or `Infinity` | +| `ERR_INVALID_OPTIONS` | an unknown option, a wrong type, a path that names no place of the schema or a place of another kind (`sparse` on a non-struct, a layout on anything but a collection of records, `delta` on a string, `intern` on anything but a string), `layout: 'rows'` on anything but a collection, `sparse` on a union's branch, an option of one schema given to `Schema.fromModel` instead of `entities`, an `entities` key that names no entity of the snapshot or an entity's own `entity`, `entities` or `metaschema`, a snapshot of several entities without `entity`, `entity` other than a metaschema Schema's own, a `fingerprint` that is neither `'fnv1a32'`, `'fnv1a64'` nor a function, or a function that gives no non-empty string, an `aliases` path of `resolve` that names no field of the reader, a `maxLength` that is not a positive integer or `Infinity` | | `ERR_SCHEMA_EXPECTED` | `Schema.from` got neither a metaschema `Schema` or `Model`, a snapshot, a protoarray schema nor a definition with `{ metaschema }`; `encode(schema, …)` and friends got something that is not a `Schema` | | `ERR_CODEGEN_UNAVAILABLE` | `{ codegen: true }` where `new Function` is blocked | | `ERR_COMPILED_MISMATCH` | a generated module's codec is of another schema than the snapshot given with it, or the module was generated by another version of protoarray | | `ERR_INCOMPATIBLE` | `resolve` cannot read the writer as the reader: a narrowing type change, a required reader field the writer lacks, a union split between branches, a chain of references too long for the stack | -| `ERR_ID_COLLISION` | a registry got two schemas of different signatures with one id: the fingerprint is too narrow for it | +| `ERR_ID_COLLISION` | a registry got two schemas of different signatures with one id: the fingerprint is too narrow for it (`{ fingerprint: 'fnv1a64' }`) | | `ERR_UNKNOWN_SCHEMA` | a registry's `encode` got `as` an id it does not know | diff --git a/docs/guide/evolution.md b/docs/guide/evolution.md index 90fbe62..15b93a2 100644 --- a/docs/guide/evolution.md +++ b/docs/guide/evolution.md @@ -91,12 +91,12 @@ pass `{ strict: true }` to `decode` or `parse`: extra elements, values or column ## Telling versions apart -`schema.id` is a 32-bit FNV-1a fingerprint of the schema's signature: its canonical form -(`schema.toString()`: every position, its type and its key, with the layout), followed by the -canonical form of every other entity it references by name (see -[Defining a Schema](/guide/schema#properties)). Two sides whose ids match agree on the wire, the -entities their references reach included: a change to `Address` changes the id of a `User` that -references it, so a registry or a handshake tells the two versions apart. +`schema.id` is an FNV-1a fingerprint of the schema's signature, 32 bits as eight hex digits by +default (`'fnv1a64'` below widens it): its canonical form (`schema.toString()`: every position, +its type and its key, with the layout), followed by the canonical form of every other entity it +references by name (see [Defining a Schema](/guide/schema#properties)). Two sides whose ids match +agree on the wire, the entities their references reach included: a change to `Address` changes +the id of a `User` that references it, so a registry or a handshake tells the two versions apart. Exchange it in a handshake (an HTTP header, the `hello` of an RPC session, a field in an envelope) and map ids to schemas on the receiving side when a breaking change has to coexist with the old version for a while: [`SchemaRegistry`](/guide/resolve#the-registry) keeps that map and @@ -111,14 +111,36 @@ The id includes keys, so renaming a field changes it even though the wire stays is deliberate: a swap of two same-typed fields is a change of meaning, and the id should say so. If only wire compatibility matters, compare `toString()` after stripping keys on your side. -### Your own fingerprint +### A wider fingerprint Thirty-two bits are plenty for a handshake between two sides, where the only question is "do we have the same schema?": two different schemas collide with probability 2⁻³². They are not enough -for a registry that keys many schemas by id, where a collision becomes likely around 65,000 -entries. The `fingerprint` option replaces the hash with any synchronous function of the -signature, which is the canonical form itself for a schema that references no other entity, and -`schema.list` inherits it: +for a registry that keys many schemas by id. Among K schemas, some two share a b-bit id with +probability P ≈ 1 − exp(−K(K−1)/2ᵇ⁺¹), the birthday bound: + +| Schemas | 32 bits | 64 bits | +| --- | ---: | ---: | +| 2¹⁰ (1,024) | 0.012% | 2.8 × 10⁻¹⁴ | +| 2¹² (4,096) | 0.2% | 4.6 × 10⁻¹³ | +| 2¹⁴ (16,384) | 3.1% | 7.3 × 10⁻¹² | +| 2¹⁶ (65,536) | 39% | 1.2 × 10⁻¹⁰ | +| 2¹⁸ (262,144) | ≈ 100% | 1.9 × 10⁻⁹ | +| 2²⁰ (1,048,576) | ≈ 100% | 3.0 × 10⁻⁸ | + +The `fingerprint` option names the hash: `'fnv1a32'` is the default, `'fnv1a64'` the 64-bit +FNV-1a of the same signature, as sixteen hex digits, computed as cheaply and without a +dependency. A registry of thousands of schemas takes it on every schema it holds: the registry +refuses two schemas of different signatures under one id (`ERR_ID_COLLISION`) rather than mix +them up. What a schema derives hashes with the same name (`schema.list`, the entities its +references reach, a resolved schema), and a standalone module carries it. + +```js +const V1 = Schema.from({ id: 'number', name: 'string' }, { metaschema, fingerprint: 'fnv1a64' }); +V1.id; // '2eb324893863d0cf' — 64-bit FNV-1a of '{id:number,name:string}'; '030d98ef' with 32 +``` + +The option also takes any synchronous function of the signature, which is the canonical form +itself for a schema that references no other entity, for a hash of your own: ```js const { createHash } = require('node:crypto'); @@ -131,7 +153,9 @@ User.id; // '17ad5aba43b19ad4' — 64 bits of SHA-256 of '{id:number,name:string The function must be deterministic and derive the id from its argument only: both sides compute it on their own from the same schema, which is what makes the handshake work. A random or time-based identifier (a cuid, a ulid) would differ between two copies of the same schema; if you -need hand-written version ids, keep a map from id to schema in the application instead. +need hand-written version ids, keep a map from id to schema in the application instead. A +function has no source, so a [standalone module](/guide/standalone#what-it-does-not-write) +carries the ids it computed and hashes what it derives with the default; a name travels. ### Past 31 fields or names diff --git a/docs/guide/resolve.md b/docs/guide/resolve.md index 64afa32..29f35d5 100644 --- a/docs/guide/resolve.md +++ b/docs/guide/resolve.md @@ -147,6 +147,7 @@ know, or one of a version the reader cannot resolve, a `DecodeError` `schema` wi `error.id` (never in the message: the payload gave it) and, for the second, the `ERR_INCOMPATIBLE` as `cause`. `decode` and `parse` throw nothing else, and a pair that cannot resolve is tried once. Writing `as` an id the registry does not know is `SchemaDefinitionError` `ERR_UNKNOWN_SCHEMA`. Two schemas of different signatures under one id throw -`ERR_ID_COLLISION` when added: the 32-bit fingerprint is too narrow for that registry, and a wider one is the remedy (see -[Your own fingerprint](/guide/evolution#your-own-fingerprint)). The registry is a subpath of its +`ERR_ID_COLLISION` when added: the 32-bit fingerprint is too narrow for that registry, and +`{ fingerprint: 'fnv1a64' }` on its schemas is the remedy (see +[A wider fingerprint](/guide/evolution#a-wider-fingerprint)). The registry is a subpath of its own, so a program that does not use it does not carry it. diff --git a/docs/guide/schema.md b/docs/guide/schema.md index b56818e..a08ea8d 100644 --- a/docs/guide/schema.md +++ b/docs/guide/schema.md @@ -200,7 +200,7 @@ by a reference keeps the options given for it in `entities`. | `delta` | `[]` | The paths of integer, number, date or decimals columns and lists that travel as differences. See [Types](/guide/types#delta) | | `intern` | `false` | `true` for every string, or the paths of strings: `[table, body]`. See [Types](/guide/types#interned-strings) | | `stringify` | `'auto'` | How `stringify` writes the text: `'auto'`, `'builder'` or `'native'`. See [Runtimes](/guide/runtimes#how-stringify-writes-the-text) | -| `fingerprint` | FNV-1a 32-bit | `(signature) => string`, computes `id` from the schema's signature (below). See [Schema Evolution](/guide/evolution#your-own-fingerprint) | +| `fingerprint` | `'fnv1a32'` | What `id` is of the schema's signature (below): `'fnv1a32'` (eight hex digits), `'fnv1a64'` (sixteen, for a registry of many schemas) or `(signature) => string`. See [Schema Evolution](/guide/evolution#a-wider-fingerprint) | | `maxDepth` | `Infinity` | The levels of a recursion one payload may nest: deeper is `DecodeError` `structure` or `EncodeError` `depth`, at the same depth on either backend. By default nothing is counted and the call stack is the limit, which differs between backends; `512` is a depth both reach everywhere. See [Errors](/guide/errors#depth) | | `references` | `'kind'` | How every reference travels: `'kind'`, `'embed'` or `'id'`. See [References](#references) | | `types` | `{}` | The wire type of each custom type with a check of its own: `{ decimal: 'string' }` | @@ -269,7 +269,7 @@ changes nothing on the wire. ```js User.toString(); // '{id:number,name:string,active:boolean,role:enum(admin|editor|viewer),address:{city:string,street:string,zip:?string},tags:[string],orders:[{sku:string,qty:number}],createdAt:date,perms:flags(read|write|admin),balance:?number=0,manager:?@User,email:?string}' -User.id; // 'cecbac56' — FNV-1a of its signature, the same on both sides of the wire +User.id; // 'cecbac56' — 32-bit FNV-1a of its signature, the same on both sides of the wire ``` `id` is the fingerprint of the schema's **signature**: its canonical form, followed by @@ -297,7 +297,8 @@ so does the optionality of a root scalar (`'?string'` prints as `?string`). The the one protoarray 1.x gave the same shape, and so is the id of a schema that references no other entity, so a 2.0 schema and a 1.x schema agree on the wire; one that references another entity by name has a new id in 2.0 (1.x fingerprinted its canonical form alone). The `fingerprint` option -replaces the hash (see [Schema Evolution](/guide/evolution#your-own-fingerprint)). +names the hash, `'fnv1a64'` for a registry of many schemas, or replaces it with a function of +the signature (see [Schema Evolution](/guide/evolution#a-wider-fingerprint)). ## A snapshot is input diff --git a/docs/guide/standalone.md b/docs/guide/standalone.md index edc87cf..62a6118 100644 --- a/docs/guide/standalone.md +++ b/docs/guide/standalone.md @@ -60,7 +60,8 @@ fs.writeFileSync('schemas.compiled.js', toSource({ User, Address }, { format: 'e ``` The options a schema was built with travel with it (`sparse`, `layout`, `delta`, `intern`, -`references`, `types`, `validate`, `stringify`). +`references`, `types`, `validate`, `stringify`, `maxDepth`, and `fingerprint` when it names a +built-in hash). `toSource(schema)` writes one schema as the default export, `toSource({ A, B })` named exports. `register()` from the same subpath adds `schema.toSource(options)`. @@ -99,8 +100,9 @@ would otherwise use is about as fast as `JSON` (×1.5 and ×2 slower than the mo - A schema from `resolve`: it is built at runtime from two schemas; write the writer and the reader, and resolve them at runtime. - `schema.list`: write the schema, and take its `list` at runtime. -- A custom `fingerprint` function: the module carries each schema's `id` as it was computed, and - the schemas it derives at runtime (`list`) use the default fingerprint. +- A `fingerprint` function: the module carries each schema's `id` as it was computed, and the + schemas it derives at runtime (`list`) use the default fingerprint. A built-in hash + (`'fnv1a64'`) travels by name, and `list` hashes with it. - The metaschema schema: a standalone schema's `source` is `null`. Values are validated where metaschema runs, if anywhere; the module itself encodes and decodes. diff --git a/docs/guide/wire-format.md b/docs/guide/wire-format.md index 156b04d..8c4cba4 100644 --- a/docs/guide/wire-format.md +++ b/docs/guide/wire-format.md @@ -110,5 +110,6 @@ as a JSON string (`{"a:b":number}`), and so does a string enum value that reads its UTF-16 code units (not UTF-8 bytes, as most FNV-1a implementations): the canonical form, then `;Name=` for every other entity the schema references by name (`@Name`), sorted by name as code units (the name printed as in the form); a reference to its own name adds nothing, -nor does the schema itself as the record of its `list`. The `fingerprint` option supplies another -function of the signature. +nor does the schema itself as the record of its `list`. `{ fingerprint: 'fnv1a64' }` is the +64-bit FNV-1a of the same code units, as sixteen hex digits, and a `fingerprint` function +computes the id from the signature itself. diff --git a/index.d.ts b/index.d.ts index b05f249..92fe781 100644 --- a/index.d.ts +++ b/index.d.ts @@ -164,11 +164,13 @@ export interface SchemaOptions { */ readonly stringify?: 'auto' | 'builder' | 'native'; /** - * Computes `id` from the schema's signature: its canonical form (`toString()`), then - * `;Name=` for every other entity it references, by name. The default is a - * 32-bit FNV-1a hash as eight hex digits. It must be synchronous and deterministic. + * What `id` is of the schema's signature: its canonical form (`toString()`), then + * `;Name=` for every other entity it references, by name. `'fnv1a32'` (the + * default) is a 32-bit FNV-1a hash as eight hex digits, enough for a handshake; `'fnv1a64'` the + * 64-bit one as sixteen, for a registry of many schemas; a function computes it from the + * signature, and must be synchronous and deterministic. */ - readonly fingerprint?: (signature: string) => string; + readonly fingerprint?: 'fnv1a32' | 'fnv1a64' | ((signature: string) => string); /** * The levels of a recursion one payload may nest, a positive integer: deeper is `DecodeError` * `structure` or `EncodeError` `depth`, at the same level on either backend and in generated diff --git a/src/plan.js b/src/plan.js index 79257e3..11f78e2 100644 --- a/src/plan.js +++ b/src/plan.js @@ -196,10 +196,10 @@ const signature = (text, entities) => { return out; }; -// FNV-1a, 32-bit, over UTF-16 code units: enough to tell schema versions -// apart in a handshake, and cheap enough to compute on every page load. It -// hashes a text a piece at a time, so the default id of a signature is -// computed without joining it. +// FNV-1a over UTF-16 code units, 32-bit by default: enough to tell schema +// versions apart in a handshake, and cheap enough to compute on every page +// load. Both widths hash a text a piece at a time, so the id of a signature +// is computed without joining it. const OFFSET = 0x811c9dc5; const hashed = (hash, text) => { @@ -213,18 +213,49 @@ const hashed = (hash, text) => { const hex = (hash) => (hash >>> 0).toString(16).padStart(8, '0'); -const fingerprint = (text) => hex(hashed(OFFSET, text)); +// The 64-bit width, for a registry of many schemas, where 32 bits collide. +// On a BigInt: it runs once per schema, when its id is first read. +const OFFSET64 = 0xcbf29ce484222325n; +const PRIME64 = 0x100000001b3n; +const MASK64 = 0xffffffffffffffffn; -// The id of a schema: its fingerprint function of the signature, which the -// default one hashes piece by piece. +const hashed64 = (hash, text) => { + let out = hash; + for (let i = 0; i < text.length; i++) { + out = ((out ^ BigInt(text.charCodeAt(i))) * PRIME64) & MASK64; + } + return out; +}; + +const hex64 = (hash) => hash.toString(16).padStart(16, '0'); + +// A built-in hash, by the name the `fingerprint` option takes: its seed, its +// step over a piece of text and the hex digits of the result. +const HASHES = Object.assign(Object.create(null), { + fnv1a32: { seed: OFFSET, step: hashed, digits: hex }, + fnv1a64: { seed: OFFSET64, step: hashed64, digits: hex64 }, +}); + +// The built-in fingerprints, each a function of the whole text. +const FINGERPRINTS = Object.create(null); +for (const name of Object.keys(HASHES)) { + const { seed, step, digits } = HASHES[name]; + FINGERPRINTS[name] = (text) => digits(step(seed, text)); +} + +const fingerprint = FINGERPRINTS.fnv1a32; + +// The id of a schema: its fingerprint function of the signature, or the +// built-in hash it names, which hashes the signature piece by piece. const identify = (text, entities, hash) => { - if (hash !== fingerprint) return hash(signature(text, entities)); - let out = hashed(OFFSET, text); + if (typeof hash === 'function') return hash(signature(text, entities)); + const { seed, step, digits } = HASHES[hash]; + let out = step(seed, text); for (const [name, schema] of entities) { - out = hashed(hashed(hashed(out, ';'), print(name)), '='); - out = hashed(out, String(schema)); + out = step(step(step(out, ';'), print(name)), '='); + out = step(out, String(schema)); } - return hex(out); + return digits(out); }; module.exports = { @@ -240,6 +271,7 @@ module.exports = { canonical, referenced, signature, + FINGERPRINTS, fingerprint, identify, }; diff --git a/src/registry.js b/src/registry.js index bdafb3b..65a583e 100644 --- a/src/registry.js +++ b/src/registry.js @@ -47,9 +47,10 @@ class SchemaRegistry { const { id } = schema; const known = this.#byId.get(id); if (known !== undefined && !known[SAME](schema)) { + const remedy = "use a wider fingerprint ({ fingerprint: 'fnv1a64' })"; throw new SchemaDefinitionError( 'ERR_ID_COLLISION', - `Schemas "${known.name}" and "${schema.name}" share the id ${id}: use a wider fingerprint`, + `Schemas "${known.name}" and "${schema.name}" share the id ${id}: ${remedy}`, schema.name, ); } diff --git a/src/schema.js b/src/schema.js index 8ba3e61..dfd6ccb 100644 --- a/src/schema.js +++ b/src/schema.js @@ -23,7 +23,7 @@ const { canonical, referenced, signature, - fingerprint, + FINGERPRINTS, identify, } = require('./plan.js'); const { SCALARS } = require('./types.js'); @@ -111,7 +111,8 @@ const VALID = Object.assign(Object.create(null), { stringify: (value) => STRINGIFY.includes(value), compiled: isCompiled, layout: isLayouts, - fingerprint: (value) => typeof value === 'function', + fingerprint: (value) => + typeof value === 'function' || (typeof value === 'string' && FINGERPRINTS[value] !== undefined), maxDepth: (value) => value === Infinity || (Number.isSafeInteger(value) && value > 0), references: (value) => REFERENCES.includes(value), types: isWireTypes, @@ -133,6 +134,7 @@ const invalidOption = (key, value) => { const path = Object.keys(value).find((place) => !isLayouts(value[place])); return `Invalid option "layout" at "${path}": rows, flat or columns`; } + if (key === 'fingerprint') return 'Invalid option "fingerprint": fnv1a32, fnv1a64 or a function'; return `Invalid option "${key}"`; }; @@ -145,7 +147,7 @@ const readOptions = (options, name) => { delta: [], stringify: 'auto', layout: 'rows', - fingerprint, + fingerprint: 'fnv1a32', maxDepth: Infinity, references: 'kind', types: Object.create(null), diff --git a/src/standalone.js b/src/standalone.js index 353132b..a23f136 100644 --- a/src/standalone.js +++ b/src/standalone.js @@ -229,9 +229,11 @@ const planSource = (plan, settings, names) => { const codecSource = (schema, names) => planSource(schema[PLAN], schema[SETTINGS], names); // The options a schema of the module is built with again, without its -// compiled codecs. A fingerprint function has no source: the id travels -// instead. A limit of the nesting is written only when there is one, since -// JSON has no Infinity. +// compiled codecs. A built-in fingerprint travels by name when it is not the +// default, so what the module derives at runtime (`list`) has the same id as +// here; a fingerprint function has no source: the id travels instead. A limit +// of the nesting is written only when there is one, since JSON has no +// Infinity. const OPTIONS = ['validate', 'sparse', 'layout', 'intern', 'delta', 'stringify', 'references']; const settingsSource = (schema) => { @@ -239,6 +241,10 @@ const settingsSource = (schema) => { const options = OPTIONS.map((key) => `${key}:${JSON.stringify(settings[key])}`); options.push(`types:${JSON.stringify(settings.types)}`); if (settings.maxDepth !== Infinity) options.push(`maxDepth:${settings.maxDepth}`); + const { fingerprint } = settings; + if (typeof fingerprint === 'string' && fingerprint !== 'fnv1a32') { + options.push(`fingerprint:${JSON.stringify(fingerprint)}`); + } return options; }; diff --git a/tests/types/index.test-d.ts b/tests/types/index.test-d.ts index 99c22bb..7a410ce 100644 --- a/tests/types/index.test-d.ts +++ b/tests/types/index.test-d.ts @@ -118,6 +118,9 @@ expectError(Schema.from(UserDefinition, { intern: 'tags' })); expectError(Schema.from(UserDefinition, { types: { money: 'float' } })); expectError(Schema.from(UserDefinition, { schemas: {} })); expectError(Schema.from(UserDefinition, { maxDepth: '512' })); +Schema.from(UserDefinition, { fingerprint: 'fnv1a64' }); +Schema.from(UserDefinition, { fingerprint: 'fnv1a32' }); +expectError(Schema.from(UserDefinition, { fingerprint: 'sha1' })); // The free functions follow the schema's methods. expectType(encode(User, decoded)); diff --git a/tests/unit/registry.test.js b/tests/unit/registry.test.js index 2adc7aa..97d181f 100644 --- a/tests/unit/registry.test.js +++ b/tests/unit/registry.test.js @@ -32,6 +32,9 @@ test('Registry: schemas by id and by name', () => { assert.throws(() => registry.add({}), { code: 'ERR_SCHEMA_EXPECTED' }); }); +// What a collision's message tells to do. +const REMEDY = "use a wider fingerprint ({ fingerprint: 'fnv1a64' })"; + test('Registry: a narrow fingerprint is a collision, not a silent mix-up', () => { const narrow = () => 'x'; const A = named('A', { a: 'number' }, { fingerprint: narrow }); @@ -41,7 +44,7 @@ test('Registry: a narrow fingerprint is a collision, not a silent mix-up', () => name: 'SchemaDefinitionError', code: 'ERR_ID_COLLISION', schema: 'B', - message: 'Schemas "A" and "B" share the id x: use a wider fingerprint', + message: `Schemas "A" and "B" share the id x: ${REMEDY}`, }); }); @@ -242,7 +245,7 @@ test('Registry: one id for two versions is a collision, whatever tells them apar for (const other of versions.slice(1)) { assert.throws(() => new SchemaRegistry([versions[0], other]), { code: 'ERR_ID_COLLISION', - message: 'Schemas "A" and "A" share the id same: use a wider fingerprint', + message: `Schemas "A" and "A" share the id same: ${REMEDY}`, }); } // The same version again changes nothing. diff --git a/tests/unit/resolve.test.js b/tests/unit/resolve.test.js index 2a1ac48..682faba 100644 --- a/tests/unit/resolve.test.js +++ b/tests/unit/resolve.test.js @@ -696,6 +696,16 @@ test("Resolve: a resolved schema has the writer's id, whatever its fingerprint", assert.strictEqual(resolved[SIGNATURE], writer[SIGNATURE]); }); +test("Resolve: a resolved schema hashes with the writer's built-in fingerprint", () => { + const writer = define({ a: 'number' }, { fingerprint: 'fnv1a64' }); + const resolved = resolve(writer, define({ a: 'number', 'b?': 'string' })); + assert.strictEqual(resolved.id, writer.id); + assert.match(resolved.id, /^[0-9a-f]{16}$/); + // What it derives hashes with the same width. + assert.strictEqual(resolved.list.id, writer.list.id); + assert.match(resolved.list.id, /^[0-9a-f]{16}$/); +}); + test('Resolve: a chain too long for the stack is an error, not a RangeError', () => { const chain = (extra) => { const entities = {}; diff --git a/tests/unit/schema.test.js b/tests/unit/schema.test.js index 295415e..36c9b8c 100644 --- a/tests/unit/schema.test.js +++ b/tests/unit/schema.test.js @@ -5,6 +5,7 @@ const { BACKENDS } = require('../support/backends.js'); const { ms, define, model } = require('../support/schema.js'); const CODEC = Symbol.for('alexify.protoarray.codec'); +const PLAN = Symbol.for('alexify.protoarray.plan'); const BRAND = Symbol.for('alexify.metaschema.brand'); // A snapshot of one struct entity `T`, written by hand: what metaschema @@ -1006,6 +1007,63 @@ test('Schema: the list of a recursive schema has the id of its canonical form', assert.strictEqual(User.list.id, fingerprint(User.list.toString())); }); +test('Schema: the built-in fingerprints', () => { + const { FINGERPRINTS, fingerprint, signature, identify } = require('../../src/plan.js'); + assert.deepStrictEqual(Object.keys(FINGERPRINTS), ['fnv1a32', 'fnv1a64']); + assert.strictEqual(FINGERPRINTS.fnv1a32, fingerprint); + // The reference vectors of FNV-1a over one-byte texts, where code units + // and UTF-8 bytes agree, then a code unit past one byte, hashed as one. + const vectors = [ + ['', '811c9dc5', 'cbf29ce484222325'], + ['a', 'e40c292c', 'af63dc4c8601ec8c'], + ['foobar', 'bf9cf968', '85944171f73967e8'], + ['a\u20acb', 'ba777fc6', '77a39118a6897166'], + ]; + for (const [text, narrow, wide] of vectors) { + assert.strictEqual(FINGERPRINTS.fnv1a32(text), narrow, text); + assert.strictEqual(FINGERPRINTS.fnv1a64(text), wide, text); + } + // A named hash of a signature, piece by piece, is the hash of the text. + const entities = { + Address: { Struct: {}, city: 'string' }, + User: { Struct: {}, home: 'Address', 'boss?': 'User' }, + }; + const SIGNATURE = Symbol.for('alexify.protoarray.signature'); + const wide = model(entities, { fingerprint: 'fnv1a64' }); + const text = '{home:@Address,boss:?@User};Address={city:string}'; + assert.strictEqual(wide.User[SIGNATURE], text); + assert.strictEqual(wide.User.id, FINGERPRINTS.fnv1a64(text)); + assert.match(wide.User.id, /^[0-9a-f]{16}$/); + const reached = [['Address', wide.Address]]; + const canonical = wide.User.toString(); + assert.strictEqual(signature(canonical, reached), text); + assert.strictEqual(identify(canonical, reached, 'fnv1a64'), FINGERPRINTS.fnv1a64(text)); + assert.strictEqual(identify(canonical, reached, 'fnv1a32'), fingerprint(text)); + // The 32-bit one is the default, by name or not. + const plain = model(entities).User; + assert.strictEqual(model(entities, { fingerprint: 'fnv1a32' }).User.id, plain.id); + assert.strictEqual(plain.id, fingerprint(text)); + assert.match(plain.id, /^[0-9a-f]{8}$/); + // What a schema derives takes the name: its list, a collection in a layout + // of its own. + assert.strictEqual(wide.User.list.id, FINGERPRINTS.fnv1a64(wide.User.list[SIGNATURE])); + assert.match(wide.User.list.id, /^[0-9a-f]{16}$/); + const Outer = define( + { items: { array: { s: 'string' } } }, + { layout: { items: 'columns' }, fingerprint: 'fnv1a64' }, + ); + const nested = Outer[PLAN].fields[0].node.slot.schema; + assert.strictEqual(nested.toString(), 'columns[{s:string}]'); + assert.strictEqual(nested.id, FINGERPRINTS.fnv1a64(nested.toString())); + // Anything else is an invalid option, a name of the prototype included. + for (const value of ['sha256', 'FNV1A64', 'fnv1a128', '', 'constructor', 'toString', 1]) { + invalid(() => define({ a: 'number' }, { fingerprint: value }), { + code: 'ERR_INVALID_OPTIONS', + message: 'Invalid option "fingerprint": fnv1a32, fnv1a64 or a function', + }); + } +}); + test('Schema: a fingerprint gives a non-empty string', () => { for (const id of [42, '', null]) { const schema = define({ a: 'number' }, { fingerprint: () => id }); diff --git a/tests/unit/standalone.test.js b/tests/unit/standalone.test.js index c86fc7d..105c574 100644 --- a/tests/unit/standalone.test.js +++ b/tests/unit/standalone.test.js @@ -364,6 +364,33 @@ test('Standalone: an import module runs in this process too', () => { agree(outer, Outer, { items: [{ s: 'a' }, { s: 'a', n: 2 }], tag: 'a' }, 'nested'); }); +test('Standalone: a built-in fingerprint travels by name, a function by its id', () => { + const { FINGERPRINTS } = require('../../src/plan.js'); + const entities = { + Address: { Struct: {}, city: 'string' }, + User: { Struct: {}, home: 'Address', 'boss?': 'User' }, + }; + const { User } = model(entities, { fingerprint: 'fnv1a64' }); + assert.match(standalone.toSource({ User }), /fingerprint:"fnv1a64"/); + const copy = local({ User }).User; + assert.strictEqual(copy.id, User.id); + assert.match(copy.id, /^[0-9a-f]{16}$/); + // What the module derives at runtime hashes with the same width. + assert.strictEqual(copy.list.id, User.list.id); + assert.match(copy.list.id, /^[0-9a-f]{16}$/); + // The default is not written, by name or not; a function has no source, + // so its id travels and what the module derives takes the default. + const base = model(entities).User; + assert.doesNotMatch(standalone.toSource({ base }), /fingerprint:/); + const narrow = model(entities, { fingerprint: 'fnv1a32' }).User; + assert.doesNotMatch(standalone.toSource({ narrow }), /fingerprint:/); + assert.strictEqual(local({ narrow }).narrow.list.id, base.list.id); + const custom = define({ a: 'integer' }, { fingerprint: (text) => `v${text.length}` }); + const copied = local({ custom }).custom; + assert.strictEqual(copied.id, 'v11'); + assert.strictEqual(copied.list.id, FINGERPRINTS.fnv1a32(copied.list.toString())); +}); + test('Standalone: the inline kernel', () => { const { define: build, slot, nested, types: kernelTypes } = kernel(); const target = { codec: 'codec' }; From aa2159d4d20dc6b9ac8875cd548b86fc7296c76f Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Tue, 20 Oct 2026 22:52:58 +0300 Subject: [PATCH 04/17] feat(binary): the same schema on a binary wire (@alexify/protoarray/binary) A fifth subpath, `./binary`: `toBinary(schema, value)` gives the bytes of a value as a Uint8Array, `fromBinary(schema, bytes)` the value back, and `register()` installs them as `schema.toBinary` and `schema.fromBinary`, for a schema or its `list`. The same plan is compiled a second time, on first use, into a writer and a reader of bytes kept on the schema, by the backend the schema runs: generated code through `codegen.compile`, the one primitive that turns source text into a function (src/binary/generate.js mirrors codegen.js), or closures (src/binary/closures.js, the reference the tests hold the generated code to: same bytes, same errors and paths, on the vectors and random schemas, with corrupted values, corrupted bytes and every truncation). Decode checks the wire as the schema's `validate` says, and bounds every read, count and bitmap by the message. Encode trusts its values as the text wire does, and refuses what the writer cannot measure: a string slot without a string, a bytes slot without a Uint8Array. The wire, little-endian: a record is a header of presence bits (required fields too, then the booleans' values) and its fields in declaration order; integers, dates and decimals are zigzag varints exact over the safe integers; a number a float64; an enum index and a union tag a byte up to 256 values; a string its doubled byte length with the form in the low bit, UTF-8 up to 21 characters and UTF-16 past them or for a lone surrogate, copied rather than transcoded, which is what keeps long strings ahead of JSON. The runtime twins carry the string functions next to base64. Both sides need the same schema; interned strings, lists of differences, the flat and columns layouts and a schema from resolve have no binary form (SchemaDefinitionError ERR_NO_BINARY_FORM, naming the place). Measured in the comparison (bench/results/compare.json, Apple M3 Max, Node 24): a round trip of one record 1.05M/s against 942k of the text wire and 720k of protobufjs, 32 doubles 1.66M/s against 270k and avsc's 1.20M, long strings 1.02M/s against JSON's 715k; the subpath adds 7.7 KB min+gzip, the core is unchanged under its 22 KB gate. The docs record how the decision was made: two spikes under bench/binary/ (removed; bench/results/binary.json is the second one's run) and criteria written before the run. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 22 + CLAUDE.md | 60 +- README.md | 21 +- bench/compare/adapters.js | 18 + bench/report.js | 91 +- bench/results/binary.json | 1208 +++++++++++++++++ bench/results/compare.json | 151 ++- bench/results/size.json | 25 +- binary.d.ts | 40 + binary.js | 1 + docs/.vitepress/config.mts | 1 + docs/api/exports.md | 24 +- docs/comparison.json | 2 +- docs/guide/benchmarks.md | 198 ++- docs/guide/binary.md | 136 ++ docs/guide/comparison.md | 2 +- docs/guide/why.md | 9 +- docs/guide/wire-format.md | 37 + docs/public/benchmarks/coldstart.svg | 48 +- .../benchmarks/compare-numeric-bytes.svg | 159 ++- docs/public/benchmarks/compare-numeric.svg | 174 +-- .../benchmarks/compare-packets-bytes.svg | 151 ++- docs/public/benchmarks/compare-packets.svg | 164 +-- .../benchmarks/compare-records-bytes.svg | 143 +- docs/public/benchmarks/compare-records.svg | 188 +-- .../benchmarks/compare-single-bytes.svg | 143 +- docs/public/benchmarks/compare-single.svg | 170 +-- .../benchmarks/compare-sparse-bytes.svg | 147 +- docs/public/benchmarks/compare-sparse.svg | 172 +-- .../benchmarks/compare-strings-bytes.svg | 63 +- docs/public/benchmarks/compare-strings.svg | 170 +-- .../public/benchmarks/compare-typed-bytes.svg | 143 +- docs/public/benchmarks/compare-typed.svg | 174 +-- docs/public/benchmarks/size.svg | 98 +- package.json | 8 +- scripts/size-compare.js | 9 + scripts/size.js | 1 + src/binary.js | 186 +++ src/binary/closures.js | 596 ++++++++ src/binary/generate.js | 499 +++++++ src/binary/wire.js | 360 +++++ src/codegen.js | 16 +- src/errors.js | 1 + src/runtime/browser.js | 48 +- src/runtime/node.js | 19 +- src/types.js | 2 + tests/types/binary.test-d.ts | 30 + tests/unit/binary.test.js | 864 ++++++++++++ tests/unit/export-parity.test.js | 2 + tests/unit/platform.test.js | 34 + 50 files changed, 5876 insertions(+), 1152 deletions(-) create mode 100644 bench/results/binary.json create mode 100644 binary.d.ts create mode 100644 binary.js create mode 100644 docs/guide/binary.md create mode 100644 src/binary.js create mode 100644 src/binary/closures.js create mode 100644 src/binary/generate.js create mode 100644 src/binary/wire.js create mode 100644 tests/types/binary.test-d.ts create mode 100644 tests/unit/binary.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c2be09..40e223a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **The binary wire**, `@alexify/protoarray/binary`: `toBinary(schema, value)` gives the bytes of + a value as a `Uint8Array`, `fromBinary(schema, bytes)` the value back, `register()` installs + them as `schema.toBinary` and `schema.fromBinary`, for a schema or its `list`. The same plan is + compiled a second time, by the backend the schema runs (generated code through the one compile + primitive of `codegen.js`, closures elsewhere; both write the same bytes), with the schema's + `validate` on decode and the same errors and paths. A record is a header of presence bits (the + booleans in it) and its fields; integers, dates and decimals are zigzag varints, a `number` a + float64, an enum index a byte, a string UTF-8 up to 21 characters and UTF-16 past them (copied, + not transcoded), the rest as in the wire format page. Both sides need the same schema. Interned + strings, lists of differences, the `flat` and `columns` layouts and a schema from `resolve` have + no binary form (`SchemaDefinitionError` `ERR_NO_BINARY_FORM`, naming the place); the registry, + streaming and standalone modules stay on the text wire. See The Binary Wire in the guide and the + benchmarks. + +### Changed + +- The runtime twins (`src/runtime/node.js`, `browser.js`) carry the string functions of the + binary wire next to base64 (Buffer's in Node.js, `TextDecoder` and loops in a browser), and + `DecodeError` `structure` has a `bytes` detail ("is not a Uint8Array"). + ## [2.0.0] - 2026-10-16 2.0 hands the definition of a schema to `@alexify/metaschema` and keeps the wire to protoarray. diff --git a/CLAUDE.md b/CLAUDE.md index c4027f8..7848e30 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -52,7 +52,7 @@ pre-merge gate. There is no build step. `import`/`export`, no `'use strict'`. - The root `index.js` is a one-line shim over `src/index.js`. `browser.js` intentionally mirrors it. Do not deduplicate the two (see "Platform"). The subpaths `registry.js`, `resolve.js`, - `stream.js` and `standalone.js` are one-line shims over `src/` modules of the same names, each + `stream.js`, `standalone.js` and `binary.js` are one-line shims over `src/` modules of the same names, each with its own d.ts; they hold what the core does not need, as free functions with the schema first, and `register()` installs the methods (`Schema.prototype`, or `Schema.resolve`, non-enumerable). Their d.ts declare those methods by module augmentation of `Schema`. @@ -92,7 +92,7 @@ devDependencies are fine (metaschema for the tests, the comparative benchmark's ``` index.js browser.js index.d.ts root entry shims + hand-written types -registry.js resolve.js stream.js standalone.js (+ .d.ts) subpath shims + types +registry.js resolve.js stream.js standalone.js binary.js (+ .d.ts) subpath shims + types bin/protoarray.js the CLI: `protoarray compile` (Node, outside src/) src/ index.js public barrel (explicit named exports) @@ -104,7 +104,8 @@ src/ resolve.js writer plan × reader plan → resolved plan; resolve()/register() (./resolve) types.js scalar type table (a closed kernel): checks, conversions, bridges, interned strings, delta lists, recipes; shared by both backends - runtime/ base64 twin for `bytes`: node.js (Buffer) and browser.js (a closed kernel) + runtime/ platform twin: base64 for `bytes` and the strings of the binary wire (UTF-8 + read, UTF-16 both ways); node.js (Buffer) and browser.js (a closed kernel) codegen.js new Function backend (the only file that may compile source text), the builder closures.js no-eval backend, same semantics, used as the fallback and as the reference layouts.js flat/columns layouts, delta columns, the interned root (a closed kernel) @@ -113,6 +114,11 @@ src/ registry.js SchemaRegistry and the [id, payload] envelope (./registry) stream.js NDJSON, TransformStreams, stringifyItems/parseItems (./stream) standalone.js toSource, the import runtime, the inline kernel, register (./standalone) + binary.js toBinary/fromBinary/register: the same schema on the binary wire (./binary); + the codec kept on the schema, the backend the schema runs, refusals + binary/ wire.js (Writer/Reader kernel over the runtime twin: varints, header bitsets, + strings), generate.js (generated backend through codegen.compile), + closures.js (the no-eval twin and reference, same bytes and errors) tests/ unit/*.test.js node:test suites, one per area, every behavioural suite runs per backend support/ BACKENDS + normalize, define/named/model (metaschema injected), the @@ -124,7 +130,8 @@ scripts/size-compare.js cross-library bundle sizes; scripts/comparison.js rend bench/ engine harness (bench.js), comparison (compare.js + compare/: adapters, scenarios, measure, releases-worker), releases.js, report.js + report/ (model.mjs: the chart models, svg.mjs: README images); baseline.json and results/*.json are - snapshots + snapshots (results/binary.json is the run of the spike that decided ./binary, + kept as it was for the docs) docs/ VitePress site (protoarray.vercel.app), not published; comparison.json is the data of the feature matrix, .vitepress/theme/components/Bench*.vue the charts .claude/skills/ metaskills v1.0.5 skills (js-conventions, npm-publish adapted) @@ -224,6 +231,33 @@ records take `JSON.stringify`. `schema.js` sets `builder` for `stringify: 'build `'auto'` only on records of converted types (`converted`), the only shape that wins against the native path on every engine measured. Parity tests force it on random schemas. +**The binary wire** (`binary.js`, the `./binary` subpath; `register()` installs `toBinary` and +`fromBinary` on `Schema.prototype`): the same plan compiled a second time, into a writer and a +reader of bytes, on first use, kept on the schema under `Symbol.for('alexify.protoarray.binary')` +as `{ write, read, numbers, writer, reader, busy }`. The backend follows `schema.backend`: +`binary/generate.js` mirrors `codegen.js` (one `eN(w,o)`/`dN(r)` per struct, an object literal +on decode, a union as a switch, a slot as `H[k].write`/`.read` of the referenced schema's own +entry, registered before it is built so a recursion resolves) and compiles through +`codegen.compile`, the one primitive that turns source text into a function (the platform test's +count of `new Function(` stays two); `binary/closures.js` is the no-eval twin and the reference +`binary.test.js` holds it to on random schemas: same bytes, same errors and paths, with and +without validation, on corrupted values and bytes and every truncation. `validate` comes from the +schema; a union's tag is checked in every mode; decode bounds every read, count (by the bytes or +bits left) and bitmap, so no message allocates past its length. Encode trusts values but refuses +a string slot without a string and a `bytes` slot without a `Uint8Array` (`type`): the writer +measures them, and garbage would corrupt its position. A `null` root is `required`. `inspect` +refuses, before anything is built, what has no binary form (`ERR_NO_BINARY_FORM`, the place by +its path: `intern`, `delta`/`deltas`, a layout other than `rows`, a resolved plan); `readsNumbers` +walks the plan and the schemas its slots reach so the reader makes a DataView only when a float64 +is read. One Writer and one Reader per schema, reset per call; `busy` is saved and restored around +a call so a re-entrant call (a `toJSON` of a json value) takes fresh ones. The wire: a struct +header of presence bits (required fields too) then the booleans' values, in whole bytes; zigzag +LEB128 exact over the safe integers (the sign in the low bit, never forming 2n); enum and tag as +a byte up to 256 values; a string's length doubled with its form in the low bit, UTF-8 up to 21 +characters (`3 × 21 × 2 < 128`, one length byte), UTF-16 past them or for a lone surrogate, +read in either form at any length. Both sides need the same schema: nothing is trimmed or +tolerated. The registry envelope, streams, resolve and standalone modules stay on the text wire. + **Standalone modules** (`standalone.js`): `errors.js`, `types.js`, `layouts.js` and `runtime/browser.js` are closed functions (`kernel`) run once, so their source can be embedded. Every scalar node and helper function of `types.js` carries a `recipe`, the expression that makes @@ -289,7 +323,10 @@ checks they agree with and without the native methods. `pnpm size` bundles `brow `platform: 'browser'`, so a leaked builtin fails CI. The subpaths follow the same rule: `stream.js` uses only the globals `TextDecoder`, -`TransformStream` and `ReadableStream` (all in Node.js 18), `standalone.js` requires the browser +`TransformStream` and `ReadableStream` (all in Node.js 18), `binary/wire.js` is a closed kernel +over the runtime twin (whose Node half reaches Buffer's `utf8Slice`, `ucs2Write` and `ucs2Slice` +through `.call` on a plain Uint8Array, and whose browser half uses `TextDecoder` and loops, a +lone surrogate kept), `standalone.js` requires the browser half of the base64 twin directly for its kernel, and only `bin/` (outside `src/`) uses Node builtins. A kernel must stay closed: nothing but its parameters and standard globals, since its source runs on its own in an inline module (`standalone.test.js` and `bundle.test.js` run such @@ -384,6 +421,14 @@ change. **This file wins** if they ever disagree. the raw line separator. - `pnpm test` passes the glob unquoted (`tests/unit/*.test.js`) so the shell expands it on Node 18/20; Node ≥ 22 expands it itself, which is why CI runs Windows only on 22/24. +- `tests/unit/binary.test.js` is the binary wire's suite: the vectors and the random schemas on + both backends (identical bytes; corrupted values, corrupted bytes and every truncation give the + same outcome; validation on and off), the wire kernel over both runtime twins, the refusals by + path, the register and the codegen fallback (`codegen.compile` stubbed through the module + object). A count read from the wire is bounded by the bytes (or bits) left: without it a + corrupted count overflowed `(c + 7) >> 3` and looped for minutes, and garbage in a string or + bytes slot made the writer's position NaN and doubled its buffer until the process was killed, + which is why encode checks those two slots. - `tests/unit/bundle.test.js` builds `browser.js` and `index.js` with esbuild in memory (minified and not) and round-trips a schema through both backends, comparing as JSON because the browser bundle runs in another realm. That realm is a `vm` context, which has no `btoa`/`atob` unless @@ -413,8 +458,9 @@ published to npm, and lint/format do not cover it. the docs true to the code: the examples come from `tests/support/vectors.js`. Examples define schemas in metaschema (`import { Schema as Definition } from '@alexify/metaschema'`). - The benchmark tables in README and `docs/guide/benchmarks.md` sit between - `` markers (`compare`, `coldstart`, `size`, `releases`, `engine`) - and are generated by `pnpm bench:report` from `bench/results/*.json` and `bench/baseline.json`, + `` markers (`compare`, `coldstart`, `size`, `releases`, `engine`; + `binary` in the docs only) and are generated by `pnpm bench:report` from + `bench/results/*.json` and `bench/baseline.json`, with the SVG charts in `docs/public/benchmarks/` that the README embeds; do not edit them by hand. The docs render the same models (`bench/report/model.mjs`) interactively through the `Bench*.vue` components, so a chart and its table cannot disagree. Every adapter of the diff --git a/README.md b/README.md index a76c7e2..fde85c1 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,8 @@ const User = Schema.from(await (await fetch('/schemas/user')).json()); messages it encodes faster than protobufjs, protobuf-es, avsc, msgpackr and cbor-x, and it has the fastest round trip of one record; msgpackr with shared structures decodes faster (and avsc, cbor-x or protobufjs on some shapes), and the binary codecs win on mostly numeric payloads (see - [benchmarks](#benchmarks)). + [benchmarks](#benchmarks)). Where a channel carries bytes, `@alexify/protoarray/binary` puts + the same schema on a [binary wire](https://protoarray.vercel.app/guide/binary) of its own. - **Still JSON.** The payload is a plain JSON array, so it goes through any transport and storage that already handle JSON, compresses with gzip, brotli and zstd like JSON, and stays readable in logs. @@ -86,7 +87,8 @@ const User = Schema.from(await (await fetch('/schemas/user')).json()); it recognises a metaschema schema by its brand and reads its snapshot, and takes the metaschema module as an option (`{ metaschema }`) only to read a raw definition. CommonJS with ESM named imports, no build step, one package for Node.js 18+ and browsers; the registry, resolve, - streaming and standalone extras are subpaths of 1.8–2.5 KB each. Generated code where the + streaming and standalone extras are subpaths of 1.8–2.5 KB each, the binary wire one of + 7.7 KB. Generated code where the runtime allows it, and under a strict CSP or on edge runtimes either a closures fallback or a standalone module generated at build time. @@ -121,6 +123,7 @@ protoarray alone. | Migrations | `resolve(writer, reader)` (`@alexify/protoarray/resolve`) reads and writes another version by name, with `aliases`; `SchemaRegistry` keeps versions by id and wraps payloads as `[id, payload]` | | Streaming | `encodeEach`/`decodeEach`; NDJSON, `TransformStream`s and the items of a large array read as they arrive (`@alexify/protoarray/stream`) | | Standalone modules | `protoarray compile` writes the generated code of your schemas into a module, for a strict CSP and edge runtimes, with no metaschema at runtime (`@alexify/protoarray/standalone`) | +| Binary wire | `toBinary(schema, value)` and `fromBinary(schema, bytes)` put the same schema on bytes: a header of presence bits, varints, float64s, UTF-16 for long strings; for channels that carry bytes and numeric payloads (`@alexify/protoarray/binary`) | | Validation | the wire on decode by default: `DecodeError` with codes, paths like `orders[1].sku` and the schema name; the values before encode with metaschema's `check` and `parse` | | Evolution | append-only positions; `{ strict: true }` rejects elements beyond the schema; `schema.id` or your own `fingerprint` tells versions apart | @@ -186,6 +189,7 @@ text or bytes and back. | **protoarray**
codegen + metaschema check · v2.0.0 | 689,000 | 624 | 1,630,000 | 1,780,000 | 809,000 | 20,200 | 231,000 | | **protoarray**
columns layout · v2.0.0 | — | 879 | — | — | — | — | — | | **protoarray**
columns, delta + intern · v2.0.0 | — | 958 | — | — | — | — | — | +| **protoarray**
binary wire · v2.0.0 | 1,050,000 | 1,300 | 4,800,000 | 3,310,000 | 1,020,000 | 42,200 | 1,660,000 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 717,000 | 675 | 3,530,000 | 2,920,000 | 270,000 | 14,600 | 520,000 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 261,000 | 301 | 720,000 | 889,000 | 209,000 | 6,400 | 163,000 | | avsc
Type.forSchema · v5.7.9 | 840,000 | 786 | 1,040,000 | 1,680,000 | 300,000 | 14,300 | 1,190,000 | @@ -199,7 +203,7 @@ text or bytes and back. | devalue
stringify · parse · v6.0.3 | 184,000 | 171 | 1,010,000 | 383,000 | 344,000 | 5,960 | 123,000 | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 615,000 | 577 | 3,560,000 | 1,820,000 | 664,000 | — | 395,000 | -_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-16. A round trip is one encode and one decode of the payload; medians, rounded to three significant figures; — where the scenario does not apply._ +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. A round trip is one encode and one decode of the payload; medians, rounded to three significant figures; — where the scenario does not apply._ @@ -233,6 +237,7 @@ module has its codec written already. | **protoarray**
closures, no eval · v2.0.0 | 11.6 ms | 3.8 ms | 0.11 ms | 0.17 ms | 15.7 ms | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 11.1 ms | 1.5 ms | 3.1 ms | 0.35 ms | 16.2 ms | | **protoarray**
codegen · v2.0.0 | 11.7 ms | 4.5 ms | 0.14 ms | 0.16 ms | 16.5 ms | +| **protoarray**
binary wire · v2.0.0 | 13.9 ms | 4.5 ms | 1.4 ms | 0.24 ms | 20.0 ms | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 21.0 ms | 4.5 ms | 0.26 ms | 0.01 ms | 25.7 ms | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 24.5 ms | 0.38 ms | 0.80 ms | 0.06 ms | 25.9 ms | @@ -253,15 +258,16 @@ tree-shaken; `pnpm size:compare`). | devalue
stringify · parse · v6.0.3 | 10.7 KB | 4.0 KB | | | superjson
stringify · parse · v2.2.6 | 11.1 KB | 4.0 KB | | | @msgpack/msgpack
Encoder · Decoder · v3.1.3 | 20.7 KB | 5.8 KB | | -| **protoarray**
standalone module, no eval · v2.0.0 | 18.3 KB | 6.9 KB | the module protoarray compile writes for this schema, with its inline runtime | +| **protoarray**
standalone module, no eval · v2.0.0 | 18.7 KB | 7.1 KB | the module protoarray compile writes for this schema, with its inline runtime | | msgpackr
records, shared structures · v2.1.0 | 27.4 KB | 10.3 KB | | | cbor-x
records · v1.6.6 | 28.6 KB | 10.6 KB | | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 66.6 KB | 18.2 KB | the runtime and the descriptor generated code embeds | -| **protoarray**
codegen, from a snapshot · v2.0.0 | 60.9 KB | 22.0 KB | the schema built from its snapshot: a client without metaschema | | BSON
serialize · deserialize · v7.3.3 | 74.8 KB | 22.2 KB | | +| **protoarray**
codegen, from a snapshot · v2.0.0 | 61.5 KB | 22.3 KB | the schema built from its snapshot: a client without metaschema | | protobufjs
protobufjs/light, Root.fromJSON · v8.8.1 | 105.7 KB | 29.2 KB | reflection; static code from pbjs with protobufjs/minimal is smaller, plus the generated code | +| **protoarray**
binary wire, from a snapshot · v2.0.0 | 84.3 KB | 30.2 KB | the core and the ./binary subpath, the schema built from its snapshot | | avsc
Type.forSchema · v5.7.9 | 115.4 KB | 34.3 KB | avsc's browser build of types, with the buffer and util polyfills it requires | -| **protoarray**
codegen, metaschema definition · v2.0.0 | 120.0 KB | 42.6 KB | protoarray and @alexify/metaschema, which reads the definition and validates values | +| **protoarray**
codegen, metaschema definition · v2.0.0 | 120.6 KB | 42.9 KB | protoarray and @alexify/metaschema, which reads the definition and validates values | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 217.4 KB | 62.9 KB | encode only; JSON.parse decodes | @@ -388,7 +394,7 @@ table shows. Static type from the schema✅ from the metaschema schema (Standard Schema types)— schemaless⚠️ third-party: TypeBox, json-schema-to-ts— schemaless— schemaless; parse returns any— schemaless— schemaless; decompress<T> is a cast✅ generated by pbts from static code✅ generated by protoc-gen-es✅ generated interfaces⚠️ third-party @ovotech/avro-ts codegen✅ generated d.ts (js:ts)✅ generated by flatc --ts✅ generated by capnp-es -ots✅ generated by bebopc✅ the TS type is the schema❌ none— no schema— no schema— no schema— no schema— schemaless— schemaless Values checked before encode✅ metaschema check of the same definition12— schemaless❌ coerces: '12' → 12, null → 013— schemaless— schemaless— schemaless— schemaless✅ Type.verify; encode coerces silently⚠️ toBinary range-checks numbers; strings coerced14⚠️ writer range-checks numbers; no verify15✅ type.isValid(v, { errorHook }); toBuffer throws❌ string in i32 → 0, 2^40 → 016❌ coerces: string in int → 0, overflow wraps17❌ coerces: string in UInt32 → 018❌ trusts: string in uint32 → 019✅ assertEncode, assertStringify, typia.assert⚠️ typeof and range checks, on by default20— no schema— no schema— no schema— no schema— schemaless— schemaless Wire format -Wire formatJSON text (positional arrays)JSON textJSON textJSON text + metadata ({ json, meta })JSON text (flat value table, index refs)21JSON text ([n, …keys, …values])JSON array ([values[], rootKey], encoded strings)binary (Protobuf); ProtoJSON via ext/protojsonbinary (Protobuf); ProtoJSON toJson/fromJsonbinary (Protobuf); JSON via toJSON/fromJSONbinary (Avro); Avro JSON via toStringbinary (Binary/Compact) or JSON (TJSONProtocol)binary (FlatBuffers: vtables + offsets)binary (Cap'n Proto, optional packing)binary (Bebop)JSON text (json.*) or binary Protobuf (protobuf.*)binary (custom, big-endian, varints)binary (MessagePack + record extension)binary (MessagePack)binary (CBOR + record tags)binary (BSON)binary (V8 structured clone)Ion text or Ion binary +Wire formatJSON text (positional arrays); bytes through ./binaryJSON textJSON textJSON text + metadata ({ json, meta })JSON text (flat value table, index refs)21JSON text ([n, …keys, …values])JSON array ([values[], rootKey], encoded strings)binary (Protobuf); ProtoJSON via ext/protojsonbinary (Protobuf); ProtoJSON toJson/fromJsonbinary (Protobuf); JSON via toJSON/fromJSONbinary (Avro); Avro JSON via toStringbinary (Binary/Compact) or JSON (TJSONProtocol)binary (FlatBuffers: vtables + offsets)binary (Cap'n Proto, optional packing)binary (Bebop)JSON text (json.*) or binary Protobuf (protobuf.*)binary (custom, big-endian, varints)binary (MessagePack + record extension)binary (MessagePack)binary (CBOR + record tags)binary (BSON)binary (V8 structured clone)Ion text or Ion binary Human-readable payload✅ JSON arrays✅ JSON✅ JSON✅ JSON✅ JSON, values referenced by index22✅ JSON⚠️ JSON, but base62 numbers and refs❌ binary❌ binary❌ binary⚠️ binary; Avro JSON via type.toString()⚠️ binary; TJSONProtocol is JSON❌ binary❌ binary❌ binary✅ JSON path; protobuf path is binary❌ binary❌ binary❌ binary❌ binary❌ binary (EJSON for text)❌ binary⚠️ Ion text (JSON superset, not JSON) Travels over a JSON or text channel as is✅ plain JSON✅ it is JSON✅ it is JSON✅ it is JSON✅ JSON string✅ JSON string✅ JSON.stringify(compress(x))❌ binary; base64 to travel as text❌ binary; base64 to travel as text❌ binary; base64 to travel as text✅ only as Avro JSON (toString/fromString)✅ only with TJSONProtocol❌ needs base64❌ needs base64❌ needs base64✅ JSON path; protobuf needs base64❌ Buffer; base64 needed❌ binary, needs base64❌ binary, needs base64❌ binary, needs base64❌ binary; EJSON is a separate JSON form❌ needs base64✅ Ion text is a string Field names on the wire❌ positional✅ every message✅ every message✅ every message (and in meta paths)✅ every object⚠️ once per collection (header)⚠️ once per distinct key set❌ tag numbers❌ tag numbers❌ tag numbers❌ positional (schema order)❌ tag numbers (field ids)❌ vtable slots (field ids)❌ fixed offsets❌ positional structs; 1-byte tags in messages✅ JSON: every message; protobuf: tag numbers❌ positional (sorted key order)⚠️ once per structure (new Packr() records)✅ every object⚠️ once per structure (new Encoder() records)✅ every document✅ every object, repeated✅ text: each time; binary: once (symbol table) @@ -524,6 +530,7 @@ const { resolve } = require('@alexify/protoarray/resolve'); const { SchemaRegistry } = require('@alexify/protoarray/registry'); const { parseLines, parseItems /* … */ } = require('@alexify/protoarray/stream'); const { toSource } = require('@alexify/protoarray/standalone'); +const { toBinary, fromBinary } = require('@alexify/protoarray/binary'); ``` ```ts diff --git a/bench/compare/adapters.js b/bench/compare/adapters.js index 5ae540a..9ec07f0 100644 --- a/bench/compare/adapters.js +++ b/bench/compare/adapters.js @@ -327,6 +327,24 @@ const ADAPTERS = { return text(schema); }, }, + // The same schema on the binary wire (`@alexify/protoarray/binary`): the + // codec is built on the first call, which the cold start counts. + 'protoarray-binary': { + name: 'protoarray', + variant: 'binary wire', + package: '@alexify/protoarray', + family: 'binary', + load: () => ({ ...loadProtoarray(), binary: require(path.join(ROOT, 'binary.js')) }), + setup: (scenario, lib) => { + const schema = fromDefinition(lib, scenario); + const { toBinary, fromBinary } = lib.binary; + return { + encode: (value) => toBinary(schema, value), + decode: (bytes) => fromBinary(schema, bytes), + normalize: plain, + }; + }, + }, protobufjs: { name: 'protobufjs', variant: 'reflection, Root.fromJSON', diff --git a/bench/report.js b/bench/report.js index ede2091..e0d3140 100644 --- a/bench/report.js +++ b/bench/report.js @@ -11,8 +11,10 @@ * * The inputs: bench/results/compare.json (pnpm bench:compare --save), * bench/results/size.json (pnpm size:compare --save), - * bench/results/releases.json (pnpm bench:releases) and - * bench/baseline.json (pnpm bench --save). + * bench/results/releases.json (pnpm bench:releases), + * bench/results/binary.json (the run of the binary-wire spike that decided + * the ./binary subpath, kept as it was, docs only) and bench/baseline.json + * (pnpm bench --save). * * Run: pnpm bench:report */ @@ -71,6 +73,7 @@ const main = async () => { const compare = readJson(path.join(RESULTS, 'compare.json')); const size = readJson(path.join(RESULTS, 'size.json')); const releases = readJson(path.join(RESULTS, 'releases.json')); + const binary = readJson(path.join(RESULTS, 'binary.json')); const baseline = readJson(path.join(__dirname, 'baseline.json')); fs.rmSync(SVG_DIR, { recursive: true, force: true }); @@ -87,26 +90,18 @@ const main = async () => { // name → { readme, docs } markdown const regions = {}; - if (compare) { - const caption = model.compareCaption(compare); - let headline = ''; - const more = []; - const sizes = []; - for (const scenario of model.compareScenarios(compare)) { - const speed = model.compareModel(compare, scenario.id, 'roundtrip'); - const img = chartImage(`compare-${scenario.id}.svg`, speed, caption); - if (scenario.id === HEADLINE) headline = img; - else more.push(img); - const bytes = model.compareModel(compare, scenario.id, 'bytes'); - sizes.push(chartImage(`compare-${scenario.id}-bytes.svg`, bytes, caption)); - } - const scenarios = compare.scenarios; - const ok = (result) => result !== undefined && result.error === undefined; - const roundTrip = (result) => 1 / (1 / result.encode.median + 1 / result.decode.median); - const summary = table( + const ok = (result) => result !== undefined && result.error === undefined; + const roundTrip = (result) => 1 / (1 / result.encode.median + 1 / result.decode.median); + + // The tables of a comparison (compare.json, or the binary prototype's + // binary.json of the same shape): the round trips of every library on every + // scenario, and per scenario, encode, decode and the sizes. + const summaryTable = (report) => { + const scenarios = report.scenarios; + return table( ['Library', ...scenarios.map((scenario) => `${scenario.title}
round trips/s`)], ['---', ...scenarios.map(() => '---:')], - compare.libraries + report.libraries .filter((library) => scenarios.some((scenario) => ok(scenario.results[library.id]))) .map((library) => [ libraryCell(library), @@ -116,13 +111,15 @@ const main = async () => { }), ]), ); - const note = - `_${caption}. A round trip is one encode and one decode of the payload; medians, ` + - 'rounded to three significant figures; — where the scenario does not apply._'; - const full = scenarios.map((scenario) => { + }; + const summaryNote = (caption) => + `_${caption}. A round trip is one encode and one decode of the payload; medians, ` + + 'rounded to three significant figures; — where the scenario does not apply._'; + const scenarioTables = (report) => + report.scenarios.map((scenario) => { const json = scenario.results.json; const relative = (value, base) => (ok(json) ? ` (×${(value / base).toFixed(2)})` : ''); - const rows = compare.libraries + const rows = report.libraries .filter((library) => ok(scenario.results[library.id])) .map((library) => { const result = scenario.results[library.id]; @@ -144,11 +141,29 @@ const main = async () => { ), ].join('\n\n'); }); - const fullNote = - `_${caption}. Medians; (×n) against JSON; bytes are the wire size of one payload ` + - `(binary formats before any text encoding), gzip at level ${compare.method.gzipLevel}, ` + - `brotli at quality ${compare.method.brotliQuality}. The spread of the rounds is in ` + - 'bench/results/compare.json._'; + const scenarioNote = (report, caption, file) => + `_${caption}. Medians; (×n) against JSON; bytes are the wire size of one payload ` + + `(binary formats before any text encoding), gzip at level ${report.method.gzipLevel}, ` + + `brotli at quality ${report.method.brotliQuality}. The spread of the rounds is in ` + + `bench/results/${file}._`; + + if (compare) { + const caption = model.compareCaption(compare); + let headline = ''; + const more = []; + const sizes = []; + for (const scenario of model.compareScenarios(compare)) { + const speed = model.compareModel(compare, scenario.id, 'roundtrip'); + const img = chartImage(`compare-${scenario.id}.svg`, speed, caption); + if (scenario.id === HEADLINE) headline = img; + else more.push(img); + const bytes = model.compareModel(compare, scenario.id, 'bytes'); + sizes.push(chartImage(`compare-${scenario.id}-bytes.svg`, bytes, caption)); + } + const summary = summaryTable(compare); + const note = summaryNote(caption); + const full = scenarioTables(compare); + const fullNote = scenarioNote(compare, caption, 'compare.json'); regions.compare = { readme: [ headline, @@ -244,6 +259,22 @@ const main = async () => { }; } + // The binary-wire spike that decided the ./binary subpath, as it ran: its + // tables go to the docs only, under the criteria written there before it. + if (binary) { + const caption = model.compareCaption(binary); + regions.binary = { + docs: [ + summaryTable(binary), + summaryNote(caption), + details('Every scenario: encode, decode and the payload sizes', [ + ...scenarioTables(binary), + scenarioNote(binary, caption, 'binary.json'), + ]), + ].join('\n\n'), + }; + } + if (baseline) { const caption = model.engineCaption(baseline); const images = model.ENGINE_OPS.map((op) => diff --git a/bench/results/binary.json b/bench/results/binary.json new file mode 100644 index 0000000..ff87113 --- /dev/null +++ b/bench/results/binary.json @@ -0,0 +1,1208 @@ +{ + "generatedAt": "2026-10-18T18:05:26.134Z", + "machine": { + "cpu": "Apple M3 Max", + "cores": 16, + "memoryGb": 64, + "os": "darwin 25.6.0 (arm64)", + "node": "v24.14.1", + "v8": "13.6.233.17-node.44" + }, + "method": { + "warmup": 500, + "rounds": 5, + "roundMs": 400, + "coldStartRuns": 10, + "gzipLevel": 6, + "brotliQuality": 5 + }, + "libraries": [ + { + "id": "json", + "name": "JSON", + "variant": "stringify · parse", + "package": "JSON (built in)", + "version": "24.14.1", + "family": "text" + }, + { + "id": "protoarray", + "name": "protoarray", + "variant": "codegen", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "text", + "commit": "5a482c5" + }, + { + "id": "protoarray-closures", + "name": "protoarray", + "variant": "closures, no eval", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "text", + "commit": "5a482c5" + }, + { + "id": "protoarray-binary", + "name": "protoarray", + "variant": "binary wire (prototype)", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "binary", + "commit": "5a482c5" + }, + { + "id": "protoarray-binary-ucs2", + "name": "protoarray", + "variant": "binary wire, two-byte long strings (prototype)", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "binary", + "commit": "5a482c5" + }, + { + "id": "protobufjs", + "name": "protobufjs", + "variant": "reflection, Root.fromJSON", + "package": "protobufjs", + "version": "8.8.1", + "family": "binary" + }, + { + "id": "avsc", + "name": "avsc", + "variant": "Type.forSchema", + "package": "avsc", + "version": "5.7.9", + "family": "binary" + }, + { + "id": "msgpackr-records", + "name": "msgpackr", + "variant": "records, shared structures", + "package": "msgpackr", + "version": "2.1.0", + "family": "binary" + } + ], + "scenarios": [ + { + "id": "single", + "title": "One record", + "description": "one record: 10 fields, nested address, tags, 2 orders", + "results": { + "json": { + "encode": { + "median": 1406221, + "min": 1399177, + "max": 1406973, + "spread": 1 + }, + "decode": { + "median": 956607, + "min": 944556, + "max": 961170, + "spread": 2 + }, + "bytes": { + "raw": 266, + "gzip": 212, + "brotli": 178 + } + }, + "protoarray": { + "encode": { + "median": 2070093, + "min": 2062372, + "max": 2077299, + "spread": 1 + }, + "decode": { + "median": 1730231, + "min": 1709211, + "max": 1734810, + "spread": 1 + }, + "bytes": { + "raw": 135, + "gzip": 134, + "brotli": 110 + } + }, + "protoarray-closures": { + "encode": { + "median": 1496754, + "min": 1490058, + "max": 1516601, + "spread": 2 + }, + "decode": { + "median": 1055561, + "min": 1047187, + "max": 1069680, + "spread": 2 + }, + "bytes": { + "raw": 135, + "gzip": 134, + "brotli": 110 + } + }, + "protoarray-binary": { + "encode": { + "median": 2242292, + "min": 2209664, + "max": 2248111, + "spread": 2 + }, + "decode": { + "median": 1991344, + "min": 1976294, + "max": 1999979, + "spread": 1 + }, + "bytes": { + "raw": 125, + "gzip": 128, + "brotli": 121 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 2222749, + "min": 2200515, + "max": 2230504, + "spread": 1 + }, + "decode": { + "median": 1946489, + "min": 1935506, + "max": 1957854, + "spread": 1 + }, + "bytes": { + "raw": 125, + "gzip": 128, + "brotli": 120 + } + }, + "protobufjs": { + "encode": { + "median": 1405798, + "min": 1380432, + "max": 1407671, + "spread": 2 + }, + "decode": { + "median": 1475189, + "min": 1462196, + "max": 1479926, + "spread": 1 + }, + "bytes": { + "raw": 115, + "gzip": 127, + "brotli": 117 + } + }, + "avsc": { + "encode": { + "median": 1753528, + "min": 1750982, + "max": 1755379, + "spread": 0 + }, + "decode": { + "median": 1817469, + "min": 1804193, + "max": 1828832, + "spread": 1 + }, + "bytes": { + "raw": 97, + "gzip": 111, + "brotli": 101 + } + }, + "msgpackr-records": { + "encode": { + "median": 1481899, + "min": 1433398, + "max": 1489820, + "spread": 4 + }, + "decode": { + "median": 2644916, + "min": 2623301, + "max": 2654217, + "spread": 1 + }, + "bytes": { + "raw": 108, + "gzip": 124, + "brotli": 112 + } + } + } + }, + { + "id": "records", + "title": "1,000 records", + "description": "1000 records of the same shape", + "results": { + "json": { + "encode": { + "median": 1529, + "min": 1524, + "max": 1556, + "spread": 2 + }, + "decode": { + "median": 902, + "min": 886, + "max": 907, + "spread": 2 + }, + "bytes": { + "raw": 271215, + "gzip": 33793, + "brotli": 29133 + } + }, + "protoarray": { + "encode": { + "median": 1987, + "min": 1960, + "max": 2001, + "spread": 2 + }, + "decode": { + "median": 1460, + "min": 1440, + "max": 1463, + "spread": 2 + }, + "bytes": { + "raw": 140738, + "gzip": 30666, + "brotli": 26946 + } + }, + "protoarray-closures": { + "encode": { + "median": 1422, + "min": 1412, + "max": 1430, + "spread": 1 + }, + "decode": { + "median": 979, + "min": 977, + "max": 980, + "spread": 0 + }, + "bytes": { + "raw": 140738, + "gzip": 30666, + "brotli": 26946 + } + }, + "protoarray-binary": { + "encode": { + "median": 3170, + "min": 3165, + "max": 3179, + "spread": 0 + }, + "decode": { + "median": 2255, + "min": 2237, + "max": 2288, + "spread": 2 + }, + "bytes": { + "raw": 129625, + "gzip": 31004, + "brotli": 28428 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 3034, + "min": 3004, + "max": 3048, + "spread": 1 + }, + "decode": { + "median": 2195, + "min": 2179, + "max": 2199, + "spread": 1 + }, + "bytes": { + "raw": 129625, + "gzip": 30995, + "brotli": 28457 + } + }, + "protobufjs": { + "encode": { + "median": 1307, + "min": 1295, + "max": 1315, + "spread": 2 + }, + "decode": { + "median": 1393, + "min": 1388, + "max": 1403, + "spread": 1 + }, + "bytes": { + "raw": 120459, + "gzip": 31414, + "brotli": 28985 + } + }, + "avsc": { + "encode": { + "median": 1416, + "min": 1403, + "max": 1421, + "spread": 1 + }, + "decode": { + "median": 1805, + "min": 1783, + "max": 1820, + "spread": 2 + }, + "bytes": { + "raw": 101839, + "gzip": 29895, + "brotli": 27334 + } + }, + "msgpackr-records": { + "encode": { + "median": 1548, + "min": 1535, + "max": 1562, + "spread": 2 + }, + "decode": { + "median": 2857, + "min": 2828, + "max": 2885, + "spread": 2 + }, + "bytes": { + "raw": 112274, + "gzip": 30957, + "brotli": 28370 + } + } + } + }, + { + "id": "sparse", + "title": "Sparse record", + "description": "20 optional fields, 3 present", + "results": { + "json": { + "encode": { + "median": 6853739, + "min": 6747133, + "max": 6886984, + "spread": 2 + }, + "decode": { + "median": 4568304, + "min": 4492586, + "max": 4601632, + "spread": 2 + }, + "bytes": { + "raw": 33, + "gzip": 53, + "brotli": 36 + } + }, + "protoarray": { + "encode": { + "median": 7299108, + "min": 7083366, + "max": 7367307, + "spread": 4 + }, + "decode": { + "median": 7544052, + "min": 7393377, + "max": 7649839, + "spread": 3 + }, + "bytes": { + "raw": 23, + "gzip": 43, + "brotli": 27 + } + }, + "protoarray-closures": { + "encode": { + "median": 3031008, + "min": 2965152, + "max": 3072558, + "spread": 4 + }, + "decode": { + "median": 2409518, + "min": 2383808, + "max": 2426106, + "spread": 2 + }, + "bytes": { + "raw": 23, + "gzip": 43, + "brotli": 27 + } + }, + "protoarray-binary": { + "encode": { + "median": 12962469, + "min": 12861512, + "max": 13094728, + "spread": 2 + }, + "decode": { + "median": 7850196, + "min": 7792620, + "max": 7869110, + "spread": 1 + }, + "bytes": { + "raw": 18, + "gzip": 34, + "brotli": 22 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 12849384, + "min": 12798374, + "max": 12905530, + "spread": 1 + }, + "decode": { + "median": 8086091, + "min": 8015910, + "max": 8103296, + "spread": 1 + }, + "bytes": { + "raw": 18, + "gzip": 34, + "brotli": 22 + } + }, + "protobufjs": { + "encode": { + "median": 5450409, + "min": 5265748, + "max": 5520426, + "spread": 5 + }, + "decode": { + "median": 9597120, + "min": 9591467, + "max": 9621948, + "spread": 0 + }, + "bytes": { + "raw": 11, + "gzip": 31, + "brotli": 15 + } + }, + "avsc": { + "encode": { + "median": 1445461, + "min": 1432916, + "max": 1450473, + "spread": 1 + }, + "decode": { + "median": 3621499, + "min": 3594603, + "max": 3626941, + "spread": 1 + }, + "bytes": { + "raw": 28, + "gzip": 35, + "brotli": 23 + } + }, + "msgpackr-records": { + "encode": { + "median": 6052760, + "min": 5946798, + "max": 6058131, + "spread": 2 + }, + "decode": { + "median": 12235411, + "min": 12160297, + "max": 12308771, + "spread": 1 + }, + "bytes": { + "raw": 9, + "gzip": 29, + "brotli": 13 + } + } + } + }, + { + "id": "typed", + "title": "Typed record", + "description": "enum, boolean and date heavy record", + "results": { + "json": { + "encode": { + "median": 837791, + "min": 835992, + "max": 842124, + "spread": 1 + }, + "decode": { + "median": 2918908, + "min": 2897945, + "max": 2952798, + "spread": 2 + }, + "bytes": { + "raw": 123, + "gzip": 120, + "brotli": 94 + } + }, + "protoarray": { + "encode": { + "median": 4711181, + "min": 4695702, + "max": 4761356, + "spread": 1 + }, + "decode": { + "median": 4812177, + "min": 4728014, + "max": 4828008, + "spread": 2 + }, + "bytes": { + "raw": 36, + "gzip": 46, + "brotli": 36 + } + }, + "protoarray-closures": { + "encode": { + "median": 3598327, + "min": 3576053, + "max": 3612172, + "spread": 1 + }, + "decode": { + "median": 3150768, + "min": 3145035, + "max": 3171297, + "spread": 1 + }, + "bytes": { + "raw": 36, + "gzip": 46, + "brotli": 36 + } + }, + "protoarray-binary": { + "encode": { + "median": 6616242, + "min": 6586410, + "max": 6623561, + "spread": 1 + }, + "decode": { + "median": 6697672, + "min": 6608825, + "max": 6712150, + "spread": 2 + }, + "bytes": { + "raw": 26, + "gzip": 38, + "brotli": 30 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 6655370, + "min": 6622318, + "max": 6684491, + "spread": 1 + }, + "decode": { + "median": 6571614, + "min": 6527975, + "max": 6628396, + "spread": 2 + }, + "bytes": { + "raw": 26, + "gzip": 38, + "brotli": 30 + } + }, + "protobufjs": { + "encode": { + "median": 3711548, + "min": 3549749, + "max": 3749889, + "spread": 5 + }, + "decode": { + "median": 12220629, + "min": 11787137, + "max": 12251547, + "spread": 4 + }, + "bytes": { + "raw": 13, + "gzip": 33, + "brotli": 17 + } + }, + "avsc": { + "encode": { + "median": 1917877, + "min": 1896201, + "max": 2005238, + "spread": 6 + }, + "decode": { + "median": 7881899, + "min": 7863402, + "max": 7899043, + "spread": 0 + }, + "bytes": { + "raw": 12, + "gzip": 32, + "brotli": 16 + } + }, + "msgpackr-records": { + "encode": { + "median": 3477755, + "min": 3457508, + "max": 3496460, + "spread": 1 + }, + "decode": { + "median": 5190129, + "min": 5146188, + "max": 5211259, + "spread": 1 + }, + "bytes": { + "raw": 24, + "gzip": 45, + "brotli": 28 + } + } + } + }, + { + "id": "strings", + "title": "Long strings", + "description": "long Cyrillic strings", + "results": { + "json": { + "encode": { + "median": 1522856, + "min": 1504459, + "max": 1558508, + "spread": 4 + }, + "decode": { + "median": 1346931, + "min": 1342229, + "max": 1362474, + "spread": 2 + }, + "bytes": { + "raw": 816, + "gzip": 266, + "brotli": 229 + } + }, + "protoarray": { + "encode": { + "median": 1806954, + "min": 1786906, + "max": 1816899, + "spread": 2 + }, + "decode": { + "median": 1720687, + "min": 1711216, + "max": 1723601, + "spread": 1 + }, + "bytes": { + "raw": 785, + "gzip": 230, + "brotli": 207 + } + }, + "protoarray-closures": { + "encode": { + "median": 1695879, + "min": 1677466, + "max": 1710626, + "spread": 2 + }, + "decode": { + "median": 1573136, + "min": 1558429, + "max": 1587353, + "spread": 2 + }, + "bytes": { + "raw": 785, + "gzip": 230, + "brotli": 207 + } + }, + "protoarray-binary": { + "encode": { + "median": 907396, + "min": 900390, + "max": 910579, + "spread": 1 + }, + "decode": { + "median": 1133018, + "min": 1129966, + "max": 1146427, + "spread": 1 + }, + "bytes": { + "raw": 774, + "gzip": 228, + "brotli": 205 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 1879701, + "min": 1843875, + "max": 1908632, + "spread": 3 + }, + "decode": { + "median": 2241510, + "min": 2229004, + "max": 2267624, + "spread": 2 + }, + "bytes": { + "raw": 821, + "gzip": 265, + "brotli": 247 + } + }, + "protobufjs": { + "encode": { + "median": 571717, + "min": 563380, + "max": 580623, + "spread": 3 + }, + "decode": { + "median": 508360, + "min": 502700, + "max": 509096, + "spread": 1 + }, + "bytes": { + "raw": 777, + "gzip": 233, + "brotli": 211 + } + }, + "avsc": { + "encode": { + "median": 500024, + "min": 497215, + "max": 503655, + "spread": 1 + }, + "decode": { + "median": 778250, + "min": 772073, + "max": 785690, + "spread": 2 + }, + "bytes": { + "raw": 774, + "gzip": 231, + "brotli": 210 + } + }, + "msgpackr-records": { + "encode": { + "median": 881678, + "min": 869083, + "max": 883749, + "spread": 2 + }, + "decode": { + "median": 241388, + "min": 240062, + "max": 242458, + "spread": 1 + }, + "bytes": { + "raw": 775, + "gzip": 230, + "brotli": 206 + } + } + } + }, + { + "id": "packets", + "title": "100 wrpc packets", + "description": "100 wrpc packets: calls, callbacks and events", + "results": { + "json": { + "encode": { + "median": 51772, + "min": 51485, + "max": 52406, + "spread": 2 + }, + "decode": { + "median": 24043, + "min": 23774, + "max": 24197, + "spread": 2 + }, + "bytes": { + "raw": 8834, + "gzip": 852, + "brotli": 630 + } + }, + "protoarray": { + "encode": { + "median": 67870, + "min": 66972, + "max": 67950, + "spread": 1 + }, + "decode": { + "median": 50529, + "min": 50358, + "max": 50847, + "spread": 1 + }, + "bytes": { + "raw": 4332, + "gzip": 685, + "brotli": 501 + } + }, + "protoarray-closures": { + "encode": { + "median": 48917, + "min": 48126, + "max": 49083, + "spread": 2 + }, + "decode": { + "median": 27079, + "min": 26924, + "max": 27146, + "spread": 1 + }, + "bytes": { + "raw": 4332, + "gzip": 685, + "brotli": 501 + } + }, + "protoarray-binary": { + "encode": { + "median": 107251, + "min": 103586, + "max": 108155, + "spread": 4 + }, + "decode": { + "median": 68197, + "min": 67782, + "max": 69141, + "spread": 2 + }, + "bytes": { + "raw": 3064, + "gzip": 653, + "brotli": 539 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 107579, + "min": 105315, + "max": 108344, + "spread": 3 + }, + "decode": { + "median": 68411, + "min": 66398, + "max": 68746, + "spread": 3 + }, + "bytes": { + "raw": 3064, + "gzip": 656, + "brotli": 540 + } + }, + "protobufjs": { + "encode": { + "median": 23943, + "min": 23800, + "max": 24177, + "spread": 2 + }, + "decode": { + "median": 37437, + "min": 37187, + "max": 37614, + "spread": 1 + }, + "bytes": { + "raw": 3691, + "gzip": 778, + "brotli": 609 + } + }, + "avsc": { + "encode": { + "median": 20736, + "min": 20150, + "max": 21232, + "spread": 5 + }, + "decode": { + "median": 46589, + "min": 46334, + "max": 46917, + "spread": 1 + }, + "bytes": { + "raw": 2833, + "gzip": 633, + "brotli": 521 + } + }, + "msgpackr-records": { + "encode": { + "median": 29835, + "min": 29596, + "max": 29932, + "spread": 1 + }, + "decode": { + "median": 91199, + "min": 89897, + "max": 91864, + "spread": 2 + }, + "bytes": { + "raw": 3697, + "gzip": 725, + "brotli": 566 + } + } + } + }, + { + "id": "numeric", + "title": "32 doubles", + "description": "32 doubles (where binary formats shine)", + "results": { + "json": { + "encode": { + "median": 320721, + "min": 318824, + "max": 324195, + "spread": 2 + }, + "decode": { + "median": 485627, + "min": 482444, + "max": 489712, + "spread": 1 + }, + "bytes": { + "raw": 764, + "gzip": 406, + "brotli": 392 + } + }, + "protoarray": { + "encode": { + "median": 424447, + "min": 417839, + "max": 425849, + "spread": 2 + }, + "decode": { + "median": 738887, + "min": 736106, + "max": 745242, + "spread": 1 + }, + "bytes": { + "raw": 582, + "gzip": 309, + "brotli": 295 + } + }, + "protoarray-closures": { + "encode": { + "median": 367319, + "min": 363732, + "max": 368473, + "spread": 1 + }, + "decode": { + "median": 572652, + "min": 570211, + "max": 573987, + "spread": 1 + }, + "bytes": { + "raw": 582, + "gzip": 309, + "brotli": 295 + } + }, + "protoarray-binary": { + "encode": { + "median": 2497902, + "min": 2432797, + "max": 2518236, + "spread": 3 + }, + "decode": { + "median": 5022385, + "min": 4897009, + "max": 5033477, + "spread": 3 + }, + "bytes": { + "raw": 260, + "gzip": 283, + "brotli": 262 + } + }, + "protoarray-binary-ucs2": { + "encode": { + "median": 2515262, + "min": 2500544, + "max": 2527607, + "spread": 1 + }, + "decode": { + "median": 5054823, + "min": 5014794, + "max": 5067420, + "spread": 1 + }, + "bytes": { + "raw": 260, + "gzip": 283, + "brotli": 262 + } + }, + "protobufjs": { + "encode": { + "median": 723688, + "min": 714289, + "max": 727934, + "spread": 2 + }, + "decode": { + "median": 1944379, + "min": 1918989, + "max": 1949028, + "spread": 2 + }, + "bytes": { + "raw": 305, + "gzip": 328, + "brotli": 304 + } + }, + "avsc": { + "encode": { + "median": 1950178, + "min": 1914124, + "max": 1967303, + "spread": 3 + }, + "decode": { + "median": 3115226, + "min": 3086008, + "max": 3141099, + "spread": 2 + }, + "bytes": { + "raw": 256, + "gzip": 279, + "brotli": 260 + } + }, + "msgpackr-records": { + "encode": { + "median": 1419571, + "min": 1414265, + "max": 1424695, + "spread": 1 + }, + "decode": { + "median": 3426719, + "min": 3323529, + "max": 3449207, + "spread": 4 + }, + "bytes": { + "raw": 289, + "gzip": 312, + "brotli": 280 + } + } + } + } + ], + "coldStart": { + "scenario": "single", + "results": {} + } +} diff --git a/bench/results/compare.json b/bench/results/compare.json index 5d445a4..d254935 100644 --- a/bench/results/compare.json +++ b/bench/results/compare.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-16T20:34:06.440Z", + "generatedAt": "2026-10-18T21:15:18.239Z", "machine": { "cpu": "Apple M3 Max", "cores": 16, @@ -88,6 +88,15 @@ "family": "text", "commit": "72941bf" }, + { + "id": "protoarray-binary", + "name": "protoarray", + "variant": "binary wire", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "binary", + "commit": "5a482c5" + }, { "id": "protobufjs", "name": "protobufjs", @@ -513,6 +522,25 @@ "gzip": 212, "brotli": 178 } + }, + "protoarray-binary": { + "encode": { + "median": 2250702, + "min": 2230346, + "max": 2262398, + "spread": 1 + }, + "decode": { + "median": 1975521, + "min": 1948973, + "max": 2004565, + "spread": 3 + }, + "bytes": { + "raw": 125, + "gzip": 128, + "brotli": 120 + } } } }, @@ -881,6 +909,25 @@ "gzip": 33793, "brotli": 29133 } + }, + "protoarray-binary": { + "encode": { + "median": 3152, + "min": 3085, + "max": 3183, + "spread": 3 + }, + "decode": { + "median": 2215, + "min": 2182, + "max": 2268, + "spread": 4 + }, + "bytes": { + "raw": 129625, + "gzip": 30995, + "brotli": 28457 + } } } }, @@ -1211,6 +1258,25 @@ "gzip": 53, "brotli": 36 } + }, + "protoarray-binary": { + "encode": { + "median": 12291110, + "min": 12263482, + "max": 12615704, + "spread": 3 + }, + "decode": { + "median": 7879352, + "min": 7813400, + "max": 7920137, + "spread": 1 + }, + "bytes": { + "raw": 18, + "gzip": 34, + "brotli": 22 + } } } }, @@ -1541,6 +1607,25 @@ "gzip": 107, "brotli": 83 } + }, + "protoarray-binary": { + "encode": { + "median": 6636444, + "min": 6608522, + "max": 6654509, + "spread": 1 + }, + "decode": { + "median": 6594808, + "min": 6535206, + "max": 6629859, + "spread": 1 + }, + "bytes": { + "raw": 26, + "gzip": 38, + "brotli": 30 + } } } }, @@ -1871,6 +1956,25 @@ "gzip": 266, "brotli": 229 } + }, + "protoarray-binary": { + "encode": { + "median": 1849724, + "min": 1817832, + "max": 1874090, + "spread": 3 + }, + "decode": { + "median": 2276200, + "min": 2229300, + "max": 2283172, + "spread": 2 + }, + "bytes": { + "raw": 821, + "gzip": 265, + "brotli": 247 + } } } }, @@ -2182,6 +2286,25 @@ "gzip": 1721, "brotli": 1345 } + }, + "protoarray-binary": { + "encode": { + "median": 110336, + "min": 108068, + "max": 111133, + "spread": 3 + }, + "decode": { + "median": 68447, + "min": 67465, + "max": 68867, + "spread": 2 + }, + "bytes": { + "raw": 3064, + "gzip": 656, + "brotli": 540 + } } } }, @@ -2512,6 +2635,25 @@ "gzip": 406, "brotli": 392 } + }, + "protoarray-binary": { + "encode": { + "median": 2492508, + "min": 2400375, + "max": 2498713, + "spread": 4 + }, + "decode": { + "median": 5009602, + "min": 4959766, + "max": 5025558, + "spread": 1 + }, + "bytes": { + "raw": 260, + "gzip": 283, + "brotli": 262 + } } } } @@ -2630,6 +2772,13 @@ "encode": 0.255, "decode": 0.012, "total": 25.748 + }, + "protoarray-binary": { + "load": 13.856, + "build": 4.506, + "encode": 1.447, + "decode": 0.239, + "total": 20.002 } } } diff --git a/bench/results/size.json b/bench/results/size.json index 34233f2..7b2f7f3 100644 --- a/bench/results/size.json +++ b/bench/results/size.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-10-16T20:47:11.991Z", + "generatedAt": "2026-10-18T21:15:48.374Z", "scenario": "single", "libraries": [ { @@ -9,8 +9,8 @@ "package": "@alexify/protoarray", "version": "2.0.0", "family": "text", - "min": 62390, - "gzip": 22538, + "min": 62963, + "gzip": 22802, "note": "the schema built from its snapshot: a client without metaschema" }, { @@ -20,10 +20,21 @@ "package": "@alexify/protoarray", "version": "2.0.0", "family": "text", - "min": 122929, - "gzip": 43642, + "min": 123502, + "gzip": 43886, "note": "protoarray and @alexify/metaschema, which reads the definition and validates values" }, + { + "id": "protoarray-binary", + "name": "protoarray", + "variant": "binary wire, from a snapshot", + "package": "@alexify/protoarray", + "version": "2.0.0", + "family": "binary", + "min": 86325, + "gzip": 30875, + "note": "the core and the ./binary subpath, the schema built from its snapshot" + }, { "id": "protoarray-standalone", "name": "protoarray", @@ -31,8 +42,8 @@ "package": "@alexify/protoarray", "version": "2.0.0", "family": "text", - "min": 18712, - "gzip": 7069, + "min": 19151, + "gzip": 7264, "note": "the module protoarray compile writes for this schema, with its inline runtime" }, { diff --git a/binary.d.ts b/binary.d.ts new file mode 100644 index 0000000..54a3b34 --- /dev/null +++ b/binary.d.ts @@ -0,0 +1,40 @@ +/** + * Hand-written typings for @alexify/protoarray/binary: the same schema on a binary wire. The + * bytes are a `Uint8Array`; both sides need the same schema (see Wire Format in the guide). + */ + +import type { AnySchema } from './index.js'; + +type Value = Parameters[0]; +type Decoded = ReturnType; + +/** + * The bytes of a value on the binary wire, a `Uint8Array` of their own. Throws `EncodeError` for + * a value with no form (the wrong shape, `null` at the root, a union value outside the union, a + * decimal past the safe integers), `SchemaDefinitionError` `ERR_NO_BINARY_FORM` for a schema + * with interned strings, lists of differences, a flat or columns layout, or one from `resolve`. + */ +export function toBinary(schema: S, value: Value): Uint8Array; + +/** + * The value of the bytes `toBinary` of the same schema gave. Throws `DecodeError` for a message + * that is not of the schema (truncated, an index out of range, a required field missing) when + * the schema validates, and for anything but a `Uint8Array`. + */ +export function fromBinary(schema: S, bytes: Uint8Array): Decoded; + +/** Installs `toBinary` and `fromBinary` as methods of Schema. Once per process. */ +export function register(): void; + +declare module './index.js' { + interface Schema { + /** `toBinary` of `@alexify/protoarray/binary`, once its `register()` has run. */ + toBinary(value: Input): Uint8Array; + /** `fromBinary` of `@alexify/protoarray/binary`, once its `register()` has run. */ + fromBinary(bytes: Uint8Array): Output; + } + interface Collection { + toBinary(value: readonly Input[]): Uint8Array; + fromBinary(bytes: Uint8Array): Output[]; + } +} diff --git a/binary.js b/binary.js new file mode 100644 index 0000000..1ae20bf --- /dev/null +++ b/binary.js @@ -0,0 +1 @@ +module.exports = require('./src/binary.js'); diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 9adb90a..77a5f91 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -145,6 +145,7 @@ export default withMermaid( { text: 'TypeScript', link: '/guide/typescript' }, { text: 'Runtimes and CSP', link: '/guide/runtimes' }, { text: 'Standalone Modules', link: '/guide/standalone' }, + { text: 'The Binary Wire', link: '/guide/binary' }, { text: 'Compression', link: '/guide/compression' }, { text: 'With wrpc', link: '/guide/wrpc' }, { text: 'Benchmarks', link: '/guide/benchmarks' }, diff --git a/docs/api/exports.md b/docs/api/exports.md index 9238806..cc7a427 100644 --- a/docs/api/exports.md +++ b/docs/api/exports.md @@ -8,7 +8,7 @@ const { } = require('@alexify/protoarray'); ``` -Four subpaths carry what the core does not need, so a program that does not use them does not +Five subpaths carry what the core does not need, so a program that does not use them does not bundle them: | Subpath | Exports | Page | @@ -17,6 +17,7 @@ bundle them: | `@alexify/protoarray/registry` | `SchemaRegistry` | [Migrations and the Registry](/guide/resolve#the-registry) | | `@alexify/protoarray/stream` | `stringifyLines`, `parseLines`, `stringifyStream`, `parseStream`, `stringifyItems`, `parseItems`, `register` | [Streaming](/guide/streaming) | | `@alexify/protoarray/standalone` | `toSource`, `runtime`, `register` | [Standalone Modules](/guide/standalone) | +| `@alexify/protoarray/binary` | `toBinary`, `fromBinary`, `register` | [The Binary Wire](/guide/binary) | The package also installs a `protoarray` command: `protoarray compile ` (see [Standalone Modules](/guide/standalone#protoarray-compile)). None of it requires @@ -177,3 +178,24 @@ See [TypeScript](/guide/typescript) for how a schema takes its types from metasc | `Infer`, `Input`, `Encoded`, `Definition` and the other definition types | `Schema`, `Decoded`, `InputOf`, `OutputOf` | See [Migrating from 1.x](/guide/migrating-from-1). + +## `toBinary(schema, value)`, `fromBinary(schema, bytes)` + +```ts +import { toBinary, fromBinary, register } from '@alexify/protoarray/binary'; + +function toBinary(schema: S, value: Parameters[0]): Uint8Array; +function fromBinary(schema: S, bytes: Uint8Array): ReturnType; +function register(): void; // installs schema.toBinary(value) and schema.fromBinary(bytes) +``` + +The same schema on the [binary wire](/guide/binary): `toBinary` gives a `Uint8Array` of the value, +`fromBinary` the value back, for a schema or a collection (`schema.list`). The codec is compiled +from the schema's plan on first use and kept on the schema, by the backend the schema runs, and +checks the wire on decode as the schema's `validate` says. Throws `EncodeError` for a value with +no form (`null` at the root, the wrong shape, a union value outside the union, a decimal past the +safe integers, a string or `bytes` slot of another type), `DecodeError` for a message that is not +of the schema (truncated, an index out of range, a required field missing; anything but a +`Uint8Array` is `structure`), and `SchemaDefinitionError` `ERR_NO_BINARY_FORM` for a schema with +interned strings, lists of differences, a `flat` or `columns` layout, or one from `resolve`. Both +sides need the same schema. diff --git a/docs/comparison.json b/docs/comparison.json index c925941..881dd90 100644 --- a/docs/comparison.json +++ b/docs/comparison.json @@ -311,7 +311,7 @@ }, "wire": { "mark": "", - "text": "JSON text (positional arrays)" + "text": "JSON text (positional arrays); bytes through `./binary`" }, "readable": { "mark": "✅", diff --git a/docs/guide/benchmarks.md b/docs/guide/benchmarks.md index 23db221..f0da528 100644 --- a/docs/guide/benchmarks.md +++ b/docs/guide/benchmarks.md @@ -53,6 +53,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 1,500,853 (×1.08) | 1,065,375 (×1.10) | 135 | 134 | 110 | | **protoarray**
standalone module, no eval · v2.0.0 | 2,038,604 (×1.46) | 1,606,745 (×1.66) | 135 | 134 | 110 | | **protoarray**
codegen + metaschema check · v2.0.0 | 1,148,641 (×0.82) | 1,723,478 (×1.78) | 135 | 134 | 110 | +| **protoarray**
binary wire · v2.0.0 | 2,250,702 (×1.62) | 1,975,521 (×2.04) | 125 | 128 | 120 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 1,398,695 (×1.00) | 1,472,331 (×1.52) | 115 | 127 | 117 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 487,218 (×0.35) | 560,158 (×0.58) | 115 | 127 | 117 | | avsc
Type.forSchema · v5.7.9 | 1,626,234 (×1.17) | 1,738,513 (×1.79) | 97 | 111 | 101 | @@ -77,6 +78,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
codegen + metaschema check · v2.0.0 | 1,090 (×0.74) | 1,462 (×1.64) | 140,738 | 30,666 | 26,946 | | **protoarray**
columns layout · v2.0.0 | 2,059 (×1.40) | 1,535 (×1.72) | 138,758 | 27,412 | 21,053 | | **protoarray**
columns, delta + intern · v2.0.0 | 2,245 (×1.53) | 1,670 (×1.87) | 114,965 | 24,574 | 19,799 | +| **protoarray**
binary wire · v2.0.0 | 3,152 (×2.15) | 2,215 (×2.49) | 129,625 | 30,995 | 28,457 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 1,301 (×0.89) | 1,403 (×1.57) | 120,459 | 31,414 | 28,985 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 617 (×0.42) | 587 (×0.66) | 120,459 | 31,414 | 28,985 | | avsc
Type.forSchema · v5.7.9 | 1,399 (×0.95) | 1,793 (×2.01) | 101,839 | 29,895 | 27,334 | @@ -99,6 +101,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 3,029,115 (×0.45) | 2,382,353 (×0.52) | 23 | 43 | 27 | | **protoarray**
standalone module, no eval · v2.0.0 | 6,712,097 (×0.99) | 7,171,599 (×1.57) | 23 | 43 | 27 | | **protoarray**
codegen + metaschema check · v2.0.0 | 2,089,158 (×0.31) | 7,340,173 (×1.61) | 23 | 43 | 27 | +| **protoarray**
binary wire · v2.0.0 | 12,291,110 (×1.81) | 7,879,352 (×1.73) | 18 | 34 | 22 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 5,568,923 (×0.82) | 9,670,688 (×2.12) | 11 | 31 | 15 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 966,102 (×0.14) | 2,827,757 (×0.62) | 11 | 31 | 15 | | avsc
Type.forSchema · v5.7.9 | 1,457,467 (×0.21) | 3,623,007 (×0.79) | 28 | 35 | 23 | @@ -121,6 +124,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 3,514,161 (×4.28) | 3,118,202 (×1.08) | 36 | 46 | 36 | | **protoarray**
standalone module, no eval · v2.0.0 | 4,666,737 (×5.68) | 4,696,261 (×1.62) | 36 | 46 | 36 | | **protoarray**
codegen + metaschema check · v2.0.0 | 2,803,310 (×3.41) | 4,851,985 (×1.67) | 36 | 46 | 36 | +| **protoarray**
binary wire · v2.0.0 | 6,636,444 (×8.08) | 6,594,808 (×2.28) | 26 | 38 | 30 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 3,837,319 (×4.67) | 12,143,747 (×4.19) | 13 | 33 | 17 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 1,272,891 (×1.55) | 2,952,236 (×1.02) | 13 | 33 | 17 | | avsc
Type.forSchema · v5.7.9 | 2,135,877 (×2.60) | 7,911,258 (×2.73) | 12 | 32 | 16 | @@ -143,6 +147,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 1,673,013 (×1.07) | 1,557,716 (×1.17) | 785 | 230 | 207 | | **protoarray**
standalone module, no eval · v2.0.0 | 1,758,113 (×1.13) | 1,674,028 (×1.25) | 785 | 230 | 207 | | **protoarray**
codegen + metaschema check · v2.0.0 | 1,548,540 (×0.99) | 1,693,591 (×1.27) | 785 | 230 | 207 | +| **protoarray**
binary wire · v2.0.0 | 1,849,724 (×1.19) | 2,276,200 (×1.70) | 821 | 265 | 247 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 575,285 (×0.37) | 507,626 (×0.38) | 777 | 233 | 211 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 407,763 (×0.26) | 427,814 (×0.32) | 777 | 233 | 211 | | avsc
Type.forSchema · v5.7.9 | 491,511 (×0.32) | 769,652 (×0.58) | 774 | 231 | 210 | @@ -165,6 +170,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 48,093 (×0.93) | 26,769 (×1.11) | 4,332 | 685 | 501 | | **protoarray**
standalone module, no eval · v2.0.0 | 65,648 (×1.26) | 47,442 (×1.97) | 4,332 | 685 | 501 | | **protoarray**
codegen + metaschema check · v2.0.0 | 33,902 (×0.65) | 49,890 (×2.07) | 4,332 | 685 | 501 | +| **protoarray**
binary wire · v2.0.0 | 110,336 (×2.13) | 68,447 (×2.84) | 3,064 | 656 | 540 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 24,172 (×0.47) | 37,015 (×1.54) | 3,691 | 778 | 609 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 10,961 (×0.21) | 15,383 (×0.64) | 3,691 | 778 | 609 | | avsc
Type.forSchema · v5.7.9 | 20,632 (×0.40) | 46,294 (×1.92) | 2,833 | 633 | 521 | @@ -186,6 +192,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | **protoarray**
closures, no eval · v2.0.0 | 360,928 (×1.12) | 566,586 (×1.17) | 582 | 309 | 295 | | **protoarray**
standalone module, no eval · v2.0.0 | 404,218 (×1.25) | 731,900 (×1.51) | 582 | 309 | 295 | | **protoarray**
codegen + metaschema check · v2.0.0 | 336,196 (×1.04) | 737,943 (×1.53) | 582 | 309 | 295 | +| **protoarray**
binary wire · v2.0.0 | 2,492,508 (×7.71) | 5,009,602 (×10.36) | 260 | 283 | 262 | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 709,089 (×2.19) | 1,944,328 (×4.02) | 305 | 328 | 304 | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 252,782 (×0.78) | 455,256 (×0.94) | 305 | 328 | 304 | | avsc
Type.forSchema · v5.7.9 | 1,937,808 (×5.99) | 3,115,855 (×6.45) | 256 | 279 | 260 | @@ -199,7 +206,7 @@ call, the ratio to JSON, the spread of the rounds and the exact version. | devalue
stringify · parse · v6.0.3 | 230,286 (×0.71) | 265,668 (×0.55) | 853 | 454 | 443 | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 2,243,379 (×6.94) | 479,670 (×0.99) | 764 | 406 | 392 | -_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-16. Medians; (×n) against JSON; bytes are the wire size of one payload (binary formats before any text encoding), gzip at level 6, brotli at quality 5. The spread of the rounds is in bench/results/compare.json._ +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. Medians; (×n) against JSON; bytes are the wire size of one payload (binary formats before any text encoding), gzip at level 6, brotli at quality 5. The spread of the rounds is in bench/results/compare.json._ @@ -219,9 +226,12 @@ _Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each booleans). Over a round trip protoarray is first on one record, long strings and tagged messages, second behind msgpackr on lists and sparse records and behind protobufjs on typed ones. -- **Numbers** are the honest exception: a double prints as up to 17 characters and parses back - through a decimal conversion, where avsc and protobufjs read 8 bytes. Use a binary codec for - dense numeric payloads. +- **Numbers** are the honest exception of the text wire: a double prints as up to 17 characters + and parses back through a decimal conversion, where avsc and protobufjs read 8 bytes. The + `binary wire` rows are [`@alexify/protoarray/binary`](/guide/binary), the same schema on bytes, + for channels that carry them: ahead of the text wire on every payload but long strings, where + the two are close, and ahead of the binary codecs on most (how it was decided is + [recorded below](#the-binary-wire-measured)). - **Validation.** The wire is checked on decode and the values given to encode are trusted, so protoarray encodes the way protobufjs and msgpackr do, without checks, and decodes with them. Checking the values is metaschema's: the `codegen + metaschema check` row runs metaschema's @@ -271,6 +281,7 @@ process, loading itself, building the schema of one record, encoding it and deco | **protoarray**
closures, no eval · v2.0.0 | 11.6 ms | 3.8 ms | 0.11 ms | 0.17 ms | 15.7 ms | | protobufjs
reflection, Root.fromJSON · v8.8.1 | 11.1 ms | 1.5 ms | 3.1 ms | 0.35 ms | 16.2 ms | | **protoarray**
codegen · v2.0.0 | 11.7 ms | 4.5 ms | 0.14 ms | 0.16 ms | 16.5 ms | +| **protoarray**
binary wire · v2.0.0 | 13.9 ms | 4.5 ms | 1.4 ms | 0.24 ms | 20.0 ms | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 21.0 ms | 4.5 ms | 0.26 ms | 0.01 ms | 25.7 ms | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 24.5 ms | 0.38 ms | 0.80 ms | 0.06 ms | 25.9 ms | @@ -297,15 +308,16 @@ by esbuild for the browser (minified, tree-shaken) and checked to give the recor | devalue
stringify · parse · v6.0.3 | 10.7 KB | 4.0 KB | | | superjson
stringify · parse · v2.2.6 | 11.1 KB | 4.0 KB | | | @msgpack/msgpack
Encoder · Decoder · v3.1.3 | 20.7 KB | 5.8 KB | | -| **protoarray**
standalone module, no eval · v2.0.0 | 18.3 KB | 6.9 KB | the module protoarray compile writes for this schema, with its inline runtime | +| **protoarray**
standalone module, no eval · v2.0.0 | 18.7 KB | 7.1 KB | the module protoarray compile writes for this schema, with its inline runtime | | msgpackr
records, shared structures · v2.1.0 | 27.4 KB | 10.3 KB | | | cbor-x
records · v1.6.6 | 28.6 KB | 10.6 KB | | | protobuf-es
create · toBinary · fromBinary · v2.16.0 | 66.6 KB | 18.2 KB | the runtime and the descriptor generated code embeds | -| **protoarray**
codegen, from a snapshot · v2.0.0 | 60.9 KB | 22.0 KB | the schema built from its snapshot: a client without metaschema | | BSON
serialize · deserialize · v7.3.3 | 74.8 KB | 22.2 KB | | +| **protoarray**
codegen, from a snapshot · v2.0.0 | 61.5 KB | 22.3 KB | the schema built from its snapshot: a client without metaschema | | protobufjs
protobufjs/light, Root.fromJSON · v8.8.1 | 105.7 KB | 29.2 KB | reflection; static code from pbjs with protobufjs/minimal is smaller, plus the generated code | +| **protoarray**
binary wire, from a snapshot · v2.0.0 | 84.3 KB | 30.2 KB | the core and the ./binary subpath, the schema built from its snapshot | | avsc
Type.forSchema · v5.7.9 | 115.4 KB | 34.3 KB | avsc's browser build of types, with the buffer and util polyfills it requires | -| **protoarray**
codegen, metaschema definition · v2.0.0 | 120.0 KB | 42.6 KB | protoarray and @alexify/metaschema, which reads the definition and validates values | +| **protoarray**
codegen, metaschema definition · v2.0.0 | 120.6 KB | 42.9 KB | protoarray and @alexify/metaschema, which reads the definition and validates values | | fast-json-stringify
compiled stringify, JSON.parse · v7.1.0 | 217.4 KB | 62.9 KB | encode only; JSON.parse decodes | @@ -368,6 +380,178 @@ once, as sending it would (see [Runtimes](/guide/runtimes#how-stringify-writes-t resolved schema, an inline standalone module, `parseItems` against parsing the whole text, and `Schema.from` with each backend. +## The binary wire, measured + +protoarray writes JSON text because the engines print and parse it natively, and the honest +exception above is numbers: a double is up to 17 characters of text against 8 bytes. The +`binary wire` rows of the comparison above are +[`@alexify/protoarray/binary`](/guide/binary), the same schema on bytes, and this section is the +record of how it was decided: two research spikes under `bench/binary/` (since removed; the +subpath is their product) and criteria written before the second one ran. The first spike +(October 17) compiled the plan into closures and came 11% short of protobufjs on one record, +which left the question to a sketch with generated code. The second compiled the plan the way +`src/codegen.js` does, through `new Function`: one encode and one decode function per record, a +decoded record built as one object literal, a union as a switch over its tag, a referenced +schema called through a slot, with the first spike's closures as the reference that writes the +same bytes. Its wire is the subpath's, with one difference: strings were UTF-8, where the subpath +writes a string past 21 characters as UTF-16. + +Two notes on fairness. protobufjs's decode returns a `Message` (its `toObject` runs outside the +timing); the binary codec returns plain objects, as the product does. And `typed` declares its +ids as `number`, eight bytes here where protobuf and Avro write an `int32`. + +The decision was written down before the run, a round trip being one encode and one decode: + +1. **C1, the speed on the question.** On 32 doubles the prototype's round trip is at least twice + the generated text code's. +2. **C2, competitive.** On 32 doubles its round trip reaches 60% of avsc's. +3. **C3, not a slower protobuf.** On one record its round trip is at least protobufjs's, in the + same run. +4. **C4, strings.** On long strings its round trip is at least 0.8 of JSON's: every binary codec + written in JavaScript transcodes UTF-8 where `JSON.parse` reads the engine's own strings, and + the only levers are where the loop ends and the platform's call begins. +5. **C5, lists.** On 1,000 records it decodes at least as fast as avsc. + +Reported, not gated: the round trip on 1,000 records against msgpackr's shared structures, the +sparse and typed records, the wrpc packets, what the codec would add to the core bundle, and a +variant outside the wire above, with a string past 24 characters written as UTF-16, the +engine's own two-byte form copied rather than transcoded. **The rule:** a `./binary` subpath is +built in a later minor only if C3 and C4 hold (C1, C2 and C5 make the case for it); if either +fails, the numbers are recorded here and the question is closed. + + + + +| Library | One record
round trips/s | 1,000 records
round trips/s | Sparse record
round trips/s | Typed record
round trips/s | Long strings
round trips/s | 100 wrpc packets
round trips/s | 32 doubles
round trips/s | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 569,000 | 567 | 2,740,000 | 651,000 | 715,000 | 16,400 | 193,000 | +| **protoarray**
codegen · v2.0.0 | 942,000 | 842 | 3,710,000 | 2,380,000 | 881,000 | 29,000 | 270,000 | +| **protoarray**
closures, no eval · v2.0.0 | 619,000 | 580 | 1,340,000 | 1,680,000 | 816,000 | 17,400 | 224,000 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 1,050,000 | 1,320 | 4,890,000 | 3,330,000 | 504,000 | 41,700 | 1,670,000 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 1,040,000 | 1,270 | 4,960,000 | 3,310,000 | 1,020,000 | 41,800 | 1,680,000 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 720,000 | 674 | 3,480,000 | 2,850,000 | 269,000 | 14,600 | 527,000 | +| avsc
Type.forSchema · v5.7.9 | 892,000 | 794 | 1,030,000 | 1,540,000 | 304,000 | 14,300 | 1,200,000 | +| msgpackr
records, shared structures · v2.1.0 | 950,000 | 1,000 | 4,050,000 | 2,080,000 | 190,000 | 22,500 | 1,000,000 | + +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. A round trip is one encode and one decode of the payload; medians, rounded to three significant figures; — where the scenario does not apply._ + +
+Every scenario: encode, decode and the payload sizes + +**One record** — one record: 10 fields, nested address, tags, 2 orders + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 1,406,221 (×1.00) | 956,607 (×1.00) | 266 | 212 | 178 | +| **protoarray**
codegen · v2.0.0 | 2,070,093 (×1.47) | 1,730,231 (×1.81) | 135 | 134 | 110 | +| **protoarray**
closures, no eval · v2.0.0 | 1,496,754 (×1.06) | 1,055,561 (×1.10) | 135 | 134 | 110 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 2,242,292 (×1.59) | 1,991,344 (×2.08) | 125 | 128 | 121 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 2,222,749 (×1.58) | 1,946,489 (×2.03) | 125 | 128 | 120 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 1,405,798 (×1.00) | 1,475,189 (×1.54) | 115 | 127 | 117 | +| avsc
Type.forSchema · v5.7.9 | 1,753,528 (×1.25) | 1,817,469 (×1.90) | 97 | 111 | 101 | +| msgpackr
records, shared structures · v2.1.0 | 1,481,899 (×1.05) | 2,644,916 (×2.76) | 108 | 124 | 112 | + +**1,000 records** — 1000 records of the same shape + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 1,529 (×1.00) | 902 (×1.00) | 271,215 | 33,793 | 29,133 | +| **protoarray**
codegen · v2.0.0 | 1,987 (×1.30) | 1,460 (×1.62) | 140,738 | 30,666 | 26,946 | +| **protoarray**
closures, no eval · v2.0.0 | 1,422 (×0.93) | 979 (×1.09) | 140,738 | 30,666 | 26,946 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 3,170 (×2.07) | 2,255 (×2.50) | 129,625 | 31,004 | 28,428 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 3,034 (×1.98) | 2,195 (×2.43) | 129,625 | 30,995 | 28,457 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 1,307 (×0.85) | 1,393 (×1.54) | 120,459 | 31,414 | 28,985 | +| avsc
Type.forSchema · v5.7.9 | 1,416 (×0.93) | 1,805 (×2.00) | 101,839 | 29,895 | 27,334 | +| msgpackr
records, shared structures · v2.1.0 | 1,548 (×1.01) | 2,857 (×3.17) | 112,274 | 30,957 | 28,370 | + +**Sparse record** — 20 optional fields, 3 present + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 6,853,739 (×1.00) | 4,568,304 (×1.00) | 33 | 53 | 36 | +| **protoarray**
codegen · v2.0.0 | 7,299,108 (×1.06) | 7,544,052 (×1.65) | 23 | 43 | 27 | +| **protoarray**
closures, no eval · v2.0.0 | 3,031,008 (×0.44) | 2,409,518 (×0.53) | 23 | 43 | 27 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 12,962,469 (×1.89) | 7,850,196 (×1.72) | 18 | 34 | 22 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 12,849,384 (×1.87) | 8,086,091 (×1.77) | 18 | 34 | 22 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 5,450,409 (×0.80) | 9,597,120 (×2.10) | 11 | 31 | 15 | +| avsc
Type.forSchema · v5.7.9 | 1,445,461 (×0.21) | 3,621,499 (×0.79) | 28 | 35 | 23 | +| msgpackr
records, shared structures · v2.1.0 | 6,052,760 (×0.88) | 12,235,411 (×2.68) | 9 | 29 | 13 | + +**Typed record** — enum, boolean and date heavy record + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 837,791 (×1.00) | 2,918,908 (×1.00) | 123 | 120 | 94 | +| **protoarray**
codegen · v2.0.0 | 4,711,181 (×5.62) | 4,812,177 (×1.65) | 36 | 46 | 36 | +| **protoarray**
closures, no eval · v2.0.0 | 3,598,327 (×4.30) | 3,150,768 (×1.08) | 36 | 46 | 36 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 6,616,242 (×7.90) | 6,697,672 (×2.29) | 26 | 38 | 30 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 6,655,370 (×7.94) | 6,571,614 (×2.25) | 26 | 38 | 30 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 3,711,548 (×4.43) | 12,220,629 (×4.19) | 13 | 33 | 17 | +| avsc
Type.forSchema · v5.7.9 | 1,917,877 (×2.29) | 7,881,899 (×2.70) | 12 | 32 | 16 | +| msgpackr
records, shared structures · v2.1.0 | 3,477,755 (×4.15) | 5,190,129 (×1.78) | 24 | 45 | 28 | + +**Long strings** — long Cyrillic strings + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 1,522,856 (×1.00) | 1,346,931 (×1.00) | 816 | 266 | 229 | +| **protoarray**
codegen · v2.0.0 | 1,806,954 (×1.19) | 1,720,687 (×1.28) | 785 | 230 | 207 | +| **protoarray**
closures, no eval · v2.0.0 | 1,695,879 (×1.11) | 1,573,136 (×1.17) | 785 | 230 | 207 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 907,396 (×0.60) | 1,133,018 (×0.84) | 774 | 228 | 205 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 1,879,701 (×1.23) | 2,241,510 (×1.66) | 821 | 265 | 247 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 571,717 (×0.38) | 508,360 (×0.38) | 777 | 233 | 211 | +| avsc
Type.forSchema · v5.7.9 | 500,024 (×0.33) | 778,250 (×0.58) | 774 | 231 | 210 | +| msgpackr
records, shared structures · v2.1.0 | 881,678 (×0.58) | 241,388 (×0.18) | 775 | 230 | 206 | + +**100 wrpc packets** — 100 wrpc packets: calls, callbacks and events + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 51,772 (×1.00) | 24,043 (×1.00) | 8,834 | 852 | 630 | +| **protoarray**
codegen · v2.0.0 | 67,870 (×1.31) | 50,529 (×2.10) | 4,332 | 685 | 501 | +| **protoarray**
closures, no eval · v2.0.0 | 48,917 (×0.94) | 27,079 (×1.13) | 4,332 | 685 | 501 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 107,251 (×2.07) | 68,197 (×2.84) | 3,064 | 653 | 539 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 107,579 (×2.08) | 68,411 (×2.85) | 3,064 | 656 | 540 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 23,943 (×0.46) | 37,437 (×1.56) | 3,691 | 778 | 609 | +| avsc
Type.forSchema · v5.7.9 | 20,736 (×0.40) | 46,589 (×1.94) | 2,833 | 633 | 521 | +| msgpackr
records, shared structures · v2.1.0 | 29,835 (×0.58) | 91,199 (×3.79) | 3,697 | 725 | 566 | + +**32 doubles** — 32 doubles (where binary formats shine) + +| Library | encode ops/s | decode ops/s | bytes | gzip | brotli | +| --- | ---: | ---: | ---: | ---: | ---: | +| JSON
stringify · parse · Node 24.14.1 | 320,721 (×1.00) | 485,627 (×1.00) | 764 | 406 | 392 | +| **protoarray**
codegen · v2.0.0 | 424,447 (×1.32) | 738,887 (×1.52) | 582 | 309 | 295 | +| **protoarray**
closures, no eval · v2.0.0 | 367,319 (×1.15) | 572,652 (×1.18) | 582 | 309 | 295 | +| **protoarray**
binary wire (prototype) · v2.0.0 | 2,497,902 (×7.79) | 5,022,385 (×10.34) | 260 | 283 | 262 | +| **protoarray**
binary wire, two-byte long strings (prototype) · v2.0.0 | 2,515,262 (×7.84) | 5,054,823 (×10.41) | 260 | 283 | 262 | +| protobufjs
reflection, Root.fromJSON · v8.8.1 | 723,688 (×2.26) | 1,944,379 (×4.00) | 305 | 328 | 304 | +| avsc
Type.forSchema · v5.7.9 | 1,950,178 (×6.08) | 3,115,226 (×6.41) | 256 | 279 | 260 | +| msgpackr
records, shared structures · v2.1.0 | 1,419,571 (×4.43) | 3,426,719 (×7.06) | 289 | 312 | 280 | + +_Apple M3 Max · 16 cores · Node v24.14.1 · median of 5 × 400 ms rounds, each library and scenario in a fresh process · 2026-10-18. Medians; (×n) against JSON; bytes are the wire size of one payload (binary formats before any text encoding), gzip at level 6, brotli at quality 5. The spread of the rounds is in bench/results/binary.json._ + +
+ + + +**The verdict, from the run above:** four of the five held and C4 did not. On 32 doubles the +prototype's round trip was 1,670,000 a second, ×6.2 the generated text code's (C1) and ×1.39 +avsc's (C2), at 260 bytes against 582. On one record it was 1,050,000 a second against +protobufjs's 720,000 (C3, ×1.47), ahead of the text wire's 942,000 and of every codec here. On +1,000 records it decoded 2,260 a second against avsc's 1,810 (C5) and msgpackr's 2,860, and its +round trip, 1,320 a second, was the fastest of the table. On long strings it was 504,000 a +second, 0.70 of JSON's 715,000, and C4 failed: what remained was transcoding 1.2 KB of UTF-8 per +round trip where `JSON.parse` and `JSON.stringify` copy the engine's own strings. The variant +measured next to it carried a string past 24 characters as UTF-16, copied rather than transcoded: +1,020,000 round trips a second on long strings, ×1.43 JSON's and ahead of the text wire's +881,000, at 6% more bytes before compression and 16% after gzip, every other scenario unchanged. +On that wire all five criteria held and the prototype led every scenario of the comparison, so +that is the wire the subpath ships, with the threshold at 21 characters so that a short string's +doubled length still fits one byte; the `binary wire` rows above are the shipped codec on the +same machine. The rows reported, not gated: the sparse record ×1.32 the text wire's round trip +and ahead of msgpackr; the typed record ×1.40 and ahead of protobufjs; the wrpc packets ×1.44. + ## Running them ```bash diff --git a/docs/guide/binary.md b/docs/guide/binary.md new file mode 100644 index 0000000..ac571b7 --- /dev/null +++ b/docs/guide/binary.md @@ -0,0 +1,136 @@ +# The Binary Wire + +protoarray's wire is JSON text: a positional array the engines print and parse natively, which is +what makes it fast on records, strings and tagged messages, readable in logs and at home on any +channel that carries JSON. Where a channel carries bytes and the payload is numbers, booleans, +enums and dates, text is the slower form: a double prints as up to 17 characters where 8 bytes +would do, and every value is parsed back from decimal text. `@alexify/protoarray/binary` puts the +same schema on a binary wire. Nothing about the schema changes: the same plan is compiled a second +time, into a writer and a reader of bytes, with the same checks on decode and the same errors. + +```js +import { Schema } from '@alexify/protoarray'; +import { toBinary, fromBinary } from '@alexify/protoarray/binary'; +import { Schema as Definition } from '@alexify/metaschema'; + +const Reading = Schema.from( + new Definition('Reading', { sensor: 'string', at: 'date', values: { array: 'number' } }), +); + +const bytes = toBinary(Reading, { sensor: 'a1', at: new Date(), values: [20.5, 20.7, 21.1] }); +// Uint8Array(…): the header, 'a1', the timestamp as a varint, three float64s +fromBinary(Reading, bytes); +// { sensor: 'a1', at: 2026-10-19T…, values: [20.5, 20.7, 21.1] } +``` + +`toBinary` gives a `Uint8Array` of its own, `fromBinary` takes one (a Node.js `Buffer` is one). +The codec is compiled from the schema's plan on first use and kept on the schema, by the backend +the schema runs: generated code, or closures where the runtime has none (see +[Runtimes](/guide/runtimes)). `register()` installs the two as methods: + +```js +import { register } from '@alexify/protoarray/binary'; + +register(); // once per process +Reading.toBinary(value); +Reading.fromBinary(bytes); +Reading.list.toBinary([value, value]); // the rows of a list, through the same codec +``` + +## When to use it + +The choice is the channel's and the payload's, not a question of which wire is "faster": over a +round trip the binary wire is ahead of the text wire on every payload of the +[comparison](/guide/benchmarks#the-binary-wire-measured) except long strings, and ahead of the +binary codecs written in JavaScript on most, at fewer bytes before compression. What the text wire +keeps is everything that makes it JSON. + +- **Text** when the payload has to be readable (logs, a debugger, a REST client), travels over a + channel that carries text (a JSON API, a WebSocket text frame, NDJSON, `localStorage`, a + JSON-RPC envelope), is read by a side that only has JSON and the schema, or streams: the + [registry](/guide/resolve#the-registry), [resolve](/guide/resolve), + [streaming](/guide/streaming) and [standalone modules](/guide/standalone) are the text wire's. + Through a text channel the bytes would need base64, which costs a third more and the time to + write and read it, and after gzip or brotli the two wires are close in size (on a list of + records the text wire is even smaller). +- **Binary** when the channel carries bytes (a WebSocket binary frame, `fetch` with an + `ArrayBuffer`, a file, an IPC pipe) and the payload is numeric, or many short records a second: + 32 doubles are 260 bytes against 582 of text and six times the round trip; a record of enums, + booleans and dates 26 bytes against 36. +- **In a browser** the text wire leans on the engine's native `JSON.parse`, while the binary + wire's strings go through `TextDecoder`; on records of strings the text wire stays ahead or + level there, and the binary wire wins on numbers. The numbers of the comparison are Node's, + where strings go through `Buffer`, as protobufjs and avsc do. + +## The wire + +The normative table is in [Wire Format](/guide/wire-format#the-binary-wire). In short, little-endian: + +- a **record** is a header bitset (a presence bit per field, then the value of every boolean + field) in whole bytes, and its present fields in declaration order; a `sparse` record is no + different, since every record has a header; +- a **union** is its tag (a byte, a varint past 256 values) and the branch's record without the + discriminator; +- a **collection** is its count, a presence bitmap when its items may be `null`, and its items; a + map its entries, a dictionary its keys and values, a tuple a bitmap and its elements; +- a **`number`** is a float64; an `integer`, a `date` (milliseconds) and a `decimals` number + (scaled) are zigzag varints; an enum index is a byte (a varint past 256 values); a flags mask one + varint per 31 names; a boolean outside a record a byte; +- a **string** travels behind its byte length, doubled, whose low bit says its form: UTF-8 for + one of up to 21 characters, UTF-16 for a longer one, copied rather than transcoded, which is what + keeps long strings within reach of the text wire; a lone surrogate is kept, as JSON keeps it; + `bytes` travel as they are behind their length, a `bigint` as its decimal digits, `json` as its + text; `null` writes nothing. + +## Both sides need the schema + +Positions, header bits and tags are the schema's, so the reader needs the writer's schema, as an +Avro reader needs the writer's. There is no trimming of trailing optionals and no tolerance of +extra elements: a message of another version of the schema is a `DecodeError` (or garbage without +validation), not a partially understood one. Hand the snapshot to the other side, or agree on it +through `schema.id` in a handshake (see [Schema Evolution](/guide/evolution#telling-versions-apart)). +Reading another version through `resolve`, and the `[id, payload]` envelope of the registry, are +the text wire's in this release. + +## Validation and errors + +Decode checks the wire as the text wire does, behind the schema's `validate` option (`true` by +default): the bounds of every read (a truncated message is a `DecodeError` `structure`), the +range of an enum index, the fit of a bitmap, a string or a count, a required field's presence +(`required`), a date within its range (`type`); a union's tag is checked whatever `validate` +says. Errors carry the schema's name and a path, as `decode` writes them: `orders[1].sku`. + +Encode trusts its values, as `encode` does (checking them is metaschema's), and throws +`EncodeError` only for what has no form on the wire: `null` at the root (`required`), a record +that is not an object, a collection that is not an array, a `Set` or a `Map` (`object`, `array`, +`set`, `map`), a `null` item of a list of required items (`type`), a union value outside the +union (`union`), a `decimals` number past the safe integers (`type`), and, unlike the text wire, +which carries anything as it is, a string slot that does not hold a string or a `bytes` slot that +does not hold a `Uint8Array` (`type`). A value JSON has no text for in a `json` slot (a function) +travels as `null`, as on the text wire. + +## What has no binary form + +The options that shape the text wire shape text: an interned string (`intern`), a list or a +column of differences (`delta`), the `flat` and `columns` layouts (a collection nested with a +layout of its own included) and a schema from `resolve`. `toBinary` and `fromBinary` of such a +schema throw `SchemaDefinitionError` with code `ERR_NO_BINARY_FORM`, naming the place by its path: + +```js +const Users = Schema.from(definition, { metaschema, layout: 'columns', intern: ['[].city'] }); +toBinary(Users, users); +// SchemaDefinitionError [ERR_NO_BINARY_FORM]: The binary wire has no form for the columns layout +``` + +Numbers need no `delta` on this wire, booleans and small enums are already bits and bytes, and a +`number` is eight bytes whatever its value. + +## Backends, runtimes and size + +The binary codec follows `schema.backend`: generated code where the schema has it, closures +elsewhere, both writing the same bytes (the test suite holds them to each other on random +schemas, as it does the text backends). Under a strict CSP the closures are what runs; the +binary wire is not written into [standalone modules](/guide/standalone) in this release. In +Node.js the strings go through `Buffer`'s own UTF-8 and UTF-16, in a browser through +`TextDecoder` and a loop, by the same runtime twin that handles base64. The subpath adds about +7.7 KB min+gzip to a bundle that uses it; the core is unchanged. diff --git a/docs/guide/comparison.md b/docs/guide/comparison.md index a187fac..6fe72c9 100644 --- a/docs/guide/comparison.md +++ b/docs/guide/comparison.md @@ -128,7 +128,7 @@ what each one does. Static type from the schema✅ from the metaschema schema (Standard Schema types)— schemaless⚠️ third-party: TypeBox, json-schema-to-ts— schemaless— schemaless; parse returns any— schemaless— schemaless; decompress<T> is a cast✅ generated by pbts from static code✅ generated by protoc-gen-es✅ generated interfaces⚠️ third-party @ovotech/avro-ts codegen✅ generated d.ts (js:ts)✅ generated by flatc --ts✅ generated by capnp-es -ots✅ generated by bebopc✅ the TS type is the schema❌ none— no schema— no schema— no schema— no schema— schemaless— schemaless Values checked before encode✅ metaschema check of the same definition12— schemaless❌ coerces: '12' → 12, null → 013— schemaless— schemaless— schemaless— schemaless✅ Type.verify; encode coerces silently⚠️ toBinary range-checks numbers; strings coerced14⚠️ writer range-checks numbers; no verify15✅ type.isValid(v, { errorHook }); toBuffer throws❌ string in i32 → 0, 2^40 → 016❌ coerces: string in int → 0, overflow wraps17❌ coerces: string in UInt32 → 018❌ trusts: string in uint32 → 019✅ assertEncode, assertStringify, typia.assert⚠️ typeof and range checks, on by default20— no schema— no schema— no schema— no schema— schemaless— schemaless Wire format -Wire formatJSON text (positional arrays)JSON textJSON textJSON text + metadata ({ json, meta })JSON text (flat value table, index refs)21JSON text ([n, …keys, …values])JSON array ([values[], rootKey], encoded strings)binary (Protobuf); ProtoJSON via ext/protojsonbinary (Protobuf); ProtoJSON toJson/fromJsonbinary (Protobuf); JSON via toJSON/fromJSONbinary (Avro); Avro JSON via toStringbinary (Binary/Compact) or JSON (TJSONProtocol)binary (FlatBuffers: vtables + offsets)binary (Cap'n Proto, optional packing)binary (Bebop)JSON text (json.*) or binary Protobuf (protobuf.*)binary (custom, big-endian, varints)binary (MessagePack + record extension)binary (MessagePack)binary (CBOR + record tags)binary (BSON)binary (V8 structured clone)Ion text or Ion binary +Wire formatJSON text (positional arrays); bytes through ./binaryJSON textJSON textJSON text + metadata ({ json, meta })JSON text (flat value table, index refs)21JSON text ([n, …keys, …values])JSON array ([values[], rootKey], encoded strings)binary (Protobuf); ProtoJSON via ext/protojsonbinary (Protobuf); ProtoJSON toJson/fromJsonbinary (Protobuf); JSON via toJSON/fromJSONbinary (Avro); Avro JSON via toStringbinary (Binary/Compact) or JSON (TJSONProtocol)binary (FlatBuffers: vtables + offsets)binary (Cap'n Proto, optional packing)binary (Bebop)JSON text (json.*) or binary Protobuf (protobuf.*)binary (custom, big-endian, varints)binary (MessagePack + record extension)binary (MessagePack)binary (CBOR + record tags)binary (BSON)binary (V8 structured clone)Ion text or Ion binary Human-readable payload✅ JSON arrays✅ JSON✅ JSON✅ JSON✅ JSON, values referenced by index22✅ JSON⚠️ JSON, but base62 numbers and refs❌ binary❌ binary❌ binary⚠️ binary; Avro JSON via type.toString()⚠️ binary; TJSONProtocol is JSON❌ binary❌ binary❌ binary✅ JSON path; protobuf path is binary❌ binary❌ binary❌ binary❌ binary❌ binary (EJSON for text)❌ binary⚠️ Ion text (JSON superset, not JSON) Travels over a JSON or text channel as is✅ plain JSON✅ it is JSON✅ it is JSON✅ it is JSON✅ JSON string✅ JSON string✅ JSON.stringify(compress(x))❌ binary; base64 to travel as text❌ binary; base64 to travel as text❌ binary; base64 to travel as text✅ only as Avro JSON (toString/fromString)✅ only with TJSONProtocol❌ needs base64❌ needs base64❌ needs base64✅ JSON path; protobuf needs base64❌ Buffer; base64 needed❌ binary, needs base64❌ binary, needs base64❌ binary, needs base64❌ binary; EJSON is a separate JSON form❌ needs base64✅ Ion text is a string Field names on the wire❌ positional✅ every message✅ every message✅ every message (and in meta paths)✅ every object⚠️ once per collection (header)⚠️ once per distinct key set❌ tag numbers❌ tag numbers❌ tag numbers❌ positional (schema order)❌ tag numbers (field ids)❌ vtable slots (field ids)❌ fixed offsets❌ positional structs; 1-byte tags in messages✅ JSON: every message; protobuf: tag numbers❌ positional (sorted key order)⚠️ once per structure (new Packr() records)✅ every object⚠️ once per structure (new Encoder() records)✅ every document✅ every object, repeated✅ text: each time; binary: once (symbol table) diff --git a/docs/guide/why.md b/docs/guide/why.md index 0fddfd8..ad197e1 100644 --- a/docs/guide/why.md +++ b/docs/guide/why.md @@ -70,7 +70,9 @@ small. The payload is still plain JSON. cbor-x, and decodes faster than every one of them on long strings. Two honest exceptions: payloads that are mostly numbers (a double prints as up to 17 characters, binary stores 8 bytes), and msgpackr with shared record structures, which decodes records faster than - `JSON.parse` can read them (avsc, cbor-x and protobufjs do too on some shapes). + `JSON.parse` can read them (avsc, cbor-x and protobufjs do too on some shapes). Where a + channel carries bytes, [the binary wire](/guide/binary) (`@alexify/protoarray/binary`) puts the + same schema on bytes and takes the numeric payloads too. - **Readable on the wire.** The payload can be logged, inspected and edited as is, and travels over channels that only carry text without base64. - **Schema evolution with the same discipline.** Positions are append-only, like protobuf field @@ -91,8 +93,9 @@ serialization libraries, feature by feature. - The other side is not JavaScript and already speaks protobuf or Avro: protoarray's wire is specified, but JavaScript is its only implementation. -- The payload is dense numeric data (sensor readings, vectors): a typed array in a binary frame is - the right tool. +- The payload is dense numeric data (sensor readings, vectors) and the channel carries text: the + [binary wire](/guide/binary) needs a channel that carries bytes, and base64 would eat its gain; + a typed array in a binary frame is the right tool there. - You need to read one field of a large message without decoding the rest: FlatBuffers and Cap'n Proto read in place. - The data has no fixed shape at all: `json` fields pass anything through, but a schema with diff --git a/docs/guide/wire-format.md b/docs/guide/wire-format.md index 156b04d..ed32a57 100644 --- a/docs/guide/wire-format.md +++ b/docs/guide/wire-format.md @@ -91,6 +91,43 @@ throws only for what has no wire at all. | A payload nested deeper than the call stack (or than `maxDepth` levels of a recursion) | `DecodeError` `structure` ("nested too deeply"), always; with `maxDepth` at the same depth on either backend | | Keys on the decoded object | every key of the schema, in schema order | +## The binary wire + +`@alexify/protoarray/binary` ([The Binary Wire](/guide/binary)) carries the same schema as bytes, +little-endian. A varint is unsigned LEB128; a zigzag varint interleaves negatives (the sign in the +low bit), exact over the safe integers. Both sides need the same schema: positions, header bits +and tags are the schema's, nothing is trimmed and nothing beyond the schema is tolerated. + +| Construct | On the wire | +| --- | --- | +| Struct `{ a, b, c }`, sparse or not | a header, then the present fields in declaration order; the header holds a presence bit per field (bit *i* is field *i*, required ones too), then the value of each boolean field in field order, in whole bytes, low bit first | +| Missing, `undefined` or `null` field | its presence bit clear, nothing else | +| Nested struct, a schema used as a field, a reference held as the record | the referenced schema's own form in place | +| `{ union: [...], discriminator }` | the tag (a byte; a varint past 256 values), then the branch's struct without the discriminator | +| `{ array: T }`, `{ set: T }` | the count as a varint; a presence bitmap of count bits when the items may be `null`; the present items in order | +| `{ map: { K: V } }` | the count, then each entry: the key (a string, or a float64 for a `number` key), a presence byte (0 or 1) when the value may be `null`, the value | +| `{ object: { string: V } }` | the count, then each key as a string and its value, a presence byte before it when it may be `null` | +| Tuple | a presence bitmap of its elements, then the present elements | +| `number` | float64 | +| `integer`, `date` (epoch milliseconds), `{ type: 'number', decimals: n }` (`Math.round(value × 10ⁿ)`) | a zigzag varint | +| `boolean` | a bit of the header as a struct's field; a byte (0 or 1) anywhere else | +| `{ enum: [...] }` | the index as a byte; a varint past 256 values | +| `{ flags: [...] }` | one varint per 31 names, the words of the text wire's mask | +| `string` | its byte length doubled plus its form, as a varint, then the bytes: form 0 is UTF-8, form 1 UTF-16LE; this implementation writes UTF-8 up to 21 characters and UTF-16 past them or for a lone surrogate, and reads either form at any length | +| `bytes` | the length as a varint, then the bytes | +| `bigint` | its decimal digits as a string | +| `json`, `any`, `unknown` | its JSON text as a string (`null` for a value JSON has no text for) | +| `null` | nothing; its presence bit is never set | +| Root struct or union | as above, with no envelope; `null` has no form (`EncodeError`) | +| Root scalar | a presence byte (0 or 1), then the value | +| Root collection or tuple | as above | +| An interned string, a `delta` list or column, a `flat` or `columns` collection, a schema from `resolve` | no form: `SchemaDefinitionError` `ERR_NO_BINARY_FORM` | + +Decode checks the bounds of every read, the range of an enum index, the fit of a bitmap, a string +and a count, a required field's presence and a date's range, with validation; a union's tag is +checked always. Encode trusts its values, checks the shapes the wire is built from as the text +wire does, and refuses a string slot without a string and a `bytes` slot without a `Uint8Array`. + ## Canonical form `schema.toString()` prints the schema as `{key:type,key:?type=default,...}` with `[item]` for diff --git a/docs/public/benchmarks/coldstart.svg b/docs/public/benchmarks/coldstart.svg index 472cafa..d51c42d 100644 --- a/docs/public/benchmarks/coldstart.svg +++ b/docs/public/benchmarks/coldstart.svg @@ -1,4 +1,4 @@ - +