[EthanThatOneKid]

What

The shared benchmark from #414 (item 1 of its proposed path). deno task bench:shacl generates a seeded graph per size, runs each case in its own process, and prints a Markdown table (--json for raw results). Details are in bench/shacl/README.md.

• bench/shacl/fixture.ts: a deterministic people/organizations graph as N-Triples. The shapes cover the constraint kinds Performance: SHACL validation at scale (10^6+ quads) across Wiki and Worlds — shared benchmark, shapes memoization, delta-scoped validation #414 flags (class targets, sh:class, a two-step sequence path, an inverse path), and about 1% of people carry a planted violation.
• bench/shacl/case.ts: one case per process. It reports the total, named phases, OS peak RSS (VmHWM on Linux, PeakWorkingSet64 on Windows), and the SHACL result count.
• bench/shacl/run.ts: the orchestrator. --sizes, --cases, --timeout, --json, --out, and --emit DIR, which writes only data.nt and shapes.ttl for a Worlds-side harness.
• tests/bench_fixture_test.ts: checks the fixture is deterministic, lands near its target size, and that the shapes find exactly the planted violations.
• deno.json: the bench:shacl task, plus bench/ in check, fmt, and lint. The benchmark isn't in publish.include or package.json files.

Finding for wazootech/worlds-sdk-ts#151

Validating a small Δ-plus-neighbourhood graph isn't enough on its own. The neighbourhood contains typed nodes that aren't part of the write, such as the person someone knows. The shapes' class targets validate them without their neighbourhood, so they report violations that aren't real: 542 results over 200 writes, of which 5 were real. The delta case therefore re-targets the shapes with sh:targetNode on exactly the touched nodes. #151's scoped validation needs the same thing.

Baseline results

Windows 11, x86_64, Deno from deno task bench:shacl, 15.5 GB RAM with only about 2–3.5 GB free during the run. This was two invocations: every case at 10³–10⁵, and at 10⁶ every case except the two that run OWL-RL inference (wiki-phases, wiki-check-full). Timeout was 1,200 s per case. Peak RSS is the OS's PeakWorkingSet64 for the case's process, including about 170 MB of Deno and module baseline.

Validator alone (the part Wiki and Worlds share)

Quads	validator-full	Peak RSS	Results
1,007	136 ms (validate 121 ms)	177 MB	4
10,006	425 ms (validate 397 ms)	249 MB	24
100,007	3.1 s (validate 2.9 s)	449 MB	194
1,000,017	34.4 s (validate 31.0 s)	2,657 MB	2,078

Write-scoped validation (the Worlds proxy, 200 writes)

Quads	Validate p50 / p95	Neighbourhood fetch p50	Index build	Peak RSS	Results
1,007	7 / 12 ms	0 ms	10 ms	197 MB	5
10,006	7 / 10 ms	0 ms	63 ms	240 MB	0
100,007	7 / 9 ms	0 ms	401 ms	328 MB	3
1,000,017	10 / 20 ms	0 ms	4.5 s	1,972 MB	2

The fetch is an in-memory n3 index lookup, not D1.

Wiki

Quads	wiki-check-full	wiki-phases: loadGraph / loadShapes / validate	scoped-1	scoped-50	Peak RSS (full / scoped-50)
1,007	2.2 s	1.9 s / 1 ms / 76 ms	380 ms	2.0 s	220 / 174 MB
10,006	45.1 s	43.4 s / 7 ms / 381 ms	485 ms	2.1 s	566 / 327 MB
100,007	timeout (>1,200 s)	timeout (stopped at 2,892 s)	3.4 s	3.9 s	n/a / 586 MB
1,000,017	not run	not run	14.3 s	16.2 s	n/a / 2,932 MB

At 10⁵ the first run had the runner report both inference cases as "exit 1, no stderr". That was the Windows timeout kill being mislabelled, fixed in 1745d85. The wiki-phases process took 2,892 s to stop, well past its 1,200 s limit. That fits a machine short on memory, but I didn't confirm the cause.

What the numbers say

1. SHACL validation isn't the wall. rdf-validate-shacl scales about linearly once the fixed startup cost is small (×3, ×7, ×11 per 10× data) and validates a million quads in 31 s.
2. OWL-RL inference in loadGraph is the wall for the Wiki. It grows ×23 from 10³ to 10⁴ (1.9 s → 43.4 s) and doesn't finish within 20 minutes at 10⁵. A full wiki check is capped well below 10⁵ quads by inference, not SHACL. Performance: incremental / single-file wiki render, cache OWL inference #22 and Performance: cross-process graph reuse for wiki render #30 cover inference caching; this is the measurement behind them. Options worth costing are validating against the uninferred graph, as scoped mode already does, or making inference incremental.
3. Write-scoped validation stays flat: p50 7 ms from 10³ to 10⁵ and 10 ms at 10⁶, which is evidence for feat: validate every write against a world's SHACL shapes, on by default with a per-world opt-out worlds-sdk-ts#151's approach. This holds only with shapes re-targeted at the touched nodes (sh:targetNode). Without that, typed neighbours report violations that aren't there (542 results over 200 writes when 5 were real).
4. Memory is about 2.5 KB per quad for full validation ((2,657 − 177) MB / 10⁶ quads). That rules out full-graph validation inside a 128 MB Worker beyond a few tens of thousands of quads, even before Worker overhead. This is a rough figure from Deno, not measured in a Worker.
5. Scoped Wiki checks still scale with the corpus. At 10⁶, one page takes 14.3 s and 2.9 GB, because scoped mode loads the whole uninferred graph to find the shapes. perf(check): load SHACL shapes once per scoped check #415 removed the per-file repeat, but the single load remains. Extracting shapes without loading all the data (shapes live in shape pages and sh: triples) would make a scoped check independent of corpus size.

Not done

• The 10⁶ inference cases, which were left out on purpose. At 10⁵ they already exceed 20 minutes.
• The Worlds half: D1 neighbourhood fetch latency, and the validator bundle size and CPU under Workers limits. They need a Worker harness, and --emit provides the fixture for it.
• wiki build, since its check is measured here and the rest isn't SHACL.

Checks

deno task check, deno task lint, deno task fmt:check, and the new fixture tests pass locally, and CI runs the full suite. The timeout fix was verified with --timeout 1, which now reports timeout after 1 s. This PR adds files only under bench/ and tests/, plus the deno.json task and include entries. It adds no machine-specific numbers to the repository; those are in this PR and #414.

Refs #414

🤖 Generated with Claude Code