Repository navigation
perf(check): load SHACL shapes once per scoped check - #415
Merged
Merged
Conversation
A scoped `wiki check FILE...` (also `build --only` and the watcher) called `checkShaclFile` once per file, and each call re-extracted the shapes graph with `loadShapes`, which scans every quad in the corpus several times. A scoped check of k files cost O(k x corpus) for the same shapes every time. `runCheck` now builds the shapes once with the new `loadWikiShapes` and passes them to `checkShaclFile`, which takes an optional `shapesGraph` and still defaults to loading the shapes itself, so existing callers are unchanged. On a generated 2,000-page wiki, a 50-file scoped check went from 73-103 s to 12-15 s. Refs #414 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A scoped
wiki check FILE...(whichbuild --onlyand the file watcher also use) calledcheckShaclFileonce per file. Each call re-extracted the shapes graph withloadShapes, which scans every quad in the corpus several times (shacl.ts:88-154). A scoped check of k files cost O(k × corpus) for shapes that never change between files.shacl.ts: newloadWikiShapes(config)(the uninferred graph's shapes, the same expressioncheckShaclFileused inline).checkShaclFiletakes an optional thirdshapesGraph, which defaults toloadWikiShapes(config), so existing callers and the public API behave the same.audit.ts:runCheckbuilds the shapes once per scoped check and passes them to each file.mod.ts: exportsloadWikiShapesalongsidecheckShaclFile, so in-process callers can do the same.This is item 1 of "Where Wiki's cost grows with corpus size" in #414.
Measured
A generated wiki of 2,000
Personpages plus one node shape,wiki checkon 50 of them, two runs each, Windows:main@a7074eeA single-file scoped check on the same wiki takes about 10 s, which is mostly the graph load the cache already covers. So almost all of the remaining time for 50 files is fixed cost, not per-file cost. I also checked by hand that an invalid page still reports
shacl_violationalongside a valid one.Checks
deno task check,deno task lint,deno task fmt:check, anddeno task test(574 passed, 1 ignored). The new test asserts that a passed-in shapes graph is used as given: the wiki's own shapes give the same result as the default, and an empty graph has nothing to violate.Not changed
checkShaclAllalready loads shapes once. Building the validator dataset only once (Performance: SHACL validation at scale (10^6+ quads) across Wiki and Worlds — shared benchmark, shapes memoization, delta-scoped validation #414 item 2) and validating only what changed (item 3) are separate follow-ups.Refs #414
🤖 Generated with Claude Code