Skip to content

feat: a local-first sync engine: replication, schema, queries, signed ops, full-text search - #1

Merged
Alex-Dolid merged 67 commits into
mainfrom
feature/v1
Oct 23, 2026
Merged

Alex-Dolid merged 67 commits into
mainfrom
feature/v1

Conversation

@Alex-Dolid

Copy link
Copy Markdown
Contributor

Summary

This branch is the whole of @alexify/syncom as it stands, from the migration of the globalstorage files to a
replicated, schema-validated, queryable storage engine with signed operations. main holds only the initial
commit, so this PR is effectively the package.

The version is 0.0.1 — a development placeholder. Nothing is published to npm, and there is no release until the
package does what it is meant to; the first release version is chosen then (CHANGELOG.md keeps everything under
[Unreleased]). Zero runtime dependencies, CommonJS, no build step.

What is in it

Replication

  • Every write is an operation in a persisted, hash-linkable log; peers exchange ops over an injected link (wrpc over
    WebSocket or WebRTC), resume from the last event id, and bootstrap from a streamed snapshot when they cannot.
  • A coordinator-free mesh: every node dials and hosts; hubs dial each other; ops relay hop by hop, de-duplicated by
    version vectors kept per origin and scope.
  • Push recovery on the link's next sign of life, an outbox bound, per-peer push limits, a failed page treated as a
    break in the stream (retry from the saved cursor, then stuck).

Data model — what converges

  • Last-writer-wins registers per field, per-origin counters (inc), observed-remove sets (add/remove),
    tombstones a newer write revives; a field is one kind only.
  • purge(id) — the right to be forgotten: wins at any stamp, scrubs the payload from every log it reaches.

Storage

  • Injected backends (node:fs, OPFS, IndexedDB via @alexify/syncom/idb, memory), exclusive open, a bounded cache,
    an in-memory record catalog (list()/hasRecord() without I/O), encryption at rest, an optional chain
    (chain: false for hubs), repair(id) from the log.

Schema and queries

  • Injected validators (metaschema, any Standard Schema, ajv), schema versions on every op, reading older records
    through a lens, migrate().
  • Mango cursors with indexes, unique keys, top-k, watch(), and full-text search ($text) — fuzzy, TF-IDF
    ranked trigram search through the @alexify/syncom/search subpath (0.9 KB).

Trust

  • Every op is signed with the node's Ed25519 key, and the node id is that key: any peer verifies an op through
    any relay. sync.signatures: 'verify' | 'require' | false.
  • The host checks ids, op shape, clock skew and scopes before anything reads an op; snapshots and feeds serve only
    the granted scopes.

Operability

  • Hooks inside writes, reads and peer batches; contained events; injected OpenTelemetry (spans, metrics, log events
    as a surface); a SyncomError with a code for every operational failure; stats(), peers(), hosted().

Docs and benchmarks

  • A VitePress site: guides for storage, sync (incl. signatures), schema, queries (incl. full-text search), hooks,
    backends, scaling, WebRTC, telemetry, architecture, why, a 40-feature comparison with 18 libraries, and benchmarks
    against nine of them. Every number comes from a script in the repository.

Breaking, relative to globalstorage

The full list is under Changed (breaking) in CHANGELOG.md. The ones that change how code is written:

  • open() is exclusive by default (LOCKED); smart contracts live in @alexify/syncom/contracts.
  • Record ids are checked ([A-Za-z0-9_@-][A-Za-z0-9._@-]{0,127}); an entity definition takes only known keys.
  • set() is a whole-record write; a field is a register, a counter or a set, never two.
  • The node id is the Ed25519 public key; a UUID in an existing node.json is rekeyed once (identity.rekeyed).
  • Node.js ≥ 22.

Not in this PR

  • A signed hello challenge (proof that a peer holds the key it greets with).
  • List and text CRDTs; git-like branching; master/backup/cache server roles.
  • Framework coupling (Impress, Metarhia) — out of scope by design: transport, backend, validator and telemetry are
    injected.

Testing

  • pnpm lint, pnpm format:check, pnpm test:coverage (720 tests: 719 pass, 1 skipped; 99.6 % lines, 96.4 %
    branches), pnpm test:types, pnpm size (browser entry 40.7 KB min+gzip of a 41 KB budget; the search subpath
    0.9 KB), pnpm run docs:build — all green.
  • Cross-platform behaviour runs on both halves (Node, and the browser halves under Node with an OPFS mock and
    webcrypto); the wrpc and metaschema integration tests ran; one test is skipped by design (it needs
    --expose-gc).
  • Convergence: a seeded fuzz over a four-node mesh (set, update, inc, add, remove, delete), partitions, a late
    joiner bootstrapping from a compacted host, and a three-node mesh under signatures: 'require' where every logged
    op verifies against its origin.
  • Not checked by hand: a real browser (two tabs, WebRTC between them).

Performance note

Signing is the main cost in memory. In the cross-library benchmarks (re-measured on this branch) an in-memory
insert went from ~37,500/s to ~15,600/s, and replicating a store in memory from ~108,000 to ~9,000 records/s — the
replica verifies every op at ~0.1 ms each. On disk inserts are unchanged (~1,500/s; the file writes dominate),
replication went from ~7,000 to ~4,000/s. sync.signatures: false removes both costs. Verifying on Node's
thread pool instead of the main thread is the obvious next step and is not in this PR.

Reviewing

Commit by commit: each one is a single change with every gate green. CLAUDE.md lists the load-bearing rules the
code follows (the three write paths, the op model, the trust boundary, signatures, the catalog seam).

🤖 Generated with Claude Code

Alex-Dolid and others added 30 commits September 4, 2026 16:11
The migration from metarhia/globalstorage copied files but adapted nothing:
require('@alexify/syncom') threw on a missing metautil dependency, browser.js
was an 810-line hand-duplicated bundle importing a nonexistent ./metautil.js,
npm test matched zero files, and bench/docs/scripts were empty directories
already wired into six npm scripts.

- Vendor the 5 helpers syncom's src/ needs (Emitter, generateUUID, between,
  fileExists, directoryExists) into src/utils.js / utils.browser.js, keeping
  the zero-runtime-dependency promise honest
- Replace browser.js with a 2-line shim over a real src/index.browser.js
  barrel; rename browser modules to the *.browser.js pattern
- Fix two real bugs surfaced while wiring this up: Storage.validate() threw
  on no arguments instead of validating the whole chain, and Emitter.emit
  silently dropped every argument after the first (Record's `update` event
  never delivered its delta)
- Rename tests/*.js to *.test.js and fix stale ../lib/ paths so `pnpm test`
  actually runs (0 -> 130 passing tests, 97%+ coverage)
- Add scripts/size.js (bundle-size budget + self-contained bundle check),
  bench/ (harness + 5 benchmarks), and tests/package/consistency.test.js
  (10 checks tying package.json/exports/files/docs together)
- Add a VitePress docs site, rewritten README, CHANGELOG, SECURITY,
  CONTRIBUTING, CLAUDE.md, and a 3-job GitHub Actions CI workflow
- Remove emulation/ and the broken root example.js (legacy dead code)

No new features — this only makes the copied code actually run and gives it
the same tooling wrpc has.
Mirrors kerberos's getCallId pattern: Storage.generateId() is the public
default (crypto.randomUUID), overridable per instance with a sync or async
getId function. Lets callers use ULID, CUID2, nanoid or server-issued ids
without adding a runtime dependency to the package.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Turns storage.sync from a registry-only stub into working replication.
Every write becomes an operation (write/set/inc/delete) appended to a
persisted, WAL-ordered log; a field is a last-writer-wins register or a
per-origin counter, never both, so ops converge in any order and any
number of times. SyncManager dials registered peers, pushes what a host
lacks, subscribes to live changes, relays across a mesh de-duplicated by
version vector, and bootstraps a late joiner from a streamed snapshot.

The transport is injected, never a dependency: @alexify/syncom/wrpc
adapts a WrpcClient to the small SyncLink contract, leaving reconnect,
backoff and subscription resume to wrpc itself. @alexify/wrpc is a
devDependency for one skip-guarded integration test only.

Also: encryption at rest is now a hybrid AES-256-GCM + RSA-OAEP/SHA-256
envelope, identical (and portable) on both platform halves, replacing
the broken un-awaited browser path and the ~200-byte RSA ceiling. Fixed
along the way: apply()'s write/delete handling, Blockchain.validate on
an unknown hash, loadKeys with a missing private half, and the node
registry's host-collision bug. sync.browser.js is gone — one sync.js
over fsi serves both platforms.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A local write was silently dropped (no op, no event, no error) whenever a
stamp in the record's metadata was above the node's clock. Two paths led
there: a snapshot joined from a peer whose clock ran ahead — its field
clocks, base, tombstone and counter stamps were never observed — and a
reopen, where the clock was rebuilt only from the unsealed log tail, so a
rolled log or a wall clock corrected backwards left the node stamping
below its own earlier writes.

- ops.js: latestClock(meta), the newest stamp an entry holds, tolerant of
  raw peer-shaped meta.
- storage.js: [COMMIT] observes the record's meta (and the joined meta for
  a merge) before stamping; #init seeds the clock from log.latestTs.
- log.js: latestTs, maintained on load and append.
- Tests for both paths and an end-to-end case in the sync-engine suite;
  a "Clocks" section in docs/guide/sync.md and a rule in CLAUDE.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…context

`open({ telemetry })` takes the same object wrpc's option does — `{ api }` or
`{ tracer, meter }` — and everything is duck-typed; @opentelemetry/* stays a
devDependency. Six INTERNAL spans (syncom.commit, apply, replay, push,
bootstrap, log.roll) and nine metrics cover what only the engine can see:
the local write path, what a push came to, replication lag, log rolls,
bootstraps as one unit, pushes as one batch. Nothing wrpc already emits is
repeated, and syncom puts no trace context into ops: its spans open with
startActiveSpan and nest under wrpc's, so a write on one node and its
application on the next are one trace through the socket.

Merge commits are counted, not spanned. Identity (record id, peer url/node)
rides on spans behind includeIdentity and is never a metric label. The
browser budget is ratcheted 14 → 17 KB for it (16.1 measured).

Two bugs on the same seam: applyRemote emitted `error` unguarded, so a
refused op with no listener failed the push before the vector moved and the
peer offered the same page forever; OpLog never received the storage logger.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`src/hooks.js` (no requires — it ships to the browser) brings HookError,
normalizeHooks, runHook and notify. Every event the engine emits now goes
out through notify(): invoked at once, never awaited, contained. A `change`
listener that throws no longer fails the write it observed, one that writes
back queues behind it instead of deadlocking the mutation queue, and an
`error` with nobody listening is silent rather than an Emitter throw. The
fire-and-forget emits in Record and SyncManager stop leaking rejections.

HookError is exported now so the hook commits that follow can land their
types alongside — the consistency test wants every runtime export declared.

Also: a peer hung up while its page was applying no longer writes its
registry file after stop(), which raced whoever reopened the directory next.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
#saveNode dropped itself from the pending map before its write began, so
stop() only waited for writes that had not started, and a second save
during one in flight wrote the same file concurrently — a reopen could read
half a JSON document. Writes now chain per node, stop() awaits the chain,
and a peer that was disconnected mid-push or mid-bootstrap does not write
its file afterwards.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`open({ hooks })` — one function per name, awaited inside the operation.
beforeWrite sees the op being made and the record it lands on, may replace
the data of a write or a set, and refuses the write by throwing (HookError;
nothing stamped, logged or written). afterWrite sees the durable op; a throw
fails the call but the op stays and is announced. beforeRead/afterRead wrap
get(): the second may redact or hide what the caller sees, while swap,
snapshot and the cache keep reading raw data.

Hooks run inside the mutation queue, so a write issued from a hook is
refused rather than left to deadlock. The first encrypted write now goes
through the same commit path (an `encrypted` option), so the hooks see
every write and "one write path" is literally true.

Budget ratcheted 17 → 18 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
beforeApply runs per remote op, after the duplicate/gap checks and before
the op is applied. `false` skips it HERE only: the op is still logged and
relayed and the vector moves — a hole in this node's log would be a gap for
every peer resuming from it, and a vector that does not move would have the
sender offer the page forever. The log flags such entries so a reopen's
replay of the tail leaves them alone too. A throw aborts the batch before
anything is durable; the sender sees a failed push and offers it again.

afterApply sees the durable batch; whatever it does, the record views, the
new `apply` event and the `change` listeners hear about every op. The
result grows `skipped`, and applyRemote takes a `peer` descriptor
({ url, node }) so hooks know who sent the page without the link object.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`open({ sync: { hooks } })`. beforeConnect sees a registry entry about to
be dialled; `false` leaves it registered but not dialled. beforeBootstrap
runs when a host's log cannot resume us and its whole state is about to be
joined; `false` refuses — the peer stays connected for pushes and receives
nothing until the next handshake asks again. A throw in either is reported
through the manager's `error` event, as any failure on the way out is.

There is deliberately no hook on the ops that leave for a peer: seqs run
contiguously per origin and scope, so an op withheld from one peer is a
gap that stops every later op of that stream at it. Per-peer visibility
is what scopes are for.

New events: `start`, `stop`, `node` ({ url, action }), `connect` (a link
accepted, before hello — `open` used to say nothing) and, on the hosting
side, `hello` ({ node, scopes }) once a peer is authorized. SyncOptions
gains the `connect`, `autoStart` and `pageSize` it always accepted.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ose event

StorageEvents, RecordEvents and SyncEvents type every event with its
payload, and on/once/off/listenerCount are generic over them on Storage,
SyncomRecord and SyncManager. SyncManager was declared with six members
while the code and the docs had eleven: started, enableNode, sync, peers,
unit and the listeners are declared now, with PeerView, SyncUnitOptions
and SyncUnit; Node gains the cursor and remote it always carried.

Storage now emits `close` — once, after the log is sealed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new guide page (docs/guide/hooks.md): what a hook may return and what a
throw does, per hook; the order of a local write and of a peer's page; why
a skipped op is still logged and relayed; why there is no hook on what
leaves for a peer; the events of each class with their payloads. Every
example was run before it was written down. The API reference, the sync
and storage guides, the README (features, bundle table) and the changelog
follow; CLAUDE.md gets the rules that bite.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Six of metaschema's seven metaskills v1.0.5 skills. data-structures,
error-handling, js-data-structures and js-gof are verbatim copies.
js-conventions (120 columns, 'use strict', no runtime dependencies, the
engine's error and hot-path rules) and npm-publish (CI gates, hand-synced docs
nav version, first-release notes) are adapted to syncom. metautil-data-structures
is left out: syncom takes no runtime dependencies. CLAUDE.md gains a Skills
section.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r, idField, $type immutability, check()

An entity definition is `{ schema?, idField?, validate?, index?, unique?,
fields? }`. `schema` is validated through one of three shapes, none a
dependency: a Standard Schema (`~standard` — an entity of a metaschema
Model, zod, valibot, arktype), a compiled function (ajv), or a plain
object compiled by `open({ validator })`. src/schema.js (no requires)
normalizes them to `async (record) => validated`, reads the id field
(`id`, metaschema's `<entityName>Id`, or none) and the indexes off the
schema, and projects the record the validator sees.

A local write is validated after `applyOp` and before `#log.append`, as
the whole record it leaves; what the validator returns is folded back into
the op (a `set` keeps only its own keys plus the ones the record lacked),
so peers receive the validated record. A peer's op is never rewritten: an
op the schema refuses is skipped like a `beforeApply` false and reported
through the `error` event. `$type` is immutable on the local path,
`Collection.set()` stamps its own, `storage.check()` validates without
writing, and a non-function `getId` is refused at open.

`@alexify/metaschema` joins as a devDependency for one skip-guarded
integration test. Browser budget 18 → 20 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A write that landed before the handshake's push went out travelled with
it as a 'handshake' push and the relay span the test looks for was never
made — a race the test lost about one run in six.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…v, schema.json, hello.schema, drift

`open({ schema: { version, snapshot } })`: the version is stamped on every
op this node makes as `v` (the 13th slot of the packed shape; a reader of
the 12-slot shape ignores it), and a record remembers the version of its
last whole-record write as `meta.v` — a `set` that creates the entry takes
its op's, a merge takes the higher. `snapshot` is any JSON describing the
schema (metaschema's `model.toJSON()`), hashed into `<path>/schema.json`
`{ version, migrated, hash }`; `storage.schemaInfo` is `{ version, hash,
previous }`, `previous` what the file held before this open when that
differed — a fact the caller reads after open, where an event at open would
find no listener yet.

`hello` carries `schema: { version, hash } | null` both ways: `authorize`
sees it, `listNodes()[0].remote.schema` keeps it, the `peer` and `hello`
events carry it, and the same version with another hash is reported on
both sides as the storage's `schema` event `{ kind: 'drift' }` plus a
warning — never refused. `WriteContext.v`. Browser budget 20 → 20.5 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… ahead 'hold' with held replay

A record of an older schema version is read through a lens, computed once
per cached entry and never written back: `schema.migrations` (keyed by the
version migrated to, storage-wide or per entity, `up(data, { entity, id,
from, to })`), then the entity's Standard Schema validator — its aliases and
defaults read a renamed field with no migration written, which is what
metaschema's evolution model offers. `schema.lens` picks the steps. `get()`,
`Record` and `swap()` read the view; `update()` is validated through it and
folded against how the record read before; `snapshot()`, `getCachedData()`
and replication stay raw. `storage.migrate({ entities?, batch? })` rewrites
old records as whole-record writes of their view with the counters added
back, idempotently.

`schema.ahead`: `'apply'` applies an op from a newer version as it is and
emits `schema { kind: 'ahead', held: false }`; `'hold'` keeps it in the log
(`LogEntry.held`, `OpLog.held()`, `droppedHeld` persisted in the index) until
an open runs that version, which applies it and reports
`schemaInfo.held = { replayed, lost }`. A hold needs retention of 1000 ops
or more and refuses a downgrade. Browser budget 20.5 → 22 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ncom/contracts

`SmartContract.load` reconstructs code from the chain with `new Function`;
on the package root every node that only stores and replicates carried that
executor, and a hub accepting blocks had it in core. The subpath keeps the
feature and takes it off the root: a process that never imports
`@alexify/syncom/contracts` never runs a block's code. `contracts.js`,
`contracts.d.ts` and its tsd test, an `exports`/`files`/size row, the
barrels and types without it (~0.4 KB back on the main entry; the budget
stays), docs and SECURITY.md say so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…createMemoryFsi, durable writes

The five fsi functions were a module-level require in storage, log, chain,
crypto and sync: in the browser that meant OPFS and nothing else, and no
test, SSR render or benchmark could run a node without touching a disk.
`open({ fsi })` now takes any object with `read`, `write`, `exists`,
`remove` and `list`, checked at open by the method it lacks, and every file
of the storage — records, log, chain, keys, registry — goes through it;
`storage.fsi` is what it uses, `loadKeys`, `Blockchain` and `OpLog` take
`{ fsi }` too. src/backend.js (no requires) holds the contract, `isFsi`,
`createMemoryFsi()` (a Map; two storages over one share it) and `withSync`,
which `open({ durable: true })` wraps a backend in: every write followed by
its `sync` — fsync on node:fs, a debug line on a backend without one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`createIdbFsi({ name, store, indexedDB })` is the fsi contract over one
object store keyed by `[dir, name]` with an index on `dir`: a directory
listing is one index lookup, every other call one request, and a write
resolves on the transaction's `complete`. The database opens on the first
call, lets go when another tab upgrades or deletes it (`versionchange`) and
reopens on the next, and `close()` is the backend's own — a storage never
closes a backend it was handed. Browser-only by nature, so a subpath with
its own size row; `fake-indexeddb` joins the devDependencies for the shared
fsi suite, the storage and the sync-engine suite over it, and the hostile
cases (a request that fails, a transaction that errors or aborts, a database
deleted under us).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two storages over one directory both read the log's vector and both mint
the next `origin:seq`: the same number for two different ops, which every
peer drops as a duplicate and nothing reconciles. `open()` now holds the
directory first — the Web Locks API in a browser (released with the tab;
Node's own `navigator.locks` is per process and is not used), a
`<path>/lock.json` with the pid elsewhere, stale once that process is gone
or, without a pid, after 30 minutes — and a second `open()` rejects with
`code: 'LOCKED'`. `lock: 'wait'` polls until it is free, `lock: false` opts
out (the bench does). `close()` releases it, a failed open releases it, a
lock that is not ours is left alone. src/lock.js has no requires.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The lint failure slipped through a pipeline whose exit status was tail's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… a bounded cache, integrity

`list()` was a scan of `data/` with four reads and a hash per record, the
entry cache grew with every id ever touched (absent ones included), and
`hasRecord()` depended on what happened to be cached. src/catalog.js keeps
every live record's id, collection and indexed field values in memory —
put at the one seam both write paths share (`#writeEntry`, plus the
encrypted pair), indexing the lensed view and never failing a write — and
persists them as `<path>/index/catalog.json` stamped with the log position
they reflect, every `catalog.flushEvery` changes and at close. A reopen
re-reads only what the log changed since the stamp, or rebuilds from one
scan when the log cannot say (first open, another epoch, below the floor,
other indexes). `candidates()` and `uniqueConflicts()` are ready for the
query layer.

`cache: { max }` bounds the entry cache (LRU, default 5000; a live Record
pins its own; absent ids are not cached), `integrity: 'lazy'` trusts a
record file on the cold read. Browser budget 23 → 24.5 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`collection.select(query)` is a cursor over a Mango-shaped query —
equality with dotted paths (an array field holds a scalar), `$eq` `$ne`
`$in` `$nin` `$gt` `$gte` `$lt` `$lte` `$exists` `$regex`/`$options`,
`$and` `$or` `$not` — compiled once in src/query.js, with `order`/`desc`,
`limit`/`offset`, `filter`, `projection`, `fetch`/`count`/`one`, `for await`
and `watch(listener)`, which runs now and after every change to the
collection, coalesced, until stopped. A cursor plans: a query whose fields,
order and projection are all indexed is answered from the catalog alone,
an equality on an indexed field narrows the records read to its posting
list, an unordered limit stops at enough matches. It reads through the
storage's `[SOURCE]` seam — the catalog and a lensed, hook-free load, like
`snapshot()` and `swap()`.

Unique keys: an entity's `unique` fields (single or composite) are refused
on a local write with a `ValidationError` issue `{ code: 'unique',
conflict }`, after validation and the fold, before anything is logged. A
peer's op that leaves two records on one key cannot be refused — it is in
the peer's log — so it is applied and reported: the `conflict` event
`{ kind: 'unique', entity, fields, id, other, op }` and the
`syncom.conflicts` metric; both records stay until the application decides.
Browser budget 24.5 → 26.5 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Who wrote an op is an audit question the log answers, not the record:
`open({ actor })` — a string, or a function asked per write for the
signed-in user — and a per-write `{ actor }` on every write method, the
collection's, the Record's and `migrate()` stamp the op with `actor`, the
14th packed slot (a reader of 13 ignores it, a merge and an advance carry
none). It rides into every peer's log, `change` and `beforeApply`, and never
into the record or its meta. It is a claim, not a signature: a host that
knows who a peer is can hold the claim to it in `beforeApply`.
`WriteContext.actor`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…contract, a hub-mesh test

A hub had no say over what a peer pushed: `unit({ limits: { opsPerPush,
bytesPerPush, opsPerSecond } })` refuses a page over the first two with
code 413 and a peer over the third — a token bucket per peer, refilled at
that rate, holding one second's worth — with 429, in `push`, the one place
that knows the ops and the bytes whatever transport carried them. A code,
never a hang-up: the dialer reports it (`error.code`), drops that outbox and
recomputes from the vector at its next handshake. `syncom.ops.limited`
counts refusals by reason. `unit({ meta })` is copied onto all four
procedures, so a wrpc `onRequest` hook can read a hub-wide policy as
`context.procedure.meta` while the unit stays a plain object.

`isSharedLog()` and the `SharedLog` type reserve the contract of the
broker-backed hub mode — structurally wrpc's `BrokerLog`, so a broker's
`.log` fits with no adapter — and `open({ sync: { sharedLog } })` is refused
until that mode exists. tests/hub.mesh.test.js runs two hubs and two
clients: ops cross the hub edge, a hub restarts and its clients resume, a
client moves between hubs. Browser budget 26.5 → 27.5 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… form

`delete()` leaves a tombstone and the record's history in the log, which is
what lets a peer catch up; `purge(id)` is the other thing. A `purge` op —
the 7th type, `meta.purged` — wins over every other op at any stamp, leaves
a canonical empty entry whatever arrived before, and nothing but another
purge or a merge touches it again, so a later or concurrent write on any
node cannot bring the record back. `OpLog.scrub(id)` strips every entry of
the id of its payload, in memory and in the segments and open files that
hold them (`scrubbed: true`; position, origin and stamp kept, so the vector
and a resuming peer are unaffected). A peer that receives the purge, or
anything about the id afterwards, scrubs its own log once the batch is
durable and before the relay. The chain keeps hashes only.

The limit is honest: a file is rewritten in place, so a crash mid-scrub
tears it and a peer resuming through it bootstraps; the id stays marked
until a later purge finishes the rewrite. Browser budget 27.5 → 28 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…wser that hosts

No transport code: a PeerLink's `remote` from `@alexify/wrpc/webrtc` is a
WrpcClient, which `createWrpcLink()` takes as it is, and `storage.sync.unit()`
mounts on a browser's PeerHost as on a server — so a browser node hosts for
its peers, which the docs used to deny. The devDependency moves to
`@alexify/wrpc` ^2.0.1 (the existing suite passes unchanged), and the
integration suite gains the raw-channel level every WrpcPeer link rests on,
over a fake data-channel pair (tests/utils/fake-channel.js): a channel
attached to an RpcServer (both directions, a bootstrap streamed over the
channel, a close), a channel factory wrpc asks on every reopen (the
subscription resumes, syncom greets again, no snapshot), and a channel on a
PeerHost (`authorize` reading the `trust: 'link'` pseudo-session, 403 for
another peer, 403 before hello under `trust: 'none'`, the unit's limits).
A WebRTC guide with the dial, host, signaling, mesh, reconnect and
raw-channel recipes; `webrtc` joins the keywords.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Alex-Dolid and others added 28 commits October 21, 2026 02:53
…d not

- API reference: three subpaths, not two (`createIdbFsi` named); `migrate()`
  takes `actor`; the `fetch()` example returns rows, not ids; `schemaInfo`
  carries `held`; a registry node's `remote` carries `schema`; the write
  hooks see `purge`; `storage.logger` documented
- types: `storage.logger`, `OpLog.latestTs` and the `Collection`/`Record`
  constructor types existed at runtime but not in `index.d.ts`;
  `listenerCount`, `listeners`, `eventNames`, `toPromise` and `clear` typed
  on the three emitters; the package consistency test checks the subpaths'
  runtime exports against their `.d.ts` too
- comments: `packOps` is the wire shape, the log stores keyed JSON
  (CLAUDE.md); a streamed snapshot page is collected whole; a torn open log
  file is dropped at reopen, a torn segment skipped; the mesh test helper
  waits three polls; two tautological ternaries in crdt.js
- SECURITY.md: the threat model names what a peer cannot do to a dialer's
  cursor and memory, and that events and hooks never carry the link or the
  token

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…obs, guards, caches

- `src/emitter.js`, `src/logger.js`, `src/keys.js`: the Emitter, the logger
  writer and `loadKeys` lived once per platform half, identical; the two
  `utils` halves now re-export them and keep only the platform's own bits.
  `src/scope.js` defines "in scope" once for the log, the storage and the
  sync manager. `between` and `vectorDominates` had no callers
- the directory lock prefers `fsi.create` — exclusive create, on `node:fs`
  and the memory backend (`withSync` forwards it) — so openers racing for a
  free lock lose deterministically; `open({ lock })` takes an object:
  `mode`, `staleAfter`, `poll`, `timeout` (a `'wait'` that gives up with
  `LOCKED`)
- `open({ maxListeners })`, a hundred by default: ten threw at the eleventh
  `change` listener, on the storage and on its sync manager
- `$regex` capped at 1000 characters, `projection(['__proto__'])` refused
- a torn registry file is reported (`sync.node.unreadable`) and skipped at
  open; a torn `node.json`, `schema.json` or `.blockchain.json` rejects with
  `code: 'CORRUPT'` naming the file
- the OPFS backend keeps its directory handles (walking from the root was
  the cost of every call), the browser crypto keeps its imported keys; the
  log's `latestTs` rides in the index and its open builds the dedup set once
- `examples/` is linted and formatted and closes its storage; the wrpc
  integration test skips only on `MODULE_NOT_FOUND`; the mesh test helper
  no longer awaits a condition that is always true

Left for their own change, each a decision or a refactor of its own: the
split of storage.js (M-1), an error model with codes everywhere (M-6), the
Reference magic and crdt.js (M-9, M-8), a repair of an unreadable record
(R-11), the query engine's top-k and parallel loads (P-6), the benches that
do not exist yet (P-9).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ths over one core

storage.js was 1 900 lines and one class: option checks, the Record
objects, the cache and record files, the local write path, a peer's
batch, the bootstrap page and the host's router unit. It is now the
facade (open sequence, mutation queue, public API) and lends a `core`
— its log, clock, chain, keys, entries, catalog and the lensed readers
— to three path modules: commit.js (prepare, validate, persist,
announce), apply.js (admit, apply one by one, persist, announce) and
bootstrap.js (snapshot pages, merge pages, adoptVector). EntryStore
(entries.js) owns data/, the bounded cache and the catalog row every
change leaves — the one seam that calls catalog.put. record.js holds
Reference, Record and Collection over the public API and the three
symbols (symbols.js); options.js checks open()'s plain options; shape.js
checks what enters (plain objects, ids, stream names, a peer's op).
sync.js keeps the dialing side and delegates unit() to hosting.js
(hello, push, snapshot, changes as procedures over a host context, the
limits and grant()); protocol.js holds what both halves share.

Pure extraction: the same order of effects, hook calls and events on
every path; no error message, event payload, hook signature or on-disk
format changed. Every function in the engine is under 60 lines; the
held replay keeps the skipped flag out of its filter (held entries carry
both). CLAUDE.md names the new files and the rules now point at them.

The browser budget goes 33.5 → 35.5 KB (35.3 measured): a class's
private names minify to one letter, the property names of a seam shared
between modules do not. 626 tests pass; coverage 99.5/96.07/99.66/99.5.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…on on every refusal

A caller could not tell "Storage is closed" from "not found" from "is a
counter; use inc()" without parsing the message, and the codes that did
exist came in four styles. Every error syncom raises on purpose now has
a shape by kind. An operational failure — one that can happen to correct
calling code given the data, the files or the peers — is a SyncomError
(src/errors.js, exported from the root) with a code from one union
(CLOSED, IN_HOOK, NOT_FOUND, FIELD_KIND, ENCRYPTED, INTEGRITY, CORRUPT,
DOWNGRADE, LOCKED, KEYS_INCOMPLETE, BAD_ENVELOPE, BLOCK_NOT_FOUND,
NOT_CONTRACT, NODE_EXISTS, NODE_NOT_FOUND, BAD_HELLO, HANDSHAKE_LOOP,
PAGE_FAILED, STUCK, STREAMS_UNSUPPORTED, SKEW, OP_UNREADABLE, OP_REFUSED,
OP_INVALID), the fields the code documents (field/kind, id, url, hash,
cursor/attempt, op) and `cause`. HookError and ValidationError extend it
and keep their names, codes and fields. A wrong argument, option or hook
result stays a TypeError without a code; typed() keeps the three whose
value is data or configuration (INVALID_ID, TYPE_IMMUTABLE, NO_CHAIN).
A host's wire refusal keeps the numeric code wrpc carries and gains
`reason` (HELLO_FIRST, BAD_HELLO, PROTOCOL, SELF, UNAUTHORIZED, SCOPES,
OPS_PER_PUSH, BYTES_PER_PUSH, OPS_PER_SECOND), logged with the refusal.

No message changed; three registry argument errors became TypeErrors.
index.d.ts declares SyncomError and SyncomErrorCode (the storage `error`
event is typed as one, `op` required), browser.d.ts re-exports them, the
tsd cases cover both. tests/errors.test.js reaches every code the way an
application would; the existing assertions name the codes they expect.
docs/reference/api.md has the table (code → who raises it → fields),
CLAUDE.md the rule. Browser budget 35.5 → 36 KB (35.7 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…scopes, benches for apply and sync

Queries: an ordered cursor with a limit keeps only the best offset + limit
rows as the candidates go by (TopK — a binary search and a bounded splice
per match, stable like the sort it replaces) instead of sorting every
match; a loaded cursor reads its candidates in parallel, a batch as wide
as what is still wanted and sixteen at most, in candidate order (one()
still stops at the first match); Catalog.candidates narrows through every
$and branch and through an $or whose branches all name an indexed
equality; ids() is cached until the collection changes, indexed() is one
set per collection. Cursor#matching is the one place that reads records.

repair(id): a record whose file fails its integrity check or is torn is
rebuilt from this node's log — OpLog.entriesOf(id), replayed from the
first whole-record op (write, merge, delete, or a purge even scrubbed),
written through #persistBatch with a chain block of its own, back into
the catalog and the cache, a live Record refreshed. UNRECOVERABLE when no
such op is retained (compacted away, never written here, encrypted). Not
implicit on read: an integrity failure is evidence first.

The hosted feed: OpLog.read(eventId, { scopes, limit }) answers
{ entries, last } — last the seq of the last entry examined, in scope or
not, a real entry's position; since() is read().entries. When a wake
brings nothing for the peer, `changes` moves its cursor there without a
page, so an op of another scope is looked at once rather than on every
wake. Never lastEventId, which runs ahead of the entries during an
append — the earlier attempt lost the batch in flight.

Benches: bench/apply.js (applyOp, packOps/unpackOps, applyRemote pages
of 100 and 500 writes, sets and incs, with and without the chain; one
origin per stream, rows count ops) and bench/sync.js over the memory
link (a write applied one and two hops away, a snapshot page, a
2 000-record bootstrap from a compacted host, a scoped feed under another
scope's appends); the cold-read row of bench/storage.js closes the
instance it opens.

Types, tsd, docs (query plan, Repair, the API tables, the sync guide, the
log-event table) and CLAUDE.md follow. 634 tests pass; coverage
99.55/96.16/99.83/99.55. Browser budget 36 → 37 KB (36.5 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…hteen, why and architecture pages

Benchmarks: bench/compare.js runs syncom (with and without the chain), Yjs,
Automerge, Loro, TinyBase, PouchDB, RxDB, Fireproof, GUN and OrbitDB on the
same scenarios — insert, update, read, a converging counter, a batch of
1,000, an empty replica pulling a store of 1,000 — in memory and on disk,
plus a cold start; each (library, scenario) pair in a fresh process, every
answer checked before anything is timed, results merged into
bench/results/compare.json after every scenario, a failing pair retried once
and then recorded as failed. The adapters (bench/compare/adapters) use each
library the way its docs show. The libraries live in bench/compare's own
package.json (pnpm bench:compare:install), never in syncom's
devDependencies. bench/compare/size.js measures what a browser app ships to
store and sync records, WebAssembly included. pnpm bench --save records
syncom's own scripts in bench/results/engine.json; run-all skips compare.js
and report.js.

Charts: bench/report/model.mjs decides what every chart shows; the docs
render it as interactive bars (BenchCompare, BenchFigure, BenchEngine) and
bench/report.js as SVGs for the README plus the tables between the
bench:<name> markers. Brand colours checked for separation in both themes.

Comparison: docs/comparison.json — syncom and eighteen alternatives (CRDT
libraries, local-first databases, p2p and event-sourced stores, sync
services) across forty features, every cell decided from the published
source and official docs with its source kept in the data, the strongest
claims checked twice; scripts/comparison.js (pnpm docs:comparison) renders
it into docs/guide/comparison.md with the matrix styles of mailer.

Pages: why, comparison, benchmarks and architecture (with diagrams of the
facade, the three write paths, a local write and replication) under
Introduction; the home page repositioned as a sync engine with new tiles and
the benchmark chart; getting-started, chain, llms.txt, the README and the
site description brought up to date (the chain is optional, storage
replicates through ops, the bundle is 37 KB). Multi-line Mermaid labels no
longer lose their last line to the .vp-doc paragraph line height.

The numbers say where syncom stands: the fastest cold start (34 ms),
replication in memory level with Yjs and Loro, writes on disk an order of
magnitude behind LevelDB-backed libraries — half of that the catalog saved
whole after writes, which the benchmarks page names as the open improvement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalog is a whole file and it was saved every catalog.flushEvery (64)
puts whatever its size, so every write paid for a sixty-fourth of every row:
an insert into 20,000 records in memory ran at a third of the speed of one
into 1,000, and Catalog.save was half of the CPU. Storage#flushThreshold
makes flushEvery the floor: a catalog is saved every sixteenth of its size
as it grows, never more than half the log's retention.ops apart, so a node
stopped without close() still finds its stamp in the log and catches up
instead of rebuilding from a scan. OpLog.retainedOps says what the log keeps
(Infinity when no count bounds it), typed and tsd-tested; a suite test on
both halves checks that a growing catalog saves less often but never more
than half the retained log behind.

Measured again (bench/results, the charts and the docs numbers refreshed):
inserts in memory 11,100 → 37,500 a second (44,300 without the chain) and
flat to 40,000 records; a batch of 1,000 in memory 36,000 → 41,000; an empty
replica in memory 101,000 → 108,000 records a second; on disk 1,390 →
1,490 inserts a second, where the file writes are now what is left. The
browser entry is 36.6 KB of its 37 KB budget.

Version 1.0.0: package.json, the docs nav label, the syncom column of the
matrix and the measured results carry it; "pre-1.0" is gone from the
README, why, comparison, llms.txt and SECURITY.md (it is still not on npm,
and the pages say so); the changelog's [Unreleased] names 1.0.0 as the first
release, and the npm-publish skill knows the version was chosen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d merge rules

A third field kind next to the register and the counter: `add` and
`remove` ops (codes `+`/`-`, payload in the key/value slots like `inc`)
keep `meta.sets[field][elementKey][origin] = [added, removed]`, the ts of
that origin's newest add and the newest add ts a remove had observed. Both
ops are joins on the slot, so they commute and are idempotent; a reset
drops a slot below `base` and clamps a surviving component below it, so
meta — not only data — agrees whatever the order. `data[field]` is the
live elements by the earliest live add, then by key. The kind rule gains
`reason: 'set'`; `mergeEntry` joins slots and lets a register, then a
counter, win over a set. Browser budget 37 → 38 KB (37.7 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`storage.add(id, field, elements)` and `storage.remove(...)` (and the
same on a live Record) commit `add`/`remove` ops; a remove's op carries
the tags this node observed, read inside the mutation queue. The kind
rule gets its third hint (`a set; use add() or remove()`); a set op is
validated as the record it leaves, like an inc, never folded; `migrate`
adds a record's sets back after the whole-record write and reports
`sets`. A set reads sorted by element text — it has no insertion order,
and a same-origin re-add moves the element's stamp — so the order is one
rule on every node. Hooks see `key`/`value` as the elements; purge scrubs
them with the payload; the convergence suite covers a partition, the
seeded fuzz and a late joiner's snapshot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…yped surface

`SyncOpType` gains `add` and `remove`; `SyncOp.value` and
`WriteContext.value` take the elements (and, for a remove, the observed
tags as `SetRemoval`); `EntryMeta.sets` is declared; the refusal reason is
the `FieldKind` union (`register | counter | set`); `Storage` and
`SyncomRecord` get `add`/`remove`; `migrate` reports `sets`. tsd cases
for each, both barrels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…arrows by element

`{ tags: 'bug' }` on an indexed array field used to narrow to records whose
value IS `'bug'` and miss every array holding it, while the predicate
matched any element — a covered count and a loaded fetch disagreed. The
catalog now posts an array value under its own key and under each element's
(a superset the predicate filters: a nested array's element can share a key
with a value); unique keys still compare the whole array. The first thing a
set field needs from the catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ply bench rows

The sync guide's "Registers, counters and sets" (the example run before it
was written), the kind-clash limit named next to the counter/reset one, the
storage and schema guides, the API reference rows (`add`, `remove`,
`FIELD_KIND`, `ENCRYPTED`, `migrate`), hooks, architecture, telemetry
labels, llms.txt, README (features, the size table) and the comparison
matrix's `lists` cell; CLAUDE.md's field-kind and `packOps` rules (a new
op type after 1.0.0 is a PROTOCOL bump) and the fuzz note. `bench/apply.js`
gains an add and a remove on a 20-element set and 100 adds on cached
records; results and charts regenerated. An add or a remove on a
20-element set costs ~4 µs in `applyOp` — the entry's slots are deep-copied
per op and the field re-read sorted — invisible under `applyRemote`, where
the record write dominates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A second pair next to the RSA one: `sign-public.pem`/`sign-private.pem`,
loaded by `loadSigningKeys` (`loadKeyPair` takes the file names now; half
a pair is refused per pair), whose `id` is the raw 32-byte public key in
base64url — what a node will be known by. `sign(message, privateKey)` and
`verify(message, signature, id)` are synchronous in node:crypto and
promise-returning in Web Crypto, like the three before them; `verify` is
false, never a throw, for a malformed id or signature. The browser import
cache is keyed by usage as well as PEM, so one PEM never comes back as the
other kind of key. Cross-half tests: a signature made on one half verifies
on the other with either half's keys, and one pair on disk is one id on
both. Browser budget 38 → 38.5 KB (38.3 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`open()` loads the signing pair next to the RSA one and the node id is its
public key — 43 base64url characters, a canonical last character
(`isKeyId`), a valid origin — so a signature will verify against its op's
origin with no key to look up. `node.json` still names the id for the
operator: a directory from before signing (a UUID there) is rekeyed once
with an `identity.rekeyed` warning; a file naming another signing key is
refused with `IDENTITY_MISMATCH` (`id`, `key`), never changed in silence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`src/signing.js`: what a signature covers — the header (origin, seq, ts,
type, id, scope, from, v, actor) as a positional JSON array under a version
number, plus a digest of the payload (data, meta, key, value, vector)
serialized with sorted keys and salted with a nonce — `signOp` and
`verifyOp` (ok / unsigned / forged). The split lets a purge scrub the
payload and the nonce and leave a header that still verifies, and the
nonce keeps a scrubbed op's digest from giving a low-entropy payload away.
`packOps` carries `sig`, `digest` and `nonce` in slots 15–17; `wellFormed`
checks their shape; `OpLog.scrub` deletes the nonce with the payload. The
suite runs over both crypto halves. Browser budget 38.5 → 39 KB.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`core.sign` signs an op in place once it is final: in `persistLocal` after
validation rewrote its data and before the log append, in `mergeBatch`
after each merge's seq is assigned, and in `adopt` for the advance — so a
snapshot joined from a peer is re-signed by the joiner as its own (a
snapshot record carries no provenance). A validator's default is under the
digest, since it is in the op the peers receive. After a purge the scrubbed
entries keep their signature and digest, lose the nonce, and still verify,
on both halves. Browser budget 39 → 39.5 KB (39.3 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`open({ sync: { signatures } })`: `'verify'` (the default) signs this
node's ops and checks a peer's op when it carries a signature, taking an
unsigned one; `'require'` refuses unsigned ops and keyless origins too —
the one mode that resists a relay stripping signatures; `false` signs and
checks nothing. The node id is the key in every mode. `applyRemote` runs a
pre-pass over the page, sixty-four verifications at a time under
Promise.all (the browser's async verify is four times faster that way),
before the sequential `admit`; a refused op is counted per op (`forged`,
`unsigned` in the result and the metric) and reported once per page as an
`error` event (`BAD_SIGNATURE` / `UNSIGNED`, with the first op and the
count), neither logged nor relayed, the vector unmoved — a gap for its
stream until it comes again intact. The suite runs on both halves; a
three-node mesh under 'require' converges with every logged op verifying
against its origin. Browser budget 39.5 → 40 KB (39.8 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…a keyless hello

`push` announces what the apply refused for its signature through the
`refused` event (`reason: 'forged' | 'unsigned'`), tying the refusal to the
client that pushed; the metric was counted per op in applyRemote already,
so the host only announces. Under `sync.signatures: 'require'`, `hello`
with a node id that is no signing key is refused with 403 `UNSIGNED` — such
a peer could never push, and would have every op refused in silence at
every handshake otherwise (not a proof of possession: a signed hello
challenge would be). Telemetry: `syncom.ops.received{refused}`,
`syncom.ops.rejected{forged|unsigned}`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…del, the reference; bench rows

What a signature covers and what it proves, the three modes in a table, the
refusals, the cost, and the honest list of what it does not prove (`actor`
a claim, a bootstrap re-signed by the joiner, stripping under 'verify',
delivery, key possession at hello); SECURITY.md's threat model and scope;
the API reference (crypto table, `nodeId`/`signatures`, `scrub`, the
codes); storage (keys on disk), browser (five async functions), scaling,
telemetry (outcomes, reasons, events), README (feature row, limits), why,
the comparison matrix's `signing` cell (✅ with a note), llms.txt, the
CLAUDE.md rule and the CHANGELOG entry. `bench/apply.js` gains a page of
signed writes and the verify pass alone: verification costs ~0.1 ms per
op and halves the throughput of a page of new records (7.0k → 3.8k ops/s
here); results and charts regenerated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…contract, $text

An entity may name `search: ['title', 'body']`; the catalog joins those
fields' strings (and string elements of arrays) into the record's row as it
reads, persists them as a seventh cell (format 3 — an older file is rebuilt
once), and hands the text to an injected index on every put — inside
`#insert`/`#remove`, under a try/catch that logs `catalog.search`, since
derived state never fails a write — and replays every row into it at load.
`open({ search })` is a function `(entity) => TextIndex` (`put`, `remove`,
`query` → `[{ id, score }]` best first), checked at open; an entity with
search fields and no index is refused there. `$text: string`, at the top
level of a query only, compiles to no predicate: the catalog seeds the
candidates with the index's ranking and every equality narrows that order,
so an unordered cursor hands rows out best first, `limit` stops early and
`count()` counts them all. Browser budget 40 → 41 KB (40.7 measured).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`createSearch({ minOverlap })` hands `open({ search })` a fresh trigram
index per searchable entity. Text is normalized (NFD, combining marks
stripped, lower case, anything but letters and digits a word break) and
cut into per-word, space-padded trigrams, so a word's start and end count
and an exact word outranks a substring of a longer one. A record's score
is the sum of qtf · tf · idf over the trigrams it shares with the query,
over the square root of its trigram count, with idf = ln(1 + N/df) — never
zero, so a one-record store finds its record; it matches when it holds
`minOverlap` (0.3) of the query's distinct trigrams, which is what makes a
typo at the end of a word still match. Ties break by id. Postings live in
memory; the catalog persists the text. Platform-neutral, no requires, its
own 2 KB budget (0.9 measured) off the main entry. Shims, exports, files,
keywords (`full-text-search`, mirrored in the docs config), the size row,
the consistency check, unit tests and tsd.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through the storage, on node and on the browser halves: search fields with
no `open({ search })`, a bad option and a bad index refused at open; every
write path reaching the index (insert, update, delete, a revival, purge, a
peer's op); `$text` ranking the short exact title first, an equality
narrowing it, an unordered limit and `one()` reading one record, `order()`
re-sorting, a covered projection reading none, a typo still matching; the
lensed view being what is searched (a v1 `headline` found as v2's `title`),
an encrypted record never matched, `$text` on a collection without search
fields refused at select; a reopen bringing the text back from
catalog.json with no record read, and other search fields rebuilding it;
`watch()` waking on a matching insert; a throwing index logged as
`catalog.search` with the write landed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`bench/search.js` over a deterministic 2000-record corpus (a 3-word title
and a 40-word body each, both searched): `index.put` of a 43-word record
(~60 µs), a one-word, two-word and typo query of the index (~0.6, 0.7 and
0.13 ms), an insert with two search fields, `select` with `$text` and a
limit, with an equality narrowing it, `count`, and an open that reloads
the catalog and replays the text into the index (~85 ms). Inserts go to a
store of their own, so the queries run over exactly the corpus. The
report model gains the `search` subject; results and charts regenerated
(the other charts change only their date).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the rules

The query guide's "Full-text search" section — the example run before it
was written, rank order against `order()`, what is searched (the lensed
record, never encrypted or deleted), how it matches (normalization,
padded per-word trigrams, `minOverlap`, TF-IDF over length), what it
costs (a put per write, the text in catalog.json, rewritten whole on
OPFS), its limits and a TextIndex of your own — plus the forms-table row
and the TypeError sentence. The reference: the fourth subpath and its
section, `open({ search })`, the Cursor link pointing at the query guide.
Schema's entity row, storage's catalog and on-disk notes, the telemetry
`catalog.search` event, architecture's module table (search.js and
signing.js), llms.txt, "about 41 KB" in why/index/README, README's feature
row and the six-row size table, the CLAUDE.md `$text` rule and suite list,
and the CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`verifyPage` ran 64 checks under Promise.all, but on Node each one was the
synchronous `crypto.verify`, so they ran one after another on the main
thread and held a replica to ~9,000 records a second in memory. The
engine now verifies through `verifyAsync` — the callback form of
`crypto.verify`, which runs on libuv's thread pool (UV_THREADPOOL_SIZE,
4 by default) — so a page's checks run in parallel: the `verifyOp` bench
row went from ~10,100 to ~36,700 ops/s, and replicating a store in the
comparison from ~9,000 to ~25,200 records/s in memory (~4,000 → ~5,800 on
disk). Signing a local write stays on the main thread.

`verifyAsync` is the engine's, not the API's: both barrels drop it, the
public `verify` stays synchronous in Node.js (the async-asymmetry rule in
CLAUDE.md now names the exception), and in the browser it is `verify`
itself. It answers `false`, never a throw, for a malformed id or
signature, on both halves. Sync and Scaling describe the cost and
`UV_THREADPOOL_SIZE`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The package goes to main unpublished: there is no npm release until it
does what it is meant to, so `package.json` carries 0.0.1, a development
placeholder, and the first release version is chosen when it is cut. The
docs nav label, the comparison matrix, CHANGELOG ([Unreleased]), the
npm-publish skill, SECURITY.md, README, llms.txt and the Why page say so;
the install instructions note that it is not on npm yet.

Docs that had fallen behind the code: the result of `applyRemote` and the
`sync`/`apply` events (`unsigned`, `forged`) in hooks, sync, the API
reference, telemetry and CHANGELOG's breaking list; the `syncom.apply` span
attributes; the local-write sequence (the op is signed), the host's checks
and the files on disk in Architecture; both key pairs in Getting started;
three field kinds in CONTRIBUTING and the CRDT page; sets, signatures and
full-text search on the home tiles and the Why page; the entity
definition's `search` key in CHANGELOG.

syncom re-measured — the comparison (`--only syncom,syncom-nochain`), the
bundle sizes and every engine bench — since the published numbers predated
signed ops, sets and search. Signing is the main cost in memory: an insert
went from ~37,500 to ~16,200/s (on disk unchanged at ~1,500/s), and
replication in memory from ~108,000 to ~25,200/s with verification on the
thread pool; the browser bundle in the size comparison is 41.3 KB. The
benchmark prose, Why and the home page quote the new numbers, and no longer
claim replication level with Yjs and Loro.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lone

A peer's page moves the version vector at the log append, and its record
files are written after that, inside the same applyRemote. settle() took
equal vectors as convergence, so on a slow disk a test could read a record
before its file landed: `sync.resilience.test.js:108` saw `null` instead of
`{ n: 2 }` on the Node 24 CI job. Reproduced deterministically with a
replica whose data writes lag 80 ms (5 of 5 reads missed); with settle()
also requiring `storage.stats().queue === 0` on every node, 0 of 5. The
signatures suite's own wait for the record is no longer needed and goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Alex-Dolid
Alex-Dolid merged commit 1a753f7 into main Oct 23, 2026
4 checks passed
@Alex-Dolid
Alex-Dolid deleted the feature/v1 branch October 23, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant