From eda6ad8582726fdfcd7fcf7da08f04fc78901699 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Fri, 4 Sep 2026 16:11:56 +0300 Subject: [PATCH 01/67] feat: migrate from globalstorage files to syncom --- README.md | 70 +++ browser.js | 815 ++++++++++++++++++++++++++++++++++- emulation/data.js | 392 +++++++++++++++++ emulation/example.js | 147 +++++++ emulation/storage.js | 360 ++++++++++++++++ example.js | 91 ++++ examples/domain.js | 32 ++ examples/main.js | 30 ++ index.d.ts | 239 ++++++++++ index.js | 10 +- package.json | 9 +- src/chain.js | 75 ++++ src/contract.js | 75 ++++ src/crdt.js | 103 +++++ src/crypto-browser.js | 96 +++++ src/crypto.js | 51 +++ src/fsi-browser.js | 47 ++ src/fsi.js | 21 + src/storage.js | 354 +++++++++++++++ src/sync-browser.js | 56 +++ src/sync.js | 79 ++++ tests/chain.js | 91 ++++ tests/contract.js | 146 +++++++ tests/crdt.js | 207 +++++++++ tests/crypto-browser.js | 38 ++ tests/fsi-browser.js | 18 + tests/fsi.js | 13 + tests/integration.js | 51 +++ tests/keys.js | 17 + tests/storage.js | 384 +++++++++++++++++ tests/suites/crypto-keys.js | 49 +++ tests/suites/fsi.js | 68 +++ tests/suites/sync-manager.js | 63 +++ tests/sync-browser.js | 10 + tests/sync.js | 16 + tests/utils/mock-opfs.js | 107 +++++ tests/utils/temp.js | 26 ++ 37 files changed, 4449 insertions(+), 7 deletions(-) create mode 100644 emulation/data.js create mode 100644 emulation/example.js create mode 100644 emulation/storage.js create mode 100644 example.js create mode 100644 examples/domain.js create mode 100644 examples/main.js create mode 100644 index.d.ts create mode 100644 src/chain.js create mode 100644 src/contract.js create mode 100644 src/crdt.js create mode 100644 src/crypto-browser.js create mode 100644 src/crypto.js create mode 100644 src/fsi-browser.js create mode 100644 src/fsi.js create mode 100644 src/storage.js create mode 100644 src/sync-browser.js create mode 100644 src/sync.js create mode 100644 tests/chain.js create mode 100644 tests/contract.js create mode 100644 tests/crdt.js create mode 100644 tests/crypto-browser.js create mode 100644 tests/fsi-browser.js create mode 100644 tests/fsi.js create mode 100644 tests/integration.js create mode 100644 tests/keys.js create mode 100644 tests/storage.js create mode 100644 tests/suites/crypto-keys.js create mode 100644 tests/suites/fsi.js create mode 100644 tests/suites/sync-manager.js create mode 100644 tests/sync-browser.js create mode 100644 tests/sync.js create mode 100644 tests/utils/mock-opfs.js create mode 100644 tests/utils/temp.js diff --git a/README.md b/README.md index 142ab38..72aec75 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,72 @@ # syncom Fast, zero-dependency distributed data sync engine ๐ŸŒ + +[![npm](https://img.shields.io/npm/v/%40alexify%2Fsyncom)](https://www.npmjs.com/package/@alexify/syncom) +[![CI](https://github.com/Alexis-Technologies/syncom/actions/workflows/ci.yml/badge.svg)](https://github.com/Alexis-Technologies/syncom/actions/workflows/ci.yml) +[![node](https://img.shields.io/node/v/%40alexify%2Fsyncom)](#installation) +[![dependencies](https://img.shields.io/badge/runtime_dependencies-0-brightgreen)](#bundle-size) +[![docs](https://img.shields.io/badge/docs-online-blue)](https://syncom.vercel.app/) +[![license](https://img.shields.io/npm/l/%40alexify%2Fsyncom)](./LICENSE) + +## The Concept + +๐Ÿ“ฆ `syncom` is a modular distributed data sync engine for JavaScript and the Webโ€”featuring blockchain integrity, local-first sync, CRDTs, and smart contracts. + +### Key Features + +- **Distributed architecture** with global unique IDs and deterministic replication +- **Blockchain log** for tamper-proof operation history +- **JavaScript Smart Contracts** executed within your app +- **CRDT synchronization** for offline-first and P2P support +- **Local-first storage**: supports `fs`, `IndexedDB`, and `OPFS` +- **Metaschema-driven domain modeling** for validation and sync +- **CAS operations** for consistency and concurrency safety +- **Web-ready**: works in Web browsers and Node.js runtime +- **Secure transport** via `metacom` protocol and WebSocket +- **Crypto support**: RSA, SHA-256, and more +- **Built-in integration** with [Impress Application Server](https://github.com/metarhia/impress) + +### Install + +Requires **Node.js 18+**. + +```bash +npm install @alexify/syncom +``` + +### Usage + +- **Node.js** loads the CommonJS entry `syncom.js`. +- **Browser** exports `syncom.mjs` (ESM). +- **TypeScript** types for development are maintained in [`syncom.d.ts`](./syncom.d.ts). + +### Ideal For + +- Local-first apps +- Real-time collaboration tools +- P2P distributed systems +- Full-stack JavaScript projects +- Blockchain-based data pipelines + +### Additional features + +- Built-in database engine will minimize interprocess communication to access data +- Minimize data transformations: all data should be stored, transmitted, handled and placed in RAM in the same format +- Maximum memory usage, read-ahead and lazy-write, minimizing data conversion +- Using metadata everywhere, special declarative format for subject domein representation (including fields, relations, and indices), memory structures, GUI, API, etc. +- The same API for client-side runtime and server-side runtime: + - server-side storage engine + - client-side storage engine (multiple implementations for different platforms including mobile, desktop and browser) + - sharding for distributed storage of large data amounts, geo-distribution, save backup copies, access load balancing + - allows user to exchange data in P2P mode +- Synchronization between client and server in realtime (close to realtime) and in lazy mode +- Offline-first approach: applications can work in online and offline (with locally stored data) +- Bidirectional data sync with CRDT and hierarchical versioning like git have +- Global data structures unification for applications working with [Metarhia](https://github.com/metarhia/Metarhia) technological stack and [Impress](https://github.com/metarhia/impress) application server +- Ability to work with non-unified data structures (custom schemas), specific to certain subject domain +- Data structures have global distributed identification system, so data can be inserted anywhere and will not bring ID conflicts +- Data reliability is provided by distributed storage facilities, so each data structure should have master server and multiple backup and cache servers; using those servers globalstorage supports addressing, versioning and branching + +## Contributors + +See GitHub for full [contributors list](https://github.com/metarhia/globalstorage/graphs/contributors) diff --git a/browser.js b/browser.js index e551299..4e6d47f 100644 --- a/browser.js +++ b/browser.js @@ -1,5 +1,810 @@ -// Browser entry โ€” intentionally mirrors index.js. Do not deduplicate: the -// package.json "exports" browser condition targets this file, and bundlers -// substitute src/runtime/node.js -> src/runtime/browser.js via the -// package.json "browser" field map. -module.exports = require('./src/index.js'); +import { Emitter, generateUUID } from './metautil.js'; + +// crdt.js + +const diff = (source, target) => { + if (typeof source !== 'object') source = {}; + if (typeof target !== 'object') target = {}; + const delta = {}; + const sourceKeys = new Set(Object.keys(source)); + const targetKeys = new Set(Object.keys(target)); + + for (const key of targetKeys) { + if (!sourceKeys.has(key)) { + delta[key] = target[key]; + } else if (Array.isArray(target[key]) || target[key] instanceof Set) { + const sourceArr = Array.isArray(source[key]) ? source[key] : source[key]; + const targetArr = Array.isArray(target[key]) ? target[key] : target[key]; + const sourceSet = new Set(sourceArr); + const targetSet = new Set(targetArr); + const diffSet = new Set([...targetSet].filter((x) => !sourceSet.has(x))); + if (diffSet.size > 0) { + delta[key] = Array.from(diffSet); + } + } else if (typeof target[key] === 'object' && target[key] !== null) { + const nestedDiff = diff(source[key], target[key]); + if (Object.keys(nestedDiff).length > 0) { + delta[key] = nestedDiff; + } + } else if (source[key] !== target[key]) { + delta[key] = target[key]; + } + } + + for (const key of sourceKeys) { + if (!targetKeys.has(key)) { + delta[key] = undefined; + } + } + + return delta; +}; + +const merge = (source, delta) => { + if (!delta || typeof delta !== 'object') return source; + if (source === null || typeof source !== 'object') return delta; + + const result = Array.isArray(source) ? [...source] : { ...source }; + + for (const key in delta) { + if (delta[key] === undefined) { + delete result[key]; + continue; + } + + if (Array.isArray(delta[key]) || delta[key] instanceof Set) { + const sourceSet = new Set( + Array.isArray(result[key]) ? result[key] : result[key] || [], + ); + const deltaSet = new Set( + Array.isArray(delta[key]) ? delta[key] : delta[key], + ); + result[key] = Array.from(new Set([...sourceSet, ...deltaSet])); + } else if (typeof delta[key] === 'object' && delta[key] !== null) { + if (typeof result[key] === 'object' && result[key] !== null) { + result[key] = merge(result[key], delta[key]); + } else { + result[key] = delta[key]; + } + } else if ( + typeof delta[key] === 'number' && + typeof result[key] === 'number' + ) { + result[key] += delta[key]; + } else { + result[key] = delta[key]; + } + } + + return result; +}; + +const apply = (source, history) => { + if (!Array.isArray(history)) return source; + let result = { ...source }; + + const handlers = { + write: (record, state) => ({ ...state }), + delta: (record, state) => merge(state, record.data), + delete: (record, state) => ({ ...state, [record.key]: undefined }), + inc: (record, state) => merge(state, { [record.key]: record.value }), + dec: (record, state) => merge(state, { [record.key]: -record.value }), + }; + + for (const record of history) { + if (!record || typeof record !== 'object' || !record.type) continue; + const handler = handlers[record.type]; + if (handler) { + result = handler(record, result); + } + } + + return result; +}; + +// contract.js + +const deserializeFunction = (source) => { + const arrowIndex = source.indexOf('=>'); + const header = source.slice(0, arrowIndex).trim(); + let body = source.slice(arrowIndex + 2).trim(); + const argsStart = header.indexOf('('); + const argsEnd = header.lastIndexOf(')'); + const args = header.slice(argsStart + 1, argsEnd); + if (body.startsWith('{')) body = body.slice(1, -1); + else body = `return ${body};`; + body = `return async (${args}) => {${body}}`; + return new Function('reader', 'args', body)(); +}; + +class DataReader { + #storage; + + constructor(storage) { + this.#storage = storage; + } + + async get(id) { + return this.#storage.loadData(id); + } +} + +class SmartContract { + #storage; + #chain; + #proc; + + constructor(name, proc, { storage, chain }) { + this.name = name; + this.#storage = storage; + this.#chain = chain; + this.#proc = proc; + } + + async execute(args) { + const reader = new DataReader(this.#storage); + try { + const result = await this.#proc(reader, args); + await this.#storage.saveData(args.id, result); + return result; + } catch (error) { + const contract = this.name; + const timestamp = Date.now(); + const block = { contract, args, error: error.message, timestamp }; + await this.#chain.addBlock(block); + throw error; + } + } + + static async save(name, chain, proc) { + const source = proc.toString(); + const block = { type: 'smart contract', name, source }; + const hash = await chain.addBlock(block); + return hash; + } + + static async load(hash, { storage, chain }) { + const block = await chain.readBlock(hash); + if (!block) throw new Error('Block not found'); + if (block.data.type !== 'smart contract') { + throw new Error('Not a smart contract block'); + } + const { name, source } = block.data; + const proc = deserializeFunction(source); + const contract = new SmartContract(name, proc, { storage, chain }); + return contract; + } +} + +// fsi-browser.js + +const resolveDir = async (dirPath) => { + const root = await navigator.storage.getDirectory(); + const parts = dirPath.split('/').filter(Boolean); + let dir = root; + for (const part of parts) { + dir = await dir.getDirectoryHandle(part, { create: true }); + } + return dir; +}; + +const read = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + const fileHandle = await dirHandle.getFileHandle(filename); + const file = await fileHandle.getFile(); + return file.text(); +}; + +const write = async (dir, filename, content) => { + const dirHandle = await resolveDir(dir); + const fileHandle = await dirHandle.getFileHandle(filename, { create: true }); + const writable = await fileHandle.createWritable(); + await writable.write(content); + await writable.close(); +}; + +const exists = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + try { + await dirHandle.getFileHandle(filename); + return true; + } catch { + return false; + } +}; + +const remove = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + try { + await dirHandle.removeEntry(filename); + } catch { + // File may not exist + } +}; + +// crypto-browser.js + +const PUBLIC_KEY_HEADER = '-----BEGIN PUBLIC KEY-----'; +const PUBLIC_KEY_FOOTER = '-----END PUBLIC KEY-----'; +const PRIVATE_KEY_HEADER = '-----BEGIN PRIVATE KEY-----'; +const PRIVATE_KEY_FOOTER = '-----END PRIVATE KEY-----'; + +const ALG = { name: 'RSA-OAEP', hash: 'SHA-256' }; + +const calculateHash = async (data) => { + const block = JSON.stringify(data); + const dataBuffer = new TextEncoder().encode(block); + const hashBuffer = await crypto.subtle.digest('SHA-256', dataBuffer); + const hashArray = Array.from(new Uint8Array(hashBuffer)); + return hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); +}; + +const toBase64UrlSafe = (buffer) => { + const base64 = btoa(String.fromCharCode(...new Uint8Array(buffer))); + return base64.replaceAll('+', '-').replaceAll('/', '_').replaceAll('=', ''); +}; + +const base64ToBytes = (base64) => { + const normalized = base64.replaceAll('-', '+').replaceAll('_', '/'); + const binary = atob(normalized); + const bytes = new Uint8Array(binary.length); + for (let i = 0; i < binary.length; i++) { + bytes[i] = binary.charCodeAt(i); + } + return bytes; +}; + +const cleanPemKey = (pemKey, header, footer) => { + const s = pemKey.replace(header, '').replace(footer, ''); + return s.replaceAll(/\s/g, ''); +}; + +const generateKeys = async () => { + const keyPair = await crypto.subtle.generateKey( + { ...ALG, modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]) }, + true, + ['encrypt', 'decrypt'], + ); + const pubBuf = await crypto.subtle.exportKey('spki', keyPair.publicKey); + const privBuf = await crypto.subtle.exportKey('pkcs8', keyPair.privateKey); + const pub = toBase64UrlSafe(pubBuf); + const priv = toBase64UrlSafe(privBuf); + return { + publicKey: [PUBLIC_KEY_HEADER, pub, PUBLIC_KEY_FOOTER].join('\n'), + privateKey: [PRIVATE_KEY_HEADER, priv, PRIVATE_KEY_FOOTER].join('\n'), + }; +}; + +const importPublicKey = async (pemKey) => { + const base64 = cleanPemKey(pemKey, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER); + const bytes = base64ToBytes(base64); + return crypto.subtle.importKey('spki', bytes, ALG, false, ['encrypt']); +}; + +const importPrivateKey = async (pemKey) => { + const base64 = cleanPemKey(pemKey, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER); + const bytes = base64ToBytes(base64); + return crypto.subtle.importKey('pkcs8', bytes, ALG, false, ['decrypt']); +}; + +const encrypt = async (data, publicKey) => { + const key = await importPublicKey(publicKey); + const dataBuffer = new TextEncoder().encode(JSON.stringify(data)); + const encryptedBuffer = await crypto.subtle.encrypt(ALG, key, dataBuffer); + return toBase64UrlSafe(encryptedBuffer); +}; + +const decrypt = async (data, privateKey) => { + const key = await importPrivateKey(privateKey); + const bytes = base64ToBytes(data); + const buffer = await crypto.subtle.decrypt(ALG, key, bytes); + return JSON.parse(new TextDecoder().decode(buffer)); +}; + +const loadKeys = async (basePath = 'keys') => { + let publicKey, privateKey; + if (await fsi.exists(basePath, 'public.pem')) { + publicKey = await fsi.read(basePath, 'public.pem'); + privateKey = await fsi.read(basePath, 'private.pem'); + } else { + const keys = await generateKeys(); + ({ publicKey, privateKey } = keys); + await fsi.write(basePath, 'public.pem', publicKey); + await fsi.write(basePath, 'private.pem', privateKey); + } + return { publicKey, privateKey }; +}; + +// sync-browser.js + +class SyncManager { + #nodes = new Set(); + + constructor() { + return this.#init(); + } + + async #init() { + await this.start(); + return this; + } + + async start() { + // TODO: Implement metacom-based sync (WebSocket client) + // Future: WebRTC transport for peer-to-peer sync + const uplinks = Array.from(this.#nodes).filter( + (node) => node.type === 'uplink' && node.enabled, + ); + if (uplinks.length === 0) return; + for (const uplink of uplinks) { + try { + // Sync here + } catch (error) { + console.error(`Sync failed with ${uplink.url}:`, error.message); + } + } + } + + async addNode(nodeData) { + const { url, type, token } = nodeData; + if (!url) throw new Error('Node URL is required'); + if (type !== 'uplink' && type !== 'downlink') { + throw new Error('Node type must be uplink or downlink'); + } + const existing = Array.from(this.#nodes).find((n) => n.url === url); + if (existing) throw new Error(`Node with URL ${url} already exists`); + const now = Date.now(); + const node = { type, url, token, enabled: true, lastSync: now }; + this.#nodes.add(node); + await this.start(); + } + + async removeNode(url) { + const node = Array.from(this.#nodes).find((n) => n.url === url); + if (!node) throw new Error(`Node with URL ${url} not found`); + this.#nodes.delete(node); + } + + listNodes() { + return Array.from(this.#nodes); + } +} + +// chain.js + +const BLOCKCHAIN = '.blockchain.json'; + +class Blockchain { + constructor(basePath) { + this.path = basePath; + this.tailHash = null; + return this.#init(); + } + + async #init() { + const exists = await fsi.exists(this.path, BLOCKCHAIN); + if (!exists) { + const block = { prev: '0', timestamp: Date.now(), data: {} }; + this.tailHash = await this.writeBlock(block); + await this.writeChain(); + } else { + const raw = await fsi.read(this.path, BLOCKCHAIN); + const chain = JSON.parse(raw); + this.tailHash = chain.tailHash; + } + return this; + } + + async writeChain() { + const chain = { tailHash: this.tailHash }; + await fsi.write(this.path, BLOCKCHAIN, JSON.stringify(chain)); + } + + async addBlock(data) { + const prev = this.tailHash; + const timestamp = Date.now(); + const block = { prev, timestamp, data }; + const hash = await this.writeBlock(block); + this.tailHash = hash; + await this.writeChain(); + return hash; + } + + async writeBlock(block) { + const hash = await calculateHash(block); + await fsi.write(this.path, `${hash}.json`, JSON.stringify(block)); + return hash; + } + + async readBlock(hash) { + const exists = await fsi.exists(this.path, `${hash}.json`); + if (!exists) throw new Error(`Block not found: ${hash}`); + const raw = await fsi.read(this.path, `${hash}.json`); + return JSON.parse(raw); + } + + async validate({ last, from } = {}) { + let currentHash = from || this.tailHash; + if (!currentHash) return false; + let count = 0; + while (currentHash) { + const block = await this.readBlock(currentHash); + const expectedHash = await calculateHash(block); + const valid = currentHash === expectedHash; + if (!valid) return false; + count++; + if (last && count >= last) break; + if (block.prev === '0') break; + currentHash = block.prev; + } + return true; + } +} + +// storage.js + +const calculateDelta = (oldData, newData) => { + const delta = {}; + for (const key in newData) { + if (oldData[key] !== newData[key]) { + delta[key] = newData[key]; + } + } + return delta; +}; + +class Reference { + #id; + #storage = null; + + constructor(id, storage) { + this.#id = id; + this.#storage = storage; + } + + async record() { + return await this.#storage.record(this.#id); + } +} + +class Record extends Emitter { + #id; + #storage = null; + #data = null; + #changes = {}; + + constructor(id, storage) { + super(); + this.#id = id; + this.#storage = storage; + return this.#load(); + } + + async #load() { + if (this.#data !== null) return this; + const rawData = await this.#storage.get(this.#id); + if (rawData) { + this.#data = JSON.parse(JSON.stringify(rawData)); + this.#convertToReferences(this.#data); + this.#defineProperties(); + } + return this; + } + + #convertToReferences(data) { + if (!data || typeof data !== 'object') return; + for (const key in data) { + const value = data[key]; + if (typeof value === 'string') { + if (this.#storage.hasRecord(value)) { + data[key] = new Reference(value, this.#storage); + } + } else if (value && typeof value === 'object') { + const isReference = value instanceof Reference; + if (!isReference) { + this.#convertToReferences(value); + } + } + } + } + + get id() { + return this.#id; + } + + #defineProperties() { + for (const key in this.#data) { + if (key === 'id') continue; + if (Object.prototype.hasOwnProperty.call(this, key)) continue; + Object.defineProperty(this, key, { + get() { + const value = this.#data?.[key]; + if (value instanceof Reference) { + return value; + } + return value; + }, + set(newValue) { + const oldValue = this.#data?.[key]; + if (oldValue === newValue) return; + this.#data[key] = newValue; + this.#changes[key] = newValue; + const delta = { [key]: newValue }; + this.emit('update', this.#data, delta); + }, + enumerable: true, + configurable: true, + }); + } + } + + async data() { + return this.#data; + } + + delta() { + return { ...this.#changes }; + } + + async save() { + if (Object.keys(this.#changes).length === 0) return; + const delta = this.delta(); + const data = this.#data; + await this.#storage.saveData(this.#id, data); + this.#changes = {}; + this.#storage.addUpdate({ + type: 'delta', + id: this.#id, + timestamp: Date.now(), + data: delta, + }); + } + + async delete() { + await this.#storage.delete(this.#id); + this.#data = null; + this.#changes = {}; + } +} + +class Collection { + #storage; + #name; + + constructor(storage, name) { + this.#storage = storage; + this.#name = name; + } + + get name() { + return this.#name; + } + + async insert(data) { + const id = generateUUID(); + const record = { ...data, $type: this.#name }; + await this.#storage.saveData(id, record); + return this.#storage.record(id); + } + + async get(id) { + return this.#storage.get(id); + } + + async delete(id) { + return this.#storage.delete(id); + } + + async update(id, delta) { + return this.#storage.update(id, delta); + } + + async record(id) { + return this.#storage.record(id); + } +} + +class Storage { + #path; + #keys = null; + #chain = null; + #records = null; + #sync = null; + #updates = []; + #cache = new Map(); + #schema = null; + + constructor(options = {}) { + this.#path = options.path || options.name || 'gs'; + this.#records = new Map(); + if (options.schema) this.#schema = options.schema; + return this.#init(); + } + + async #init() { + this.#keys = await loadKeys(`${this.#path}/keys`); + this.#chain = await new Blockchain(`${this.#path}/blockchain`); + this.#sync = await new SyncManager(this.#path, this, this.#chain); + this.#initCollections(); + return this; + } + + #initCollections() { + const entities = this.#schema?.entities; + if (!entities) return; + for (const [name] of entities.entries?.() ?? Object.entries(entities)) { + this[name] = new Collection(this, name); + } + } + + get schema() { + return this.#schema; + } + + #emitUpdate(id, data, delta) { + const emitter = this.#records.get(id); + if (emitter) { + emitter.emit('update', data, delta); + } + } + + addUpdate(update) { + this.#updates.push(update); + } + + async saveData(id, data, options = {}) { + const { encrypted = false } = options; + const record = encrypted ? encrypt(data, this.#keys.publicKey) : data; + const timestamp = Date.now(); + + const hash = await calculateHash(record); + const blockHash = await this.#chain.addBlock({ id, hash }); + + const entry = { data: record, encrypted, timestamp, block: blockHash }; + await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); + this.#cache.set(id, data); + + this.#updates.push({ type: 'write', id, timestamp, data }); + } + + async loadData(id) { + if (this.#cache.has(id)) return this.#cache.get(id); + const dataDir = `${this.#path}/data`; + const exists = await fsi.exists(dataDir, `${id}.json`); + if (!exists) { + this.#cache.set(id, null); + return null; + } + const raw = await fsi.read(dataDir, `${id}.json`); + const { data, encrypted, block } = JSON.parse(raw); + const isValid = await this.validate(id, data, block); + if (!isValid) throw new Error(`Storage record ${id} is invalid`); + const decrypted = encrypted ? decrypt(data, this.#keys.privateKey) : data; + this.#cache.set(id, decrypted); + return decrypted; + } + + async validate(idOrOptions, data, blockHash) { + const isChainValidation = + !data && !blockHash && ('last' in idOrOptions || 'from' in idOrOptions); + if (isChainValidation) { + return await this.#chain.validate(idOrOptions); + } + const id = idOrOptions; + try { + const block = await this.#chain.readBlock(blockHash); + const expectedHash = await calculateHash(data); + if (!block) return false; + return block.data.hash === expectedHash && block.data.id === id; + } catch { + return false; + } + } + + async insert(data) { + const id = generateUUID(); + await this.saveData(id, data); + return id; + } + + async has(id) { + return fsi.exists(`${this.#path}/data`, `${id}.json`); + } + + hasRecord(id) { + return this.#cache.has(id) || this.#records.has(id); + } + + async get(id) { + return await this.loadData(id); + } + + async set(id, data) { + const oldData = await this.get(id); + await this.saveData(id, data); + if (oldData) { + const delta = calculateDelta(oldData, data); + this.#emitUpdate(id, data, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + } + + async delete(id) { + await fsi.remove(`${this.#path}/data`, `${id}.json`); + this.#records.delete(id); + this.#cache.delete(id); + this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); + } + + async update(id, delta) { + const data = await this.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const updated = { ...data, ...delta }; + await this.saveData(id, updated); + this.#emitUpdate(id, updated, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + + async swap(id, changes, prev) { + const data = await this.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const matches = Object.keys(prev).every((key) => data[key] === prev[key]); + if (!matches) return false; + const updated = { ...data, ...changes }; + await this.saveData(id, updated); + this.#emitUpdate(id, updated, changes); + return true; + } + + async record(id) { + if (!this.#records.has(id)) { + const record = await new Record(id, this); + this.#records.set(id, record); + } + return this.#records.get(id); + } + + getCachedData(id) { + return this.#cache.has(id) ? this.#cache.get(id) : null; + } + + get sync() { + return this.#sync; + } +} + +Storage.Collection = Collection; +Storage.Record = Record; + +const open = (options) => new Storage(options); + +export { + diff, + merge, + apply, + DataReader, + SmartContract, + read, + write, + exists, + remove, + calculateHash, + generateKeys, + encrypt, + decrypt, + loadKeys, + SyncManager, + Blockchain, + Storage, + open, +}; diff --git a/emulation/data.js b/emulation/data.js new file mode 100644 index 0000000..f15f759 --- /dev/null +++ b/emulation/data.js @@ -0,0 +1,392 @@ +'use strict'; + +const { generateUUID } = require('metautil'); + +const createInitialData = () => { + const now = new Date().toISOString(); + const author1Id = generateUUID(); + const author2Id = generateUUID(); + const author3Id = generateUUID(); + const author4Id = generateUUID(); + const feed1Id = generateUUID(); + const feed2Id = generateUUID(); + const chat1Id = generateUUID(); + const chat2Id = generateUUID(); + const file1Id = generateUUID(); + const file2Id = generateUUID(); + const file3Id = generateUUID(); + const folder1Id = generateUUID(); + const folder2Id = generateUUID(); + const message1Id = generateUUID(); + const message2Id = generateUUID(); + const message3Id = generateUUID(); + const message4Id = generateUUID(); + const post1Id = generateUUID(); + const post2Id = generateUUID(); + const post3Id = generateUUID(); + const post4Id = generateUUID(); + const node1Id = generateUUID(); + const node2Id = generateUUID(); + const peer1Id = generateUUID(); + const peer2Id = generateUUID(); + + const data = {}; + + data[author1Id] = { + nick: 'marcus_aurelius', + name: 'Marcus Aurelius Antoninus', + bio: 'Roman Emperor, Stoic philosopher, author of Meditations', + icon: 'โš”๏ธ', + photo: file1Id, + status: 'online', + created: now, + updated: now, + settings: { theme: 'dark', notifications: true }, + nodes: [node1Id, node2Id], + }; + + data[author2Id] = { + nick: 'faustina', + name: 'Faustina the Younger', + bio: 'Empress of Rome, wife of Marcus Aurelius', + icon: '๐Ÿ‘‘', + photo: file2Id, + status: 'away', + created: now, + updated: now, + settings: { theme: 'light', notifications: true }, + nodes: [node1Id], + }; + + data[author3Id] = { + nick: 'lucius_verus', + name: 'Lucius Verus', + bio: 'Roman co-emperor, brother of Marcus Aurelius', + icon: '๐Ÿ›ก๏ธ', + photo: null, + status: 'online', + created: now, + updated: now, + settings: { theme: 'dark', notifications: false }, + nodes: [node2Id], + }; + + data[author4Id] = { + nick: 'commodus', + name: 'Commodus Antoninus', + bio: 'Roman Emperor, son of Marcus Aurelius', + icon: '๐Ÿ—ก๏ธ', + photo: file3Id, + status: 'dnd', + created: now, + updated: now, + settings: { theme: 'dark', notifications: true }, + nodes: [node1Id], + }; + + data[folder1Id] = { + name: 'Imperial Documents', + owner: author1Id, + icon: '๐Ÿ“œ', + description: 'Official imperial decrees and philosophical works', + created: now, + }; + + data[folder2Id] = { + name: 'Personal Correspondence', + owner: author2Id, + icon: '๐Ÿ“ฌ', + description: 'Private letters and messages', + created: now, + }; + + data[file1Id] = { + name: 'marcus_aurelius_portrait.jpg', + owner: author1Id, + folder: folder1Id, + icon: '๐Ÿ–ผ๏ธ', + description: 'Official imperial portrait', + mime: 'image/jpeg', + size: 245678, + checksum: + 'a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456', + created: now, + modified: now, + visibility: 'public', + }; + + data[file2Id] = { + name: 'faustina_bust.sculpture', + owner: author2Id, + folder: folder2Id, + icon: '๐Ÿ—ฟ', + description: 'Marble bust sculpture', + mime: 'image/sculpture', + size: 1024000, + checksum: + 'b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef12345678', + created: now, + modified: now, + visibility: 'unlisted', + }; + + data[file3Id] = { + name: 'meditations_book.pdf', + owner: author1Id, + folder: folder1Id, + icon: '๐Ÿ“–', + description: 'Meditations by Marcus Aurelius', + mime: 'application/pdf', + size: 512000, + checksum: + 'c3d4e5f6789012345678901234567890abcdef1234567890abcdef1234567890', + created: now, + modified: now, + visibility: 'public', + }; + + data[feed1Id] = { + name: 'Senatus Populusque Romanus', + owner: author1Id, + icon: '๐Ÿ›๏ธ', + description: 'Official announcements from the Senate and People of Rome', + created: now, + visibility: 'public', + joinPolicy: 'open', + pinned: [post1Id], + }; + + data[feed2Id] = { + name: 'Stoic Philosophy', + owner: author1Id, + icon: '๐Ÿ“š', + description: 'Discussions on Stoic philosophy and wisdom', + created: now, + visibility: 'public', + joinPolicy: 'open', + pinned: [post2Id], + }; + + data[post1Id] = { + title: 'On the Duty of an Emperor', + subtitle: 'Meditations Book I', + content: + 'From my grandfather Verus: I learned good morals and the government ' + + 'of my temper. From the reputation and remembrance of my father: ' + + 'modesty and a manly character. From my mother: piety and ' + + 'beneficence, and abstinence, not only from evil deeds, but even ' + + 'from evil thoughts; and further, simplicity in my way of living, ' + + 'far removed from the habits of the rich.', + feed: feed1Id, + author: author1Id, + created: now, + edited: null, + published: now, + deleted: null, + status: 'published', + reactions: { 'โš”๏ธ': [author2Id, author3Id], '๐Ÿ“š': [author4Id] }, + pinned: true, + attachments: [file3Id], + }; + + data[post2Id] = { + title: 'The Universe is Change', + subtitle: 'Meditations Book IV', + content: + 'The universe is transformation: life is opinion. If a man will take ' + + 'to heart that all things which happen have been from eternity, and ' + + 'in like manner will be to all eternity, he will more easily bear ' + + 'himself contentedly.', + feed: feed2Id, + author: author1Id, + created: now, + edited: null, + published: now, + deleted: null, + status: 'published', + reactions: { '๐Ÿ“š': [author2Id, author3Id], '๐Ÿ’ญ': [author4Id] }, + pinned: true, + attachments: [], + }; + + data[post3Id] = { + title: 'Preparations for the Parthian Campaign', + subtitle: 'Military Strategy', + content: + 'Lucius Verus and I shall lead the legions east. The Parthians have ' + + 'crossed our borders. We must act with haste but maintain discipline. ' + + 'The soldiers trust in our leadership.', + feed: feed1Id, + author: author1Id, + created: now, + edited: null, + published: now, + deleted: null, + status: 'published', + reactions: { 'โš”๏ธ': [author3Id], '๐Ÿ›ก๏ธ': [author2Id] }, + pinned: false, + attachments: [], + }; + + data[post4Id] = { + title: 'The Duties of an Empress', + subtitle: 'Personal Reflections', + content: + 'As empress, my role extends beyond the palace walls. I must support ' + + 'my husband in his duties while maintaining the dignity of the ' + + 'imperial household. The people look to us as examples.', + feed: feed2Id, + author: author2Id, + created: now, + edited: null, + published: null, + deleted: null, + status: 'draft', + reactions: {}, + pinned: false, + attachments: [file2Id], + }; + + data[chat1Id] = { + name: 'Imperial Council', + owner: author1Id, + icon: '๐Ÿ‘‘', + description: 'Private discussions of the imperial family and advisors', + created: now, + lastActivity: now, + members: [author1Id, author2Id, author3Id, author4Id], + admins: [author1Id], + moderators: [author2Id], + mute: [], + ban: [], + pinned: [message1Id], + }; + + data[chat2Id] = { + name: 'Military Command', + owner: author1Id, + icon: 'โš”๏ธ', + description: 'Strategic discussions and military planning', + created: now, + lastActivity: now, + members: [author1Id, author3Id], + admins: [author1Id], + moderators: [], + mute: [], + ban: [], + pinned: [message3Id], + }; + + data[message1Id] = { + chat: chat1Id, + author: author1Id, + content: + 'Greetings to all. We must discuss the upcoming campaigns and the ' + + 'administration of the provinces. Your counsel is valued.', + created: now, + edited: null, + deleted: null, + replyTo: null, + forwarded: null, + reactions: { '๐Ÿ‘': [author2Id, author3Id] }, + pinned: true, + attachments: [], + }; + + data[message2Id] = { + chat: chat1Id, + author: author2Id, + content: + 'The preparations for the victory celebrations are underway. The ' + + 'people expect great spectacle. We must not disappoint them.', + created: now, + edited: null, + deleted: null, + replyTo: message1Id, + forwarded: null, + reactions: { '๐Ÿ‘‘': [author1Id] }, + pinned: false, + attachments: [], + }; + + data[message3Id] = { + chat: chat2Id, + author: author3Id, + content: + 'The legions are ready. We have sufficient supplies for the march. ' + + 'When do we depart for the eastern provinces?', + created: now, + edited: null, + deleted: null, + replyTo: null, + forwarded: null, + reactions: { 'โš”๏ธ': [author1Id] }, + pinned: true, + attachments: [], + }; + + data[message4Id] = { + chat: chat2Id, + author: author1Id, + content: + 'We march at dawn in three days. Ensure the cavalry is prepared. ' + + 'The Parthians favor mounted combat - we must match their mobility.', + created: now, + edited: null, + deleted: null, + replyTo: message3Id, + forwarded: null, + reactions: { '๐Ÿ›ก๏ธ': [author3Id] }, + pinned: false, + attachments: [], + }; + + data[node1Id] = { + name: 'Roma Capital', + domain: 'roma.imperium', + ip: '192.168.1.100', + ports: [8001, 8002], + }; + + data[node2Id] = { + name: 'Antioch Province', + domain: 'antioch.imperium', + ip: '192.168.1.101', + ports: [8001], + }; + + data[peer1Id] = { + name: 'Athens Academy', + domain: 'athens.imperium', + ip: '192.168.1.102', + ports: [8001], + }; + + data[peer2Id] = { + name: 'Alexandria Library', + domain: 'alexandria.imperium', + ip: '192.168.1.103', + ports: [8001, 8002], + }; + + return data; +}; + +let cachedData = null; +let cachedIds = null; + +const getInitialDataWithIds = () => { + if (!cachedData) { + cachedData = createInitialData(); + const ids = Object.keys(cachedData); + cachedIds = { + authors: ids.filter((id) => cachedData[id].nick), + posts: ids.filter((id) => cachedData[id].title), + chats: ids.filter((id) => cachedData[id].members), + messages: ids.filter((id) => cachedData[id].chat), + }; + } + return { data: cachedData, ids: cachedIds }; +}; + +module.exports = { createInitialData, getInitialDataWithIds }; diff --git a/emulation/example.js b/emulation/example.js new file mode 100644 index 0000000..c876443 --- /dev/null +++ b/emulation/example.js @@ -0,0 +1,147 @@ +'use strict'; + +const globalStorage = require('./storage.js'); +const { getInitialDataWithIds } = require('./data.js'); + +const formatAuthor = (author) => + `${author.icon} ${author.name} (@${author.nick}) - ${author.bio}`; + +const formatMessage = (message, authors) => { + const author = authors.get(message.author); + const authorName = author ? author.name : 'Unknown'; + const reactions = Object.entries(message.reactions || {}) + .map(([emoji, authorIds]) => { + const names = authorIds + .map((id) => { + const a = authors.get(id); + return a ? a.name : 'Unknown'; + }) + .join(', '); + return `${emoji} (${names})`; + }) + .join(' | '); + const reactionsText = reactions ? `\n Reactions: ${reactions}` : ''; + return ` ${authorName}: ${message.content}${reactionsText}`; +}; + +const formatPost = (post, authors) => { + const author = authors.get(post.author); + const authorName = author ? author.name : 'Unknown'; + const status = post.status === 'published' ? 'โœ“' : 'โ—‹'; + const reactions = Object.entries(post.reactions || {}) + .map(([emoji, authorIds]) => { + const count = authorIds ? authorIds.length : 0; + return `${emoji} ${count}`; + }) + .join(' '); + const reactionsText = reactions ? ` [${reactions}]` : ''; + const preview = post.content.substring(0, 100); + const dots = post.content.length > 100 ? '...' : ''; + const byLine = `\n by ${authorName}\n ${preview}${dots}`; + return `[${status}] ${post.title}${reactionsText}${byLine}`; +}; + +const main = async () => { + const storage = await globalStorage.open(); + + const authors = new Map(); + const posts = []; + const messages = []; + const chats = new Map(); + + const { ids } = getInitialDataWithIds(); + + for (const authorId of ids.authors) { + const author = await storage.get(authorId); + if (author) authors.set(authorId, author); + } + + for (const postId of ids.posts) { + const post = await storage.get(postId); + if (post) posts.push(post); + } + + for (const chatId of ids.chats) { + const chat = await storage.get(chatId); + if (chat) chats.set(chatId, chat); + } + + for (const messageId of ids.messages) { + const message = await storage.get(messageId); + if (message) messages.push(message); + } + + console.log('\nIMPERIAL AUTHOR DIRECTORY\n'); + + authors.forEach((author) => { + console.log(formatAuthor(author)); + }); + + console.log('\nRECENT POSTS\n'); + + posts.forEach((post) => { + console.log(formatPost(post, authors)); + console.log(''); + }); + + console.log('\nMESSAGES\n'); + + chats.forEach((chat, chatId) => { + const desc = chat.description || ''; + console.log(` conversation: ${chat.name} (${desc})`); + const chatMessages = messages.filter((m) => m.chat === chatId); + chatMessages.forEach((message) => { + console.log(formatMessage(message, authors)); + }); + console.log(''); + }); + + console.log('\nADDING REACTIONS\n'); + + const { ids: initialIds } = getInitialDataWithIds(); + const firstPostId = initialIds.posts[0]; + const firstPost = await storage.get(firstPostId); + if (firstPost) { + const currentReactions = firstPost.reactions || {}; + const marcusId = initialIds.authors[0]; + if (!currentReactions['๐Ÿ‘']) { + currentReactions['๐Ÿ‘'] = []; + } + if (!currentReactions['๐Ÿ‘'].includes(marcusId)) { + currentReactions['๐Ÿ‘'].push(marcusId); + } + await storage.update(firstPostId, { reactions: currentReactions }); + console.log('โœ… Added ๐Ÿ‘ reaction to post:', firstPost.title); + } + + const firstMessageId = initialIds.messages[0]; + const firstMessage = await storage.get(firstMessageId); + if (firstMessage) { + const currentReactions = firstMessage.reactions || {}; + const faustinaId = initialIds.authors[1]; + if (!currentReactions['๐Ÿ’ฌ']) { + currentReactions['๐Ÿ’ฌ'] = []; + } + if (!currentReactions['๐Ÿ’ฌ'].includes(faustinaId)) { + currentReactions['๐Ÿ’ฌ'].push(faustinaId); + } + await storage.update(firstMessageId, { reactions: currentReactions }); + console.log('โœ… Added ๐Ÿ’ฌ reaction to message'); + } + + console.log('\nUPDATED DATA\n'); + + const updatedPost = await storage.get(firstPostId); + if (updatedPost) { + console.log(formatPost(updatedPost, authors)); + } + + const updatedMessage = await storage.get(firstMessageId); + if (updatedMessage) { + console.log('\n' + formatMessage(updatedMessage, authors)); + } + + console.log('\n'); +}; + +main().catch(console.error); diff --git a/emulation/storage.js b/emulation/storage.js new file mode 100644 index 0000000..c73c611 --- /dev/null +++ b/emulation/storage.js @@ -0,0 +1,360 @@ +'use strict'; + +const { Emitter, generateUUID } = require('metautil'); +const { getInitialDataWithIds } = require('./data.js'); + +const calculateDelta = (oldData, newData) => { + const delta = {}; + for (const key in newData) { + if (oldData[key] !== newData[key]) { + delta[key] = newData[key]; + } + } + return delta; +}; + +class Reference { + #id; + #storage = null; + + constructor(id, storage) { + this.#id = id; + this.#storage = storage; + } + + async record() { + return await this.#storage.record(this.#id); + } +} + +class Record extends Emitter { + #id; + #storage = null; + #data = null; + #changes = {}; + + constructor(id, storage) { + super(); + this.#id = id; + this.#storage = storage; + return this.#load(); + } + + async #load() { + if (this.#data !== null) return this; + const rawData = await this.#storage.get(this.#id); + if (rawData) { + this.#data = JSON.parse(JSON.stringify(rawData)); + this.#convertToReferences(this.#data); + this.#defineProperties(); + } + return this; + } + + #convertToReferences(data) { + if (!data || typeof data !== 'object') return; + for (const key in data) { + const value = data[key]; + if (typeof value === 'string') { + if (this.#storage.hasRecord(value)) { + data[key] = new Reference(value, this.#storage); + } + } else if (value && typeof value === 'object') { + const isReference = value instanceof Reference; + if (!isReference) { + this.#convertToReferences(value); + } + } + } + } + + get id() { + return this.#id; + } + + #defineProperties() { + for (const key in this.#data) { + if (key === 'id') continue; + if (Object.prototype.hasOwnProperty.call(this, key)) continue; + Object.defineProperty(this, key, { + get() { + const value = this.#data?.[key]; + if (value instanceof Reference) { + return value; + } + return value; + }, + set(newValue) { + const oldValue = this.#data?.[key]; + if (oldValue === newValue) return; + this.#data[key] = newValue; + this.#changes[key] = newValue; + const delta = { [key]: newValue }; + this.emit('update', this.#data, delta); + }, + enumerable: true, + configurable: true, + }); + } + } + + async data() { + return this.#data; + } + + delta() { + return { ...this.#changes }; + } + + async save() { + if (Object.keys(this.#changes).length === 0) return; + const delta = this.delta(); + const data = this.#data; + await this.#storage.saveData(this.#id, data); + this.#changes = {}; + this.#storage.addUpdate({ + type: 'delta', + id: this.#id, + timestamp: Date.now(), + data: delta, + }); + } + + async delete() { + await this.#storage.delete(this.#id); + this.#data = null; + this.#changes = {}; + } +} + +class SyncManager { + #nodes = new Set(); + + async addNode(nodeData) { + const { url, type, token } = nodeData; + if (!url) throw new Error('Node URL is required'); + if (type !== 'uplink' && type !== 'downlink') { + throw new Error('Node type must be uplink or downlink'); + } + const now = Date.now(); + const node = { type, url, token, enabled: true, lastSync: now }; + this.#nodes.add(node); + } + + async removeNode(url) { + const node = Array.from(this.#nodes).find((n) => n.url === url); + if (!node) throw new Error(`Node with URL ${url} not found`); + this.#nodes.delete(node); + } + + listNodes() { + return Array.from(this.#nodes); + } + + // eslint-disable-next-line class-methods-use-this + async start() { + return Promise.resolve(); + } +} + +class Collection { + #storage; + #name; + + constructor(storage, name) { + this.#storage = storage; + this.#name = name; + } + + get name() { + return this.#name; + } + + async insert(data) { + const id = generateUUID(); + const record = { ...data, $type: this.#name }; + await this.#storage.saveData(id, record); + return this.#storage.record(id); + } + + async get(id) { + return this.#storage.get(id); + } + + async delete(id) { + return this.#storage.delete(id); + } + + async update(id, delta) { + return this.#storage.update(id, delta); + } + + async record(id) { + return this.#storage.record(id); + } +} + +class Storage { + #data = new Map(); + #records = new Map(); + #sync = null; + #cache = new Map(); + #updates = []; + #schema = null; + + constructor(options = {}) { + if (options.schema) this.#schema = options.schema; + this.#sync = new SyncManager(); + return this.#init(); + } + + async #init() { + const { data } = getInitialDataWithIds(); + for (const [id, item] of Object.entries(data)) { + this.#data.set(id, item); + this.#cache.set(id, item); + } + this.#initCollections(); + return this; + } + + #initCollections() { + const entities = this.#schema?.entities; + if (!entities) return; + for (const [name] of entities.entries?.() ?? Object.entries(entities)) { + this[name] = new Collection(this, name); + } + } + + get schema() { + return this.#schema; + } + + #emitUpdate(id, data, delta) { + const emitter = this.#records.get(id); + if (emitter) { + emitter.emit('update', data, delta); + } + } + + // eslint-disable-next-line no-unused-vars + async saveData(id, data, options = {}) { + this.#data.set(id, { ...data }); + this.#cache.set(id, data); + this.#updates.push({ type: 'write', id, timestamp: Date.now(), data }); + } + + async insert(data) { + const id = generateUUID(); + await this.saveData(id, data); + return id; + } + + async has(id) { + return this.#data.has(id); + } + + async loadData(id) { + if (this.#cache.has(id)) { + return this.#cache.get(id); + } + const data = this.#data.get(id); + if (data) { + this.#cache.set(id, data); + return { ...data }; + } + this.#cache.set(id, null); + return null; + } + + async get(id) { + return this.loadData(id); + } + + getCachedData(id) { + return this.#cache.has(id) ? this.#cache.get(id) : null; + } + + hasRecord(id) { + return this.#cache.has(id) || this.#records.has(id); + } + + addUpdate(update) { + this.#updates.push(update); + } + + async set(id, data) { + const oldData = this.#data.get(id); + await this.saveData(id, data); + if (oldData) { + const delta = calculateDelta(oldData, data); + this.#emitUpdate(id, data, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + } + + async delete(id) { + this.#data.delete(id); + this.#records.delete(id); + this.#cache.delete(id); + this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); + } + + async update(id, delta) { + const data = this.#data.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const updated = { ...data, ...delta }; + await this.saveData(id, updated); + this.#emitUpdate(id, updated, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + + async swap(id, changes, prev) { + const data = this.#data.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const matches = Object.keys(prev).every((key) => data[key] === prev[key]); + if (!matches) return false; + const updated = { ...data, ...changes }; + this.#data.set(id, updated); + this.#emitUpdate(id, updated, changes); + return true; + } + + async record(id) { + if (!this.#records.has(id)) { + const record = await new Record(id, this); + this.#records.set(id, record); + } + return this.#records.get(id); + } + + // eslint-disable-next-line no-unused-vars + async validate(idOrOptions, _data, _blockHash) { + if (typeof idOrOptions === 'object' && idOrOptions !== null) { + return true; + } + const id = idOrOptions; + return this.#data.has(id); + } + + get sync() { + return this.#sync; + } +} + +Storage.Collection = Collection; +Storage.Record = Record; + +const open = async (options = {}) => new Storage(options); + +module.exports = { Storage, open, SyncManager }; diff --git a/example.js b/example.js new file mode 100644 index 0000000..ac571f5 --- /dev/null +++ b/example.js @@ -0,0 +1,91 @@ +'use strict'; + +const globalStorage = require('./globalstorage.js'); +const xxii = require('xxii-schema'); + +const main = async () => { + const xxiiSchema = await xxii.load(); + console.log(xxiiSchema); + const storage = await globalStorage.open({ schema: xxiiSchema }); + + await storage.sync + .addNode({ + type: 'uplink', + url: 'ws://example.com:8001', + name: 'Main Server', + token: 'secret-token-123', + }) + .then(() => { + console.log('Added uplink node'); + }) + .catch((error) => { + console.error('Error adding uplink node:', error.message); + }); + + await storage.sync + .addNode({ + type: 'downlink', + url: 'ws://client.example.com:8002', + name: 'Client Node', + token: 'secret-token-456', + }) + .then(() => { + console.log('Added downlink node'); + }) + .catch((error) => { + console.error('Error adding downlink node:', error.message); + }); + + const nodes = storage.sync.listNodes(); + console.log('List nodes', nodes); + + // Implicit API - Record with dynamic getters/setters + + const authorData = { name: 'Timur', email: 'timur@example.com' }; + const authorId = await storage.insert(authorData); + const authorRecord = await storage.record(authorId); + authorRecord.email = 'timur.shemsedinov@gmail.com'; + await authorRecord.save(); + + const post = { title: 'Example', content: 'Text', author: authorId }; + const postRecord = await storage.Post.insert(post); + + postRecord.on('update', (data, delta) => { + console.log('Record updated:', { id: postRecord.id, data, delta }); + }); + + console.log('Title:', postRecord.title); + console.log('Content:', postRecord.content); + console.log('Author:', postRecord.author); + + postRecord.content += 'example'; + postRecord.title = 'Updated Example'; + + const delta = postRecord.delta(); + console.log('Delta before save:', delta); + + await postRecord.save(); + + console.log('Author name:', postRecord.author.name); + + // Explicit API (still available) + + const exists = await storage.has(postRecord.id); + if (exists) { + const data = await storage.get(postRecord.id); + data.content += ' more changes'; + await storage.set(postRecord.id, data); + } + + await storage.sync.start(); + console.log('Synced'); + + const valid = await storage.validate({ last: 10 }); + console.log('Storage valid after adding:', valid); + + await storage.sync.removeNode('ws://client.example.com:8002'); + console.log('Removed downlink node'); + console.log('Remaining nodes:', storage.sync.listNodes()); +}; + +main(); diff --git a/examples/domain.js b/examples/domain.js new file mode 100644 index 0000000..fed1818 --- /dev/null +++ b/examples/domain.js @@ -0,0 +1,32 @@ +'use strict'; + +// Domain logic: message + +let storage = null; + +const init = (storageProvider) => { + storage = storageProvider; +}; + +const create = async ({ author, title, content, feed }) => { + const authorExists = await storage.has(author); + if (!authorExists) throw new Error('Author not found'); + + const feedExists = await storage.has(feed); + if (!feedExists) throw new Error('Feed not found'); + + const created = new Date().getTime(); + const status = 'draft'; + const post = { title, content, author, created, status }; + const postId = await storage.insert(post); + + return { postId }; +}; + +const publish = async ({ postId }) => { + const existing = await storage.get(postId); + if (!existing) throw new Error('Post not found'); + await storage.update(postId, { status: 'published' }); +}; + +module.exports = { init, create, publish }; diff --git a/examples/main.js b/examples/main.js new file mode 100644 index 0000000..25ebf24 --- /dev/null +++ b/examples/main.js @@ -0,0 +1,30 @@ +'use strict'; + +const globalStorage = require('../globalstorage.js'); +const domain = require('./domain.js'); + +const main = async () => { + const storage = await globalStorage.open(); + domain.init(storage); + + const authorId = await storage.insert({ + name: 'Example Author', + email: 'author@example.com', + }); + const feedId = await storage.insert({ title: 'Example feed' }); + + const title = 'Example'; + const content = 'Text'; + const { postId } = await domain.create({ + author: authorId, + title, + content, + feed: feedId, + }); + console.log('Post created:', postId); + + await domain.publish({ postId }); + console.log('Post published'); +}; + +main(); diff --git a/index.d.ts b/index.d.ts new file mode 100644 index 0000000..f06bb9f --- /dev/null +++ b/index.d.ts @@ -0,0 +1,239 @@ +export type JSONSafe = + | string + | number + | boolean + | null + | { readonly [key: string]: JSONSafe } + | readonly JSONSafe[]; + +export interface KeyPair { + readonly publicKey: string; + readonly privateKey: string; +} + +export function generateKeys(): Promise; +export function encrypt(data: unknown, publicKey: string): string; +export function decrypt(data: string, privateKey: string): unknown; +export function loadKeys(basePath: string): Promise; + +export type CrdtDelta = Record; + +export type CrdtHistoryRecord = + | { type: 'write'; id?: string; timestamp?: number; data?: unknown } + | { type: 'delta'; data: unknown } + | { type: 'delete'; key: string } + | { type: 'inc'; key: string; value: number } + | { type: 'dec'; key: string; value: number }; + +export function diff(source: unknown, target: unknown): CrdtDelta; +export function merge(source: unknown, delta: unknown): unknown; +export function apply( + source: unknown, + history: readonly CrdtHistoryRecord[], +): unknown; + +export interface Block { + prev: string; + timestamp: number; + data: T; +} + +export interface ChainDataEntry { + id: string; + hash: string; +} + +export function calculateHash(data: JSONSafe): string; + +export interface Blockchain { + readonly path: string; + tailHash: string; + + writeChain(): Promise; + addBlock(data: unknown): Promise; + writeBlock(block: Block): Promise; + readBlock(hash: string): Promise; + validate(options?: { last?: number; from?: string }): Promise; +} + +export interface BlockchainConstructor { + readonly prototype: Blockchain; + new (basePath: string): Promise; +} + +export const Blockchain: BlockchainConstructor; + +export interface ContractContext { + storage: Storage; + chain: Blockchain; +} + +export interface ContractBlock { + type: 'smart contract'; + name: string; + source: string; +} + +export interface ErrorBlock { + contract: string; + args: unknown; + error: string; + timestamp: number; +} + +export type ContractExecuteArgs = { id: string } & Record; + +export type ContractProcedure = ( + reader: DataReader, + args: ContractExecuteArgs, +) => unknown | Promise; + +export class DataReader { + constructor(storage: Storage); + get(id: string): Promise; +} + +export class SmartContract { + constructor(name: string, proc: ContractProcedure, context: ContractContext); + readonly name: string; + + execute(args: ContractExecuteArgs): Promise; + + static save( + name: string, + chain: Blockchain, + proc: ContractProcedure, + ): Promise; + static load(hash: string, context: ContractContext): Promise; +} + +export type SchemaEntity = { readonly [fieldName: string]: unknown }; +export type SchemaEntities = + | { readonly [entityName: string]: SchemaEntity } + | Map; + +export interface Schema { + entities?: SchemaEntities; + readonly [key: string]: unknown; +} + +export interface StorageOptions { + path?: string; + name?: string; + schema?: Schema; +} + +export interface StorageDataOptions { + encrypted?: boolean; +} + +export interface StorageEntry { + data: unknown; + encrypted: boolean; + timestamp: number; + block: string; +} + +export interface Node { + type: 'uplink' | 'downlink'; + url: string; + token?: string; + enabled: boolean; + lastSync: number; +} + +export interface NodeData { + url: string; + type: 'uplink' | 'downlink'; + token?: string; +} + +export interface SyncManager { + addNode(nodeData: NodeData): Promise; + removeNode(url: string): Promise; + listNodes(): Node[]; + start(): Promise; +} + +export interface SyncManagerConstructor { + readonly prototype: SyncManager; + new ( + basePath?: string, + storage?: Storage, + chain?: Blockchain, + ): Promise; +} + +export const SyncManager: SyncManagerConstructor; + +export interface StorageUpdate { + type: string; + id: string; + timestamp: number; + data: unknown; +} + +export interface Storage { + readonly schema: Schema | null; + readonly sync: SyncManager; + + saveData( + id: string, + data: unknown, + options?: StorageDataOptions, + ): Promise; + loadData(id: string): Promise; + validate(options: { last?: number; from?: string }): Promise; + validate(id: string, data: unknown, blockHash: string): Promise; + insert(data: unknown): Promise; + has(id: string): Promise; + get(id: string): Promise; + set(id: string, data: unknown): Promise; + delete(id: string): Promise; + update(id: string, delta: unknown): Promise; + swap(id: string, changes: unknown, prev: unknown): Promise; + record(id: string): Promise; + getCachedData(id: string): unknown | null; + hasRecord(id: string): boolean; + addUpdate(update: StorageUpdate): void; +} + +export interface GsRecord { + readonly id: string; + on(event: 'update', listener: (data: unknown, delta: unknown) => void): void; + data(): Promise; + delta(): Record; + save(): Promise; + delete(): Promise; +} + +declare class GsCollection { + constructor(storage: Storage, name: string); + readonly name: string; + insert(data: unknown): Promise; + get(id: string): Promise; + delete(id: string): Promise; + update(id: string, delta: unknown): Promise; + record(id: string): Promise; +} + +declare const GsRecordCtor: { + readonly prototype: GsRecord; + new (id: string, storage: Storage): Promise; +}; + +export interface StorageConstructor { + readonly prototype: Storage; + new (options?: StorageOptions): Promise; + Collection: typeof GsCollection; + Record: typeof GsRecordCtor; +} + +export const Storage: StorageConstructor; + +export namespace Storage { + export type StorageRecord = GsRecord; + export type StorageCollection = GsCollection; +} + +export function open(options?: StorageOptions): Promise; diff --git a/index.js b/index.js index 220c3fa..0401c05 100644 --- a/index.js +++ b/index.js @@ -1 +1,9 @@ -module.exports = require('./src/index.js'); +'use strict'; + +const chain = require('./src/chain.js'); +const crypto = require('./src/crypto.js'); +const crdt = require('./src/crdt.js'); +const contract = require('./src/contract.js'); +const storage = require('./src/storage.js'); + +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...storage }; diff --git a/package.json b/package.json index 669d3f9..bb4a264 100644 --- a/package.json +++ b/package.json @@ -53,7 +53,14 @@ "replication", "conflict-resolution", "CRDT", - "event-sourcing" + "event-sourcing", + "database", + "globalstorage", + "localstorage", + "storage", + "global", + "warehouse", + "distributed" ], "author": "Alex Dolid ", "license": "MIT", diff --git a/src/chain.js b/src/chain.js new file mode 100644 index 0000000..bc6405c --- /dev/null +++ b/src/chain.js @@ -0,0 +1,75 @@ +'use strict'; + +const fsi = require('./fsi.js'); +const { calculateHash } = require('./crypto.js'); + +const BLOCKCHAIN = '.blockchain.json'; + +class Blockchain { + constructor(basePath) { + this.path = basePath; + this.tailHash = null; + return this.#init(); + } + + async #init() { + const exists = await fsi.exists(this.path, BLOCKCHAIN); + if (!exists) { + const block = { prev: '0', timestamp: Date.now(), data: {} }; + this.tailHash = await this.writeBlock(block); + await this.writeChain(); + } else { + const raw = await fsi.read(this.path, BLOCKCHAIN); + const chain = JSON.parse(raw); + this.tailHash = chain.tailHash; + } + return this; + } + + async writeChain() { + const chain = { tailHash: this.tailHash }; + await fsi.write(this.path, BLOCKCHAIN, JSON.stringify(chain)); + } + + async addBlock(data) { + const prev = this.tailHash; + const timestamp = Date.now(); + const block = { prev, timestamp, data }; + const hash = await this.writeBlock(block); + this.tailHash = hash; + await this.writeChain(); + return hash; + } + + async writeBlock(block) { + const hash = await calculateHash(block); + await fsi.write(this.path, `${hash}.json`, JSON.stringify(block)); + return hash; + } + + async readBlock(hash) { + const exists = await fsi.exists(this.path, `${hash}.json`); + if (!exists) throw new Error(`Block not found: ${hash}`); + const raw = await fsi.read(this.path, `${hash}.json`); + return JSON.parse(raw); + } + + async validate({ last, from } = {}) { + let currentHash = from || this.tailHash; + if (!currentHash) return false; + let count = 0; + while (currentHash) { + const block = await this.readBlock(currentHash); + const expectedHash = await calculateHash(block); + const valid = currentHash === expectedHash; + if (!valid) return false; + count++; + if (last && count >= last) break; + if (block.prev === '0') break; + currentHash = block.prev; + } + return true; + } +} + +module.exports = { Blockchain }; diff --git a/src/contract.js b/src/contract.js new file mode 100644 index 0000000..fde0b67 --- /dev/null +++ b/src/contract.js @@ -0,0 +1,75 @@ +'use strict'; + +const deserializeFunction = (source) => { + const arrowIndex = source.indexOf('=>'); + const header = source.slice(0, arrowIndex).trim(); + let body = source.slice(arrowIndex + 2).trim(); + const argsStart = header.indexOf('('); + const argsEnd = header.lastIndexOf(')'); + const args = header.slice(argsStart + 1, argsEnd); + if (body.startsWith('{')) body = body.slice(1, -1); + else body = `return ${body};`; + body = `return async (${args}) => {${body}}`; + return new Function('reader', 'args', body)(); +}; + +class DataReader { + #storage; + + constructor(storage) { + this.#storage = storage; + } + + async get(id) { + return this.#storage.loadData(id); + } +} + +class SmartContract { + #storage; + #chain; + #proc; + + constructor(name, proc, { storage, chain }) { + this.name = name; + this.#storage = storage; + this.#chain = chain; + this.#proc = proc; + } + + async execute(args) { + const reader = new DataReader(this.#storage); + try { + const result = await this.#proc(reader, args); + await this.#storage.saveData(args.id, result); + return result; + } catch (error) { + const contract = this.name; + const timestamp = Date.now(); + const block = { contract, args, error: error.message, timestamp }; + await this.#chain.addBlock(block); + throw error; + } + } + + static async save(name, chain, proc) { + const source = proc.toString(); + const block = { type: 'smart contract', name, source }; + const hash = await chain.addBlock(block); + return hash; + } + + static async load(hash, { storage, chain }) { + const block = await chain.readBlock(hash); + if (!block) throw new Error('Block not found'); + if (block.data.type !== 'smart contract') { + throw new Error('Not a smart contract block'); + } + const { name, source } = block.data; + const proc = deserializeFunction(source); + const contract = new SmartContract(name, proc, { storage, chain }); + return contract; + } +} + +module.exports = { DataReader, SmartContract }; diff --git a/src/crdt.js b/src/crdt.js new file mode 100644 index 0000000..11f8f06 --- /dev/null +++ b/src/crdt.js @@ -0,0 +1,103 @@ +'use strict'; + +const diff = (source, target) => { + if (typeof source !== 'object') source = {}; + if (typeof target !== 'object') target = {}; + const delta = {}; + const sourceKeys = new Set(Object.keys(source)); + const targetKeys = new Set(Object.keys(target)); + + for (const key of targetKeys) { + if (!sourceKeys.has(key)) { + delta[key] = target[key]; + } else if (Array.isArray(target[key]) || target[key] instanceof Set) { + const sourceArr = Array.isArray(source[key]) ? source[key] : source[key]; + const targetArr = Array.isArray(target[key]) ? target[key] : target[key]; + const sourceSet = new Set(sourceArr); + const targetSet = new Set(targetArr); + const diffSet = new Set([...targetSet].filter((x) => !sourceSet.has(x))); + if (diffSet.size > 0) { + delta[key] = Array.from(diffSet); + } + } else if (typeof target[key] === 'object' && target[key] !== null) { + const nestedDiff = diff(source[key], target[key]); + if (Object.keys(nestedDiff).length > 0) { + delta[key] = nestedDiff; + } + } else if (source[key] !== target[key]) { + delta[key] = target[key]; + } + } + + for (const key of sourceKeys) { + if (!targetKeys.has(key)) { + delta[key] = undefined; + } + } + + return delta; +}; + +const merge = (source, delta) => { + if (!delta || typeof delta !== 'object') return source; + if (source === null || typeof source !== 'object') return delta; + + const result = Array.isArray(source) ? [...source] : { ...source }; + + for (const key in delta) { + if (delta[key] === undefined) { + delete result[key]; + continue; + } + + if (Array.isArray(delta[key]) || delta[key] instanceof Set) { + const sourceSet = new Set( + Array.isArray(result[key]) ? result[key] : result[key] || [], + ); + const deltaSet = new Set( + Array.isArray(delta[key]) ? delta[key] : delta[key], + ); + result[key] = Array.from(new Set([...sourceSet, ...deltaSet])); + } else if (typeof delta[key] === 'object' && delta[key] !== null) { + if (typeof result[key] === 'object' && result[key] !== null) { + result[key] = merge(result[key], delta[key]); + } else { + result[key] = delta[key]; + } + } else if ( + typeof delta[key] === 'number' && + typeof result[key] === 'number' + ) { + result[key] += delta[key]; + } else { + result[key] = delta[key]; + } + } + + return result; +}; + +const apply = (source, history) => { + if (!Array.isArray(history)) return source; + let result = { ...source }; + + const handlers = { + write: (record, state) => ({ ...state }), + delta: (record, state) => merge(state, record.data), + delete: (record, state) => ({ ...state, [record.key]: undefined }), + inc: (record, state) => merge(state, { [record.key]: record.value }), + dec: (record, state) => merge(state, { [record.key]: -record.value }), + }; + + for (const record of history) { + if (!record || typeof record !== 'object' || !record.type) continue; + const handler = handlers[record.type]; + if (handler) { + result = handler(record, result); + } + } + + return result; +}; + +module.exports = { diff, merge, apply }; diff --git a/src/crypto-browser.js b/src/crypto-browser.js new file mode 100644 index 0000000..574588a --- /dev/null +++ b/src/crypto-browser.js @@ -0,0 +1,96 @@ +'use strict'; + +const fsi = require('./fsi-browser.js'); + +const PUBLIC_KEY_HEADER = '-----BEGIN PUBLIC KEY-----'; +const PUBLIC_KEY_FOOTER = '-----END PUBLIC KEY-----'; +const PRIVATE_KEY_HEADER = '-----BEGIN PRIVATE KEY-----'; +const PRIVATE_KEY_FOOTER = '-----END PRIVATE KEY-----'; + +const ALG = { name: 'RSA-OAEP', hash: 'SHA-256' }; + +const calculateHash = async (data) => { + const block = JSON.stringify(data); + const dataBuffer = new TextEncoder().encode(block); + const hashBuffer = await crypto.subtle.digest('SHA-256', dataBuffer); + const hashArray = Array.from(new Uint8Array(hashBuffer)); + return hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); +}; + +const toBase64UrlSafe = (buffer) => { + const base64 = btoa(String.fromCharCode(...new Uint8Array(buffer))); + return base64.replaceAll('+', '-').replaceAll('/', '_').replaceAll('=', ''); +}; + +const base64ToBytes = (base64) => { + const normalized = base64.replaceAll('-', '+').replaceAll('_', '/'); + const binary = atob(normalized); + const bytes = new Uint8Array(binary.length); + for (let i = 0; i < binary.length; i++) { + bytes[i] = binary.charCodeAt(i); + } + return bytes; +}; + +const cleanPemKey = (pemKey, header, footer) => { + const s = pemKey.replace(header, '').replace(footer, ''); + return s.replaceAll(/\s/g, ''); +}; + +const generateKeys = async () => { + const keyPair = await crypto.subtle.generateKey( + { ...ALG, modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]) }, + true, + ['encrypt', 'decrypt'], + ); + const pubBuf = await crypto.subtle.exportKey('spki', keyPair.publicKey); + const privBuf = await crypto.subtle.exportKey('pkcs8', keyPair.privateKey); + const pub = toBase64UrlSafe(pubBuf); + const priv = toBase64UrlSafe(privBuf); + return { + publicKey: [PUBLIC_KEY_HEADER, pub, PUBLIC_KEY_FOOTER].join('\n'), + privateKey: [PRIVATE_KEY_HEADER, priv, PRIVATE_KEY_FOOTER].join('\n'), + }; +}; + +const importPublicKey = async (pemKey) => { + const base64 = cleanPemKey(pemKey, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER); + const bytes = base64ToBytes(base64); + return crypto.subtle.importKey('spki', bytes, ALG, false, ['encrypt']); +}; + +const importPrivateKey = async (pemKey) => { + const base64 = cleanPemKey(pemKey, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER); + const bytes = base64ToBytes(base64); + return crypto.subtle.importKey('pkcs8', bytes, ALG, false, ['decrypt']); +}; + +const encrypt = async (data, publicKey) => { + const key = await importPublicKey(publicKey); + const dataBuffer = new TextEncoder().encode(JSON.stringify(data)); + const encryptedBuffer = await crypto.subtle.encrypt(ALG, key, dataBuffer); + return toBase64UrlSafe(encryptedBuffer); +}; + +const decrypt = async (data, privateKey) => { + const key = await importPrivateKey(privateKey); + const bytes = base64ToBytes(data); + const buffer = await crypto.subtle.decrypt(ALG, key, bytes); + return JSON.parse(new TextDecoder().decode(buffer)); +}; + +const loadKeys = async (basePath = 'keys') => { + let publicKey, privateKey; + if (await fsi.exists(basePath, 'public.pem')) { + publicKey = await fsi.read(basePath, 'public.pem'); + privateKey = await fsi.read(basePath, 'private.pem'); + } else { + const keys = await generateKeys(); + ({ publicKey, privateKey } = keys); + await fsi.write(basePath, 'public.pem', publicKey); + await fsi.write(basePath, 'private.pem', privateKey); + } + return { publicKey, privateKey }; +}; + +module.exports = { calculateHash, generateKeys, encrypt, decrypt, loadKeys }; diff --git a/src/crypto.js b/src/crypto.js new file mode 100644 index 0000000..78df68e --- /dev/null +++ b/src/crypto.js @@ -0,0 +1,51 @@ +'use strict'; + +const crypto = require('node:crypto'); +const fsi = require('./fsi.js'); + +const calculateHash = (data) => { + const block = JSON.stringify(data); + return crypto.createHash('sha256').update(block).digest('hex'); +}; + +const generateKeys = () => + new Promise((resolve, reject) => { + const format = 'pem'; + const options = { + modulusLength: 2048, + publicKeyEncoding: { type: 'spki', format }, + privateKeyEncoding: { type: 'pkcs8', format }, + }; + crypto.generateKeyPair('rsa', options, (error, publicKey, privateKey) => { + if (error) return void reject(error); + resolve({ publicKey, privateKey }); + }); + }); + +const encrypt = (data, publicKey) => { + const buffer = Buffer.from(JSON.stringify(data)); + return crypto.publicEncrypt(publicKey, buffer).toString('base64'); +}; + +const decrypt = (data, privateKey) => { + const buffer = Buffer.from(data, 'base64'); + const str = crypto.privateDecrypt(privateKey, buffer).toString(); + return JSON.parse(str); +}; + +const loadKeys = async (basePath) => { + let publicKey, privateKey; + if (await fsi.exists(basePath, 'public.pem')) { + publicKey = await fsi.read(basePath, 'public.pem'); + privateKey = await fsi.read(basePath, 'private.pem'); + } else { + const keys = await generateKeys(); + await fsi.write(basePath, 'public.pem', keys.publicKey); + await fsi.write(basePath, 'private.pem', keys.privateKey); + publicKey = keys.publicKey; + privateKey = keys.privateKey; + } + return { publicKey, privateKey }; +}; + +module.exports = { calculateHash, generateKeys, encrypt, decrypt, loadKeys }; diff --git a/src/fsi-browser.js b/src/fsi-browser.js new file mode 100644 index 0000000..ac9858d --- /dev/null +++ b/src/fsi-browser.js @@ -0,0 +1,47 @@ +'use strict'; + +const resolveDir = async (dirPath) => { + const root = await navigator.storage.getDirectory(); + const parts = dirPath.split('/').filter(Boolean); + let dir = root; + for (const part of parts) { + dir = await dir.getDirectoryHandle(part, { create: true }); + } + return dir; +}; + +const read = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + const fileHandle = await dirHandle.getFileHandle(filename); + const file = await fileHandle.getFile(); + return file.text(); +}; + +const write = async (dir, filename, content) => { + const dirHandle = await resolveDir(dir); + const fileHandle = await dirHandle.getFileHandle(filename, { create: true }); + const writable = await fileHandle.createWritable(); + await writable.write(content); + await writable.close(); +}; + +const exists = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + try { + await dirHandle.getFileHandle(filename); + return true; + } catch { + return false; + } +}; + +const remove = async (dir, filename) => { + const dirHandle = await resolveDir(dir); + try { + await dirHandle.removeEntry(filename); + } catch { + // File may not exist + } +}; + +module.exports = { read, write, exists, remove }; diff --git a/src/fsi.js b/src/fsi.js new file mode 100644 index 0000000..5600986 --- /dev/null +++ b/src/fsi.js @@ -0,0 +1,21 @@ +'use strict'; + +const fs = require('node:fs/promises'); +const path = require('node:path'); +const { fileExists } = require('metautil'); + +const read = (dir, filename) => fs.readFile(path.join(dir, filename), 'utf8'); + +const write = async (dir, filename, content) => { + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, filename), content); +}; + +const exists = (dir, filename) => fileExists(path.join(dir, filename)); + +const remove = async (dir, filename) => { + const filePath = path.join(dir, filename); + if (await fileExists(filePath)) await fs.unlink(filePath); +}; + +module.exports = { read, write, exists, remove }; diff --git a/src/storage.js b/src/storage.js new file mode 100644 index 0000000..9e583b2 --- /dev/null +++ b/src/storage.js @@ -0,0 +1,354 @@ +'use strict'; + +const { Emitter, generateUUID } = require('metautil'); +const { encrypt, decrypt, loadKeys, calculateHash } = require('./crypto.js'); +const { Blockchain } = require('./chain.js'); +const { SyncManager } = require('./sync.js'); +const fsi = require('./fsi.js'); + +const calculateDelta = (oldData, newData) => { + const delta = {}; + for (const key in newData) { + if (oldData[key] !== newData[key]) { + delta[key] = newData[key]; + } + } + return delta; +}; + +class Reference { + #id; + #storage = null; + + constructor(id, storage) { + this.#id = id; + this.#storage = storage; + } + + async record() { + return await this.#storage.record(this.#id); + } +} + +class Record extends Emitter { + #id; + #storage = null; + #data = null; + #changes = {}; + + constructor(id, storage) { + super(); + this.#id = id; + this.#storage = storage; + return this.#load(); + } + + async #load() { + if (this.#data !== null) return this; + const rawData = await this.#storage.get(this.#id); + if (rawData) { + this.#data = JSON.parse(JSON.stringify(rawData)); + this.#convertToReferences(this.#data); + this.#defineProperties(); + } + return this; + } + + #convertToReferences(data) { + if (!data || typeof data !== 'object') return; + for (const key in data) { + const value = data[key]; + if (typeof value === 'string') { + if (this.#storage.hasRecord(value)) { + data[key] = new Reference(value, this.#storage); + } + } else if (value && typeof value === 'object') { + const isReference = value instanceof Reference; + if (!isReference) { + this.#convertToReferences(value); + } + } + } + } + + get id() { + return this.#id; + } + + #defineProperties() { + for (const key in this.#data) { + if (key === 'id') continue; + if (Object.prototype.hasOwnProperty.call(this, key)) continue; + Object.defineProperty(this, key, { + get() { + const value = this.#data?.[key]; + if (value instanceof Reference) { + return value; + } + return value; + }, + set(newValue) { + const oldValue = this.#data?.[key]; + if (oldValue === newValue) return; + this.#data[key] = newValue; + this.#changes[key] = newValue; + const delta = { [key]: newValue }; + this.emit('update', this.#data, delta); + }, + enumerable: true, + configurable: true, + }); + } + } + + async data() { + return this.#data; + } + + delta() { + return { ...this.#changes }; + } + + async save() { + if (Object.keys(this.#changes).length === 0) return; + const delta = this.delta(); + const data = this.#data; + await this.#storage.saveData(this.#id, data); + this.#changes = {}; + this.#storage.addUpdate({ + type: 'delta', + id: this.#id, + timestamp: Date.now(), + data: delta, + }); + } + + async delete() { + await this.#storage.delete(this.#id); + this.#data = null; + this.#changes = {}; + } +} + +class Collection { + #storage; + #name; + + constructor(storage, name) { + this.#storage = storage; + this.#name = name; + } + + get name() { + return this.#name; + } + + async insert(data) { + const id = generateUUID(); + const record = { ...data, $type: this.#name }; + await this.#storage.saveData(id, record); + return this.#storage.record(id); + } + + async get(id) { + return this.#storage.get(id); + } + + async delete(id) { + return this.#storage.delete(id); + } + + async update(id, delta) { + return this.#storage.update(id, delta); + } + + async record(id) { + return this.#storage.record(id); + } +} + +class Storage { + #path; + #keys = null; + #chain = null; + #records = null; + #sync = null; + #updates = []; + #cache = new Map(); + #schema = null; + + constructor(options = {}) { + this.#path = options.path || options.name || 'gs'; + this.#records = new Map(); + if (options.schema) this.#schema = options.schema; + return this.#init(); + } + + async #init() { + this.#keys = await loadKeys(`${this.#path}/keys`); + this.#chain = await new Blockchain(`${this.#path}/blockchain`); + this.#sync = await new SyncManager(this.#path, this, this.#chain); + this.#initCollections(); + return this; + } + + #initCollections() { + const entities = this.#schema?.entities; + if (!entities) return; + for (const [name] of entities.entries?.() ?? Object.entries(entities)) { + this[name] = new Collection(this, name); + } + } + + get schema() { + return this.#schema; + } + + #emitUpdate(id, data, delta) { + const emitter = this.#records.get(id); + if (emitter) { + emitter.emit('update', data, delta); + } + } + + addUpdate(update) { + this.#updates.push(update); + } + + async saveData(id, data, options = {}) { + const { encrypted = false } = options; + const record = encrypted ? encrypt(data, this.#keys.publicKey) : data; + const timestamp = Date.now(); + + const hash = await calculateHash(record); + const blockHash = await this.#chain.addBlock({ id, hash }); + + const entry = { data: record, encrypted, timestamp, block: blockHash }; + await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); + this.#cache.set(id, data); + + this.#updates.push({ type: 'write', id, timestamp, data }); + } + + async loadData(id) { + if (this.#cache.has(id)) return this.#cache.get(id); + const dataDir = `${this.#path}/data`; + const exists = await fsi.exists(dataDir, `${id}.json`); + if (!exists) { + this.#cache.set(id, null); + return null; + } + const raw = await fsi.read(dataDir, `${id}.json`); + const { data, encrypted, block } = JSON.parse(raw); + const isValid = await this.validate(id, data, block); + if (!isValid) throw new Error(`Storage record ${id} is invalid`); + const decrypted = encrypted ? decrypt(data, this.#keys.privateKey) : data; + this.#cache.set(id, decrypted); + return decrypted; + } + + async validate(idOrOptions, data, blockHash) { + const isChainValidation = + !data && !blockHash && ('last' in idOrOptions || 'from' in idOrOptions); + if (isChainValidation) { + return await this.#chain.validate(idOrOptions); + } + const id = idOrOptions; + try { + const block = await this.#chain.readBlock(blockHash); + const expectedHash = await calculateHash(data); + if (!block) return false; + return block.data.hash === expectedHash && block.data.id === id; + } catch { + return false; + } + } + + async insert(data) { + const id = generateUUID(); + await this.saveData(id, data); + return id; + } + + async has(id) { + return fsi.exists(`${this.#path}/data`, `${id}.json`); + } + + hasRecord(id) { + return this.#cache.has(id) || this.#records.has(id); + } + + async get(id) { + return await this.loadData(id); + } + + async set(id, data) { + const oldData = await this.get(id); + await this.saveData(id, data); + if (oldData) { + const delta = calculateDelta(oldData, data); + this.#emitUpdate(id, data, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + } + + async delete(id) { + await fsi.remove(`${this.#path}/data`, `${id}.json`); + this.#records.delete(id); + this.#cache.delete(id); + this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); + } + + async update(id, delta) { + const data = await this.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const updated = { ...data, ...delta }; + await this.saveData(id, updated); + this.#emitUpdate(id, updated, delta); + this.#updates.push({ + type: 'delta', + id, + timestamp: Date.now(), + data: delta, + }); + } + + async swap(id, changes, prev) { + const data = await this.get(id); + if (!data) throw new Error(`Record ${id} not found`); + const matches = Object.keys(prev).every((key) => data[key] === prev[key]); + if (!matches) return false; + const updated = { ...data, ...changes }; + await this.saveData(id, updated); + this.#emitUpdate(id, updated, changes); + return true; + } + + async record(id) { + if (!this.#records.has(id)) { + const record = await new Record(id, this); + this.#records.set(id, record); + } + return this.#records.get(id); + } + + getCachedData(id) { + return this.#cache.has(id) ? this.#cache.get(id) : null; + } + + get sync() { + return this.#sync; + } +} + +Storage.Collection = Collection; +Storage.Record = Record; + +const open = (options) => new Storage(options); + +module.exports = { Storage, open }; diff --git a/src/sync-browser.js b/src/sync-browser.js new file mode 100644 index 0000000..12cb086 --- /dev/null +++ b/src/sync-browser.js @@ -0,0 +1,56 @@ +'use strict'; + +class SyncManager { + #nodes = new Set(); + + constructor() { + return this.#init(); + } + + async #init() { + await this.start(); + return this; + } + + async start() { + // TODO: Implement metacom-based sync (WebSocket client) + // Future: WebRTC transport for peer-to-peer sync + const uplinks = Array.from(this.#nodes).filter( + (node) => node.type === 'uplink' && node.enabled, + ); + if (uplinks.length === 0) return; + for (const uplink of uplinks) { + try { + // Sync here + } catch (error) { + console.error(`Sync failed with ${uplink.url}:`, error.message); + } + } + } + + async addNode(nodeData) { + const { url, type, token } = nodeData; + if (!url) throw new Error('Node URL is required'); + if (type !== 'uplink' && type !== 'downlink') { + throw new Error('Node type must be uplink or downlink'); + } + const existing = Array.from(this.#nodes).find((n) => n.url === url); + if (existing) throw new Error(`Node with URL ${url} already exists`); + const now = Date.now(); + const node = { type, url, token, enabled: true, lastSync: now }; + this.#nodes.add(node); + await this.start(); + } + + async removeNode(url) { + const node = Array.from(this.#nodes).find((n) => n.url === url); + if (!node) throw new Error(`Node with URL ${url} not found`); + this.#nodes.delete(node); + } + + listNodes() { + return Array.from(this.#nodes); + } +} + +module.exports = { SyncManager }; diff --git a/src/sync.js b/src/sync.js new file mode 100644 index 0000000..a1c7724 --- /dev/null +++ b/src/sync.js @@ -0,0 +1,79 @@ +'use strict'; + +const fs = require('node:fs/promises'); +const path = require('node:path'); +const { between } = require('metautil'); + +class SyncManager { + #path; + #nodes = new Set(); + + constructor(basePath) { + this.#path = path.join(basePath, 'remote'); + return this.#init(); + } + + async #init() { + await fs.mkdir(this.#path, { recursive: true }); + const files = await fs.readdir(this.#path); + for (const file of files) { + if (!file.endsWith('.json')) continue; + const filePath = path.join(this.#path, file); + const raw = await fs.readFile(filePath, 'utf8'); + const node = JSON.parse(raw); + this.#nodes.add(node); + } + await this.start(); + return this; + } + + async start() { + // TODO: Implement metacom-based sync + // This would connect to the node using metacom protocol + // and exchange CRDT delta logs for sync + const uplinks = Array.from(this.#nodes).filter( + (node) => node.type === 'uplink' && node.enabled, + ); + if (uplinks.length === 0) return; + for (const uplink of uplinks) { + try { + // Sync here + } catch (error) { + console.error(`Sync failed with ${uplink.url}:`, error.message); + } + } + } + + async addNode(nodeData) { + const { url, type, token } = nodeData; + if (!url) throw new Error('Node URL is required'); + if (type !== 'uplink' && type !== 'downlink') { + throw new Error('Node type must be uplink or downlink'); + } + const existing = Array.from(this.#nodes).find((n) => n.url === url); + if (existing) throw new Error(`Node with URL ${url} already exists`); + const now = Date.now(); + const node = { type, url, token, enabled: true, lastSync: now }; + this.#nodes.add(node); + await fs.mkdir(this.#path, { recursive: true }); + const domain = between(url, '://', ':'); + const filePath = path.join(this.#path, `${domain}.json`); + await fs.writeFile(filePath, JSON.stringify(node)); + await this.start(); + } + + async removeNode(url) { + const node = Array.from(this.#nodes).find((n) => n.url === url); + if (!node) throw new Error(`Node with URL ${url} not found`); + this.#nodes.delete(node); + const domain = between(url, '://', ':'); + const filePath = path.join(this.#path, `${domain}.json`); + await fs.unlink(filePath); + } + + listNodes() { + return Array.from(this.#nodes); + } +} + +module.exports = { SyncManager }; diff --git a/tests/chain.js b/tests/chain.js new file mode 100644 index 0000000..dda10b6 --- /dev/null +++ b/tests/chain.js @@ -0,0 +1,91 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const path = require('node:path'); +const { fileExists } = require('metautil'); +const globalStorage = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +test('Chain module', async (t) => { + await t.test('calculateHash function', () => { + const data = { test: 'data' }; + const hash1 = globalStorage.calculateHash(data); + const hash2 = globalStorage.calculateHash(data); + + assert.strictEqual(typeof hash1, 'string'); + assert.strictEqual(hash1.length, 64); + assert.strictEqual(hash1, hash2); + }); + + await t.test('Blockchain constructor and initialization', async () => { + const tempDir = await createTemp(); + try { + const blockchain = await new globalStorage.Blockchain(tempDir); + + assert.strictEqual(blockchain.path, tempDir); + assert.strictEqual(typeof blockchain.tailHash, 'string'); + + const chainFile = path.join(tempDir, '.blockchain.json'); + assert.strictEqual(await fileExists(chainFile), true); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Blockchain addBlock and readBlock', async () => { + const tempDir = await createTemp(); + try { + const blockchain = await new globalStorage.Blockchain(tempDir); + const testData = { message: 'Test block data' }; + + const hash = await blockchain.addBlock(testData); + assert.strictEqual(typeof hash, 'string'); + assert.strictEqual(hash.length, 64); + + const block = await blockchain.readBlock(hash); + assert.notStrictEqual(block.prev, '0'); + assert.strictEqual(typeof block.timestamp, 'number'); + assert.deepStrictEqual(block.data, testData); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Blockchain chain validation', async () => { + const tempDir = await createTemp(); + try { + const blockchain = await new globalStorage.Blockchain(tempDir); + + await blockchain.addBlock({ data: 'block1' }); + await blockchain.addBlock({ data: 'block2' }); + await blockchain.addBlock({ data: 'block3' }); + + const isValid = await blockchain.validate(); + assert.strictEqual(isValid, true); + + const isValidLast = await blockchain.validate({ last: 2 }); + assert.strictEqual(isValidLast, true); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Blockchain writeBlock', async () => { + const tempDir = await createTemp(); + try { + const blockchain = await new globalStorage.Blockchain(tempDir); + const timestamp = Date.now(); + const data = { test: 'data' }; + const testBlock = { prev: '0', timestamp, data }; + + const hash = await blockchain.writeBlock(testBlock); + assert.strictEqual(typeof hash, 'string'); + + const blockFile = path.join(tempDir, `${hash}.json`); + assert.strictEqual(await fileExists(blockFile), true); + } finally { + await cleanupTemp(tempDir); + } + }); +}); diff --git a/tests/contract.js b/tests/contract.js new file mode 100644 index 0000000..af212c5 --- /dev/null +++ b/tests/contract.js @@ -0,0 +1,146 @@ +'use strict'; + +const path = require('node:path'); +const test = require('node:test'); +const assert = require('node:assert'); +const globalStorage = require('..'); +const { SmartContract, DataReader, Blockchain } = globalStorage; +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +test('Contract module', async (t) => { + await t.test('DataReader get reads storage', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const reader = new DataReader(storage); + + assert.strictEqual(await reader.get('missing'), null); + + await storage.saveData('doc-1', { x: 1 }); + assert.deepStrictEqual(await reader.get('doc-1'), { x: 1 }); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('SmartContract execute method', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'blockchain'); + const blockchain = await new globalStorage.Blockchain(chainPath); + const context = { storage, chain: blockchain }; + + const testProc = async (reader, args) => ({ + processed: args.input, + doubled: args.input * 2, + }); + + const contract = new SmartContract('test-contract', testProc, context); + assert.strictEqual(contract.name, 'test-contract'); + assert.ok(contract instanceof SmartContract); + + const args = { id: 'test-1', input: 42 }; + + const result = await contract.execute(args); + assert.deepStrictEqual(result, { processed: 42, doubled: 84 }); + + const savedData = await storage.loadData('test-1'); + assert.deepStrictEqual(savedData, { processed: 42, doubled: 84 }); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('SmartContract execute method with error', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'blockchain'); + const blockchain = await new globalStorage.Blockchain(chainPath); + const context = { storage, chain: blockchain }; + + const testProc = async () => { + throw new Error('Test error'); + }; + + const contract = new SmartContract('test-contract', testProc, context); + const args = { id: 'test-1', input: 42 }; + + await assert.rejects( + async () => await contract.execute(args), + /Test error/, + ); + + const blocks = []; + let currentHash = blockchain.tailHash; + while (currentHash && currentHash !== '0') { + const block = await blockchain.readBlock(currentHash); + blocks.push(block); + currentHash = block.prev; + } + + const errorBlock = blocks.find( + (block) => + block.data.contract === 'test-contract' && + block.data.error === 'Test error', + ); + assert.ok(errorBlock); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('SmartContract save and load static methods', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'blockchain'); + const blockchain = await new globalStorage.Blockchain(chainPath); + const context = { storage, chain: blockchain }; + + const testProc = async (reader, args) => ({ + result: args.input * 2, + }); + + const hash = await globalStorage.SmartContract.save( + 'test-contract', + blockchain, + testProc, + ); + assert.strictEqual(typeof hash, 'string'); + assert.strictEqual(hash.length, 64); + + const loadedContract = await SmartContract.load(hash, context); + assert.ok(loadedContract instanceof SmartContract); + assert.strictEqual(loadedContract.name, 'test-contract'); + + const result = await loadedContract.execute({ + id: 'test-1', + input: 21, + }); + assert.deepStrictEqual(result, { result: 42 }); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('SmartContract.load rejects non-contract block', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'extra-chain'); + const blockchain = await new Blockchain(chainPath); + const context = { storage, chain: blockchain }; + + const plainHash = await blockchain.addBlock({ kind: 'not-a-contract' }); + + await assert.rejects( + () => SmartContract.load(plainHash, context), + /Not a smart contract block/, + ); + } finally { + await cleanupTemp(tempDir); + } + }); +}); diff --git a/tests/crdt.js b/tests/crdt.js new file mode 100644 index 0000000..2d008b3 --- /dev/null +++ b/tests/crdt.js @@ -0,0 +1,207 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const globalStorage = require('..'); + +test('CRDT module', async (t) => { + await t.test('diff function - basic object diff', () => { + assert.deepStrictEqual(globalStorage.diff({}, {}), {}); + + const source = { a: 1, b: 2, c: 3 }; + const target = { a: 1, b: 20, d: 4 }; + const result = globalStorage.diff(source, target); + + assert.deepStrictEqual(result, { b: 20, c: undefined, d: 4 }); + }); + + await t.test('diff function - new keys added', () => { + const source = { a: 1 }; + const target = { a: 1, b: 2, c: 3 }; + const result = globalStorage.diff(source, target); + + assert.deepStrictEqual(result, { b: 2, c: 3 }); + }); + + await t.test('diff function - keys removed', () => { + const source = { a: 1, b: 2, c: 3 }; + const target = { a: 1 }; + const result = globalStorage.diff(source, target); + + assert.deepStrictEqual(result, { b: undefined, c: undefined }); + }); + + await t.test('diff function - arrays and sets', () => { + const arrDiff = globalStorage.diff( + { items: [1, 2, 3] }, + { items: [1, 2, 3, 4, 5] }, + ); + assert.deepStrictEqual(arrDiff, { items: [4, 5] }); + + const setDiff = globalStorage.diff( + { tags: new Set(['a', 'b']) }, + { tags: new Set(['a', 'b', 'c']) }, + ); + assert.deepStrictEqual(setDiff, { tags: ['c'] }); + }); + + await t.test('diff function - nested objects', () => { + const source = { user: { name: 'John', age: 30 } }; + const target = { user: { name: 'Jane', age: 30, city: 'NYC' } }; + const result = globalStorage.diff(source, target); + + assert.deepStrictEqual(result, { + user: { name: 'Jane', city: 'NYC' }, + }); + }); + + await t.test('merge function - basic merge', () => { + const source = { a: 1, b: 2 }; + const delta = { b: 20, c: 3 }; + const result = globalStorage.merge(source, delta); + + assert.deepStrictEqual(result, { a: 1, b: 22, c: 3 }); + }); + + await t.test('merge function - delete keys', () => { + const source = { a: 1, b: 2, c: 3 }; + const delta = { b: undefined }; + const result = globalStorage.merge(source, delta); + + assert.deepStrictEqual(result, { a: 1, c: 3 }); + }); + + await t.test('merge function - arrays/sets merge', () => { + const source = { items: [1, 2, 3] }; + const delta = { items: [3, 4, 5] }; + const result = globalStorage.merge(source, delta); + + assert.deepStrictEqual(result, { items: [1, 2, 3, 4, 5] }); + }); + + await t.test('merge function - counter merge', () => { + const source = { count: 10 }; + const delta = { count: 5 }; + const result = globalStorage.merge(source, delta); + + assert.strictEqual(result.count, 15); + }); + + await t.test('merge function - nested objects', () => { + const source = { user: { name: 'John', age: 30 } }; + const delta = { user: { age: 31, city: 'NYC' } }; + const result = globalStorage.merge(source, delta); + + assert.deepStrictEqual(result, { + user: { name: 'John', age: 61, city: 'NYC' }, + }); + }); + + await t.test('merge function - null/undefined delta', () => { + const source = { a: 1, b: 2 }; + const result1 = globalStorage.merge(source, null); + const result2 = globalStorage.merge(source, undefined); + + assert.deepStrictEqual(result1, source); + assert.deepStrictEqual(result2, source); + }); + + await t.test('merge function - null source', () => { + const delta = { a: 1, b: 2 }; + const result = globalStorage.merge(null, delta); + + assert.deepStrictEqual(result, delta); + }); + + await t.test('apply function - write record', () => { + const source = { a: 1, b: 2 }; + const history = [{ type: 'write', data: { c: 3, d: 4 } }]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 1, b: 2 }); + }); + + await t.test('apply function - delta record', () => { + const source = { a: 1, b: 2 }; + const history = [{ type: 'delta', data: { b: 20, c: 3 } }]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 1, b: 22, c: 3 }); + + const fromEmpty = globalStorage.apply({}, [ + { type: 'delta', data: { a: 1, b: 2 } }, + ]); + assert.deepStrictEqual(fromEmpty, { a: 1, b: 2 }); + }); + + await t.test('apply function - delete record', () => { + const source = { a: 1, b: 2, c: 3 }; + const history = [{ type: 'delete', key: 'b' }]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 1, b: undefined, c: 3 }); + }); + + await t.test('apply function - inc record', () => { + const source = { count: 10 }; + const history = [{ type: 'inc', key: 'count', value: 5 }]; + const result = globalStorage.apply(source, history); + + assert.strictEqual(result.count, 15); + }); + + await t.test('apply function - dec record', () => { + const source = { count: 10 }; + const history = [{ type: 'dec', key: 'count', value: 3 }]; + const result = globalStorage.apply(source, history); + + assert.strictEqual(result.count, 7); + }); + + await t.test('apply function - multiple records', () => { + const source = { a: 1, b: 2 }; + const history = [ + { type: 'delta', data: { b: 20 } }, + { type: 'delta', data: { c: 3 } }, + { type: 'inc', key: 'a', value: 5 }, + ]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 6, b: 22, c: 3 }); + }); + + await t.test('apply function - write preserves state', () => { + const source = { a: 1, b: 2, c: 3 }; + const history = [ + { type: 'delta', data: { d: 4 } }, + { type: 'write', data: { x: 10 } }, + ]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 1, b: 2, c: 3, d: 4 }); + }); + + await t.test('apply function - invalid history', () => { + const source = { a: 1 }; + const result1 = globalStorage.apply(source, null); + const result2 = globalStorage.apply(source, 'not-array'); + const result3 = globalStorage.apply(source, []); + + assert.deepStrictEqual(result1, source); + assert.deepStrictEqual(result2, source); + assert.deepStrictEqual(result3, source); + }); + + await t.test('apply function - invalid records', () => { + const source = { a: 1 }; + const history = [ + null, + { type: 'unknown' }, + { type: 'delta' }, + { type: 'delta', data: { b: 2 } }, + ]; + const result = globalStorage.apply(source, history); + + assert.deepStrictEqual(result, { a: 1, b: 2 }); + }); +}); diff --git a/tests/crypto-browser.js b/tests/crypto-browser.js new file mode 100644 index 0000000..b94110b --- /dev/null +++ b/tests/crypto-browser.js @@ -0,0 +1,38 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { webcrypto } = require('node:crypto'); +const { installMockOpfs } = require('./utils/mock-opfs.js'); +const { runCryptoKeysTests } = require('./suites/crypto-keys.js'); + +test('crypto-browser module', async (t) => { + const opfs = installMockOpfs(); + t.after(() => opfs.uninstall()); + + const prevCrypto = Object.getOwnPropertyDescriptor(globalThis, 'crypto'); + Object.defineProperty(globalThis, 'crypto', { + value: webcrypto, + configurable: true, + writable: true, + enumerable: true, + }); + t.after(() => { + if (prevCrypto) Object.defineProperty(globalThis, 'crypto', prevCrypto); + else delete globalThis.crypto; + }); + + if (typeof globalThis.btoa !== 'function') { + globalThis.btoa = (s) => Buffer.from(s, 'latin1').toString('base64'); + } + if (typeof globalThis.atob !== 'function') { + globalThis.atob = (s) => Buffer.from(s, 'base64').toString('latin1'); + } + + const cryptoBrowser = require('../lib/crypto-browser.js'); + + await runCryptoKeysTests(t, assert, cryptoBrowser, { + getKeysBase: async () => + `keys-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, + }); +}); diff --git a/tests/fsi-browser.js b/tests/fsi-browser.js new file mode 100644 index 0000000..026a84e --- /dev/null +++ b/tests/fsi-browser.js @@ -0,0 +1,18 @@ +'use strict'; + +const test = require('node:test'); +const { installMockOpfs } = require('./utils/mock-opfs.js'); +const { runFsiTests } = require('./suites/fsi.js'); + +test('fsi-browser module (OPFS mock)', async (t) => { + const opfs = installMockOpfs(); + t.after(() => opfs.uninstall()); + + const fsiBrowser = require('../lib/fsi-browser.js'); + + await runFsiTests(t, fsiBrowser, { + getRoot: async () => + `vfs-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, + joinPath: (root, ...parts) => [root, ...parts].join('/'), + }); +}); diff --git a/tests/fsi.js b/tests/fsi.js new file mode 100644 index 0000000..cdc07ae --- /dev/null +++ b/tests/fsi.js @@ -0,0 +1,13 @@ +'use strict'; + +const test = require('node:test'); +const fsi = require('../lib/fsi.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runFsiTests } = require('./suites/fsi.js'); + +test('fsi module', async (t) => { + await runFsiTests(t, fsi, { + getRoot: createTemp, + afterEach: cleanupTemp, + }); +}); diff --git a/tests/integration.js b/tests/integration.js new file mode 100644 index 0000000..d0c06dd --- /dev/null +++ b/tests/integration.js @@ -0,0 +1,51 @@ +'use strict'; + +const path = require('node:path'); +const test = require('node:test'); +const assert = require('node:assert'); +const globalStorage = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +test('Integration tests', async (t) => { + await t.test( + 'Full workflow: keys -> storage -> contract -> chain', + async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'blockchain'); + const blockchain = await new globalStorage.Blockchain(chainPath); + + const contractProc = async (reader, args) => { + const data = await reader.get(args.id); + const existingData = data || { count: 0 }; + return { + ...existingData, + count: existingData.count + 1, + lastUpdate: Date.now(), + }; + }; + + const contract = new globalStorage.SmartContract( + 'counter-contract', + contractProc, + { storage, chain: blockchain }, + ); + + const result1 = await contract.execute({ id: 'counter-1' }); + assert.strictEqual(result1.count, 1); + + const result2 = await contract.execute({ id: 'counter-1' }); + assert.strictEqual(result2.count, 2); + + const loadedData = await storage.loadData('counter-1'); + assert.strictEqual(loadedData.count, 2); + + const isValid = await blockchain.validate(); + assert.strictEqual(isValid, true); + } finally { + await cleanupTemp(tempDir); + } + }, + ); +}); diff --git a/tests/keys.js b/tests/keys.js new file mode 100644 index 0000000..30cc562 --- /dev/null +++ b/tests/keys.js @@ -0,0 +1,17 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const cryptoMod = require('../lib/crypto.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runCryptoKeysTests } = require('./suites/crypto-keys.js'); + +test('Keys module', async (t) => { + await runCryptoKeysTests(t, assert, cryptoMod, { + getKeysBase: async (t2) => { + const tempDir = await createTemp(); + t2.after(() => cleanupTemp(tempDir)); + return tempDir; + }, + }); +}); diff --git a/tests/storage.js b/tests/storage.js new file mode 100644 index 0000000..1c36222 --- /dev/null +++ b/tests/storage.js @@ -0,0 +1,384 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const path = require('node:path'); +const { directoryExists } = require('metautil'); +const globalStorage = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +const mockSchema = { + entities: { + Post: { title: 'string', content: 'string', author: 'Author' }, + Author: { name: 'string', email: 'string' }, + Comment: { text: 'string', post: 'Post', author: 'Author' }, + }, +}; + +test('Storage module', async (t) => { + await t.test('Storage constructor and initialization', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + assert.ok(storage instanceof globalStorage.Storage); + assert.strictEqual(await directoryExists(tempDir), true); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage saveData and loadData without encryption', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + const testData = { message: 'Hello, Storage!', number: 123 }; + await storage.saveData('test-1', testData); + + const loadedData = await storage.loadData('test-1'); + assert.deepStrictEqual(loadedData, testData); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage saveData and loadData with encryption', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + const testData = { secret: 'encrypted data', value: 456 }; + await storage.saveData('test-2', testData, { encrypted: true }); + + const loadedData = await storage.loadData('test-2'); + assert.deepStrictEqual(loadedData, testData); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage loadData with non-existent data', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + const result = await storage.loadData('nonexistent'); + assert.strictEqual(result, null); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage validate method', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + const testData = { message: 'Test validation' }; + await storage.saveData('test-3', testData); + + const filePath = path.join(tempDir, 'data', 'test-3.json'); + const raw = await fs.readFile(filePath, { encoding: 'utf8' }); + const entry = JSON.parse(raw); + + const isValid = await storage.validate('test-3', entry.data, entry.block); + assert.strictEqual(isValid, true); + + const isInvalid = await storage.validate( + 'test-3', + { modified: 'data' }, + entry.block, + ); + assert.strictEqual(isInvalid, false); + + const chainOk = await storage.validate({ last: 2 }); + assert.strictEqual(chainOk, true); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage loadData throws if tampered', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + await storage.saveData('tampered', { ok: true }); + + const filePath = path.join(tempDir, 'data', 'tampered.json'); + const entry = JSON.parse(await fs.readFile(filePath, 'utf8')); + entry.data = { ok: false }; + await fs.writeFile(filePath, JSON.stringify(entry), 'utf8'); + + const storage2 = await globalStorage.open({ path: tempDir }); + await assert.rejects(() => storage2.loadData('tampered'), /invalid/); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage insert set swap and update errors', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + const id = await storage.insert({ n: 1 }); + assert.strictEqual(typeof id, 'string'); + assert.deepStrictEqual(await storage.get(id), { n: 1 }); + + await storage.set(id, { n: 2, v: 0 }); + assert.deepStrictEqual(await storage.get(id), { n: 2, v: 0 }); + + const swapped = await storage.swap(id, { v: 1 }, { n: 2 }); + assert.strictEqual(swapped, true); + assert.deepStrictEqual(await storage.get(id), { n: 2, v: 1 }); + + const notSwapped = await storage.swap(id, { v: 9 }, { n: 999 }); + assert.strictEqual(notSwapped, false); + + await assert.rejects( + () => storage.update('missing-id', { x: 1 }), + /not found/, + ); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage with schema creates collections', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + assert.strictEqual(storage.schema, mockSchema); + assert.ok(storage.Post instanceof globalStorage.Storage.Collection); + assert.ok(storage.Author instanceof globalStorage.Storage.Collection); + assert.ok(storage.Comment instanceof globalStorage.Storage.Collection); + assert.strictEqual(storage.Post.name, 'Post'); + assert.strictEqual(storage.Author.name, 'Author'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage with Map-based schema entities', async () => { + const tempDir = await createTemp(); + try { + const mapSchema = { + entities: new Map([ + ['User', { name: 'string', email: 'string' }], + ['Article', { title: 'string', body: 'string' }], + ]), + }; + const options = { path: tempDir, schema: mapSchema }; + const storage = await globalStorage.open(options); + + assert.ok(storage.User instanceof globalStorage.Storage.Collection); + assert.ok(storage.Article instanceof globalStorage.Storage.Collection); + assert.strictEqual(storage.User.name, 'User'); + assert.strictEqual(storage.Article.name, 'Article'); + + const userData = { name: 'Test', email: 'a@b.com' }; + const user = await storage.User.insert(userData); + assert.ok(user.id); + assert.strictEqual(user.name, 'Test'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Storage without schema has no collections', async () => { + const tempDir = await createTemp(); + try { + const storage = await globalStorage.open({ path: tempDir }); + + assert.strictEqual(storage.schema, null); + assert.strictEqual(storage.Post, undefined); + assert.strictEqual(storage.Author, undefined); + } finally { + await cleanupTemp(tempDir); + } + }); +}); + +test('Collection module', async (t) => { + await t.test('Collection insert returns a Record', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const authorData = { name: 'John', email: 'john@example.com' }; + const authorRecord = await storage.Author.insert(authorData); + + assert.ok(authorRecord instanceof globalStorage.Storage.Record); + assert.ok(authorRecord.id); + assert.strictEqual(authorRecord.name, 'John'); + assert.strictEqual(authorRecord.email, 'john@example.com'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Collection insert adds $type to data', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const postData = { title: 'Hello', content: 'World' }; + const postRecord = await storage.Post.insert(postData); + + const rawData = await storage.get(postRecord.id); + assert.strictEqual(rawData.$type, 'Post'); + assert.strictEqual(rawData.title, 'Hello'); + assert.strictEqual(rawData.content, 'World'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Collection get retrieves data', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const authorRecord = await storage.Author.insert({ + name: 'Jane', + email: 'jane@example.com', + }); + + const data = await storage.Author.get(authorRecord.id); + assert.strictEqual(data.name, 'Jane'); + assert.strictEqual(data.email, 'jane@example.com'); + assert.strictEqual(data.$type, 'Author'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Collection update modifies data', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const postRecord = await storage.Post.insert({ + title: 'Original', + content: 'Content', + }); + + await storage.Post.update(postRecord.id, { title: 'Updated' }); + + const data = await storage.Post.get(postRecord.id); + assert.strictEqual(data.title, 'Updated'); + assert.strictEqual(data.content, 'Content'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Collection delete removes data', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const commentRecord = await storage.Comment.insert({ + text: 'Nice post!', + }); + const id = commentRecord.id; + + const existsBefore = await storage.has(id); + assert.strictEqual(existsBefore, true); + + await storage.Comment.delete(id); + + const existsAfter = await storage.has(id); + assert.strictEqual(existsAfter, false); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Collection record returns a Record instance', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const authorRecord = await storage.Author.insert({ + name: 'Bob', + email: 'bob@example.com', + }); + const id = authorRecord.id; + + const record = await storage.Author.record(id); + assert.ok(record instanceof globalStorage.Storage.Record); + assert.strictEqual(record.id, id); + assert.strictEqual(record.name, 'Bob'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Multiple collections work independently', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const author = await storage.Author.insert({ + name: 'Alice', + email: 'alice@example.com', + }); + + const post = await storage.Post.insert({ + title: 'My Post', + content: 'Post content', + author: author.id, + }); + + const comment = await storage.Comment.insert({ + text: 'Great post!', + post: post.id, + author: author.id, + }); + + const authorData = await storage.Author.get(author.id); + const postData = await storage.Post.get(post.id); + const commentData = await storage.Comment.get(comment.id); + + assert.strictEqual(authorData.$type, 'Author'); + assert.strictEqual(postData.$type, 'Post'); + assert.strictEqual(commentData.$type, 'Comment'); + assert.strictEqual(postData.author, author.id); + assert.strictEqual(commentData.post, post.id); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('Record save persists mutations', async () => { + const tempDir = await createTemp(); + try { + const options = { path: tempDir, schema: mockSchema }; + const storage = await globalStorage.open(options); + + const post = await storage.Post.insert({ + title: 'Draft', + content: 'Body', + }); + post.title = 'Published'; + await post.save(); + + const loaded = await storage.Post.get(post.id); + assert.strictEqual(loaded.title, 'Published'); + } finally { + await cleanupTemp(tempDir); + } + }); +}); diff --git a/tests/suites/crypto-keys.js b/tests/suites/crypto-keys.js new file mode 100644 index 0000000..a452723 --- /dev/null +++ b/tests/suites/crypto-keys.js @@ -0,0 +1,49 @@ +'use strict'; + +const runCryptoKeysTests = async ( + t, + assert, + cryptoMod, + { getKeysBase } = {}, +) => { + if (typeof getKeysBase !== 'function') { + throw new TypeError('runCryptoKeysTests requires getKeysBase'); + } + + await t.test('generateKeys function', async () => { + const keys = await Promise.resolve(cryptoMod.generateKeys()); + assert.strictEqual(typeof keys.publicKey, 'string'); + assert.strictEqual(typeof keys.privateKey, 'string'); + assert.ok(keys.publicKey.includes('-----BEGIN PUBLIC KEY-----')); + assert.ok(keys.privateKey.includes('-----BEGIN PRIVATE KEY-----')); + }); + + await t.test('encrypt and decrypt functions', async () => { + const keys = await Promise.resolve(cryptoMod.generateKeys()); + const testData = { message: 'Hello, World!', number: 42 }; + + const encrypted = await Promise.resolve( + cryptoMod.encrypt(testData, keys.publicKey), + ); + assert.strictEqual(typeof encrypted, 'string'); + assert.notStrictEqual(encrypted, JSON.stringify(testData)); + + const decrypted = await Promise.resolve( + cryptoMod.decrypt(encrypted, keys.privateKey), + ); + assert.deepStrictEqual(decrypted, testData); + }); + + await t.test('loadKeys function', async (t2) => { + const base = await getKeysBase(t2); + const keys1 = await Promise.resolve(cryptoMod.loadKeys(base)); + assert.strictEqual(typeof keys1.publicKey, 'string'); + assert.strictEqual(typeof keys1.privateKey, 'string'); + + const keys2 = await Promise.resolve(cryptoMod.loadKeys(base)); + assert.strictEqual(keys2.publicKey, keys1.publicKey); + assert.strictEqual(keys2.privateKey, keys1.privateKey); + }); +}; + +module.exports = { runCryptoKeysTests }; diff --git a/tests/suites/fsi.js b/tests/suites/fsi.js new file mode 100644 index 0000000..f45c4d8 --- /dev/null +++ b/tests/suites/fsi.js @@ -0,0 +1,68 @@ +'use strict'; + +const assert = require('node:assert'); +const path = require('node:path'); + +const runFsiTests = async (t, fsi, options = {}) => { + const { getRoot, afterEach } = options; + const joinPath = options.joinPath ?? ((r, ...p) => path.join(r, ...p)); + + const finish = async (dir) => { + if (afterEach) await afterEach(dir); + }; + + await t.test('write read exists and remove', async () => { + const dir = await getRoot(); + try { + assert.strictEqual(await fsi.exists(dir, 'a.txt'), false); + await fsi.write(dir, 'a.txt', 'hello'); + assert.strictEqual(await fsi.exists(dir, 'a.txt'), true); + assert.strictEqual(await fsi.read(dir, 'a.txt'), 'hello'); + await fsi.remove(dir, 'a.txt'); + assert.strictEqual(await fsi.exists(dir, 'a.txt'), false); + } finally { + await finish(dir); + } + }); + + await t.test('write creates nested directories', async () => { + const root = await getRoot(); + try { + const nested = joinPath(root, 'x', 'y'); + await fsi.write(nested, 'f.json', '{}'); + assert.strictEqual(await fsi.exists(nested, 'f.json'), true); + assert.strictEqual(await fsi.read(nested, 'f.json'), '{}'); + } finally { + await finish(root); + } + }); + + await t.test('remove when missing does not throw', async () => { + const dir = await getRoot(); + try { + await fsi.remove(dir, 'nope.txt'); + assert.strictEqual(await fsi.exists(dir, 'nope.txt'), false); + } finally { + await finish(dir); + } + }); + + await t.test('read when missing rejects', async () => { + const dir = await getRoot(); + try { + await assert.rejects( + () => fsi.read(dir, 'missing.txt'), + (err) => { + const isEnoent = err?.code === 'ENOENT'; + const isNotFound = err?.name === 'NotFoundError'; + const matches = isEnoent || isNotFound; + return Boolean(err && matches); + }, + ); + } finally { + await finish(dir); + } + }); +}; + +module.exports = { runFsiTests }; diff --git a/tests/suites/sync-manager.js b/tests/suites/sync-manager.js new file mode 100644 index 0000000..be660ab --- /dev/null +++ b/tests/suites/sync-manager.js @@ -0,0 +1,63 @@ +'use strict'; + +const runSyncManagerCoreTests = async (t, assert, getSync) => { + await t.test('addNode listNodes and removeNode', async (t2) => { + const sync = await getSync(t2); + + await sync.addNode({ + url: 'http://peer.example.com:9000', + type: 'uplink', + token: 't1', + }); + + const nodes = sync.listNodes(); + assert.strictEqual(nodes.length, 1); + assert.strictEqual(nodes[0].url, 'http://peer.example.com:9000'); + assert.strictEqual(nodes[0].type, 'uplink'); + + await sync.removeNode('http://peer.example.com:9000'); + assert.strictEqual(sync.listNodes().length, 0); + }); + + await t.test('addNode rejects bad input', async (t2) => { + const sync = await getSync(t2); + + await assert.rejects( + () => sync.addNode({ type: 'uplink' }), + /URL is required/, + ); + + await assert.rejects( + () => + sync.addNode({ + url: 'http://a.com:1', + type: 'relay', + }), + /uplink or downlink/, + ); + + await sync.addNode({ + url: 'http://dup.example.com:80', + type: 'downlink', + }); + + await assert.rejects( + () => + sync.addNode({ + url: 'http://dup.example.com:80', + type: 'uplink', + }), + /already exists/, + ); + }); + + await t.test('removeNode throws when unknown', async (t2) => { + const sync = await getSync(t2); + await assert.rejects( + () => sync.removeNode('http://nope.example.com:1'), + /not found/, + ); + }); +}; + +module.exports = { runSyncManagerCoreTests }; diff --git a/tests/sync-browser.js b/tests/sync-browser.js new file mode 100644 index 0000000..f66b2cc --- /dev/null +++ b/tests/sync-browser.js @@ -0,0 +1,10 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { SyncManager } = require('../lib/sync-browser.js'); +const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); + +test('SyncManager (browser)', async (t) => { + await runSyncManagerCoreTests(t, assert, async () => new SyncManager()); +}); diff --git a/tests/sync.js b/tests/sync.js new file mode 100644 index 0000000..8728276 --- /dev/null +++ b/tests/sync.js @@ -0,0 +1,16 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const globalStorage = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); + +test('SyncManager', async (t) => { + await runSyncManagerCoreTests(t, assert, async (t2) => { + const tempDir = await createTemp(); + t2.after(() => cleanupTemp(tempDir)); + const storage = await globalStorage.open({ path: tempDir }); + return storage.sync; + }); +}); diff --git a/tests/utils/mock-opfs.js b/tests/utils/mock-opfs.js new file mode 100644 index 0000000..1c8daa0 --- /dev/null +++ b/tests/utils/mock-opfs.js @@ -0,0 +1,107 @@ +'use strict'; + +class MockFileHandle { + #entry = null; + + constructor(entry) { + this.#entry = entry; + } + + async getFile() { + const content = this.#entry.content; + return { + text: async () => content, + }; + } + + async createWritable() { + const entry = this.#entry; + const chunks = []; + return { + write: async (data) => { + chunks.push(typeof data === 'string' ? data : String(data)); + }, + close: async () => { + entry.content = chunks.join(''); + }, + }; + } +} + +class MockDirectoryHandle { + #entries = null; + + constructor(entries) { + this.#entries = entries; + } + + async getDirectoryHandle(name, { create } = {}) { + if (!this.#entries.has(name)) { + if (!create) { + throw new DOMException(`Directory ${name} not found`, 'NotFoundError'); + } + this.#entries.set(name, { type: 'dir', map: new Map() }); + } + const entry = this.#entries.get(name); + if (entry.type !== 'dir') { + throw new DOMException('Not a directory', 'TypeMismatchError'); + } + return new MockDirectoryHandle(entry.map); + } + + async getFileHandle(name, { create } = {}) { + if (!this.#entries.has(name)) { + if (!create) { + throw new DOMException(`File ${name} not found`, 'NotFoundError'); + } + this.#entries.set(name, { type: 'file', content: '' }); + } + const entry = this.#entries.get(name); + if (entry.type !== 'file') { + throw new DOMException('Not a file', 'TypeMismatchError'); + } + return new MockFileHandle(entry); + } + + async removeEntry(name) { + if (!this.#entries.has(name)) { + throw new DOMException(`Entry ${name} not found`, 'NotFoundError'); + } + this.#entries.delete(name); + } +} + +const installMockOpfs = () => { + const rootMap = new Map(); + const prevNavigator = Object.getOwnPropertyDescriptor( + globalThis, + 'navigator', + ); + + Object.defineProperty(globalThis, 'navigator', { + configurable: true, + writable: true, + enumerable: true, + value: { + storage: { + getDirectory: async () => new MockDirectoryHandle(rootMap), + }, + }, + }); + + const reset = () => { + rootMap.clear(); + }; + + const uninstall = () => { + if (prevNavigator) { + Object.defineProperty(globalThis, 'navigator', prevNavigator); + } else { + delete globalThis.navigator; + } + }; + + return { reset, uninstall, rootMap }; +}; + +module.exports = { installMockOpfs }; diff --git a/tests/utils/temp.js b/tests/utils/temp.js new file mode 100644 index 0000000..7450b02 --- /dev/null +++ b/tests/utils/temp.js @@ -0,0 +1,26 @@ +'use strict'; + +const fs = require('node:fs/promises'); +const path = require('node:path'); +const os = require('node:os'); + +const createTemp = async () => { + const id = Math.random().toString(36).slice(2, 11); + const tmpRoot = os.tmpdir(); + const dirName = `gs-test-${Date.now()}-${id}`; + const tempDir = path.join(tmpRoot, dirName); + const mkdirOptions = { recursive: true }; + await fs.mkdir(tempDir, mkdirOptions); + return tempDir; +}; + +const cleanupTemp = async (dir) => { + const rmOptions = { recursive: true, force: true }; + try { + await fs.rm(dir, rmOptions); + } catch { + // ignore missing or already removed paths + } +}; + +module.exports = { createTemp, cleanupTemp }; From 62565ea956483ee17bad3a2501c5d20a9d441bf9 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Fri, 4 Sep 2026 16:27:29 +0300 Subject: [PATCH 02/67] feat: added pnpm --- pnpm-lock.yaml | 4890 +++++++++++++++++++++++++++++++++++++++++++ pnpm-workspace.yaml | 2 + 2 files changed, 4892 insertions(+) create mode 100644 pnpm-lock.yaml create mode 100644 pnpm-workspace.yaml diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml new file mode 100644 index 0000000..451a01c --- /dev/null +++ b/pnpm-lock.yaml @@ -0,0 +1,4890 @@ +lockfileVersion: '9.0' + +settings: + autoInstallPeers: true + excludeLinksFromLockfile: false + +importers: + + .: + devDependencies: + '@vercel/analytics': + specifier: ^2.0.1 + version: 2.0.1(vue@3.5.42(typescript@7.0.2)) + '@vercel/speed-insights': + specifier: ^2.0.0 + version: 2.0.0(vue@3.5.42(typescript@7.0.2)) + c8: + specifier: ^10.1.3 + version: 10.1.3 + esbuild: + specifier: ^0.28.1 + version: 0.28.2 + mermaid: + specifier: ^11.16.0 + version: 11.17.2 + oxfmt: + specifier: ^0.58.0 + version: 0.58.0 + oxlint: + specifier: ^1.73.0 + version: 1.81.0 + pino: + specifier: ^10.3.1 + version: 10.3.1 + tsd: + specifier: ^0.31.2 + version: 0.31.2 + typescript: + specifier: ^7.0.2 + version: 7.0.2 + vitepress: + specifier: ^1.6.4 + version: 1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) + vitepress-plugin-mermaid: + specifier: ^2.0.17 + version: 2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)) + +packages: + + '@algolia/abtesting@1.23.0': + resolution: {integrity: sha512-j45MBISstltys9QyQ4xf6quRiN1g7vMuwQL9VM4dx8YuRZvCQ173b9royZAx6iAbRX3IB1VnG1z//NuwyQ8jpQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/autocomplete-core@1.17.7': + resolution: {integrity: sha512-BjiPOW6ks90UKl7TwMv7oNQMnzU+t/wk9mgIDi6b1tXpUek7MW0lbNOUHpvam9pe3lVCf4xPFT+lK7s+e+fs7Q==} + + '@algolia/autocomplete-plugin-algolia-insights@1.17.7': + resolution: {integrity: sha512-Jca5Ude6yUOuyzjnz57og7Et3aXjbwCSDf/8onLHSQgw1qW3ALl9mrMWaXb5FmPVkV3EtkD2F/+NkT6VHyPu9A==} + peerDependencies: + search-insights: '>= 1 < 3' + + '@algolia/autocomplete-preset-algolia@1.17.7': + resolution: {integrity: sha512-ggOQ950+nwbWROq2MOCIL71RE0DdQZsceqrg32UqnhDz8FlO9rL8ONHNsI2R1MH0tkgVIDKI/D0sMiUchsFdWA==} + peerDependencies: + '@algolia/client-search': '>= 4.9.1 < 6' + algoliasearch: '>= 4.9.1 < 6' + + '@algolia/autocomplete-shared@1.17.7': + resolution: {integrity: sha512-o/1Vurr42U/qskRSuhBH+VKxMvkkUVTLU6WZQr+L5lGZZLYWyhdzWjW0iGXY7EkwRTjBqvN2EsR81yCTGV/kmg==} + peerDependencies: + '@algolia/client-search': '>= 4.9.1 < 6' + algoliasearch: '>= 4.9.1 < 6' + + '@algolia/client-abtesting@5.57.0': + resolution: {integrity: sha512-JVFFujiZUCguk5tz3LZr4fTQxqpIrj4/Jw3SI7kMljSqtfLxYn/s/TWH0J2s4iNfsDpxPhgFGMotCpmDI4kZ8w==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-analytics@5.57.0': + resolution: {integrity: sha512-6KqECK4ED3JJQEoDrQWnGPQzElA828xAD4qK5ceawNNyP/LcSvzAoLHjFkoTPksZ/kxj6VUtCRH+IHZesLltng==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-common@5.57.0': + resolution: {integrity: sha512-uqpGF3oXYsoCbQq5d7BzNrNTfIfuvJyGP1CKvSW27T9boUg7KOwyxsAw1AX0a3jSW2HrYEJ/NN+Z4MiGivbpeQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-insights@5.57.0': + resolution: {integrity: sha512-u5NboJVJXDEFplvNnqqX4CxkXPYysjJRj47hOSh9329H8kG5gFLKJBIiS5utMQ+GZm8xQl3Te7NInDk6elEADQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-personalization@5.57.0': + resolution: {integrity: sha512-uzc0b2LmHAK9/QID4xeo35OG84AkZl4YewkCqawqAOGLjT2eZpM/OZx45ESygMHG30Ws+ZTSdluPtMJcUnrbWQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-query-suggestions@5.57.0': + resolution: {integrity: sha512-dIAhnM6ue/ssa5PjgNfu4g8A4yTojl9ZOUzZU3wIaIKRerL2R/3Emuf9n/D6ICXXP167KC6XCeC7nliSw7cuSw==} + engines: {node: '>= 14.0.0'} + + '@algolia/client-search@5.57.0': + resolution: {integrity: sha512-2TTPTTKSJmCptvhCm4Xf3bBYMqZni+Pgc2hVdqc4l9wsBpSJNVTVIKpnd10OubUgkGcmppVDj1XQqYaf6EnPSQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/ingestion@1.57.0': + resolution: {integrity: sha512-W4JseHKt+pzOxlFV+T3MWEG0h4Z2Se5zjoXUD0ewlw8aOWMG/yjRdopUdLQsXULepB/My2tDuZjkwk2sMfsUrQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/monitoring@1.57.0': + resolution: {integrity: sha512-BrxJVE0/eLinEPICCD7BKN/2xnt0nkjge70u8zzE2ISP3fuB3tjLgcwpanUycvlHBFLI4gK0l5ol54p6IYuR/Q==} + engines: {node: '>= 14.0.0'} + + '@algolia/recommend@5.57.0': + resolution: {integrity: sha512-Gc29jkeiLKlVfHvyrIgyUHHE+aYTdXEeLfK42rjr5/1TTVsYwUJz0XkvoIBIqfMjcDg6gXeHb1jTUZ0H+SYIlQ==} + engines: {node: '>= 14.0.0'} + + '@algolia/requester-browser-xhr@5.57.0': + resolution: {integrity: sha512-PIPnPN7MP3fp2VAi01BVXhCWmD366ZB2Hkq5TlYKtThd4KxUtMmaaNDpFgVCTXtSIqWVZLJntOHRvxg/sIPd8Q==} + engines: {node: '>= 14.0.0'} + + '@algolia/requester-fetch@5.57.0': + resolution: {integrity: sha512-AX3RlOudXMdTwtwUqdAf5hAVLvXfOZZH1FZh6ALDdrhVLT0TtAIe48N6nYcWcTnwoTxK/wDIQqZ19IMjK8zJAA==} + engines: {node: '>= 14.0.0'} + + '@algolia/requester-node-http@5.57.0': + resolution: {integrity: sha512-cWZc1dKb7wy9/wPpwMtL1y89gK2G7y2A47Coa7zwf1ydtIeJm4+S+XxoQ2b/ZRiQnrC1YHavLjYUPDCdnT7Khg==} + engines: {node: '>= 14.0.0'} + + '@antfu/install-pkg@2.0.1': + resolution: {integrity: sha512-iCKVQcIC0e3oDxEfs3SHQGW+ovhBMZmS1TE+bTk50rVyMCBmCfClv7Qi3HQKlumYwvjb/iIMeWCW2i67q6kFfQ==} + + '@babel/code-frame@7.29.7': + resolution: {integrity: sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw==} + engines: {node: '>=6.9.0'} + + '@babel/helper-string-parser@7.29.7': + resolution: {integrity: sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==} + engines: {node: '>=6.9.0'} + + '@babel/helper-validator-identifier@7.29.7': + resolution: {integrity: sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==} + engines: {node: '>=6.9.0'} + + '@babel/parser@7.29.8': + resolution: {integrity: sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA==} + engines: {node: '>=6.0.0'} + hasBin: true + + '@babel/types@7.29.8': + resolution: {integrity: sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg==} + engines: {node: '>=6.9.0'} + + '@bcoe/v8-coverage@1.0.2': + resolution: {integrity: sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==} + engines: {node: '>=18'} + + '@braintree/sanitize-url@6.0.4': + resolution: {integrity: sha512-s3jaWicZd0pkP0jf5ysyHUI/RE7MHos6qlToFcGWXVp+ykHOy77OUMrfbgJ9it2C5bow7OIQwYYaHjk9XlBQ2A==} + + '@braintree/sanitize-url@7.1.2': + resolution: {integrity: sha512-jigsZK+sMF/cuiB7sERuo9V7N9jx+dhmHHnQyDSVdpZwVutaBu7WvNYqMDLSgFgfB30n452TP3vjDAvFC973mA==} + + '@chevrotain/types@11.1.2': + resolution: {integrity: sha512-U+HFai5+zmJCkK86QsaJtoITlboZHBqrVketcO2ROv865xfCMSFpELQoz1GkX5GzME8pTa+3kbKrZHQtI0gdbw==} + + '@docsearch/css@3.8.2': + resolution: {integrity: sha512-y05ayQFyUmCXze79+56v/4HpycYF3uFqB78pLPrSV5ZKAlDuIAAJNhaRi8tTdRNXh05yxX/TyNnzD6LwSM89vQ==} + + '@docsearch/js@3.8.2': + resolution: {integrity: sha512-Q5wY66qHn0SwA7Taa0aDbHiJvaFJLOJyHmooQ7y8hlwwQLQ/5WwCcoX0g7ii04Qi2DJlHsd0XXzJ8Ypw9+9YmQ==} + + '@docsearch/react@3.8.2': + resolution: {integrity: sha512-xCRrJQlTt8N9GU0DG4ptwHRkfnSnD/YpdeaXe02iKfqs97TkZJv60yE+1eq/tjPcVnTW8dP5qLP7itifFVV5eg==} + peerDependencies: + '@types/react': '>= 16.8.0 < 19.0.0' + react: '>= 16.8.0 < 19.0.0' + react-dom: '>= 16.8.0 < 19.0.0' + search-insights: '>= 1 < 3' + peerDependenciesMeta: + '@types/react': + optional: true + react: + optional: true + react-dom: + optional: true + search-insights: + optional: true + + '@esbuild/aix-ppc64@0.21.5': + resolution: {integrity: sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==} + engines: {node: '>=12'} + cpu: [ppc64] + os: [aix] + + '@esbuild/aix-ppc64@0.28.2': + resolution: {integrity: sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [aix] + + '@esbuild/android-arm64@0.21.5': + resolution: {integrity: sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==} + engines: {node: '>=12'} + cpu: [arm64] + os: [android] + + '@esbuild/android-arm64@0.28.2': + resolution: {integrity: sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==} + engines: {node: '>=18'} + cpu: [arm64] + os: [android] + + '@esbuild/android-arm@0.21.5': + resolution: {integrity: sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==} + engines: {node: '>=12'} + cpu: [arm] + os: [android] + + '@esbuild/android-arm@0.28.2': + resolution: {integrity: sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==} + engines: {node: '>=18'} + cpu: [arm] + os: [android] + + '@esbuild/android-x64@0.21.5': + resolution: {integrity: sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==} + engines: {node: '>=12'} + cpu: [x64] + os: [android] + + '@esbuild/android-x64@0.28.2': + resolution: {integrity: sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==} + engines: {node: '>=18'} + cpu: [x64] + os: [android] + + '@esbuild/darwin-arm64@0.21.5': + resolution: {integrity: sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==} + engines: {node: '>=12'} + cpu: [arm64] + os: [darwin] + + '@esbuild/darwin-arm64@0.28.2': + resolution: {integrity: sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [darwin] + + '@esbuild/darwin-x64@0.21.5': + resolution: {integrity: sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==} + engines: {node: '>=12'} + cpu: [x64] + os: [darwin] + + '@esbuild/darwin-x64@0.28.2': + resolution: {integrity: sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==} + engines: {node: '>=18'} + cpu: [x64] + os: [darwin] + + '@esbuild/freebsd-arm64@0.21.5': + resolution: {integrity: sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==} + engines: {node: '>=12'} + cpu: [arm64] + os: [freebsd] + + '@esbuild/freebsd-arm64@0.28.2': + resolution: {integrity: sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [freebsd] + + '@esbuild/freebsd-x64@0.21.5': + resolution: {integrity: sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==} + engines: {node: '>=12'} + cpu: [x64] + os: [freebsd] + + '@esbuild/freebsd-x64@0.28.2': + resolution: {integrity: sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==} + engines: {node: '>=18'} + cpu: [x64] + os: [freebsd] + + '@esbuild/linux-arm64@0.21.5': + resolution: {integrity: sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==} + engines: {node: '>=12'} + cpu: [arm64] + os: [linux] + + '@esbuild/linux-arm64@0.28.2': + resolution: {integrity: sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==} + engines: {node: '>=18'} + cpu: [arm64] + os: [linux] + + '@esbuild/linux-arm@0.21.5': + resolution: {integrity: sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==} + engines: {node: '>=12'} + cpu: [arm] + os: [linux] + + '@esbuild/linux-arm@0.28.2': + resolution: {integrity: sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==} + engines: {node: '>=18'} + cpu: [arm] + os: [linux] + + '@esbuild/linux-ia32@0.21.5': + resolution: {integrity: sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==} + engines: {node: '>=12'} + cpu: [ia32] + os: [linux] + + '@esbuild/linux-ia32@0.28.2': + resolution: {integrity: sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==} + engines: {node: '>=18'} + cpu: [ia32] + os: [linux] + + '@esbuild/linux-loong64@0.21.5': + resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==} + engines: {node: '>=12'} + cpu: [loong64] + os: [linux] + + '@esbuild/linux-loong64@0.28.2': + resolution: {integrity: sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==} + engines: {node: '>=18'} + cpu: [loong64] + os: [linux] + + '@esbuild/linux-mips64el@0.21.5': + resolution: {integrity: sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==} + engines: {node: '>=12'} + cpu: [mips64el] + os: [linux] + + '@esbuild/linux-mips64el@0.28.2': + resolution: {integrity: sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==} + engines: {node: '>=18'} + cpu: [mips64el] + os: [linux] + + '@esbuild/linux-ppc64@0.21.5': + resolution: {integrity: sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==} + engines: {node: '>=12'} + cpu: [ppc64] + os: [linux] + + '@esbuild/linux-ppc64@0.28.2': + resolution: {integrity: sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==} + engines: {node: '>=18'} + cpu: [ppc64] + os: [linux] + + '@esbuild/linux-riscv64@0.21.5': + resolution: {integrity: sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==} + engines: {node: '>=12'} + cpu: [riscv64] + os: [linux] + + '@esbuild/linux-riscv64@0.28.2': + resolution: {integrity: sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==} + engines: {node: '>=18'} + cpu: [riscv64] + os: [linux] + + '@esbuild/linux-s390x@0.21.5': + resolution: {integrity: sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==} + engines: {node: '>=12'} + cpu: [s390x] + os: [linux] + + '@esbuild/linux-s390x@0.28.2': + resolution: {integrity: sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==} + engines: {node: '>=18'} + cpu: [s390x] + os: [linux] + + '@esbuild/linux-x64@0.21.5': + resolution: {integrity: sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==} + engines: {node: '>=12'} + cpu: [x64] + os: [linux] + + '@esbuild/linux-x64@0.28.2': + resolution: {integrity: sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==} + engines: {node: '>=18'} + cpu: [x64] + os: [linux] + + '@esbuild/netbsd-arm64@0.28.2': + resolution: {integrity: sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==} + engines: {node: '>=18'} + cpu: [arm64] + os: [netbsd] + + '@esbuild/netbsd-x64@0.21.5': + resolution: {integrity: sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==} + engines: {node: '>=12'} + cpu: [x64] + os: [netbsd] + + '@esbuild/netbsd-x64@0.28.2': + resolution: {integrity: sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==} + engines: {node: '>=18'} + cpu: [x64] + os: [netbsd] + + '@esbuild/openbsd-arm64@0.28.2': + resolution: {integrity: sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openbsd] + + '@esbuild/openbsd-x64@0.21.5': + resolution: {integrity: sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==} + engines: {node: '>=12'} + cpu: [x64] + os: [openbsd] + + '@esbuild/openbsd-x64@0.28.2': + resolution: {integrity: sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==} + engines: {node: '>=18'} + cpu: [x64] + os: [openbsd] + + '@esbuild/openharmony-arm64@0.28.2': + resolution: {integrity: sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==} + engines: {node: '>=18'} + cpu: [arm64] + os: [openharmony] + + '@esbuild/sunos-x64@0.21.5': + resolution: {integrity: sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==} + engines: {node: '>=12'} + cpu: [x64] + os: [sunos] + + '@esbuild/sunos-x64@0.28.2': + resolution: {integrity: sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==} + engines: {node: '>=18'} + cpu: [x64] + os: [sunos] + + '@esbuild/win32-arm64@0.21.5': + resolution: {integrity: sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==} + engines: {node: '>=12'} + cpu: [arm64] + os: [win32] + + '@esbuild/win32-arm64@0.28.2': + resolution: {integrity: sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==} + engines: {node: '>=18'} + cpu: [arm64] + os: [win32] + + '@esbuild/win32-ia32@0.21.5': + resolution: {integrity: sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==} + engines: {node: '>=12'} + cpu: [ia32] + os: [win32] + + '@esbuild/win32-ia32@0.28.2': + resolution: {integrity: sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==} + engines: {node: '>=18'} + cpu: [ia32] + os: [win32] + + '@esbuild/win32-x64@0.21.5': + resolution: {integrity: sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==} + engines: {node: '>=12'} + cpu: [x64] + os: [win32] + + '@esbuild/win32-x64@0.28.2': + resolution: {integrity: sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==} + engines: {node: '>=18'} + cpu: [x64] + os: [win32] + + '@iconify-json/simple-icons@1.2.94': + resolution: {integrity: sha512-l8UWzVxKaqZd9ABsE/M/9p6NyGkQnmCnOoZyhQmjlXCtY5PuL2rcWxOFk2l9pk7ux3ERMPkTLE4jl6kQpTkwxA==} + + '@iconify/types@2.0.0': + resolution: {integrity: sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg==} + + '@iconify/utils@3.1.5': + resolution: {integrity: sha512-nT+AWyh5caHY6xa0PdLXpUooIz6l7qsT+adqsCTYBfbdn5Q18VrWs2fvuTa7KEoJnPBfEPqWkmAIht0EHtCi2A==} + + '@isaacs/cliui@8.0.2': + resolution: {integrity: sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==} + engines: {node: '>=12'} + + '@istanbuljs/schema@0.1.6': + resolution: {integrity: sha512-+Sg6GCR/wy1oSmQDFq4LQDAhm3ETKnorxN+y5nbLULOR3P0c14f2Wurzj3/xqPXtasLFfHd5iRFQ7AJt4KH2cw==} + engines: {node: '>=8'} + + '@jest/schemas@29.6.3': + resolution: {integrity: sha512-mo5j5X+jIZmJQveBKeS/clAueipV7KgiX1vMgCxam1RNYiqE1w62n0/tJJnHtjW8ZHcQco5gY85jA3mi0L+nSA==} + engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0} + + '@jridgewell/resolve-uri@3.1.2': + resolution: {integrity: sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==} + engines: {node: '>=6.0.0'} + + '@jridgewell/sourcemap-codec@1.6.0': + resolution: {integrity: sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw==} + + '@jridgewell/trace-mapping@0.3.31': + resolution: {integrity: sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==} + + '@mermaid-js/mermaid-mindmap@9.3.0': + resolution: {integrity: sha512-IhtYSVBBRYviH1Ehu8gk69pMDF8DSRqXBRDMWrEfHoaMruHeaP2DXA3PBnuwsMaCdPQhlUUcy/7DBLAEIXvCAw==} + + '@mermaid-js/parser@1.2.1': + resolution: {integrity: sha512-n12NohV3mrUyUL2o93IgG/ifeW9FTyeJn3zDxkhwa8MJ9Fxg3HQMlA3RiGmD/3UnJvheztkjjQAjA2T4LmUcpw==} + + '@napi-rs/lzma-linux-x64-gnu@1.5.1': + resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==} + engines: {node: ^22.20 || ^24.12 || >=25} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@nodelib/fs.scandir@2.1.5': + resolution: {integrity: sha512-vq24Bq3ym5HEQm2NKCr3yXDwjc7vTsEThRDnkp2DK9p1uqLR+DHurm/NOTo0KG7HYHU7eppKZj3MyqYuMBf62g==} + engines: {node: '>= 8'} + + '@nodelib/fs.stat@2.0.5': + resolution: {integrity: sha512-RkhPPp2zrqDAQA/2jNhnztcPAlv64XdhIp7a7454A5ovI7Bukxgt7MX7udwAu3zg1DcpPU0rz3VV1SeaqvY4+A==} + engines: {node: '>= 8'} + + '@nodelib/fs.walk@1.2.8': + resolution: {integrity: sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==} + engines: {node: '>= 8'} + + '@oxfmt/binding-android-arm-eabi@0.58.0': + resolution: {integrity: sha512-Uz62sHduGGPftXtILGyxdSW4PX82rUg+rfdNqhsgxe881g4rIoXlIqmZQ6HVKcF4f+F8qMhdD03Bx5u7gmeTdg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [android] + + '@oxfmt/binding-android-arm64@0.58.0': + resolution: {integrity: sha512-rD0lRaJp1b+9vw6X4A2dJWKukd6X8yxiicN4JxXcXayolmUypRZxk+lKR+fVOu5q/iYc0fh5fR4bgmfOfVlbaA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@oxfmt/binding-darwin-arm64@0.58.0': + resolution: {integrity: sha512-uzbPPk7O6M+w2K65vcQ1woga3wgP8zghjL1KOG5b6qJ8dvYHZJ1VShaslg2KOK6yQIwCQtcMCXqLBM6sqXUNTg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@oxfmt/binding-darwin-x64@0.58.0': + resolution: {integrity: sha512-L0nKYDxU32oxeQqJj21W9SlIMnf81VZEhyah6iDvFhf5q0oynq498Fopth7blErUJVBpVtxQ98RMCfMPqpJX6w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@oxfmt/binding-freebsd-x64@0.58.0': + resolution: {integrity: sha512-woNwfD58dC5PGS9LSLSD5JYfo/EFK5iG9vhDWkcCg3q78ag7KC8bpDqgvPHrMoXpx83OLXxoSOhu6z8FsVTHlg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@oxfmt/binding-linux-arm-gnueabihf@0.58.0': + resolution: {integrity: sha512-Sqs8nMLxuQpY21NKJ1u4stPDmO5hskBCNNh2E3AdCfI1QqWtf4m+Qn4mGEIUO4KGmuq3SWc/SZ80uy5IiwTCDw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxfmt/binding-linux-arm-musleabihf@0.58.0': + resolution: {integrity: sha512-Vd4exzBI5B5hB9m22JiTQzIL23WvHo/Pe+sNXPNeBLXSP9swCBPKCEBRwKpmpQzYhlgYaCgfPcGXPKAJBRIiZQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxfmt/binding-linux-arm64-gnu@0.58.0': + resolution: {integrity: sha512-bUWi5mHV+4Vi56RLHE1h6q/HHfwAIT3XoB9vJAVeRzfu5NriXM8y6eeJu0vlKa0C9kq2rq1sOWRClhdLHPocrg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@oxfmt/binding-linux-arm64-musl@0.58.0': + resolution: {integrity: sha512-2ZHxemzgHcjtktAuVUwSoyXmGo/t+aF5tS1ciPpPei4rhSyrz3JOqDosXXrmhN/yLUSzJjtuW7ToTWqfQpCj2w==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@oxfmt/binding-linux-ppc64-gnu@0.58.0': + resolution: {integrity: sha512-AwKkVwjVmFQ3bcO7j0McGYAqCKH2a326fswfofng/E8VewCT/raeeGQr4huVhY704deK8AWASSTlxzMj0eZc6Q==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + libc: [glibc] + + '@oxfmt/binding-linux-riscv64-gnu@0.58.0': + resolution: {integrity: sha512-xsRpTxfUnJF8D3AUKko/qyWdjw4GZVHlCVFuGlzSCTeewLmykKINW8em1+wx+axsDVtJJcMtvsiaXggXxrlHgw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [glibc] + + '@oxfmt/binding-linux-riscv64-musl@0.58.0': + resolution: {integrity: sha512-Z4AYOTcy7nYEIiXwD62PlerimyYRcfJOgUbQAEBjXz098kxKuERBlRntofGy69HHhe9E0TLVNMl1yspVNu+efw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [musl] + + '@oxfmt/binding-linux-s390x-gnu@0.58.0': + resolution: {integrity: sha512-A3nhhtZPC/TKVWOPj9q/H3p2znJDCcHWYlJBhWL8hGq/bFmBaNBHC8Np6E581yVq1w9Mi3rMDNzDalWvtUfJtQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + libc: [glibc] + + '@oxfmt/binding-linux-x64-gnu@0.58.0': + resolution: {integrity: sha512-2g+tVkgwqphw8R4hgo+kF4oz8+P5RwVOtr9+irsC7uwEp0e9j7Crw8kDGKL20uYlLPD7g02DqA61mC/UNYx98A==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@oxfmt/binding-linux-x64-musl@0.58.0': + resolution: {integrity: sha512-rc15P6AbyyB7426aN8AakLd02Trb3a6ML/mmfAQeVHJEfVofWLcWIrBdy6zDEY+DIaL/s8E4GGPboVw+oP3+EA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [musl] + + '@oxfmt/binding-openharmony-arm64@0.58.0': + resolution: {integrity: sha512-ZWoTM27/HYPOh9iq86DAbhPu9nXb8qKvvGU/h8OfliyVUFAMMNTLDkGsWDKKnDqIkqvZ9+dXlgUOsH1LYO3O7g==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@oxfmt/binding-win32-arm64-msvc@0.58.0': + resolution: {integrity: sha512-LHZnqFXe2dEfkRI4XdZS/57nEOT/I4UCRX5IyM9v4GYW9XwQCjGe1IUK59SuKw3POwvcgWQ4pme2cYXmNqTNPg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@oxfmt/binding-win32-ia32-msvc@0.58.0': + resolution: {integrity: sha512-mZKpg20TpheCJym1rarcZCUJeW1sSruw8zAAaCYWvuVfwIUDN1CXdrPU/JgCWReXTCTrEfCB8Wyo3hh9jSZ2EA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ia32] + os: [win32] + + '@oxfmt/binding-win32-x64-msvc@0.58.0': + resolution: {integrity: sha512-N/wUU4N5PZ2orBtI+Ko7MnMfYLfE7K91UrGMY/c/pYyHR3lA9kwst1XugkZx+92YcRh/Eo+iv2eTESSWXfiZPA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + + '@oxlint/binding-android-arm-eabi@1.81.0': + resolution: {integrity: sha512-IcCRsXiedJoJopY6mpZUBEeVFsUrutmrG7dZ87zMuKJlhg70Ora9bBl1WcCxZQtyI10YpnVdEso5oCg7YcfSHw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [android] + + '@oxlint/binding-android-arm64@1.81.0': + resolution: {integrity: sha512-GRrIPyTGVhx3L3h+0T5xT2A0jFAcdPv4+IfuXpGDLIdl6XeYhgg/zw72A5ILZoUgRqZuM8F1y+V/gfDriXSxzQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [android] + + '@oxlint/binding-darwin-arm64@1.81.0': + resolution: {integrity: sha512-qNQ9tXRgLuKbqSV1S2h9h4KPHjbovO7RRR2/enUOtHzTkFZ7B9X5zqqHJua8dRyc7dBy7Aoyq5pqTSLFVcAzGQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [darwin] + + '@oxlint/binding-darwin-x64@1.81.0': + resolution: {integrity: sha512-q0QTm32jWga2Gv4j7IaVZN0jYMi9UV73sWVgFtDA4iIfqwMCLLZ3ve+9KwfYtsaKZSgQhmPaogeZWqDZpcY1Pw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [darwin] + + '@oxlint/binding-freebsd-x64@1.81.0': + resolution: {integrity: sha512-/+8wVWDXEC7wHVAhOc59Fw/SkMc1arLkFD8iQCaSsmzenK1X4doFqquL9H1wrtGUzaiycVqkf/sSpcILK6W1UA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [freebsd] + + '@oxlint/binding-linux-arm-gnueabihf@1.81.0': + resolution: {integrity: sha512-4xt422FEgioRq9hAL4Tq7fujGUWnc8z1BJ+Oi8RN8vB8axaP+sdK6a2xdlcQCCYnJg9QMuMFS0AucuIFx/EacA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxlint/binding-linux-arm-musleabihf@1.81.0': + resolution: {integrity: sha512-u3vna8KdGplH4DRCW9K54D68fcMo7IxVrkCJWwXnIhwtBdnDnYrmzOUA/XjmBlPpcLsgw9Z5BNdY4za9+Dj+MQ==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm] + os: [linux] + + '@oxlint/binding-linux-arm64-gnu@1.81.0': + resolution: {integrity: sha512-3j9k+gsYsE7nv71GWotXsqsa2l9/aJenD7dVHNt/CBvsb0SgRjSMnHFeP59IXUAl1wvVFhqGl2wJNMwWU3UBlA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@oxlint/binding-linux-arm64-musl@1.81.0': + resolution: {integrity: sha512-k5iAp3dNxW0/uDCBY+WSm8jKB2szu7SkEQZdgRRpDXvuDd69vvDcqhB3A/pWCfCwXyenjNjFn9Td1fVoyAc+Yg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@oxlint/binding-linux-ppc64-gnu@1.81.0': + resolution: {integrity: sha512-TFqLja3uYmVSte6nof9GWrex9Z8WgdZrNiLC6Te5rXGDqXB2y4j/26iFhwosXiAFqDhE9JJVuuCkDKLwptTn1g==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ppc64] + os: [linux] + libc: [glibc] + + '@oxlint/binding-linux-riscv64-gnu@1.81.0': + resolution: {integrity: sha512-UEcySvGS0NOVo7h7n7CYyJL9+6gFAh7Zc/ToDXVScFvzHSTIxtzkMVU30rmQ6+nQ1LF+UdiRDdJajpDu+OylLg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [glibc] + + '@oxlint/binding-linux-riscv64-musl@1.81.0': + resolution: {integrity: sha512-H+diDbhD00+wI1IRP8Kz88x/lat+DgtoBJzoTthS16xkTJGNaEkfb8gzmd1rzc/2uDQQMl7GNl+JFUacVeWxIA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [riscv64] + os: [linux] + libc: [musl] + + '@oxlint/binding-linux-s390x-gnu@1.81.0': + resolution: {integrity: sha512-8znJ/5TekjOKg1j1Acho4PJMdiAHLtlcXuWEiipOhAMV6rQcXdmDdXCbheyDczN6TjBwiNfjcP81k4AthrKRzw==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [s390x] + os: [linux] + libc: [glibc] + + '@oxlint/binding-linux-x64-gnu@1.81.0': + resolution: {integrity: sha512-Q2Wj70yFsvn5QjlmifFzbj4H+kJy53bwqc41o1fzoM7MpLV1NIbhg/LpWXRfC6KOkSAdUx1Wd8VJsdPmhp/HRA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@oxlint/binding-linux-x64-musl@1.81.0': + resolution: {integrity: sha512-cPInHp/ddEe5qkyK2IiyQ8Q3Mp2oLLEhhsGgTK2oZx4L6+llGam1H1yBvJZ7qHfOXj8N3hxBS8sj4tO+gtFlIg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [linux] + libc: [musl] + + '@oxlint/binding-openharmony-arm64@1.81.0': + resolution: {integrity: sha512-0CQxSX4ajqm07AHBf5U33qQzXKdd7wtq/oTL/7vpY6RNNuxrRi8W4bqUV1Jyu/vj+9KmxQyDhxfeVX1nQL6kfg==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [openharmony] + + '@oxlint/binding-win32-arm64-msvc@1.81.0': + resolution: {integrity: sha512-l0hbeISm9673hVrrQU8j/p2M7YH9Ouoj7p7E/QM55NTrKVLP+P3PF8hLu+OY+x0VtGRW+ggiQKZqmdYps9H+TA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [arm64] + os: [win32] + + '@oxlint/binding-win32-ia32-msvc@1.81.0': + resolution: {integrity: sha512-ksqPP5jbFXcYreEQ7zdJh06rJQBymCTyGRCdaXjfcf2aG4f8KxUWY5wcgYHmaTK+FJ4bPG5sUAdOX+6trnH1JA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [ia32] + os: [win32] + + '@oxlint/binding-win32-x64-msvc@1.81.0': + resolution: {integrity: sha512-IZuUCwGw9emG5JtCp+fYGB+Z4OWEoeEcM8R5BA1pYw63/ieYFVdcU2ylxTpHbVHSenZnsYE+ZZ20uHAJszQ4cA==} + engines: {node: ^20.19.0 || >=22.12.0} + cpu: [x64] + os: [win32] + + '@pinojs/redact@0.4.0': + resolution: {integrity: sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==} + + '@pkgjs/parseargs@0.11.0': + resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} + engines: {node: '>=14'} + + '@rollup/rollup-android-arm-eabi@4.63.1': + resolution: {integrity: sha512-UZ8sUxPTiHWYX9QNdJedb1kDZSpS1t/VPWBWGSgqHNi9w3Cu6IXvu2mzbhiTiPvtrqgTQJ+zqiAq2iPIPilpaQ==} + cpu: [arm] + os: [android] + + '@rollup/rollup-android-arm64@4.63.1': + resolution: {integrity: sha512-cQ4nFQABN5cDvDpbvJ7bMStCpnaVxynZrRMfUJYgxcIk9Sh54FIO1vtfkg0B69REjER77ioZ/ov+eAApx/KmLQ==} + cpu: [arm64] + os: [android] + + '@rollup/rollup-darwin-arm64@4.63.1': + resolution: {integrity: sha512-FQNqd1lRy/0QhDk3xeRIkSBiCpXCiDnZO3YLVdcDKN1UBiKToNftCzcXYNLshmPDUMlu2TdeS8tGcsU6f3YF1Q==} + cpu: [arm64] + os: [darwin] + + '@rollup/rollup-darwin-x64@4.63.1': + resolution: {integrity: sha512-pvD16V939D3CloK0+qikpGaxiPrDUXTe7Y5cWOMkMSy7m1cawa8EGy/kXYi/G/cKAC4HDAbSnzCIk1WmsoOKXg==} + cpu: [x64] + os: [darwin] + + '@rollup/rollup-freebsd-arm64@4.63.1': + resolution: {integrity: sha512-pcFGeL2345VwdTnJhA6zLbew+YgWB0qBG2+dMtXjCicf6+rm6kO6cOoh5VnTe0ZMrMRgRyuHmCJxZWrIdzYuOw==} + cpu: [arm64] + os: [freebsd] + + '@rollup/rollup-freebsd-x64@4.63.1': + resolution: {integrity: sha512-mRJlqSRulVzcKq/LKA6ICSIc3K/l4fzlVn/gePn2nXIHy8seRi5z/eeRE0d/XMBxcMldiXtQTSpRj0tkkC3g8Q==} + cpu: [x64] + os: [freebsd] + + '@rollup/rollup-linux-arm-gnueabihf@4.63.1': + resolution: {integrity: sha512-YDUNvVM85TI3g/1OpnqKP1h4NeW/j64DfWMf+G3M809xNk1bJSnpFp4sh83NpmVE5DXnkh8ULor4LTVZKoYLHw==} + cpu: [arm] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-arm-musleabihf@4.63.1': + resolution: {integrity: sha512-7Mcn71p9ZuQFAj+h+dhQXy/yeLePRS2yKRnmW1DijA9thKO5qap0GNOIQK4yQ6iP3SU0Mrb/yWo8h8vgRba8lw==} + cpu: [arm] + os: [linux] + libc: [musl] + + '@rollup/rollup-linux-arm64-gnu@4.63.1': + resolution: {integrity: sha512-4YiLQTX6U4CSl0L9cluep9A9W6UmTfqBDc2/CH6wlu54pl4E7Jn3cOD8oxzvBDEGk/JMKgJ47C8g+radF7mwvg==} + cpu: [arm64] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-arm64-musl@4.63.1': + resolution: {integrity: sha512-2ra8F7w8OquwZN9z2/fKFnli69wa8PLwaVzRMIPGb13ByMJwC28Fbp8YcVGoUhlYMTt7j5j9bNgpysrN2UM+vw==} + cpu: [arm64] + os: [linux] + libc: [musl] + + '@rollup/rollup-linux-loong64-gnu@4.63.1': + resolution: {integrity: sha512-Sy20ncyhjmBP0Ml+UvQbimjlk6VFgjW5uNP+qqwHB00mTE8Bl2C1TuHTlRwK2YoXeZbee5lP2XevBWVkAQAtSQ==} + cpu: [loong64] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-loong64-musl@4.63.1': + resolution: {integrity: sha512-noITLp8oNjYliPnGWmLyelIHwULGqbHloQHGw1rtxbWhTuWooRpnZarZQJ1y9EUC4szuCusCc+HEpUtxpIwYvA==} + cpu: [loong64] + os: [linux] + libc: [musl] + + '@rollup/rollup-linux-ppc64-gnu@4.63.1': + resolution: {integrity: sha512-hlxxXd+F1mWiAcaFR7Sv9ZQT6m6UfI8+Vy/kFJzztq2pDMU/0wZ9sish0iszNZvsQDo8Gc0i5yuFEOz5dDf6fA==} + cpu: [ppc64] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-ppc64-musl@4.63.1': + resolution: {integrity: sha512-EF7OpqQTQ/BvGqLzUi4rEHuagCV9MugAUXSHemwPW5vxZ75RR+jxO/2j95Ph2dalMpFHSVECjRoioHZgA9zOYA==} + cpu: [ppc64] + os: [linux] + libc: [musl] + + '@rollup/rollup-linux-riscv64-gnu@4.63.1': + resolution: {integrity: sha512-wQO3JesW9PRkwlabQ27y7sPfVOOTLRG73I4F2UYHG5PXun3J9U3y+b7ezVKSYbsvSKGQ1k1cq8Qlun4C9kLt3w==} + cpu: [riscv64] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-riscv64-musl@4.63.1': + resolution: {integrity: sha512-ouAGwhO6wHRXdnOVCOsB0tRFkA7nhNB2Nwax6oECXN0YiN8EYUTBAOudADOB1PI+yDL61TeNx/u7MVCzksNbkQ==} + cpu: [riscv64] + os: [linux] + libc: [musl] + + '@rollup/rollup-linux-s390x-gnu@4.63.1': + resolution: {integrity: sha512-q2R38Sn+1J8RxhfJ+T54wSWmyKXWec+9jgDfqO2AtArEqHO5R2aeayp5H5OYLr5UYDVGsVaZPEFUooMhYCdz5A==} + cpu: [s390x] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-x64-gnu@4.63.1': + resolution: {integrity: sha512-gfI5T24WLLuFfSKw7Go/zDXjAAV0fny0swTaDv+WjK7vqcw4cRhFfdsyKL1n+ukI+ooBxn3bVQnyrn06WpI50w==} + cpu: [x64] + os: [linux] + libc: [glibc] + + '@rollup/rollup-linux-x64-musl@4.63.1': + resolution: {integrity: sha512-4h6XqthmB4Hspji84wvgk+ElodTsGj+dbZqHJHHtKxj4mYq0ANSEEPX9ys3moJueqsRjwpaJYH7874Itwnj2ow==} + cpu: [x64] + os: [linux] + libc: [musl] + + '@rollup/rollup-openbsd-x64@4.63.1': + resolution: {integrity: sha512-dlfCOa87o1VAYegLQ9EKilx2JCeRofiyPGhTCmqnuXZ6bMPiycO1rq1+sKoulAp7pGLIsTIw+1x5R+zgh5LhhA==} + cpu: [x64] + os: [openbsd] + + '@rollup/rollup-openharmony-arm64@4.63.1': + resolution: {integrity: sha512-cjkLbOlfcm3QGhMM1J5zaZjsw1GggbN6rw9UTSSRrPrR1KkcXnN7Uq9rPw34xImQ9VOY9GN+6u2Zj80B9ptkcw==} + cpu: [arm64] + os: [openharmony] + + '@rollup/rollup-win32-arm64-msvc@4.63.1': + resolution: {integrity: sha512-Li1KdUnWGE4N3e1F/B4RTB1ms+nG4WBgjByO46pkeBVX/2UBsY53xf5vK9WygVmnH3RwncIST7lkSdLSY6P9lg==} + cpu: [arm64] + os: [win32] + + '@rollup/rollup-win32-ia32-msvc@4.63.1': + resolution: {integrity: sha512-t4ZYOSoLTgwhuFMrmTMLx/+i1DQVK7HYqMc6kY46EApwi8X0nIVphzdNoThU3xt6n+N5urG1/gxBdCaKDLavfg==} + cpu: [ia32] + os: [win32] + + '@rollup/rollup-win32-x64-gnu@4.63.1': + resolution: {integrity: sha512-RgroPfMmKlD1RzSDxvwgcPiy2HNQKoYV7OmwIXDsk73uKW5t6B/V8KIy27SMv/FNXFo/oSBtWc9J0X7t91ezZg==} + cpu: [x64] + os: [win32] + + '@rollup/rollup-win32-x64-msvc@4.63.1': + resolution: {integrity: sha512-at8QVep6S3h5Y6gSbdGU06bRY5WJkf6WUduM9YtvYMbYhB1MOFfUgc6kehitQXzOtMSaT70q7f9ydPhpqu821w==} + cpu: [x64] + os: [win32] + + '@shikijs/core@2.5.0': + resolution: {integrity: sha512-uu/8RExTKtavlpH7XqnVYBrfBkUc20ngXiX9NSrBhOVZYv/7XQRKUyhtkeflY5QsxC0GbJThCerruZfsUaSldg==} + + '@shikijs/engine-javascript@2.5.0': + resolution: {integrity: sha512-VjnOpnQf8WuCEZtNUdjjwGUbtAVKuZkVQ/5cHy/tojVVRIRtlWMYVjyWhxOmIq05AlSOv72z7hRNRGVBgQOl0w==} + + '@shikijs/engine-oniguruma@2.5.0': + resolution: {integrity: sha512-pGd1wRATzbo/uatrCIILlAdFVKdxImWJGQ5rFiB5VZi2ve5xj3Ax9jny8QvkaV93btQEwR/rSz5ERFpC5mKNIw==} + + '@shikijs/langs@2.5.0': + resolution: {integrity: sha512-Qfrrt5OsNH5R+5tJ/3uYBBZv3SuGmnRPejV9IlIbFH3HTGLDlkqgHymAlzklVmKBjAaVmkPkyikAV/sQ1wSL+w==} + + '@shikijs/themes@2.5.0': + resolution: {integrity: sha512-wGrk+R8tJnO0VMzmUExHR+QdSaPUl/NKs+a4cQQRWyoc3YFbUzuLEi/KWK1hj+8BfHRKm2jNhhJck1dfstJpiw==} + + '@shikijs/transformers@2.5.0': + resolution: {integrity: sha512-SI494W5X60CaUwgi8u4q4m4s3YAFSxln3tzNjOSYqq54wlVgz0/NbbXEb3mdLbqMBztcmS7bVTaEd2w0qMmfeg==} + + '@shikijs/types@2.5.0': + resolution: {integrity: sha512-ygl5yhxki9ZLNuNpPitBWvcy9fsSKKaRuO4BAlMyagszQidxcpLAr0qiW/q43DtSIDxO6hEbtYLiFZNXO/hdGw==} + + '@shikijs/vscode-textmate@10.0.2': + resolution: {integrity: sha512-83yeghZ2xxin3Nj8z1NMd/NCuca+gsYXswywDy5bHvwlWL8tpTQmzGeUuHd9FC3E/SBEMvzJRwWEOz5gGes9Qg==} + + '@sinclair/typebox@0.27.12': + resolution: {integrity: sha512-hhyNJ+nbR6ZR7pToHvllEFun9TL0sbL+tk/ON75lo+Xas054uez98qRbsuNt7MBCyZKK4+8Yli/OAGZhmfBZ/g==} + + '@tsd/typescript@5.4.5': + resolution: {integrity: sha512-saiCxzHRhUrRxQV2JhH580aQUZiKQUXI38FcAcikcfOomAil4G4lxT0RfrrKywoAYP/rqAdYXYmNRLppcd+hQQ==} + engines: {node: '>=14.17'} + + '@types/d3-array@3.2.2': + resolution: {integrity: sha512-hOLWVbm7uRza0BYXpIIW5pxfrKe0W+D5lrFiAEYR+pb6w3N2SwSMaJbXdUfSEv+dT4MfHBLtn5js0LAWaO6otw==} + + '@types/d3-axis@3.0.6': + resolution: {integrity: sha512-pYeijfZuBd87T0hGn0FO1vQ/cgLk6E1ALJjfkC0oJ8cbwkZl3TpgS8bVBLZN+2jjGgg38epgxb2zmoGtSfvgMw==} + + '@types/d3-brush@3.0.6': + resolution: {integrity: sha512-nH60IZNNxEcrh6L1ZSMNA28rj27ut/2ZmI3r96Zd+1jrZD++zD3LsMIjWlvg4AYrHn/Pqz4CF3veCxGjtbqt7A==} + + '@types/d3-chord@3.0.6': + resolution: {integrity: sha512-LFYWWd8nwfwEmTZG9PfQxd17HbNPksHBiJHaKuY1XeqscXacsS2tyoo6OdRsjf+NQYeB6XrNL3a25E3gH69lcg==} + + '@types/d3-color@3.1.3': + resolution: {integrity: sha512-iO90scth9WAbmgv7ogoq57O9YpKmFBbmoEoCHDB2xMBY0+/KVrqAaCDyCE16dUspeOvIxFFRI+0sEtqDqy2b4A==} + + '@types/d3-contour@3.0.6': + resolution: {integrity: sha512-BjzLgXGnCWjUSYGfH1cpdo41/hgdWETu4YxpezoztawmqsvCeep+8QGfiY6YbDvfgHz/DkjeIkkZVJavB4a3rg==} + + '@types/d3-delaunay@6.0.4': + resolution: {integrity: sha512-ZMaSKu4THYCU6sV64Lhg6qjf1orxBthaC161plr5KuPHo3CNm8DTHiLw/5Eq2b6TsNP0W0iJrUOFscY6Q450Hw==} + + '@types/d3-dispatch@3.0.7': + resolution: {integrity: sha512-5o9OIAdKkhN1QItV2oqaE5KMIiXAvDWBDPrD85e58Qlz1c1kI/J0NcqbEG88CoTwJrYe7ntUCVfeUl2UJKbWgA==} + + '@types/d3-drag@3.0.7': + resolution: {integrity: sha512-HE3jVKlzU9AaMazNufooRJ5ZpWmLIoc90A37WU2JMmeq28w1FQqCZswHZ3xR+SuxYftzHq6WU6KJHvqxKzTxxQ==} + + '@types/d3-dsv@3.0.7': + resolution: {integrity: sha512-n6QBF9/+XASqcKK6waudgL0pf/S5XHPPI8APyMLLUHd8NqouBGLsU8MgtO7NINGtPBtk9Kko/W4ea0oAspwh9g==} + + '@types/d3-ease@3.0.2': + resolution: {integrity: sha512-NcV1JjO5oDzoK26oMzbILE6HW7uVXOHLQvHshBUW4UMdZGfiY6v5BeQwh9a9tCzv+CeefZQHJt5SRgK154RtiA==} + + '@types/d3-fetch@3.0.7': + resolution: {integrity: sha512-fTAfNmxSb9SOWNB9IoG5c8Hg6R+AzUHDRlsXsDZsNp6sxAEOP0tkP3gKkNSO/qmHPoBFTxNrjDprVHDQDvo5aA==} + + '@types/d3-force@3.0.10': + resolution: {integrity: sha512-ZYeSaCF3p73RdOKcjj+swRlZfnYpK1EbaDiYICEEp5Q6sUiqFaFQ9qgoshp5CzIyyb/yD09kD9o2zEltCexlgw==} + + '@types/d3-format@3.0.4': + resolution: {integrity: sha512-fALi2aI6shfg7vM5KiR1wNJnZ7r6UuggVqtDA+xiEdPZQwy/trcQaHnwShLuLdta2rTymCNpxYTiMZX/e09F4g==} + + '@types/d3-geo@3.1.1': + resolution: {integrity: sha512-65Emv9fQiQQqphLlRkuQ5ypPsOmWPhtBGCMv61JDPEPMvsx+gzhGf74yw1a78xFKPj6zw4AgQICJoQv0vK9M2w==} + + '@types/d3-hierarchy@3.1.7': + resolution: {integrity: sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg==} + + '@types/d3-interpolate@3.0.4': + resolution: {integrity: sha512-mgLPETlrpVV1YRJIglr4Ez47g7Yxjl1lj7YKsiMCb27VJH9W8NVM6Bb9d8kkpG/uAQS5AmbA48q2IAolKKo1MA==} + + '@types/d3-path@3.1.1': + resolution: {integrity: sha512-VMZBYyQvbGmWyWVea0EHs/BwLgxc+MKi1zLDCONksozI4YJMcTt8ZEuIR4Sb1MMTE8MMW49v0IwI5+b7RmfWlg==} + + '@types/d3-polygon@3.0.2': + resolution: {integrity: sha512-ZuWOtMaHCkN9xoeEMr1ubW2nGWsp4nIql+OPQRstu4ypeZ+zk3YKqQT0CXVe/PYqrKpZAi+J9mTs05TKwjXSRA==} + + '@types/d3-quadtree@3.0.6': + resolution: {integrity: sha512-oUzyO1/Zm6rsxKRHA1vH0NEDG58HrT5icx/azi9MF1TWdtttWl0UIUsjEQBBh+SIkrpd21ZjEv7ptxWys1ncsg==} + + '@types/d3-random@3.0.4': + resolution: {integrity: sha512-UHYId5WTCx4L4YNel7NU00XUXXgvgpgZOvp10PuvsQENjMDXhh2RyFc0KBjO7B45ne4Ha1yVH7ii0vnzKkuzWA==} + + '@types/d3-scale-chromatic@3.1.0': + resolution: {integrity: sha512-iWMJgwkK7yTRmWqRB5plb1kadXyQ5Sj8V/zYlFGMUBbIPKQScw+Dku9cAAMgJG+z5GYDoMjWGLVOvjghDEFnKQ==} + + '@types/d3-scale@4.0.9': + resolution: {integrity: sha512-dLmtwB8zkAeO/juAMfnV+sItKjlsw2lKdZVVy6LRr0cBmegxSABiLEpGVmSJJ8O08i4+sGR6qQtb6WtuwJdvVw==} + + '@types/d3-selection@3.0.11': + resolution: {integrity: sha512-bhAXu23DJWsrI45xafYpkQ4NtcKMwWnAC/vKrd2l+nxMFuvOT3XMYTIj2opv8vq8AO5Yh7Qac/nSeP/3zjTK0w==} + + '@types/d3-shape@3.2.0': + resolution: {integrity: sha512-kVd74ta9eof3eJOvbNd1vGKS/XERRyQbT26Og63hIsvDO84cjD5gEOhsXf26w3FSoNlPVz84DOFcKv/oou+fMw==} + + '@types/d3-time-format@4.0.3': + resolution: {integrity: sha512-5xg9rC+wWL8kdDj153qZcsJ0FWiFt0J5RB6LYUNZjwSnesfblqrI/bJ1wBdJ8OQfncgbJG5+2F+qfqnqyzYxyg==} + + '@types/d3-time@3.0.4': + resolution: {integrity: sha512-yuzZug1nkAAaBlBBikKZTgzCeA+k1uy4ZFwWANOfKw5z5LRhV0gNA7gNkKm7HoK+HRN0wX3EkxGk0fpbWhmB7g==} + + '@types/d3-timer@3.0.2': + resolution: {integrity: sha512-Ps3T8E8dZDam6fUyNiMkekK3XUsaUEik+idO9/YjPtfj2qruF8tFBXS7XhtE4iIXBLxhmLjP3SXpLhVf21I9Lw==} + + '@types/d3-transition@3.0.9': + resolution: {integrity: sha512-uZS5shfxzO3rGlu0cC3bjmMFKsXv+SmZZcgp0KD22ts4uGXp5EVYGzu/0YdwZeKmddhcAccYtREJKkPfXkZuCg==} + + '@types/d3-zoom@3.0.8': + resolution: {integrity: sha512-iqMC4/YlFCSlO8+2Ii1GGGliCAY4XdeG748w5vQUbevlbDu0zSjH/+jojorQVBK/se0j6DUFNPBGSqD3YWYnDw==} + + '@types/d3@7.4.3': + resolution: {integrity: sha512-lZXZ9ckh5R8uiFVt8ogUNf+pIrK4EsWrx2Np75WvF/eTpJ0FMHNhjXk8CKEx/+gpHbNQyJWehbFaTvqmHWB3ww==} + + '@types/eslint@7.29.0': + resolution: {integrity: sha512-VNcvioYDH8/FxaeTKkM4/TiTwt6pBV9E3OfGmvaw8tPl0rrHCJ4Ll15HRT+pMiFAf/MLQvAzC+6RzUMEL9Ceng==} + + '@types/estree@1.0.9': + resolution: {integrity: sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==} + + '@types/geojson@7946.0.16': + resolution: {integrity: sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==} + + '@types/hast@3.0.5': + resolution: {integrity: sha512-rp/ezSWaD1m44dPKICGhiskI13nVr7qTloFwDa/IYkhhf5nzwP+zIQcIJh3WIFSBOy/H1PzB40jPjMDksN4F+g==} + + '@types/istanbul-lib-coverage@2.0.6': + resolution: {integrity: sha512-2QF/t/auWm0lsy8XtKVPG19v3sSOQlJe/YHZgfjb/KBBHOGSV+J2q/S671rcq9uTBrLAXmZpqJiaQbMT+zNU1w==} + + '@types/json-schema@7.0.15': + resolution: {integrity: sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==} + + '@types/linkify-it@5.0.0': + resolution: {integrity: sha512-sVDA58zAw4eWAffKOaQH5/5j3XeayukzDk+ewSsnv3p4yJEZHCCzMDiZM8e0OUrRvmpGZ85jf4yDHkHsgBNr9Q==} + + '@types/markdown-it@14.2.0': + resolution: {integrity: sha512-NoQ2yGlLWj4wpxMs+TYmRKk3thDrQ97agr7sFqfLsAlvoS8SNQuTrlObhFqG9iugdTtgOE9jpJ6FNM4ZGsa5xQ==} + + '@types/mdast@4.0.4': + resolution: {integrity: sha512-kGaNbPh1k7AFzgpud/gMdvIm5xuECykRR+JnWKQno9TAXVa6WIVCGTPvYGekIDL4uwCZQSYbUxNBSb1aUo79oA==} + + '@types/mdurl@2.0.0': + resolution: {integrity: sha512-RGdgjQUZba5p6QEFAVx2OGb8rQDL/cPRG7GiedRzMcJ1tYnUANBncjbSB1NRGwbvjcPeikRABz2nshyPk1bhWg==} + + '@types/minimist@1.2.5': + resolution: {integrity: sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag==} + + '@types/normalize-package-data@2.4.4': + resolution: {integrity: sha512-37i+OaWTh9qeK4LSHPsyRC7NahnGotNuZvjLSgcPzblpHB3rrCJxAOgI5gCdKm7coonsaX1Of0ILiTcnZjbfxA==} + + '@types/trusted-types@2.0.7': + resolution: {integrity: sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==} + + '@types/unist@3.0.3': + resolution: {integrity: sha512-ko/gIFJRv177XgZsZcBwnqJN5x/Gien8qNOn0D5bQU/zAzVf9Zt3BlcUiLqhV9y4ARk0GbT3tnUiPNgnTXzc/Q==} + + '@types/web-bluetooth@0.0.21': + resolution: {integrity: sha512-oIQLCGWtcFZy2JW77j9k8nHzAOpqMHLQejDA48XXMWH6tjCQHz5RCFz1bzsmROyL6PUm+LLnUiI4BCn221inxA==} + + '@typescript/typescript-aix-ppc64@7.0.2': + resolution: {integrity: sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==} + engines: {node: '>=16.20.0'} + cpu: [ppc64] + os: [aix] + + '@typescript/typescript-darwin-arm64@7.0.2': + resolution: {integrity: sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [darwin] + + '@typescript/typescript-darwin-x64@7.0.2': + resolution: {integrity: sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [darwin] + + '@typescript/typescript-freebsd-arm64@7.0.2': + resolution: {integrity: sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [freebsd] + + '@typescript/typescript-freebsd-x64@7.0.2': + resolution: {integrity: sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [freebsd] + + '@typescript/typescript-linux-arm64@7.0.2': + resolution: {integrity: sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [linux] + + '@typescript/typescript-linux-arm@7.0.2': + resolution: {integrity: sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==} + engines: {node: '>=16.20.0'} + cpu: [arm] + os: [linux] + + '@typescript/typescript-linux-loong64@7.0.2': + resolution: {integrity: sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==} + engines: {node: '>=16.20.0'} + cpu: [loong64] + os: [linux] + + '@typescript/typescript-linux-mips64el@7.0.2': + resolution: {integrity: sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==} + engines: {node: '>=16.20.0'} + cpu: [mips64el] + os: [linux] + + '@typescript/typescript-linux-ppc64@7.0.2': + resolution: {integrity: sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==} + engines: {node: '>=16.20.0'} + cpu: [ppc64] + os: [linux] + + '@typescript/typescript-linux-riscv64@7.0.2': + resolution: {integrity: sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==} + engines: {node: '>=16.20.0'} + cpu: [riscv64] + os: [linux] + + '@typescript/typescript-linux-s390x@7.0.2': + resolution: {integrity: sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==} + engines: {node: '>=16.20.0'} + cpu: [s390x] + os: [linux] + + '@typescript/typescript-linux-x64@7.0.2': + resolution: {integrity: sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [linux] + + '@typescript/typescript-netbsd-arm64@7.0.2': + resolution: {integrity: sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [netbsd] + + '@typescript/typescript-netbsd-x64@7.0.2': + resolution: {integrity: sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [netbsd] + + '@typescript/typescript-openbsd-arm64@7.0.2': + resolution: {integrity: sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [openbsd] + + '@typescript/typescript-openbsd-x64@7.0.2': + resolution: {integrity: sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [openbsd] + + '@typescript/typescript-sunos-x64@7.0.2': + resolution: {integrity: sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [sunos] + + '@typescript/typescript-win32-arm64@7.0.2': + resolution: {integrity: sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==} + engines: {node: '>=16.20.0'} + cpu: [arm64] + os: [win32] + + '@typescript/typescript-win32-x64@7.0.2': + resolution: {integrity: sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==} + engines: {node: '>=16.20.0'} + cpu: [x64] + os: [win32] + + '@ungap/structured-clone@1.4.0': + resolution: {integrity: sha512-1mEZtMKPM09vDmQt5y7YvmN2+DFTP7Tg0EWXdic8/C6VRnpb33e4ghisCIE3WZjsE2N8mf+QV1Zqh7ZFYLWInQ==} + + '@upsetjs/venn.js@2.0.0': + resolution: {integrity: sha512-WbBhLrooyePuQ1VZxrJjtLvTc4NVfpOyKx0sKqioq9bX1C1m7Jgykkn8gLrtwumBioXIqam8DLxp88Adbue6Hw==} + + '@vercel/analytics@2.0.1': + resolution: {integrity: sha512-MTQG6V9qQrt1tsDeF+2Uoo5aPjqbVPys1xvnIftXSJYG2SrwXRHnqEvVoYID7BTruDz4lCd2Z7rM1BdkUehk2g==} + peerDependencies: + '@remix-run/react': ^2 + '@sveltejs/kit': ^1 || ^2 + next: '>= 13' + nuxt: '>= 3' + react: ^18 || ^19 || ^19.0.0-rc + svelte: '>= 4' + vue: ^3 + vue-router: ^4 + peerDependenciesMeta: + '@remix-run/react': + optional: true + '@sveltejs/kit': + optional: true + next: + optional: true + nuxt: + optional: true + react: + optional: true + svelte: + optional: true + vue: + optional: true + vue-router: + optional: true + + '@vercel/speed-insights@2.0.0': + resolution: {integrity: sha512-jwkNcrTeafWxjmWq4AHBaptSqZiJkYU5adLC9QBSqeim0GcqDMgN5Ievh8OG1rJ6W3A4l1oiP7qr9CWxGuzu3w==} + peerDependencies: + '@sveltejs/kit': ^1 || ^2 + next: '>= 13' + nuxt: '>= 3' + react: ^18 || ^19 || ^19.0.0-rc + svelte: '>= 4' + vue: ^3 + vue-router: ^4 + peerDependenciesMeta: + '@sveltejs/kit': + optional: true + next: + optional: true + nuxt: + optional: true + react: + optional: true + svelte: + optional: true + vue: + optional: true + vue-router: + optional: true + + '@vitejs/plugin-vue@5.2.4': + resolution: {integrity: sha512-7Yx/SXSOcQq5HiiV3orevHUFn+pmMB4cgbEkDYgnkUWb0WfeQ/wa2yFv6D5ICiCQOVpjA7vYDXrC7AGO8yjDHA==} + engines: {node: ^18.0.0 || >=20.0.0} + peerDependencies: + vite: ^5.0.0 || ^6.0.0 + vue: ^3.2.25 + + '@vue/compiler-core@3.5.42': + resolution: {integrity: sha512-2Ye1ilMtKXxl8qZUrQ5j0CdgenFp/HFQmta6rfRyfEsTG69L6Wk+tWuNoHYHMx9E8tF2Slvdg1FuwDvAXdy1LQ==} + + '@vue/compiler-dom@3.5.42': + resolution: {integrity: sha512-qbhQZEFmycr+ni/qyuccS4sucNN7VAbDfbkvNxWOX2VfgFm90MNs3/UhRNKoPMEIVn0F8gdlYjLPvqxHwHeQOA==} + + '@vue/compiler-sfc@3.5.42': + resolution: {integrity: sha512-fkCAFB4okcAANGMThboWnScp/gzWjU0ZSkVnjTIiplmMDq2uq0tIB3j+xVu4rhv5rvOgBySCysudmbMd6xRRqw==} + + '@vue/compiler-ssr@3.5.42': + resolution: {integrity: sha512-xmLk3wLkbizPAiLyomjgFFosf2ys9b5Ghb+oh/k2tnvipNz8OFrQOiTcWCzyK7MpBp9KkyGtfvgfLUivbmuGYA==} + + '@vue/devtools-api@7.7.10': + resolution: {integrity: sha512-KxtEpUOOpFz/qOGRrAwA36QF7DqIA+FXgCYit9mk9wjbaZt0sXOFz81ElOZtKA4HbWHUdwNjZHBFsFFyp5BZiA==} + + '@vue/devtools-kit@7.7.10': + resolution: {integrity: sha512-3WNi2Kq4tbpVbmhml7RiphmAt0279oh3fKNeWMQIrltfX8Q91b4i5PL8DtyNKdwmcsGrV4fg+erwWOmD05CLIw==} + + '@vue/devtools-shared@7.7.10': + resolution: {integrity: sha512-wOPslzB8vTvpxwdaOcR2qAbwmuSP0L+rhpoC6Cf56V3Jip+HWb7PQQXOUPgBNQARpXsbQX/+mvi8kKucmBGRwQ==} + + '@vue/reactivity@3.5.42': + resolution: {integrity: sha512-TzNNfKpb7hDxbQltwAut8VDQA5YP+BuRlxntHUuRjyKwlMvmAPbs3unhCvieijifY6vFfVBwsS7wG/C7uq+bEQ==} + + '@vue/runtime-core@3.5.42': + resolution: {integrity: sha512-9uACtuHs7vJGkm5Bp3xu4xRDLFTIYy5DgxpToVjqGIAhAEKwQfsaLvKINhM6nFVp6bZPRFGdDqd1g52MqKsotA==} + + '@vue/runtime-dom@3.5.42': + resolution: {integrity: sha512-rsCmhiWLaRxGltLwhlCWyYkFn7WAbKRh0q17eZ1A6Dq6eqc2ACQ61IIryxz0LrsvCzHSilLA9JHovVwM8CNE2g==} + + '@vue/server-renderer@3.5.42': + resolution: {integrity: sha512-2++5dUyYS4gvo7xQXSECUDhB7TS0aOl5SeVfC5qSq1Jgfhjvegw1zqhwTIR3imZ+QYPJQw9gfcFvXGAjGZ7ajQ==} + + '@vue/shared@3.5.42': + resolution: {integrity: sha512-2rPxex1jQf4jvl9MOHl6YaXCPcrNqz/FstMOEh3QWY+/OME9nQTvl9WYeCwhW7AFjaR0SnngZGlp/wkR6rkI6g==} + + '@vueuse/core@12.8.2': + resolution: {integrity: sha512-HbvCmZdzAu3VGi/pWYm5Ut+Kd9mn1ZHnn4L5G8kOQTPs/IwIAmJoBrmYk2ckLArgMXZj0AW3n5CAejLUO+PhdQ==} + + '@vueuse/integrations@12.8.2': + resolution: {integrity: sha512-fbGYivgK5uBTRt7p5F3zy6VrETlV9RtZjBqd1/HxGdjdckBgBM4ugP8LHpjolqTj14TXTxSK1ZfgPbHYyGuH7g==} + peerDependencies: + async-validator: ^4 + axios: ^1 + change-case: ^5 + drauu: ^0.4 + focus-trap: ^7 + fuse.js: ^7 + idb-keyval: ^6 + jwt-decode: ^4 + nprogress: ^0.2 + qrcode: ^1.5 + sortablejs: ^1 + universal-cookie: ^7 + peerDependenciesMeta: + async-validator: + optional: true + axios: + optional: true + change-case: + optional: true + drauu: + optional: true + focus-trap: + optional: true + fuse.js: + optional: true + idb-keyval: + optional: true + jwt-decode: + optional: true + nprogress: + optional: true + qrcode: + optional: true + sortablejs: + optional: true + universal-cookie: + optional: true + + '@vueuse/metadata@12.8.2': + resolution: {integrity: sha512-rAyLGEuoBJ/Il5AmFHiziCPdQzRt88VxR+Y/A/QhJ1EWtWqPBBAxTAFaSkviwEuOEZNtW8pvkPgoCZQ+HxqW1A==} + + '@vueuse/shared@12.8.2': + resolution: {integrity: sha512-dznP38YzxZoNloI0qpEfpkms8knDtaoQ6Y/sfS0L7Yki4zh40LFHEhur0odJC6xTHG5dxWVPiUWBXn+wCG2s5w==} + + algoliasearch@5.57.0: + resolution: {integrity: sha512-HpND7MBGctOAkd1GoQoDZCGoCpqNTS5NG1LuhElFet3RdLJkwnyTYZXZhXwtpAQPrI36fqQ3eT6KQrdKDTKu3A==} + engines: {node: '>= 14.0.0'} + + ansi-escapes@4.3.2: + resolution: {integrity: sha512-gKXj5ALrKWQLsYG9jlTRmR/xKluxHV+Z9QEwNIgCfM1/uwPMCuzVVnh5mwTd+OuBZcwSIMbqssNWRm1lE51QaQ==} + engines: {node: '>=8'} + + ansi-regex@5.0.1: + resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==} + engines: {node: '>=8'} + + ansi-regex@6.3.0: + resolution: {integrity: sha512-WpDfL7NO6j7tH88IDBNVdUJxDh9nmCteAVW9dsep846XdwF4naCBK+/tGLX3KJgcpgMRXCFlTM2hKGoK9FsdrQ==} + engines: {node: '>=12'} + + ansi-styles@4.3.0: + resolution: {integrity: sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==} + engines: {node: '>=8'} + + ansi-styles@5.2.0: + resolution: {integrity: sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA==} + engines: {node: '>=10'} + + ansi-styles@6.2.3: + resolution: {integrity: sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==} + engines: {node: '>=12'} + + array-union@2.1.0: + resolution: {integrity: sha512-HGyxoOTYUyCM6stUe6EJgnd4EoewAI7zMdfqO+kGjnlZmBDz/cR5pf8r/cR4Wq60sL/p0IkcjUEEPwS3GFrIyw==} + engines: {node: '>=8'} + + arrify@1.0.1: + resolution: {integrity: sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==} + engines: {node: '>=0.10.0'} + + atomic-sleep@1.0.0: + resolution: {integrity: sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==} + engines: {node: '>=8.0.0'} + + balanced-match@1.0.2: + resolution: {integrity: sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==} + + balanced-match@4.0.4: + resolution: {integrity: sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==} + engines: {node: 18 || 20 || >=22} + + birpc@2.9.0: + resolution: {integrity: sha512-KrayHS5pBi69Xi9JmvoqrIgYGDkD6mcSe/i6YKi3w5kekCLzrX4+nawcXqrj2tIp50Kw/mT/s3p+GVK0A0sKxw==} + + brace-expansion@2.1.4: + resolution: {integrity: sha512-hGfVzPxthbf3+2yjg/RBs60cB0FhqBS/zvdV/4wn4/BmN0bNMMHPc4V/BbFieqf1TKAGGAHnY4eSjajCl0f2Xg==} + + brace-expansion@5.0.9: + resolution: {integrity: sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==} + engines: {node: 20 || >=22} + + braces@3.0.3: + resolution: {integrity: sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==} + engines: {node: '>=8'} + + c8@10.1.3: + resolution: {integrity: sha512-LvcyrOAaOnrrlMpW22n690PUvxiq4Uf9WMhQwNJ9vgagkL/ph1+D4uvjvDA5XCbykrc0sx+ay6pVi9YZ1GnhyA==} + engines: {node: '>=18'} + hasBin: true + peerDependencies: + monocart-coverage-reports: ^2 + peerDependenciesMeta: + monocart-coverage-reports: + optional: true + + camelcase-keys@6.2.2: + resolution: {integrity: sha512-YrwaA0vEKazPBkn0ipTiMpSajYDSe+KjQfrjhcBMxJt/znbvlHd8Pw/Vamaz5EB4Wfhs3SUR3Z9mwRu/P3s3Yg==} + engines: {node: '>=8'} + + camelcase@5.3.1: + resolution: {integrity: sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg==} + engines: {node: '>=6'} + + ccount@2.0.1: + resolution: {integrity: sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==} + + chalk@4.1.2: + resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} + engines: {node: '>=10'} + + character-entities-html4@2.1.0: + resolution: {integrity: sha512-1v7fgQRj6hnSwFpq1Eu0ynr/CDEw0rXo2B61qXrLNdHZmPKgb7fqS1a2JwF0rISo9q77jDI8VMEHoApn8qDoZA==} + + character-entities-legacy@3.0.0: + resolution: {integrity: sha512-RpPp0asT/6ufRm//AJVwpViZbGM/MkjQFxJccQRHmISF/22NBtsHqAWmL+/pmkPWoIUJdWyeVleTl1wydHATVQ==} + + cliui@8.0.1: + resolution: {integrity: sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==} + engines: {node: '>=12'} + + color-convert@2.0.1: + resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==} + engines: {node: '>=7.0.0'} + + color-name@1.1.4: + resolution: {integrity: sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==} + + comma-separated-tokens@2.0.3: + resolution: {integrity: sha512-Fu4hJdvzeylCfQPp9SGWidpzrMs7tTrlu6Vb8XGaRGck8QSNZJJp538Wrb60Lax4fPwR64ViY468OIUTbRlGZg==} + + commander@7.2.0: + resolution: {integrity: sha512-QrWXB+ZQSVPmIWIhtEO9H+gwHaMGYiF5ChvoJ+K9ZGHG/sVsa6yiesAD1GC/x46sET00Xlwo1u49RVVVzvcSkw==} + engines: {node: '>= 10'} + + commander@8.3.0: + resolution: {integrity: sha512-OkTL9umf+He2DZkUq8f8J9of7yL6RJKI24dVITBmNfZBmri9zYZQrKkuXiKhyfPSu8tUhnVBB1iKXevvnlR4Ww==} + engines: {node: '>= 12'} + + convert-source-map@2.0.0: + resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} + + copy-anything@4.1.0: + resolution: {integrity: sha512-ufbM3smX/Jbnpk5wcQjzd1MgBpzmqfNETUAyZNrGwU9foRlyHoGzMMBBCRzEhQLBjZfFDE1W2ufPXX2vdWkV8Q==} + engines: {node: '>=18'} + + cose-base@1.0.3: + resolution: {integrity: sha512-s9whTXInMSgAp/NVXVNuVxVKzGH2qck3aQlVHxDCdAEPgtMKwc4Wq6/QKhgdEdgbLSi9rBTAcPoRa6JpiG4ksg==} + + cose-base@2.2.0: + resolution: {integrity: sha512-AzlgcsCbUMymkADOJtQm3wO9S3ltPfYOFD5033keQn9NJzIbtnZj+UdBJe7DYml/8TdbtHJW3j58SOnKhWY/5g==} + + cross-spawn@7.0.6: + resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==} + engines: {node: '>= 8'} + + csstype@3.2.3: + resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==} + + cytoscape-cose-bilkent@4.1.0: + resolution: {integrity: sha512-wgQlVIUJF13Quxiv5e1gstZ08rnZj2XaLHGoFMYXz7SkNfCDOOteKBE6SYRfA9WxxI/iBc3ajfDoc6hb/MRAHQ==} + peerDependencies: + cytoscape: ^3.2.0 + + cytoscape-fcose@2.2.0: + resolution: {integrity: sha512-ki1/VuRIHFCzxWNrsshHYPs6L7TvLu3DL+TyIGEsRcvVERmxokbf5Gdk7mFxZnTdiGtnA4cfSmjZJMviqSuZrQ==} + peerDependencies: + cytoscape: ^3.2.0 + + cytoscape@3.34.2: + resolution: {integrity: sha512-Cm2jaj1X/PBNlzV9yH8zcfGOxO7U+CJ/+mxSBVPSchLaugdp4jtlGx5qaHtPRZ6tgiZ5P+o1XoRfJA+ba6KM3g==} + engines: {node: '>=0.10'} + + d3-array@2.12.1: + resolution: {integrity: sha512-B0ErZK/66mHtEsR1TkPEEkwdy+WDesimkM5gpZr5Dsg54BiTA5RXtYW5qTLIAcekaS9xfZrzBLF/OAkB3Qn1YQ==} + + d3-array@3.2.4: + resolution: {integrity: sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==} + engines: {node: '>=12'} + + d3-axis@3.0.0: + resolution: {integrity: sha512-IH5tgjV4jE/GhHkRV0HiVYPDtvfjHQlQfJHs0usq7M30XcSBvOotpmH1IgkcXsO/5gEQZD43B//fc7SRT5S+xw==} + engines: {node: '>=12'} + + d3-brush@3.0.0: + resolution: {integrity: sha512-ALnjWlVYkXsVIGlOsuWH1+3udkYFI48Ljihfnh8FZPF2QS9o+PzGLBslO0PjzVoHLZ2KCVgAM8NVkXPJB2aNnQ==} + engines: {node: '>=12'} + + d3-chord@3.0.1: + resolution: {integrity: sha512-VE5S6TNa+j8msksl7HwjxMHDM2yNK3XCkusIlpX5kwauBfXuyLAtNg9jCp/iHH61tgI4sb6R/EIMWCqEIdjT/g==} + engines: {node: '>=12'} + + d3-color@3.1.0: + resolution: {integrity: sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==} + engines: {node: '>=12'} + + d3-contour@4.0.2: + resolution: {integrity: sha512-4EzFTRIikzs47RGmdxbeUvLWtGedDUNkTcmzoeyg4sP/dvCexO47AaQL7VKy/gul85TOxw+IBgA8US2xwbToNA==} + engines: {node: '>=12'} + + d3-delaunay@6.0.4: + resolution: {integrity: sha512-mdjtIZ1XLAM8bm/hx3WwjfHt6Sggek7qH043O8KEjDXN40xi3vx/6pYSVTwLjEgiXQTbvaouWKynLBiUZ6SK6A==} + engines: {node: '>=12'} + + d3-dispatch@3.0.1: + resolution: {integrity: sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==} + engines: {node: '>=12'} + + d3-drag@3.0.0: + resolution: {integrity: sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==} + engines: {node: '>=12'} + + d3-dsv@3.0.1: + resolution: {integrity: sha512-UG6OvdI5afDIFP9w4G0mNq50dSOsXHJaRE8arAS5o9ApWnIElp8GZw1Dun8vP8OyHOZ/QJUKUJwxiiCCnUwm+Q==} + engines: {node: '>=12'} + hasBin: true + + d3-ease@3.0.1: + resolution: {integrity: sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==} + engines: {node: '>=12'} + + d3-fetch@3.0.1: + resolution: {integrity: sha512-kpkQIM20n3oLVBKGg6oHrUchHM3xODkTzjMoj7aWQFq5QEM+R6E4WkzT5+tojDY7yjez8KgCBRoj4aEr99Fdqw==} + engines: {node: '>=12'} + + d3-force@3.0.0: + resolution: {integrity: sha512-zxV/SsA+U4yte8051P4ECydjD/S+qeYtnaIyAs9tgHCqfguma/aAQDjo85A9Z6EKhBirHRJHXIgJUlffT4wdLg==} + engines: {node: '>=12'} + + d3-format@3.1.2: + resolution: {integrity: sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==} + engines: {node: '>=12'} + + d3-geo@3.1.1: + resolution: {integrity: sha512-637ln3gXKXOwhalDzinUgY83KzNWZRKbYubaG+fGVuc/dxO64RRljtCTnf5ecMyE1RIdtqpkVcq0IbtU2S8j2Q==} + engines: {node: '>=12'} + + d3-hierarchy@3.1.2: + resolution: {integrity: sha512-FX/9frcub54beBdugHjDCdikxThEqjnR93Qt7PvQTOHxyiNCAlvMrHhclk3cD5VeAaq9fxmfRp+CnWw9rEMBuA==} + engines: {node: '>=12'} + + d3-interpolate@3.0.1: + resolution: {integrity: sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==} + engines: {node: '>=12'} + + d3-path@1.0.9: + resolution: {integrity: sha512-VLaYcn81dtHVTjEHd8B+pbe9yHWpXKZUC87PzoFmsFrJqgFwDe/qxfp5MlfsfM1V5E/iVt0MmEbWQ7FVIXh/bg==} + + d3-path@3.1.0: + resolution: {integrity: sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==} + engines: {node: '>=12'} + + d3-polygon@3.0.1: + resolution: {integrity: sha512-3vbA7vXYwfe1SYhED++fPUQlWSYTTGmFmQiany/gdbiWgU/iEyQzyymwL9SkJjFFuCS4902BSzewVGsHHmHtXg==} + engines: {node: '>=12'} + + d3-quadtree@3.0.1: + resolution: {integrity: sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==} + engines: {node: '>=12'} + + d3-random@3.0.1: + resolution: {integrity: sha512-FXMe9GfxTxqd5D6jFsQ+DJ8BJS4E/fT5mqqdjovykEB2oFbTMDVdg1MGFxfQW+FBOGoB++k8swBrgwSHT1cUXQ==} + engines: {node: '>=12'} + + d3-sankey@0.12.3: + resolution: {integrity: sha512-nQhsBRmM19Ax5xEIPLMY9ZmJ/cDvd1BG3UVvt5h3WRxKg5zGRbvnteTyWAbzeSvlh3tW7ZEmq4VwR5mB3tutmQ==} + + d3-scale-chromatic@3.1.0: + resolution: {integrity: sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==} + engines: {node: '>=12'} + + d3-scale@4.0.2: + resolution: {integrity: sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==} + engines: {node: '>=12'} + + d3-selection@3.0.0: + resolution: {integrity: sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==} + engines: {node: '>=12'} + + d3-shape@1.3.7: + resolution: {integrity: sha512-EUkvKjqPFUAZyOlhY5gzCxCeI0Aep04LwIRpsZ/mLFelJiUfnK56jo5JMDSE7yyP2kLSb6LtF+S5chMk7uqPqw==} + + d3-shape@3.2.0: + resolution: {integrity: sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==} + engines: {node: '>=12'} + + d3-time-format@4.1.0: + resolution: {integrity: sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==} + engines: {node: '>=12'} + + d3-time@3.1.0: + resolution: {integrity: sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==} + engines: {node: '>=12'} + + d3-timer@3.0.1: + resolution: {integrity: sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==} + engines: {node: '>=12'} + + d3-transition@3.0.1: + resolution: {integrity: sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==} + engines: {node: '>=12'} + peerDependencies: + d3-selection: 2 - 3 + + d3-zoom@3.0.0: + resolution: {integrity: sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==} + engines: {node: '>=12'} + + d3@7.9.0: + resolution: {integrity: sha512-e1U46jVP+w7Iut8Jt8ri1YsPOvFpg46k+K8TpCb0P+zjCkjkPnV7WzfDJzMHy1LnA+wj5pLT1wjO901gLXeEhA==} + engines: {node: '>=12'} + + dagre-d3-es@7.0.14: + resolution: {integrity: sha512-P4rFMVq9ESWqmOgK+dlXvOtLwYg0i7u0HBGJER0LZDJT2VHIPAMZ/riPxqJceWMStH5+E61QxFra9kIS3AqdMg==} + + dayjs@1.11.23: + resolution: {integrity: sha512-QDTCU0M0MxR3hQfnlDJfwekQiaanm1ubOD231u73WBckQ/fsamwRLiE2GBz6D3a/xF1NgfiDLJjXBa1hYOYTtQ==} + + decamelize-keys@1.1.1: + resolution: {integrity: sha512-WiPxgEirIV0/eIOMcnFBA3/IJZAZqKnwAwWyvvdi4lsr1WCN22nhdf/3db3DoZcUjTV2SqfzIwNyp6y2xs3nmg==} + engines: {node: '>=0.10.0'} + + decamelize@1.2.0: + resolution: {integrity: sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA==} + engines: {node: '>=0.10.0'} + + delaunator@5.1.0: + resolution: {integrity: sha512-AGrQ4QSgssa1NGmWmLPqN5NY2KajF5MqxetNEO+o0n3ZwZZeTmt7bBnvzHWrmkZFxGgr4HdyFgelzgi06otLuQ==} + + dequal@2.0.3: + resolution: {integrity: sha512-0je+qPKHEMohvfRTCEo3CrPG6cAzAYgmzKyxRiYSSDkS6eGJdyVJm7WaYA5ECaAD9wLB2T4EEeymA5aFVcYXCA==} + engines: {node: '>=6'} + + devlop@1.1.0: + resolution: {integrity: sha512-RWmIqhcFf1lRYBvNmr7qTNuyCt/7/ns2jbpp1+PalgE/rDQcBT0fioSMUpJ93irlUhC5hrg4cYqe6U+0ImW0rA==} + + diff-sequences@29.6.3: + resolution: {integrity: sha512-EjePK1srD3P08o2j4f0ExnylqRs5B9tJjcp9t1krH2qRi8CCdsYfwe9JgSLurFBWwq4uOlipzfk5fHNvwFKr8Q==} + engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0} + + dir-glob@3.0.1: + resolution: {integrity: sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==} + engines: {node: '>=8'} + + dompurify@3.4.14: + resolution: {integrity: sha512-dVoH9z+MY+C9IilgGCk3YfFqjLi3fChm2OiKJMzh6axrJ5qwxqWaZamgmHrpv22CN/KdbZJuGEGgfQoL00LTdg==} + + eastasianwidth@0.2.0: + resolution: {integrity: sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==} + + emoji-regex-xs@1.0.0: + resolution: {integrity: sha512-LRlerrMYoIDrT6jgpeZ2YYl/L8EulRTt5hQcYjy5AInh7HWXKimpqx68aknBFpGL2+/IcogTcaydJEgaTmOpDg==} + + emoji-regex@8.0.0: + resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} + + emoji-regex@9.2.2: + resolution: {integrity: sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==} + + entities@7.0.1: + resolution: {integrity: sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==} + engines: {node: '>=0.12'} + + error-ex@1.3.4: + resolution: {integrity: sha512-sqQamAnR14VgCr1A618A3sGrygcpK+HEbenA/HiEAkkUwcZIIB/tgWqHFxWgOyDh4nB4JCRimh79dR5Ywc9MDQ==} + + es-errors@1.3.0: + resolution: {integrity: sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==} + engines: {node: '>= 0.4'} + + es-toolkit@1.52.0: + resolution: {integrity: sha512-XTNEJQh1tY1ZJVcf6ayP/2n4ZPyaHlW2FWs7xvw5ddPuhUVjLD3olQVQS7kf58JbAB48iL0uL/jerTrjtV3lDA==} + + esbuild@0.21.5: + resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==} + engines: {node: '>=12'} + hasBin: true + + esbuild@0.28.2: + resolution: {integrity: sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==} + engines: {node: '>=18'} + hasBin: true + + escalade@3.2.0: + resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} + engines: {node: '>=6'} + + eslint-formatter-pretty@4.1.0: + resolution: {integrity: sha512-IsUTtGxF1hrH6lMWiSl1WbGaiP01eT6kzywdY1U+zLc0MP+nwEnUiS9UI8IaOTUhTeQJLlCEWIbXINBH4YJbBQ==} + engines: {node: '>=10'} + + eslint-rule-docs@1.1.235: + resolution: {integrity: sha512-+TQ+x4JdTnDoFEXXb3fDvfGOwnyNV7duH8fXWTPD1ieaBmB8omj7Gw/pMBBu4uI2uJCCU8APDaQJzWuXnTsH4A==} + + estree-walker@2.0.2: + resolution: {integrity: sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w==} + + fast-glob@3.3.3: + resolution: {integrity: sha512-7MptL8U0cqcFdzIzwOTHoilX9x5BrNqye7Z/LuC7kCMRio1EMSyqRK3BEAUD7sXRq4iT4AzTVuZdhgQ2TCvYLg==} + engines: {node: '>=8.6.0'} + + fastdom@1.0.12: + resolution: {integrity: sha512-LB+xjSTEbjHE1cWsxu+tN2Xqr1kpi+V9aADI7sVM5ZMaXyYGPHULQMzpJMYqOTULK/73pUkWVzzObFRBkPr+hg==} + + fastq@1.20.3: + resolution: {integrity: sha512-XKv5nnLs6nLF71NgiKJLIZFLkPyIEuOselLG7ujZnGrRfQK8HpvY+WqKhAJUAdLomwVHErVS4LfxFlPq0/FTAw==} + + fill-range@7.1.1: + resolution: {integrity: sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==} + engines: {node: '>=8'} + + find-up@4.1.0: + resolution: {integrity: sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw==} + engines: {node: '>=8'} + + find-up@5.0.0: + resolution: {integrity: sha512-78/PXT1wlLLDgTzDs7sjq9hzz0vXD+zn+7wypEe4fXQxCmdmqfGsEPQxmiCSQI3ajFV91bVSsvNtrJRiW6nGng==} + engines: {node: '>=10'} + + focus-trap@7.8.0: + resolution: {integrity: sha512-/yNdlIkpWbM0ptxno3ONTuf+2g318kh2ez3KSeZN5dZ8YC6AAmgeWz+GasYYiBJPFaYcSAPeu4GfhUaChzIJXA==} + + foreground-child@3.3.1: + resolution: {integrity: sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==} + engines: {node: '>=14'} + + fsevents@2.3.3: + resolution: {integrity: sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==} + engines: {node: ^8.16.0 || ^10.6.0 || >=11.0.0} + os: [darwin] + + function-bind@1.1.2: + resolution: {integrity: sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==} + + get-caller-file@2.0.5: + resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==} + engines: {node: 6.* || 8.* || >= 10.*} + + glob-parent@5.1.2: + resolution: {integrity: sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==} + engines: {node: '>= 6'} + + glob@10.5.0: + resolution: {integrity: sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==} + deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me + hasBin: true + + globby@11.1.0: + resolution: {integrity: sha512-jhIXaOzy1sb8IyocaruWSn1TjmnBVs8Ayhcy83rmxNJ8q2uWKCAj3CnJY+KpGSXCueAPc0i05kVvVKtP1t9S3g==} + engines: {node: '>=10'} + + hachure-fill@0.5.2: + resolution: {integrity: sha512-3GKBOn+m2LX9iq+JC1064cSFprJY4jL1jCXTcpnfER5HYE2l/4EfWSGzkPa/ZDBmYI0ZOEj5VHV/eKnPGkHuOg==} + + hard-rejection@2.1.0: + resolution: {integrity: sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA==} + engines: {node: '>=6'} + + has-flag@4.0.0: + resolution: {integrity: sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==} + engines: {node: '>=8'} + + hasown@2.0.4: + resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==} + engines: {node: '>= 0.4'} + + hast-util-to-html@9.0.5: + resolution: {integrity: sha512-OguPdidb+fbHQSU4Q4ZiLKnzWo8Wwsf5bZfbvu7//a9oTYoqD/fWpe96NuHkoS9h0ccGOTe0C4NGXdtS0iObOw==} + + hast-util-whitespace@3.0.0: + resolution: {integrity: sha512-88JUN06ipLwsnv+dVn+OIYOvAuvBMy/Qoi6O7mQHxdPXpjy+Cd6xRkWwux7DKO+4sYILtLBRIKgsdpS2gQc7qw==} + + hookable@5.5.3: + resolution: {integrity: sha512-Yc+BQe8SvoXH1643Qez1zqLRmbA5rCL+sSmk6TVos0LWVfNIB7PGncdlId77WzLGSIB5KaWgTaNTs2lNVEI6VQ==} + + hosted-git-info@2.8.9: + resolution: {integrity: sha512-mxIDAb9Lsm6DoOJ7xH+5+X4y1LU/4Hi50L9C5sIswK3JzULS4bwk1FvjdBgvYR4bzT4tuUQiC15FE2f5HbLvYw==} + + hosted-git-info@4.1.0: + resolution: {integrity: sha512-kyCuEOWjJqZuDbRHzL8V93NzQhwIB71oFWSyzVo+KPZI+pnQPPxucdkrOZvkLRnrf5URsQM+IJ09Dw29cRALIA==} + engines: {node: '>=10'} + + html-escaper@2.0.2: + resolution: {integrity: sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==} + + html-void-elements@3.0.0: + resolution: {integrity: sha512-bEqo66MRXsUGxWHV5IP0PUiAWwoEjba4VCzg0LjFJBpchPaTfyfCKTG6bc5F8ucKec3q5y6qOdGyYTSBEvhCrg==} + + iconv-lite@0.6.3: + resolution: {integrity: sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==} + engines: {node: '>=0.10.0'} + + ignore@5.3.2: + resolution: {integrity: sha512-hsBTNUqQTDwkWtcdYI2i06Y/nUBEsNEDJKjWdigLvegy8kDuJAS8uRlpkkcQpyEXL0Z/pjDy5HBmMjRCJ2gq+g==} + engines: {node: '>= 4'} + + import-meta-resolve@4.2.0: + resolution: {integrity: sha512-Iqv2fzaTQN28s/FwZAoFq0ZSs/7hMAHJVX+w8PZl3cY19Pxk6jFFalxQoIfW2826i/fDLXv8IiEZRIT0lDuWcg==} + + indent-string@4.0.0: + resolution: {integrity: sha512-EdDDZu4A2OyIK7Lr/2zG+w5jmbuk1DVBnEwREQvBzspBJkCEbRa8GxU1lghYcaGJCnRWibjDXlq779X1/y5xwg==} + engines: {node: '>=8'} + + internmap@1.0.1: + resolution: {integrity: sha512-lDB5YccMydFBtasVtxnZ3MRBHuaoE8GKsppq+EchKL2U4nK/DmEpPHNH8MZe5HkMtpSiTSOZwfN0tzYjO/lJEw==} + + internmap@2.0.3: + resolution: {integrity: sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==} + engines: {node: '>=12'} + + irregular-plurals@3.5.0: + resolution: {integrity: sha512-1ANGLZ+Nkv1ptFb2pa8oG8Lem4krflKuX/gINiHJHjJUKaJHk/SXk5x6K3J+39/p0h1RQ2saROclJJ+QLvETCQ==} + engines: {node: '>=8'} + + is-arrayish@0.2.1: + resolution: {integrity: sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==} + + is-core-module@2.16.2: + resolution: {integrity: sha512-evOr8xfXKxE6qSR0hSXL2r3sd7ALj8+7jQEUvPYcm5sgZFdJ+AYzT6yNmJenvIYQBgIGwfwz08sL8zoL7yq2BA==} + engines: {node: '>= 0.4'} + + is-extglob@2.1.1: + resolution: {integrity: sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==} + engines: {node: '>=0.10.0'} + + is-fullwidth-code-point@3.0.0: + resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==} + engines: {node: '>=8'} + + is-glob@4.0.3: + resolution: {integrity: sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==} + engines: {node: '>=0.10.0'} + + is-number@7.0.0: + resolution: {integrity: sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==} + engines: {node: '>=0.12.0'} + + is-plain-obj@1.1.0: + resolution: {integrity: sha512-yvkRyxmFKEOQ4pNXCmJG5AEQNlXJS5LaONXo5/cLdTZdWvsZ1ioJEonLGAosKlMWE8lwUy/bJzMjcw8az73+Fg==} + engines: {node: '>=0.10.0'} + + is-unicode-supported@0.1.0: + resolution: {integrity: sha512-knxG2q4UC3u8stRGyAVJCOdxFmv5DZiRcdlIaAQXAbSfJya+OhopNotLQrstBhququ4ZpuKbDc/8S6mgXgPFPw==} + engines: {node: '>=10'} + + isexe@2.0.0: + resolution: {integrity: sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==} + + istanbul-lib-coverage@3.2.2: + resolution: {integrity: sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==} + engines: {node: '>=8'} + + istanbul-lib-report@3.0.1: + resolution: {integrity: sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==} + engines: {node: '>=10'} + + istanbul-reports@3.2.0: + resolution: {integrity: sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA==} + engines: {node: '>=8'} + + jackspeak@3.4.3: + resolution: {integrity: sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==} + + jest-diff@29.7.0: + resolution: {integrity: sha512-LMIgiIrhigmPrs03JHpxUh2yISK3vLFPkAodPeo0+BuF7wA2FoQbkEg1u8gBYBThncu7e1oEDUfIXVuTqLRUjw==} + engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0} + + jest-get-type@29.6.3: + resolution: {integrity: sha512-zrteXnqYxfQh7l5FHyL38jL39di8H8rHoecLH3JNxH3BwOrBsNeabdap5e0I23lD4HHI8W5VFBZqG4Eaq5LNcw==} + engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0} + + js-tokens@4.0.0: + resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==} + + json-parse-even-better-errors@2.3.1: + resolution: {integrity: sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==} + + katex@0.16.47: + resolution: {integrity: sha512-Eeo8Ys1doU1z+x8AZsPpQu+p/QcZBI5PeOo7QGQdy2x2m0MU/hYagBbGOmXwr5KVbEfVuWv9LpnQWeehogurjg==} + hasBin: true + + khroma@2.1.0: + resolution: {integrity: sha512-Ls993zuzfayK269Svk9hzpeGUKob/sIgZzyHYdjQoAdQetRKpOLj+k/QQQ/6Qi0Yz65mlROrfd+Ev+1+7dz9Kw==} + + kind-of@6.0.3: + resolution: {integrity: sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==} + engines: {node: '>=0.10.0'} + + layout-base@1.0.2: + resolution: {integrity: sha512-8h2oVEZNktL4BH2JCOI90iD1yXwL6iNW7KcCKT2QZgQJR2vbqDsldCTPRU9NifTCqHZci57XvQQ15YTu+sTYPg==} + + layout-base@2.0.1: + resolution: {integrity: sha512-dp3s92+uNI1hWIpPGH3jK2kxE2lMjdXdr+DH8ynZHpd6PUlH6x6cbuXnoMmiNumznqaNO31xu9e79F0uuZ0JFg==} + + lines-and-columns@1.2.4: + resolution: {integrity: sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg==} + + locate-path@5.0.0: + resolution: {integrity: sha512-t7hw9pI+WvuwNJXwk5zVHpyhIqzg2qTlklJOf0mVxGSbe3Fp2VieZcduNYjaLDoy6p9uGpQEGWG87WpMKlNq8g==} + engines: {node: '>=8'} + + locate-path@6.0.0: + resolution: {integrity: sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw==} + engines: {node: '>=10'} + + lodash-es@4.18.1: + resolution: {integrity: sha512-J8xewKD/Gk22OZbhpOVSwcs60zhd95ESDwezOFuA3/099925PdHJ7OFHNTGtajL3AlZkykD32HykiMo+BIBI8A==} + + log-symbols@4.1.0: + resolution: {integrity: sha512-8XPvpAA8uyhfteu8pIvQxpJZ7SYYdpUivZpGy6sFsBuKRY/7rQGavedeB8aK+Zkyq6upMFVL/9AW6vOYzfRyLg==} + engines: {node: '>=10'} + + lru-cache@10.4.3: + resolution: {integrity: sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==} + + lru-cache@6.0.0: + resolution: {integrity: sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==} + engines: {node: '>=10'} + + magic-string@0.30.21: + resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==} + + make-dir@4.0.0: + resolution: {integrity: sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==} + engines: {node: '>=10'} + + map-obj@1.0.1: + resolution: {integrity: sha512-7N/q3lyZ+LVCp7PzuxrJr4KMbBE2hW7BT7YNia330OFxIf4d3r5zVpicP2650l7CPN6RM9zOJRl3NGpqSiw3Eg==} + engines: {node: '>=0.10.0'} + + map-obj@4.3.0: + resolution: {integrity: sha512-hdN1wVrZbb29eBGiGjJbeP8JbKjq1urkHJ/LIP/NY48MZ1QVXUsQBV1G1zvYFHn1XE06cwjBsOI2K3Ulnj1YXQ==} + engines: {node: '>=8'} + + mark.js@8.11.1: + resolution: {integrity: sha512-1I+1qpDt4idfgLQG+BNWmrqku+7/2bi5nLf4YwF8y8zXvmfiTBY3PV3ZibfrjBueCByROpuBjLLFCajqkgYoLQ==} + + marked@16.4.2: + resolution: {integrity: sha512-TI3V8YYWvkVf3KJe1dRkpnjs68JUPyEa5vjKrp1XEEJUAOaQc+Qj+L1qWbPd0SJuAdQkFU0h73sXXqwDYxsiDA==} + engines: {node: '>= 20'} + hasBin: true + + mdast-util-to-hast@13.2.1: + resolution: {integrity: sha512-cctsq2wp5vTsLIcaymblUriiTcZd0CwWtCbLvrOzYCDZoWyMNV8sZ7krj09FSnsiJi3WVsHLM4k6Dq/yaPyCXA==} + + meow@9.0.0: + resolution: {integrity: sha512-+obSblOQmRhcyBt62furQqRAQpNyWXo8BuQ5bN7dG8wmwQ+vwHKp/rCFD4CrTP8CsDQD1sjoZ94K417XEUk8IQ==} + engines: {node: '>=10'} + + merge2@1.4.1: + resolution: {integrity: sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==} + engines: {node: '>= 8'} + + mermaid@11.17.2: + resolution: {integrity: sha512-V6K3C8EBdEsPFZXSKMJe6ppQOENxuHARr9GvHX4hh47lAbhMRD9qf4oEK7LoaRQxULMa80/qt5gHO73aCleBBg==} + + micromark-util-character@2.1.1: + resolution: {integrity: sha512-wv8tdUTJ3thSFFFJKtpYKOYiGP2+v96Hvk4Tu8KpCAsTMs6yi+nVmGh1syvSCsaxz45J6Jbw+9DD6g97+NV67Q==} + + micromark-util-encode@2.0.1: + resolution: {integrity: sha512-c3cVx2y4KqUnwopcO9b/SCdo2O67LwJJ/UyqGfbigahfegL9myoEFoDYZgkT7f36T0bLrM9hZTAaAyH+PCAXjw==} + + micromark-util-sanitize-uri@2.0.1: + resolution: {integrity: sha512-9N9IomZ/YuGGZZmQec1MbgxtlgougxTodVwDzzEouPKo3qFWvymFHWcnDi2vzV1ff6kas9ucW+o3yzJK9YB1AQ==} + + micromark-util-symbol@2.0.1: + resolution: {integrity: sha512-vs5t8Apaud9N28kgCrRUdEed4UJ+wWNvicHLPxCa9ENlYuAY31M0ETy5y1vA33YoNPDFTghEbnh6efaE8h4x0Q==} + + micromark-util-types@2.0.2: + resolution: {integrity: sha512-Yw0ECSpJoViF1qTU4DC6NwtC4aWGt1EkzaQB8KPPyCRR8z9TWeV0HbEFGTO+ZY1wB22zmxnJqhPyTpOVCpeHTA==} + + micromatch@4.0.8: + resolution: {integrity: sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA==} + engines: {node: '>=8.6'} + + min-indent@1.0.1: + resolution: {integrity: sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg==} + engines: {node: '>=4'} + + minimatch@10.2.6: + resolution: {integrity: sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==} + engines: {node: 18 || 20 || >=22} + + minimatch@9.0.9: + resolution: {integrity: sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg==} + engines: {node: '>=16 || 14 >=14.17'} + + minimist-options@4.1.0: + resolution: {integrity: sha512-Q4r8ghd80yhO/0j1O3B2BjweX3fiHg9cdOwjJd2J76Q135c+NDxGCqdYKQ1SKBuFfgWbAUzBfvYjPUEeNgqN1A==} + engines: {node: '>= 6'} + + minipass@7.1.3: + resolution: {integrity: sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==} + engines: {node: '>=16 || 14 >=14.17'} + + minisearch@7.2.0: + resolution: {integrity: sha512-dqT2XBYUOZOiC5t2HRnwADjhNS2cecp9u+TJRiJ1Qp/f5qjkeT5APcGPjHw+bz89Ms8Jp+cG4AlE+QZ/QnDglg==} + + mitt@3.0.1: + resolution: {integrity: sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw==} + + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + + non-layered-tidy-tree-layout@2.0.2: + resolution: {integrity: sha512-gkXMxRzUH+PB0ax9dUN0yYF0S25BqeAYqhgMaLUFmpXLEk7Fcu8f4emJuOAY0V8kjDICxROIKsTAKsV/v355xw==} + + normalize-package-data@2.5.0: + resolution: {integrity: sha512-/5CMN3T0R4XTj4DcGaexo+roZSdSFW/0AOOTROrjxzCG1wrWXEsGbRKevjlIL+ZDE4sZlJr5ED4YW0yqmkK+eA==} + + normalize-package-data@3.0.3: + resolution: {integrity: sha512-p2W1sgqij3zMMyRC067Dg16bfzVH+w7hyegmpIvZ4JNjqtGOVAIvLmjBx3yP7YTe9vKJgkoNOPjwQGogDoMXFA==} + engines: {node: '>=10'} + + on-exit-leak-free@2.1.2: + resolution: {integrity: sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==} + engines: {node: '>=14.0.0'} + + oniguruma-to-es@3.1.1: + resolution: {integrity: sha512-bUH8SDvPkH3ho3dvwJwfonjlQ4R80vjyvrU8YpxuROddv55vAEJrTuCuCVUhhsHbtlD9tGGbaNApGQckXhS8iQ==} + + oxfmt@0.58.0: + resolution: {integrity: sha512-8feG/7NVEHDVwc1OUpP6Pks+TnaDFUw2jLLFIMi5bcmmwxAX2wBQvjSzj62RRTYBf2Op1Wt8xbkmagmPTR5ETg==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + peerDependencies: + svelte: ^5.0.0 + vite-plus: '*' + peerDependenciesMeta: + svelte: + optional: true + vite-plus: + optional: true + + oxlint@1.81.0: + resolution: {integrity: sha512-HyrJYqeoOCL0iqaLEzGewGT48ZX99P3hxYh8udAF9RGGIghSamkXE4ClUyBpEDNqasamThgmlPbuMOe7SAZmHg==} + engines: {node: ^20.19.0 || >=22.12.0} + hasBin: true + peerDependencies: + oxlint-tsgolint: '>=7.0.2001' + vite-plus: '*' + peerDependenciesMeta: + oxlint-tsgolint: + optional: true + vite-plus: + optional: true + + p-limit@2.3.0: + resolution: {integrity: sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w==} + engines: {node: '>=6'} + + p-limit@3.1.0: + resolution: {integrity: sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==} + engines: {node: '>=10'} + + p-locate@4.1.0: + resolution: {integrity: sha512-R79ZZ/0wAxKGu3oYMlz8jy/kbhsNrS7SKZ7PxEHBgJ5+F2mtFW2fK2cOtBh1cHYkQsbzFV7I+EoRKe6Yt0oK7A==} + engines: {node: '>=8'} + + p-locate@5.0.0: + resolution: {integrity: sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw==} + engines: {node: '>=10'} + + p-try@2.2.0: + resolution: {integrity: sha512-R4nPAVTAU0B9D35/Gk3uJf/7XYbQcyohSKdvAxIRSNghFl4e71hVoGnBNQz9cWaXxO2I10KTC+3jMdvvoKw6dQ==} + engines: {node: '>=6'} + + package-json-from-dist@1.0.1: + resolution: {integrity: sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==} + + package-manager-detector@1.8.0: + resolution: {integrity: sha512-yQA4H19AmPEoMUeavPMDIe1higySl/gH/yaQrkT/s07Qp+7pp2hYz30N3z2l5BkjVkF9Ow6o0wjJamm2y7Sn0A==} + + parse-json@5.2.0: + resolution: {integrity: sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg==} + engines: {node: '>=8'} + + path-data-parser@0.1.0: + resolution: {integrity: sha512-NOnmBpt5Y2RWbuv0LMzsayp3lVylAHLPUTut412ZA3l+C4uw4ZVkQbjShYCQ8TCpUMdPapr4YjUqLYD6v68j+w==} + + path-exists@4.0.0: + resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==} + engines: {node: '>=8'} + + path-key@3.1.1: + resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==} + engines: {node: '>=8'} + + path-parse@1.0.7: + resolution: {integrity: sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==} + + path-scurry@1.11.1: + resolution: {integrity: sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==} + engines: {node: '>=16 || 14 >=14.18'} + + path-type@4.0.0: + resolution: {integrity: sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==} + engines: {node: '>=8'} + + perfect-debounce@1.0.0: + resolution: {integrity: sha512-xCy9V055GLEqoFaHoC1SoLIaLmWctgCUaBaWxDZ7/Zx4CTyX7cJQLJOok/orfjZAh9kEYpjJa4d0KcJmCbctZA==} + + picocolors@1.1.1: + resolution: {integrity: sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==} + + picomatch@2.3.2: + resolution: {integrity: sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==} + engines: {node: '>=8.6'} + + pino-abstract-transport@3.0.0: + resolution: {integrity: sha512-wlfUczU+n7Hy/Ha5j9a/gZNy7We5+cXp8YL+X+PG8S0KXxw7n/JXA3c46Y0zQznIJ83URJiwy7Lh56WLokNuxg==} + + pino-std-serializers@7.1.0: + resolution: {integrity: sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==} + + pino@10.3.1: + resolution: {integrity: sha512-r34yH/GlQpKZbU1BvFFqOjhISRo1MNx1tWYsYvmj6KIRHSPMT2+yHOEb1SG6NMvRoHRF0a07kCOox/9yakl1vg==} + hasBin: true + + plur@4.0.0: + resolution: {integrity: sha512-4UGewrYgqDFw9vV6zNV+ADmPAUAfJPKtGvb/VdpQAx25X5f3xXdGdyOEVFwkl8Hl/tl7+xbeHqSEM+D5/TirUg==} + engines: {node: '>=10'} + + points-on-curve@0.2.0: + resolution: {integrity: sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A==} + + points-on-path@0.2.1: + resolution: {integrity: sha512-25ClnWWuw7JbWZcgqY/gJ4FQWadKxGWk+3kR/7kD0tCaDtPPMj7oHu2ToLaVhfpnHrZzYby2w6tUA0eOIuUg8g==} + + postcss@8.5.28: + resolution: {integrity: sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A==} + engines: {node: ^10 || ^12 || >=14} + + preact@10.29.8: + resolution: {integrity: sha512-ej2aVZ+vZ8WO7tvlQWRM9N63A0KzF9q4mWJfDUHgYaIofWY9hu74QdnQrjoPMmZi2/nZ5gN0bJCQF49xQqx09Q==} + peerDependencies: + preact-render-to-string: '>=5' + peerDependenciesMeta: + preact-render-to-string: + optional: true + + pretty-format@29.7.0: + resolution: {integrity: sha512-Pdlw/oPxN+aXdmM9R00JVC9WVFoCLTKJvDVLgmJ+qAffBMxsV85l/Lu7sNx4zSzPyoL2euImuEwHhOXdEgNFZQ==} + engines: {node: ^14.15.0 || ^16.10.0 || >=18.0.0} + + process-warning@5.1.0: + resolution: {integrity: sha512-jQSaVHsPgtyw60e1rQ/A+/ArPEj/S8pS/vFnyGa/gYFXrKk/6RuDkoqVDQ5NI5MmS01698ltlAk0NoDBNLujRw==} + + property-information@7.2.0: + resolution: {integrity: sha512-IAtzIB6sUiWaJYrX9smp3V46pBGbBeLFRGdh25kg1334VcBlD8HzhPeNIWQH9zhGmo2itIe25EHt9dQP7G5hmg==} + + queue-microtask@1.2.3: + resolution: {integrity: sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==} + + quick-format-unescaped@4.0.4: + resolution: {integrity: sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==} + + quick-lru@4.0.1: + resolution: {integrity: sha512-ARhCpm70fzdcvNQfPoy49IaanKkTlRWF2JMzqhcJbhSFRZv7nPTvZJdcY7301IPmvW+/p0RgIWnQDLJxifsQ7g==} + engines: {node: '>=8'} + + react-is@18.3.1: + resolution: {integrity: sha512-/LLMVyas0ljjAtoYiPqYiL8VWXzUUdThrmU5+n20DZv+a+ClRoevUzw5JxU+Ieh5/c87ytoTBV9G1FiKfNJdmg==} + + read-pkg-up@7.0.1: + resolution: {integrity: sha512-zK0TB7Xd6JpCLmlLmufqykGE+/TlOePD6qKClNW7hHDKFh/J7/7gCWGR7joEQEW1bKq3a3yUZSObOoWLFQ4ohg==} + engines: {node: '>=8'} + + read-pkg@5.2.0: + resolution: {integrity: sha512-Ug69mNOpfvKDAc2Q8DRpMjjzdtrnv9HcSMX+4VsZxD1aZ6ZzrIE7rlzXBtWTyhULSMKg076AW6WR5iZpD0JiOg==} + engines: {node: '>=8'} + + real-require@0.2.0: + resolution: {integrity: sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==} + engines: {node: '>= 12.13.0'} + + real-require@1.0.0: + resolution: {integrity: sha512-P4nbQYQfePJxRSmY+v/KINxVucm4NF3p3s7pJveMTtom52FR4YGltUQLB8idDXwDDWW+eYrWDFbuzUnjoWHF7g==} + + redent@3.0.0: + resolution: {integrity: sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg==} + engines: {node: '>=8'} + + regex-recursion@6.0.2: + resolution: {integrity: sha512-0YCaSCq2VRIebiaUviZNs0cBz1kg5kVS2UKUfNIx8YVs1cN3AV7NTctO5FOKBA+UT2BPJIWZauYHPqJODG50cg==} + + regex-utilities@2.3.0: + resolution: {integrity: sha512-8VhliFJAWRaUiVvREIiW2NXXTmHs4vMNnSzuJVhscgmGav3g9VDxLrQndI3dZZVVdp0ZO/5v0xmX516/7M9cng==} + + regex@6.1.0: + resolution: {integrity: sha512-6VwtthbV4o/7+OaAF9I5L5V3llLEsoPyq9P1JVXkedTP33c7MfCG0/5NOPcSJn0TzXcG9YUrR0gQSWioew3LDg==} + + require-directory@2.1.1: + resolution: {integrity: sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==} + engines: {node: '>=0.10.0'} + + resolve@1.22.12: + resolution: {integrity: sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==} + engines: {node: '>= 0.4'} + hasBin: true + + reusify@1.1.0: + resolution: {integrity: sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw==} + engines: {iojs: '>=1.0.0', node: '>=0.10.0'} + + rfdc@1.4.1: + resolution: {integrity: sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==} + + robust-predicates@3.0.3: + resolution: {integrity: sha512-NS3levdsRIUOmiJ8FZWCP7LG3QpJyrs/TE0Zpf1yvZu8cAJJ6QMW92H1c7kWpdIHo8RvmLxN/o2JXTKHp74lUA==} + + rollup@4.63.1: + resolution: {integrity: sha512-3Df9jsstwhccuEfmAMi9l8XUh/GOkVObmFTU7CCVBysEbcOZLl84jCtaAZMcPiMz2EGKsATzQcU+Xr3n/wU6cg==} + engines: {node: '>=18.0.0', npm: '>=8.0.0'} + hasBin: true + + roughjs@4.6.6: + resolution: {integrity: sha512-ZUz/69+SYpFN/g/lUlo2FXcIjRkSu3nDarreVdGGndHEBJ6cXPdKguS8JGxwj5HA5xIbVKSmLgr5b3AWxtRfvQ==} + + run-parallel@1.2.0: + resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} + + rw@1.3.3: + resolution: {integrity: sha512-PdhdWy89SiZogBLaw42zdeqtRJ//zFd2PgQavcICDUgJT5oW10QCRKbJ6bg4r0/UY2M6BWd5tkxuGFRvCkgfHQ==} + + safe-stable-stringify@2.5.0: + resolution: {integrity: sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==} + engines: {node: '>=10'} + + safer-buffer@2.1.2: + resolution: {integrity: sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==} + + search-insights@2.17.3: + resolution: {integrity: sha512-RQPdCYTa8A68uM2jwxoY842xDhvx3E5LFL1LxvxCNMev4o5mLuokczhzjAgGwUZBAmOKZknArSxLKmXtIi2AxQ==} + + semver@5.7.2: + resolution: {integrity: sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g==} + hasBin: true + + semver@7.8.5: + resolution: {integrity: sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==} + engines: {node: '>=10'} + hasBin: true + + shebang-command@2.0.0: + resolution: {integrity: sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==} + engines: {node: '>=8'} + + shebang-regex@3.0.0: + resolution: {integrity: sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==} + engines: {node: '>=8'} + + shiki@2.5.0: + resolution: {integrity: sha512-mI//trrsaiCIPsja5CNfsyNOqgAZUb6VpJA+340toL42UpzQlXpwRV9nch69X6gaUxrr9kaOOa6e3y3uAkGFxQ==} + + signal-exit@4.1.0: + resolution: {integrity: sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==} + engines: {node: '>=14'} + + slash@3.0.0: + resolution: {integrity: sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==} + engines: {node: '>=8'} + + sonic-boom@4.2.1: + resolution: {integrity: sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==} + + source-map-js@1.2.1: + resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} + engines: {node: '>=0.10.0'} + + space-separated-tokens@2.0.2: + resolution: {integrity: sha512-PEGlAwrG8yXGXRjW32fGbg66JAlOAwbObuqVoJpv/mRgoWDQfgH1wDPvtzWyUSNAXBGSk8h755YDbbcEy3SH2Q==} + + spdx-correct@3.2.0: + resolution: {integrity: sha512-kN9dJbvnySHULIluDHy32WHRUu3Og7B9sbY7tsFLctQkIqnMh3hErYgdMjTYuqmcXX+lK5T1lnUt3G7zNswmZA==} + + spdx-exceptions@2.5.0: + resolution: {integrity: sha512-PiU42r+xO4UbUS1buo3LPJkjlO7430Xn5SVAhdpzzsPHsjbYVflnnFdATgabnLude+Cqu25p6N+g2lw/PFsa4w==} + + spdx-expression-parse@3.0.1: + resolution: {integrity: sha512-cbqHunsQWnJNE6KhVSMsMeH5H/L9EpymbzqTQ3uLwNCLZ1Q481oWaofqH7nO6V07xlXwY6PhQdQ2IedWx/ZK4Q==} + + spdx-license-ids@3.0.23: + resolution: {integrity: sha512-CWLcCCH7VLu13TgOH+r8p1O/Znwhqv/dbb6lqWy67G+pT1kHmeD/+V36AVb/vq8QMIQwVShJ6Ssl5FPh0fuSdw==} + + speakingurl@14.0.1: + resolution: {integrity: sha512-1POYv7uv2gXoyGFpBCmpDVSNV74IfsWlDW216UPjbWufNf+bSU6GdbDsxdcxtfwb4xlI3yxzOTKClUosxARYrQ==} + engines: {node: '>=0.10.0'} + + split2@4.2.0: + resolution: {integrity: sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==} + engines: {node: '>= 10.x'} + + strictdom@1.0.1: + resolution: {integrity: sha512-cEmp9QeXXRmjj/rVp9oyiqcvyocWab/HaoN4+bwFeZ7QzykJD6L3yD4v12K1x0tHpqRqVpJevN3gW7kyM39Bqg==} + + string-width@4.2.3: + resolution: {integrity: sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==} + engines: {node: '>=8'} + + string-width@5.1.2: + resolution: {integrity: sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==} + engines: {node: '>=12'} + + stringify-entities@4.0.4: + resolution: {integrity: sha512-IwfBptatlO+QCJUo19AqvrPNqlVMpW9YEL2LIVY+Rpv2qsjCGxaDLNRgeGsQWJhfItebuJhsGSLjaBbNSQ+ieg==} + + strip-ansi@6.0.1: + resolution: {integrity: sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==} + engines: {node: '>=8'} + + strip-ansi@7.2.0: + resolution: {integrity: sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==} + engines: {node: '>=12'} + + strip-indent@3.0.0: + resolution: {integrity: sha512-laJTa3Jb+VQpaC6DseHhF7dXVqHTfJPCRDaEbid/drOhgitgYku/letMUqOXFoWV0zIIUbjpdH2t+tYj4bQMRQ==} + engines: {node: '>=8'} + + stylis@4.4.0: + resolution: {integrity: sha512-5Z9ZpRzfuH6l/UAvCPAPUo3665Nk2wLaZU3x+TLHKVzIz33+sbJqbtrYoC3KD4/uVOr2Zp+L0LySezP9OHV9yA==} + + superjson@2.2.6: + resolution: {integrity: sha512-H+ue8Zo4vJmV2nRjpx86P35lzwDT3nItnIsocgumgr0hHMQ+ZGq5vrERg9kJBo5AWGmxZDhzDo+WVIJqkB0cGA==} + engines: {node: '>=16'} + + supports-color@7.2.0: + resolution: {integrity: sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==} + engines: {node: '>=8'} + + supports-hyperlinks@2.3.0: + resolution: {integrity: sha512-RpsAZlpWcDwOPQA22aCH4J0t7L8JmAvsCxfOSEwm7cQs3LshN36QaTkwd70DnBOXDWGssw2eUoc8CaRWT0XunA==} + engines: {node: '>=8'} + + supports-preserve-symlinks-flag@1.0.0: + resolution: {integrity: sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w==} + engines: {node: '>= 0.4'} + + tabbable@6.5.0: + resolution: {integrity: sha512-wieBHXygIm7OyQOu5hQlkk62/WyCFYGlWg7L6/ZCUZwx0o398Zkn4pVmMyfYhfMG8kGrj/Krt8eIk6UKC6VzwA==} + + test-exclude@7.0.2: + resolution: {integrity: sha512-u9E6A+ZDYdp7a4WnarkXPZOx8Ilz46+kby6p1yZ8zsGTz9gYa6FIS7lj2oezzNKmtdyyJNNmmXDppga5GB7kSw==} + engines: {node: '>=18'} + + thread-stream@4.2.0: + resolution: {integrity: sha512-e2zZ96wSChazBsbENf/Pcm/4swHt2cEKQ92rhUjkL9GCKiTDJIaTBenjE/m9DXi0QBmTMDkFDdOomUy20A1tDQ==} + engines: {node: '>=20'} + + tinyexec@1.3.1: + resolution: {integrity: sha512-GCvB3aoys96IuDFBMcTB46JOR6mdMtAToqwiW8JlWhsoh1mhHi/xn9ss/Dg7N555GiJyEt2qzoG/NHCwM6h1EA==} + engines: {node: '>=18'} + + tinypool@2.1.0: + resolution: {integrity: sha512-Pugqs6M0m7Lv1I7FtxN4aoyToKg1C4tu+/381vH35y8oENM/Ai7f7C4StcoK4/+BSw9ebcS8jRiVrORFKCALLw==} + engines: {node: ^20.0.0 || >=22.0.0} + + to-regex-range@5.0.1: + resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==} + engines: {node: '>=8.0'} + + trim-lines@3.0.1: + resolution: {integrity: sha512-kRj8B+YHZCc9kQYdWfJB2/oUl9rA99qbowYYBtr4ui4mZyAQ2JpvVBd/6U2YloATfqBhBTSMhTpgBHtU0Mf3Rg==} + + trim-newlines@3.0.1: + resolution: {integrity: sha512-c1PTsA3tYrIsLGkJkzHF+w9F2EyxfXGo4UyJc4pFL++FMjnq0HJS69T3M7d//gKrFKwy429bouPescbjecU+Zw==} + engines: {node: '>=8'} + + ts-dedent@2.3.0: + resolution: {integrity: sha512-JfJeIHke7y2egdGGgRAvpCwYFUsHlM2gPcrVOxFkznt/4uzQ7HFmvE63iFHVLBJNDuyDOQgijDK/tXH/f6Msjg==} + engines: {node: '>=6.10'} + + tsd@0.31.2: + resolution: {integrity: sha512-VplBAQwvYrHzVihtzXiUVXu5bGcr7uH1juQZ1lmKgkuGNGT+FechUCqmx9/zk7wibcqR2xaNEwCkDyKh+VVZnQ==} + engines: {node: '>=14.16'} + hasBin: true + + type-fest@0.18.1: + resolution: {integrity: sha512-OIAYXk8+ISY+qTOwkHtKqzAuxchoMiD9Udx+FSGQDuiRR+PJKJHc2NJAXlbhkGwTt/4/nKZxELY1w3ReWOL8mw==} + engines: {node: '>=10'} + + type-fest@0.21.3: + resolution: {integrity: sha512-t0rzBq87m3fVcduHDUFhKmyyX+9eo6WQjZvf51Ea/M0Q7+T374Jp1aUiyUl0GKxp8M/OETVHSDvmkyPgvX+X2w==} + engines: {node: '>=10'} + + type-fest@0.6.0: + resolution: {integrity: sha512-q+MB8nYR1KDLrgr4G5yemftpMC7/QLqVndBmEEdqzmNj5dcFOO4Oo8qlwZE3ULT3+Zim1F8Kq4cBnikNhlCMlg==} + engines: {node: '>=8'} + + type-fest@0.8.1: + resolution: {integrity: sha512-4dbzIzqvjtgiM5rw1k5rEHtBANKmdudhGyBEajN01fEyhaAIhsoKNy6y7+IN93IfpFtwY9iqi7kD+xwKhQsNJA==} + engines: {node: '>=8'} + + typescript@7.0.2: + resolution: {integrity: sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==} + engines: {node: '>=16.20.0'} + hasBin: true + + unist-util-is@6.0.1: + resolution: {integrity: sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==} + + unist-util-position@5.0.0: + resolution: {integrity: sha512-fucsC7HjXvkB5R3kTCO7kUjRdrS0BJt3M/FPxmHMBOm8JQi2BsHAHFsy27E0EolP8rp0NzXsJ+jNPyDWvOJZPA==} + + unist-util-stringify-position@4.0.0: + resolution: {integrity: sha512-0ASV06AAoKCDkS2+xw5RXJywruurpbC4JZSm7nr7MOt1ojAzvyyaO+UxZf18j8FCF6kmzCZKcAgN/yu2gm2XgQ==} + + unist-util-visit-parents@6.0.2: + resolution: {integrity: sha512-goh1s1TBrqSqukSc8wrjwWhL0hiJxgA8m4kFxGlQ+8FYQ3C/m11FcTs4YYem7V664AhHVvgoQLk890Ssdsr2IQ==} + + unist-util-visit@5.1.0: + resolution: {integrity: sha512-m+vIdyeCOpdr/QeQCu2EzxX/ohgS8KbnPDgFni4dQsfSCtpz8UqDyY5GjRru8PDKuYn7Fq19j1CQ+nJSsGKOzg==} + + uuid@14.0.2: + resolution: {integrity: sha512-xZe/16rV4aa+HGSOCiY2YeLT1OybRLrrkL/Rqaq7p7GMVXjFh+6wN4oMYgjFmnSnhY8t6Xpdl2l9qmnHYuMHwQ==} + hasBin: true + + v8-to-istanbul@9.3.0: + resolution: {integrity: sha512-kiGUalWN+rgBJ/1OHZsBtU4rXZOfj/7rKQxULKlIzwzQSvMJUUNgPwJEEh7gU6xEVxC0ahoOBvN2YI8GH6FNgA==} + engines: {node: '>=10.12.0'} + + validate-npm-package-license@3.0.4: + resolution: {integrity: sha512-DpKm2Ui/xN7/HQKCtpZxoRWBhZ9Z0kqtygG8XCgNQ8ZlDnxuQmWhj566j8fN4Cu3/JmbhsDo7fcAJq4s9h27Ew==} + + vfile-message@4.0.3: + resolution: {integrity: sha512-QTHzsGd1EhbZs4AsQ20JX1rC3cOlt/IWJruk893DfLRr57lcnOeMaWG4K0JrRta4mIJZKth2Au3mM3u03/JWKw==} + + vfile@6.0.3: + resolution: {integrity: sha512-KzIbH/9tXat2u30jf+smMwFCsno4wHVdNmzFyL+T/L3UGqqk6JKfVqOFOZEpZSHADH1k40ab6NUIXZq422ov3Q==} + + vite@5.4.21: + resolution: {integrity: sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==} + engines: {node: ^18.0.0 || >=20.0.0} + hasBin: true + peerDependencies: + '@types/node': ^18.0.0 || >=20.0.0 + less: '*' + lightningcss: ^1.21.0 + sass: '*' + sass-embedded: '*' + stylus: '*' + sugarss: '*' + terser: ^5.4.0 + peerDependenciesMeta: + '@types/node': + optional: true + less: + optional: true + lightningcss: + optional: true + sass: + optional: true + sass-embedded: + optional: true + stylus: + optional: true + sugarss: + optional: true + terser: + optional: true + + vitepress-plugin-mermaid@2.0.17: + resolution: {integrity: sha512-IUzYpwf61GC6k0XzfmAmNrLvMi9TRrVRMsUyCA8KNXhg/mQ1VqWnO0/tBVPiX5UoKF1mDUwqn5QV4qAJl6JnUg==} + peerDependencies: + mermaid: 10 || 11 + vitepress: ^1.0.0 || ^1.0.0-alpha + + vitepress@1.6.4: + resolution: {integrity: sha512-+2ym1/+0VVrbhNyRoFFesVvBvHAVMZMK0rw60E3X/5349M1GuVdKeazuksqopEdvkKwKGs21Q729jX81/bkBJg==} + hasBin: true + peerDependencies: + markdown-it-mathjax3: ^4 + postcss: ^8 + peerDependenciesMeta: + markdown-it-mathjax3: + optional: true + postcss: + optional: true + + vue@3.5.42: + resolution: {integrity: sha512-4RyHQTbQvOPs3MfvUO1Sg0YRrKNnA0mAVtvpd12Tg1fKDN7OHBUl1IqSn8zGJjK9nI3NkNp8cgTpVrSZC5TTcA==} + peerDependencies: + typescript: '*' + peerDependenciesMeta: + typescript: + optional: true + + which@2.0.2: + resolution: {integrity: sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==} + engines: {node: '>= 8'} + hasBin: true + + wrap-ansi@7.0.0: + resolution: {integrity: sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==} + engines: {node: '>=10'} + + wrap-ansi@8.1.0: + resolution: {integrity: sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==} + engines: {node: '>=12'} + + y18n@5.0.8: + resolution: {integrity: sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==} + engines: {node: '>=10'} + + yallist@4.0.0: + resolution: {integrity: sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==} + + yargs-parser@20.2.9: + resolution: {integrity: sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==} + engines: {node: '>=10'} + + yargs-parser@21.1.1: + resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==} + engines: {node: '>=12'} + + yargs@17.7.3: + resolution: {integrity: sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g==} + engines: {node: '>=12'} + + yocto-queue@0.1.0: + resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==} + engines: {node: '>=10'} + + zwitch@2.0.4: + resolution: {integrity: sha512-bXE4cR/kVZhKZX/RjPEflHaKVhUVl85noU3v6b8apfQEc1x4A+zBxjZ4lN8LqGd6WZ3dl98pY4o717VFmoPp+A==} + +snapshots: + + '@algolia/abtesting@1.23.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/autocomplete-core@1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)(search-insights@2.17.3)': + dependencies: + '@algolia/autocomplete-plugin-algolia-insights': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)(search-insights@2.17.3) + '@algolia/autocomplete-shared': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0) + transitivePeerDependencies: + - '@algolia/client-search' + - algoliasearch + - search-insights + + '@algolia/autocomplete-plugin-algolia-insights@1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)(search-insights@2.17.3)': + dependencies: + '@algolia/autocomplete-shared': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0) + search-insights: 2.17.3 + transitivePeerDependencies: + - '@algolia/client-search' + - algoliasearch + + '@algolia/autocomplete-preset-algolia@1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)': + dependencies: + '@algolia/autocomplete-shared': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0) + '@algolia/client-search': 5.57.0 + algoliasearch: 5.57.0 + + '@algolia/autocomplete-shared@1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)': + dependencies: + '@algolia/client-search': 5.57.0 + algoliasearch: 5.57.0 + + '@algolia/client-abtesting@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/client-analytics@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/client-common@5.57.0': {} + + '@algolia/client-insights@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/client-personalization@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/client-query-suggestions@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/client-search@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/ingestion@1.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/monitoring@1.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/recommend@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + '@algolia/requester-browser-xhr@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + + '@algolia/requester-fetch@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + + '@algolia/requester-node-http@5.57.0': + dependencies: + '@algolia/client-common': 5.57.0 + + '@antfu/install-pkg@2.0.1': + dependencies: + package-manager-detector: 1.8.0 + tinyexec: 1.3.1 + + '@babel/code-frame@7.29.7': + dependencies: + '@babel/helper-validator-identifier': 7.29.7 + js-tokens: 4.0.0 + picocolors: 1.1.1 + + '@babel/helper-string-parser@7.29.7': {} + + '@babel/helper-validator-identifier@7.29.7': {} + + '@babel/parser@7.29.8': + dependencies: + '@babel/types': 7.29.8 + + '@babel/types@7.29.8': + dependencies: + '@babel/helper-string-parser': 7.29.7 + '@babel/helper-validator-identifier': 7.29.7 + + '@bcoe/v8-coverage@1.0.2': {} + + '@braintree/sanitize-url@6.0.4': + optional: true + + '@braintree/sanitize-url@7.1.2': {} + + '@chevrotain/types@11.1.2': {} + + '@docsearch/css@3.8.2': {} + + '@docsearch/js@3.8.2(@algolia/client-search@5.57.0)(search-insights@2.17.3)': + dependencies: + '@docsearch/react': 3.8.2(@algolia/client-search@5.57.0)(search-insights@2.17.3) + preact: 10.29.8 + transitivePeerDependencies: + - '@algolia/client-search' + - '@types/react' + - preact-render-to-string + - react + - react-dom + - search-insights + + '@docsearch/react@3.8.2(@algolia/client-search@5.57.0)(search-insights@2.17.3)': + dependencies: + '@algolia/autocomplete-core': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0)(search-insights@2.17.3) + '@algolia/autocomplete-preset-algolia': 1.17.7(@algolia/client-search@5.57.0)(algoliasearch@5.57.0) + '@docsearch/css': 3.8.2 + algoliasearch: 5.57.0 + optionalDependencies: + search-insights: 2.17.3 + transitivePeerDependencies: + - '@algolia/client-search' + + '@esbuild/aix-ppc64@0.21.5': + optional: true + + '@esbuild/aix-ppc64@0.28.2': + optional: true + + '@esbuild/android-arm64@0.21.5': + optional: true + + '@esbuild/android-arm64@0.28.2': + optional: true + + '@esbuild/android-arm@0.21.5': + optional: true + + '@esbuild/android-arm@0.28.2': + optional: true + + '@esbuild/android-x64@0.21.5': + optional: true + + '@esbuild/android-x64@0.28.2': + optional: true + + '@esbuild/darwin-arm64@0.21.5': + optional: true + + '@esbuild/darwin-arm64@0.28.2': + optional: true + + '@esbuild/darwin-x64@0.21.5': + optional: true + + '@esbuild/darwin-x64@0.28.2': + optional: true + + '@esbuild/freebsd-arm64@0.21.5': + optional: true + + '@esbuild/freebsd-arm64@0.28.2': + optional: true + + '@esbuild/freebsd-x64@0.21.5': + optional: true + + '@esbuild/freebsd-x64@0.28.2': + optional: true + + '@esbuild/linux-arm64@0.21.5': + optional: true + + '@esbuild/linux-arm64@0.28.2': + optional: true + + '@esbuild/linux-arm@0.21.5': + optional: true + + '@esbuild/linux-arm@0.28.2': + optional: true + + '@esbuild/linux-ia32@0.21.5': + optional: true + + '@esbuild/linux-ia32@0.28.2': + optional: true + + '@esbuild/linux-loong64@0.21.5': + optional: true + + '@esbuild/linux-loong64@0.28.2': + optional: true + + '@esbuild/linux-mips64el@0.21.5': + optional: true + + '@esbuild/linux-mips64el@0.28.2': + optional: true + + '@esbuild/linux-ppc64@0.21.5': + optional: true + + '@esbuild/linux-ppc64@0.28.2': + optional: true + + '@esbuild/linux-riscv64@0.21.5': + optional: true + + '@esbuild/linux-riscv64@0.28.2': + optional: true + + '@esbuild/linux-s390x@0.21.5': + optional: true + + '@esbuild/linux-s390x@0.28.2': + optional: true + + '@esbuild/linux-x64@0.21.5': + optional: true + + '@esbuild/linux-x64@0.28.2': + optional: true + + '@esbuild/netbsd-arm64@0.28.2': + optional: true + + '@esbuild/netbsd-x64@0.21.5': + optional: true + + '@esbuild/netbsd-x64@0.28.2': + optional: true + + '@esbuild/openbsd-arm64@0.28.2': + optional: true + + '@esbuild/openbsd-x64@0.21.5': + optional: true + + '@esbuild/openbsd-x64@0.28.2': + optional: true + + '@esbuild/openharmony-arm64@0.28.2': + optional: true + + '@esbuild/sunos-x64@0.21.5': + optional: true + + '@esbuild/sunos-x64@0.28.2': + optional: true + + '@esbuild/win32-arm64@0.21.5': + optional: true + + '@esbuild/win32-arm64@0.28.2': + optional: true + + '@esbuild/win32-ia32@0.21.5': + optional: true + + '@esbuild/win32-ia32@0.28.2': + optional: true + + '@esbuild/win32-x64@0.21.5': + optional: true + + '@esbuild/win32-x64@0.28.2': + optional: true + + '@iconify-json/simple-icons@1.2.94': + dependencies: + '@iconify/types': 2.0.0 + + '@iconify/types@2.0.0': {} + + '@iconify/utils@3.1.5': + dependencies: + '@antfu/install-pkg': 2.0.1 + '@iconify/types': 2.0.0 + import-meta-resolve: 4.2.0 + + '@isaacs/cliui@8.0.2': + dependencies: + string-width: 5.1.2 + string-width-cjs: string-width@4.2.3 + strip-ansi: 7.2.0 + strip-ansi-cjs: strip-ansi@6.0.1 + wrap-ansi: 8.1.0 + wrap-ansi-cjs: wrap-ansi@7.0.0 + + '@istanbuljs/schema@0.1.6': {} + + '@jest/schemas@29.6.3': + dependencies: + '@sinclair/typebox': 0.27.12 + + '@jridgewell/resolve-uri@3.1.2': {} + + '@jridgewell/sourcemap-codec@1.6.0': {} + + '@jridgewell/trace-mapping@0.3.31': + dependencies: + '@jridgewell/resolve-uri': 3.1.2 + '@jridgewell/sourcemap-codec': 1.6.0 + + '@mermaid-js/mermaid-mindmap@9.3.0': + dependencies: + '@braintree/sanitize-url': 6.0.4 + cytoscape: 3.34.2 + cytoscape-cose-bilkent: 4.1.0(cytoscape@3.34.2) + cytoscape-fcose: 2.2.0(cytoscape@3.34.2) + d3: 7.9.0 + khroma: 2.1.0 + non-layered-tidy-tree-layout: 2.0.2 + optional: true + + '@mermaid-js/parser@1.2.1': + dependencies: + '@chevrotain/types': 11.1.2 + + '@napi-rs/lzma-linux-x64-gnu@1.5.1': + optional: true + + '@nodelib/fs.scandir@2.1.5': + dependencies: + '@nodelib/fs.stat': 2.0.5 + run-parallel: 1.2.0 + + '@nodelib/fs.stat@2.0.5': {} + + '@nodelib/fs.walk@1.2.8': + dependencies: + '@nodelib/fs.scandir': 2.1.5 + fastq: 1.20.3 + + '@oxfmt/binding-android-arm-eabi@0.58.0': + optional: true + + '@oxfmt/binding-android-arm64@0.58.0': + optional: true + + '@oxfmt/binding-darwin-arm64@0.58.0': + optional: true + + '@oxfmt/binding-darwin-x64@0.58.0': + optional: true + + '@oxfmt/binding-freebsd-x64@0.58.0': + optional: true + + '@oxfmt/binding-linux-arm-gnueabihf@0.58.0': + optional: true + + '@oxfmt/binding-linux-arm-musleabihf@0.58.0': + optional: true + + '@oxfmt/binding-linux-arm64-gnu@0.58.0': + optional: true + + '@oxfmt/binding-linux-arm64-musl@0.58.0': + optional: true + + '@oxfmt/binding-linux-ppc64-gnu@0.58.0': + optional: true + + '@oxfmt/binding-linux-riscv64-gnu@0.58.0': + optional: true + + '@oxfmt/binding-linux-riscv64-musl@0.58.0': + optional: true + + '@oxfmt/binding-linux-s390x-gnu@0.58.0': + optional: true + + '@oxfmt/binding-linux-x64-gnu@0.58.0': + optional: true + + '@oxfmt/binding-linux-x64-musl@0.58.0': + optional: true + + '@oxfmt/binding-openharmony-arm64@0.58.0': + optional: true + + '@oxfmt/binding-win32-arm64-msvc@0.58.0': + optional: true + + '@oxfmt/binding-win32-ia32-msvc@0.58.0': + optional: true + + '@oxfmt/binding-win32-x64-msvc@0.58.0': + optional: true + + '@oxlint/binding-android-arm-eabi@1.81.0': + optional: true + + '@oxlint/binding-android-arm64@1.81.0': + optional: true + + '@oxlint/binding-darwin-arm64@1.81.0': + optional: true + + '@oxlint/binding-darwin-x64@1.81.0': + optional: true + + '@oxlint/binding-freebsd-x64@1.81.0': + optional: true + + '@oxlint/binding-linux-arm-gnueabihf@1.81.0': + optional: true + + '@oxlint/binding-linux-arm-musleabihf@1.81.0': + optional: true + + '@oxlint/binding-linux-arm64-gnu@1.81.0': + optional: true + + '@oxlint/binding-linux-arm64-musl@1.81.0': + optional: true + + '@oxlint/binding-linux-ppc64-gnu@1.81.0': + optional: true + + '@oxlint/binding-linux-riscv64-gnu@1.81.0': + optional: true + + '@oxlint/binding-linux-riscv64-musl@1.81.0': + optional: true + + '@oxlint/binding-linux-s390x-gnu@1.81.0': + optional: true + + '@oxlint/binding-linux-x64-gnu@1.81.0': + optional: true + + '@oxlint/binding-linux-x64-musl@1.81.0': + optional: true + + '@oxlint/binding-openharmony-arm64@1.81.0': + optional: true + + '@oxlint/binding-win32-arm64-msvc@1.81.0': + optional: true + + '@oxlint/binding-win32-ia32-msvc@1.81.0': + optional: true + + '@oxlint/binding-win32-x64-msvc@1.81.0': + optional: true + + '@pinojs/redact@0.4.0': {} + + '@pkgjs/parseargs@0.11.0': + optional: true + + '@rollup/rollup-android-arm-eabi@4.63.1': + optional: true + + '@rollup/rollup-android-arm64@4.63.1': + optional: true + + '@rollup/rollup-darwin-arm64@4.63.1': + optional: true + + '@rollup/rollup-darwin-x64@4.63.1': + optional: true + + '@rollup/rollup-freebsd-arm64@4.63.1': + optional: true + + '@rollup/rollup-freebsd-x64@4.63.1': + optional: true + + '@rollup/rollup-linux-arm-gnueabihf@4.63.1': + optional: true + + '@rollup/rollup-linux-arm-musleabihf@4.63.1': + optional: true + + '@rollup/rollup-linux-arm64-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-arm64-musl@4.63.1': + optional: true + + '@rollup/rollup-linux-loong64-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-loong64-musl@4.63.1': + optional: true + + '@rollup/rollup-linux-ppc64-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-ppc64-musl@4.63.1': + optional: true + + '@rollup/rollup-linux-riscv64-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-riscv64-musl@4.63.1': + optional: true + + '@rollup/rollup-linux-s390x-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-x64-gnu@4.63.1': + optional: true + + '@rollup/rollup-linux-x64-musl@4.63.1': + optional: true + + '@rollup/rollup-openbsd-x64@4.63.1': + optional: true + + '@rollup/rollup-openharmony-arm64@4.63.1': + optional: true + + '@rollup/rollup-win32-arm64-msvc@4.63.1': + optional: true + + '@rollup/rollup-win32-ia32-msvc@4.63.1': + optional: true + + '@rollup/rollup-win32-x64-gnu@4.63.1': + optional: true + + '@rollup/rollup-win32-x64-msvc@4.63.1': + optional: true + + '@shikijs/core@2.5.0': + dependencies: + '@shikijs/engine-javascript': 2.5.0 + '@shikijs/engine-oniguruma': 2.5.0 + '@shikijs/types': 2.5.0 + '@shikijs/vscode-textmate': 10.0.2 + '@types/hast': 3.0.5 + hast-util-to-html: 9.0.5 + + '@shikijs/engine-javascript@2.5.0': + dependencies: + '@shikijs/types': 2.5.0 + '@shikijs/vscode-textmate': 10.0.2 + oniguruma-to-es: 3.1.1 + + '@shikijs/engine-oniguruma@2.5.0': + dependencies: + '@shikijs/types': 2.5.0 + '@shikijs/vscode-textmate': 10.0.2 + + '@shikijs/langs@2.5.0': + dependencies: + '@shikijs/types': 2.5.0 + + '@shikijs/themes@2.5.0': + dependencies: + '@shikijs/types': 2.5.0 + + '@shikijs/transformers@2.5.0': + dependencies: + '@shikijs/core': 2.5.0 + '@shikijs/types': 2.5.0 + + '@shikijs/types@2.5.0': + dependencies: + '@shikijs/vscode-textmate': 10.0.2 + '@types/hast': 3.0.5 + + '@shikijs/vscode-textmate@10.0.2': {} + + '@sinclair/typebox@0.27.12': {} + + '@tsd/typescript@5.4.5': {} + + '@types/d3-array@3.2.2': {} + + '@types/d3-axis@3.0.6': + dependencies: + '@types/d3-selection': 3.0.11 + + '@types/d3-brush@3.0.6': + dependencies: + '@types/d3-selection': 3.0.11 + + '@types/d3-chord@3.0.6': {} + + '@types/d3-color@3.1.3': {} + + '@types/d3-contour@3.0.6': + dependencies: + '@types/d3-array': 3.2.2 + '@types/geojson': 7946.0.16 + + '@types/d3-delaunay@6.0.4': {} + + '@types/d3-dispatch@3.0.7': {} + + '@types/d3-drag@3.0.7': + dependencies: + '@types/d3-selection': 3.0.11 + + '@types/d3-dsv@3.0.7': {} + + '@types/d3-ease@3.0.2': {} + + '@types/d3-fetch@3.0.7': + dependencies: + '@types/d3-dsv': 3.0.7 + + '@types/d3-force@3.0.10': {} + + '@types/d3-format@3.0.4': {} + + '@types/d3-geo@3.1.1': + dependencies: + '@types/geojson': 7946.0.16 + + '@types/d3-hierarchy@3.1.7': {} + + '@types/d3-interpolate@3.0.4': + dependencies: + '@types/d3-color': 3.1.3 + + '@types/d3-path@3.1.1': {} + + '@types/d3-polygon@3.0.2': {} + + '@types/d3-quadtree@3.0.6': {} + + '@types/d3-random@3.0.4': {} + + '@types/d3-scale-chromatic@3.1.0': {} + + '@types/d3-scale@4.0.9': + dependencies: + '@types/d3-time': 3.0.4 + + '@types/d3-selection@3.0.11': {} + + '@types/d3-shape@3.2.0': + dependencies: + '@types/d3-path': 3.1.1 + + '@types/d3-time-format@4.0.3': {} + + '@types/d3-time@3.0.4': {} + + '@types/d3-timer@3.0.2': {} + + '@types/d3-transition@3.0.9': + dependencies: + '@types/d3-selection': 3.0.11 + + '@types/d3-zoom@3.0.8': + dependencies: + '@types/d3-interpolate': 3.0.4 + '@types/d3-selection': 3.0.11 + + '@types/d3@7.4.3': + dependencies: + '@types/d3-array': 3.2.2 + '@types/d3-axis': 3.0.6 + '@types/d3-brush': 3.0.6 + '@types/d3-chord': 3.0.6 + '@types/d3-color': 3.1.3 + '@types/d3-contour': 3.0.6 + '@types/d3-delaunay': 6.0.4 + '@types/d3-dispatch': 3.0.7 + '@types/d3-drag': 3.0.7 + '@types/d3-dsv': 3.0.7 + '@types/d3-ease': 3.0.2 + '@types/d3-fetch': 3.0.7 + '@types/d3-force': 3.0.10 + '@types/d3-format': 3.0.4 + '@types/d3-geo': 3.1.1 + '@types/d3-hierarchy': 3.1.7 + '@types/d3-interpolate': 3.0.4 + '@types/d3-path': 3.1.1 + '@types/d3-polygon': 3.0.2 + '@types/d3-quadtree': 3.0.6 + '@types/d3-random': 3.0.4 + '@types/d3-scale': 4.0.9 + '@types/d3-scale-chromatic': 3.1.0 + '@types/d3-selection': 3.0.11 + '@types/d3-shape': 3.2.0 + '@types/d3-time': 3.0.4 + '@types/d3-time-format': 4.0.3 + '@types/d3-timer': 3.0.2 + '@types/d3-transition': 3.0.9 + '@types/d3-zoom': 3.0.8 + + '@types/eslint@7.29.0': + dependencies: + '@types/estree': 1.0.9 + '@types/json-schema': 7.0.15 + + '@types/estree@1.0.9': {} + + '@types/geojson@7946.0.16': {} + + '@types/hast@3.0.5': + dependencies: + '@types/unist': 3.0.3 + + '@types/istanbul-lib-coverage@2.0.6': {} + + '@types/json-schema@7.0.15': {} + + '@types/linkify-it@5.0.0': {} + + '@types/markdown-it@14.2.0': + dependencies: + '@types/linkify-it': 5.0.0 + '@types/mdurl': 2.0.0 + + '@types/mdast@4.0.4': + dependencies: + '@types/unist': 3.0.3 + + '@types/mdurl@2.0.0': {} + + '@types/minimist@1.2.5': {} + + '@types/normalize-package-data@2.4.4': {} + + '@types/trusted-types@2.0.7': + optional: true + + '@types/unist@3.0.3': {} + + '@types/web-bluetooth@0.0.21': {} + + '@typescript/typescript-aix-ppc64@7.0.2': + optional: true + + '@typescript/typescript-darwin-arm64@7.0.2': + optional: true + + '@typescript/typescript-darwin-x64@7.0.2': + optional: true + + '@typescript/typescript-freebsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-freebsd-x64@7.0.2': + optional: true + + '@typescript/typescript-linux-arm64@7.0.2': + optional: true + + '@typescript/typescript-linux-arm@7.0.2': + optional: true + + '@typescript/typescript-linux-loong64@7.0.2': + optional: true + + '@typescript/typescript-linux-mips64el@7.0.2': + optional: true + + '@typescript/typescript-linux-ppc64@7.0.2': + optional: true + + '@typescript/typescript-linux-riscv64@7.0.2': + optional: true + + '@typescript/typescript-linux-s390x@7.0.2': + optional: true + + '@typescript/typescript-linux-x64@7.0.2': + optional: true + + '@typescript/typescript-netbsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-netbsd-x64@7.0.2': + optional: true + + '@typescript/typescript-openbsd-arm64@7.0.2': + optional: true + + '@typescript/typescript-openbsd-x64@7.0.2': + optional: true + + '@typescript/typescript-sunos-x64@7.0.2': + optional: true + + '@typescript/typescript-win32-arm64@7.0.2': + optional: true + + '@typescript/typescript-win32-x64@7.0.2': + optional: true + + '@ungap/structured-clone@1.4.0': {} + + '@upsetjs/venn.js@2.0.0': + optionalDependencies: + d3-selection: 3.0.0 + d3-transition: 3.0.1(d3-selection@3.0.0) + + '@vercel/analytics@2.0.1(vue@3.5.42(typescript@7.0.2))': + optionalDependencies: + vue: 3.5.42(typescript@7.0.2) + + '@vercel/speed-insights@2.0.0(vue@3.5.42(typescript@7.0.2))': + optionalDependencies: + vue: 3.5.42(typescript@7.0.2) + + '@vitejs/plugin-vue@5.2.4(vite@5.4.21)(vue@3.5.42(typescript@7.0.2))': + dependencies: + vite: 5.4.21 + vue: 3.5.42(typescript@7.0.2) + + '@vue/compiler-core@3.5.42': + dependencies: + '@babel/parser': 7.29.8 + '@vue/shared': 3.5.42 + entities: 7.0.1 + estree-walker: 2.0.2 + source-map-js: 1.2.1 + + '@vue/compiler-dom@3.5.42': + dependencies: + '@vue/compiler-core': 3.5.42 + '@vue/shared': 3.5.42 + + '@vue/compiler-sfc@3.5.42': + dependencies: + '@babel/parser': 7.29.8 + '@vue/compiler-core': 3.5.42 + '@vue/compiler-dom': 3.5.42 + '@vue/compiler-ssr': 3.5.42 + '@vue/shared': 3.5.42 + estree-walker: 2.0.2 + magic-string: 0.30.21 + postcss: 8.5.28 + source-map-js: 1.2.1 + + '@vue/compiler-ssr@3.5.42': + dependencies: + '@vue/compiler-dom': 3.5.42 + '@vue/shared': 3.5.42 + + '@vue/devtools-api@7.7.10': + dependencies: + '@vue/devtools-kit': 7.7.10 + + '@vue/devtools-kit@7.7.10': + dependencies: + '@vue/devtools-shared': 7.7.10 + birpc: 2.9.0 + hookable: 5.5.3 + mitt: 3.0.1 + perfect-debounce: 1.0.0 + speakingurl: 14.0.1 + superjson: 2.2.6 + + '@vue/devtools-shared@7.7.10': + dependencies: + rfdc: 1.4.1 + + '@vue/reactivity@3.5.42': + dependencies: + '@vue/shared': 3.5.42 + + '@vue/runtime-core@3.5.42': + dependencies: + '@vue/reactivity': 3.5.42 + '@vue/shared': 3.5.42 + + '@vue/runtime-dom@3.5.42': + dependencies: + '@vue/reactivity': 3.5.42 + '@vue/runtime-core': 3.5.42 + '@vue/shared': 3.5.42 + csstype: 3.2.3 + + '@vue/server-renderer@3.5.42': + dependencies: + '@vue/compiler-ssr': 3.5.42 + '@vue/runtime-dom': 3.5.42 + '@vue/shared': 3.5.42 + + '@vue/shared@3.5.42': {} + + '@vueuse/core@12.8.2(typescript@7.0.2)': + dependencies: + '@types/web-bluetooth': 0.0.21 + '@vueuse/metadata': 12.8.2 + '@vueuse/shared': 12.8.2(typescript@7.0.2) + vue: 3.5.42(typescript@7.0.2) + transitivePeerDependencies: + - typescript + + '@vueuse/integrations@12.8.2(focus-trap@7.8.0)(typescript@7.0.2)': + dependencies: + '@vueuse/core': 12.8.2(typescript@7.0.2) + '@vueuse/shared': 12.8.2(typescript@7.0.2) + vue: 3.5.42(typescript@7.0.2) + optionalDependencies: + focus-trap: 7.8.0 + transitivePeerDependencies: + - typescript + + '@vueuse/metadata@12.8.2': {} + + '@vueuse/shared@12.8.2(typescript@7.0.2)': + dependencies: + vue: 3.5.42(typescript@7.0.2) + transitivePeerDependencies: + - typescript + + algoliasearch@5.57.0: + dependencies: + '@algolia/abtesting': 1.23.0 + '@algolia/client-abtesting': 5.57.0 + '@algolia/client-analytics': 5.57.0 + '@algolia/client-common': 5.57.0 + '@algolia/client-insights': 5.57.0 + '@algolia/client-personalization': 5.57.0 + '@algolia/client-query-suggestions': 5.57.0 + '@algolia/client-search': 5.57.0 + '@algolia/ingestion': 1.57.0 + '@algolia/monitoring': 1.57.0 + '@algolia/recommend': 5.57.0 + '@algolia/requester-browser-xhr': 5.57.0 + '@algolia/requester-fetch': 5.57.0 + '@algolia/requester-node-http': 5.57.0 + + ansi-escapes@4.3.2: + dependencies: + type-fest: 0.21.3 + + ansi-regex@5.0.1: {} + + ansi-regex@6.3.0: {} + + ansi-styles@4.3.0: + dependencies: + color-convert: 2.0.1 + + ansi-styles@5.2.0: {} + + ansi-styles@6.2.3: {} + + array-union@2.1.0: {} + + arrify@1.0.1: {} + + atomic-sleep@1.0.0: {} + + balanced-match@1.0.2: {} + + balanced-match@4.0.4: {} + + birpc@2.9.0: {} + + brace-expansion@2.1.4: + dependencies: + balanced-match: 1.0.2 + + brace-expansion@5.0.9: + dependencies: + balanced-match: 4.0.4 + + braces@3.0.3: + dependencies: + fill-range: 7.1.1 + + c8@10.1.3: + dependencies: + '@bcoe/v8-coverage': 1.0.2 + '@istanbuljs/schema': 0.1.6 + find-up: 5.0.0 + foreground-child: 3.3.1 + istanbul-lib-coverage: 3.2.2 + istanbul-lib-report: 3.0.1 + istanbul-reports: 3.2.0 + test-exclude: 7.0.2 + v8-to-istanbul: 9.3.0 + yargs: 17.7.3 + yargs-parser: 21.1.1 + + camelcase-keys@6.2.2: + dependencies: + camelcase: 5.3.1 + map-obj: 4.3.0 + quick-lru: 4.0.1 + + camelcase@5.3.1: {} + + ccount@2.0.1: {} + + chalk@4.1.2: + dependencies: + ansi-styles: 4.3.0 + supports-color: 7.2.0 + + character-entities-html4@2.1.0: {} + + character-entities-legacy@3.0.0: {} + + cliui@8.0.1: + dependencies: + string-width: 4.2.3 + strip-ansi: 6.0.1 + wrap-ansi: 7.0.0 + + color-convert@2.0.1: + dependencies: + color-name: 1.1.4 + + color-name@1.1.4: {} + + comma-separated-tokens@2.0.3: {} + + commander@7.2.0: {} + + commander@8.3.0: {} + + convert-source-map@2.0.0: {} + + copy-anything@4.1.0: {} + + cose-base@1.0.3: + dependencies: + layout-base: 1.0.2 + + cose-base@2.2.0: + dependencies: + layout-base: 2.0.1 + + cross-spawn@7.0.6: + dependencies: + path-key: 3.1.1 + shebang-command: 2.0.0 + which: 2.0.2 + + csstype@3.2.3: {} + + cytoscape-cose-bilkent@4.1.0(cytoscape@3.34.2): + dependencies: + cose-base: 1.0.3 + cytoscape: 3.34.2 + + cytoscape-fcose@2.2.0(cytoscape@3.34.2): + dependencies: + cose-base: 2.2.0 + cytoscape: 3.34.2 + + cytoscape@3.34.2: {} + + d3-array@2.12.1: + dependencies: + internmap: 1.0.1 + + d3-array@3.2.4: + dependencies: + internmap: 2.0.3 + + d3-axis@3.0.0: {} + + d3-brush@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-drag: 3.0.0 + d3-interpolate: 3.0.1 + d3-selection: 3.0.0 + d3-transition: 3.0.1(d3-selection@3.0.0) + + d3-chord@3.0.1: + dependencies: + d3-path: 3.1.0 + + d3-color@3.1.0: {} + + d3-contour@4.0.2: + dependencies: + d3-array: 3.2.4 + + d3-delaunay@6.0.4: + dependencies: + delaunator: 5.1.0 + + d3-dispatch@3.0.1: {} + + d3-drag@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-selection: 3.0.0 + + d3-dsv@3.0.1: + dependencies: + commander: 7.2.0 + iconv-lite: 0.6.3 + rw: 1.3.3 + + d3-ease@3.0.1: {} + + d3-fetch@3.0.1: + dependencies: + d3-dsv: 3.0.1 + + d3-force@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-quadtree: 3.0.1 + d3-timer: 3.0.1 + + d3-format@3.1.2: {} + + d3-geo@3.1.1: + dependencies: + d3-array: 3.2.4 + + d3-hierarchy@3.1.2: {} + + d3-interpolate@3.0.1: + dependencies: + d3-color: 3.1.0 + + d3-path@1.0.9: {} + + d3-path@3.1.0: {} + + d3-polygon@3.0.1: {} + + d3-quadtree@3.0.1: {} + + d3-random@3.0.1: {} + + d3-sankey@0.12.3: + dependencies: + d3-array: 2.12.1 + d3-shape: 1.3.7 + + d3-scale-chromatic@3.1.0: + dependencies: + d3-color: 3.1.0 + d3-interpolate: 3.0.1 + + d3-scale@4.0.2: + dependencies: + d3-array: 3.2.4 + d3-format: 3.1.2 + d3-interpolate: 3.0.1 + d3-time: 3.1.0 + d3-time-format: 4.1.0 + + d3-selection@3.0.0: {} + + d3-shape@1.3.7: + dependencies: + d3-path: 1.0.9 + + d3-shape@3.2.0: + dependencies: + d3-path: 3.1.0 + + d3-time-format@4.1.0: + dependencies: + d3-time: 3.1.0 + + d3-time@3.1.0: + dependencies: + d3-array: 3.2.4 + + d3-timer@3.0.1: {} + + d3-transition@3.0.1(d3-selection@3.0.0): + dependencies: + d3-color: 3.1.0 + d3-dispatch: 3.0.1 + d3-ease: 3.0.1 + d3-interpolate: 3.0.1 + d3-selection: 3.0.0 + d3-timer: 3.0.1 + + d3-zoom@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-drag: 3.0.0 + d3-interpolate: 3.0.1 + d3-selection: 3.0.0 + d3-transition: 3.0.1(d3-selection@3.0.0) + + d3@7.9.0: + dependencies: + d3-array: 3.2.4 + d3-axis: 3.0.0 + d3-brush: 3.0.0 + d3-chord: 3.0.1 + d3-color: 3.1.0 + d3-contour: 4.0.2 + d3-delaunay: 6.0.4 + d3-dispatch: 3.0.1 + d3-drag: 3.0.0 + d3-dsv: 3.0.1 + d3-ease: 3.0.1 + d3-fetch: 3.0.1 + d3-force: 3.0.0 + d3-format: 3.1.2 + d3-geo: 3.1.1 + d3-hierarchy: 3.1.2 + d3-interpolate: 3.0.1 + d3-path: 3.1.0 + d3-polygon: 3.0.1 + d3-quadtree: 3.0.1 + d3-random: 3.0.1 + d3-scale: 4.0.2 + d3-scale-chromatic: 3.1.0 + d3-selection: 3.0.0 + d3-shape: 3.2.0 + d3-time: 3.1.0 + d3-time-format: 4.1.0 + d3-timer: 3.0.1 + d3-transition: 3.0.1(d3-selection@3.0.0) + d3-zoom: 3.0.0 + + dagre-d3-es@7.0.14: + dependencies: + d3: 7.9.0 + lodash-es: 4.18.1 + + dayjs@1.11.23: {} + + decamelize-keys@1.1.1: + dependencies: + decamelize: 1.2.0 + map-obj: 1.0.1 + + decamelize@1.2.0: {} + + delaunator@5.1.0: + dependencies: + robust-predicates: 3.0.3 + + dequal@2.0.3: {} + + devlop@1.1.0: + dependencies: + dequal: 2.0.3 + + diff-sequences@29.6.3: {} + + dir-glob@3.0.1: + dependencies: + path-type: 4.0.0 + + dompurify@3.4.14: + optionalDependencies: + '@types/trusted-types': 2.0.7 + + eastasianwidth@0.2.0: {} + + emoji-regex-xs@1.0.0: {} + + emoji-regex@8.0.0: {} + + emoji-regex@9.2.2: {} + + entities@7.0.1: {} + + error-ex@1.3.4: + dependencies: + is-arrayish: 0.2.1 + + es-errors@1.3.0: {} + + es-toolkit@1.52.0: {} + + esbuild@0.21.5: + optionalDependencies: + '@esbuild/aix-ppc64': 0.21.5 + '@esbuild/android-arm': 0.21.5 + '@esbuild/android-arm64': 0.21.5 + '@esbuild/android-x64': 0.21.5 + '@esbuild/darwin-arm64': 0.21.5 + '@esbuild/darwin-x64': 0.21.5 + '@esbuild/freebsd-arm64': 0.21.5 + '@esbuild/freebsd-x64': 0.21.5 + '@esbuild/linux-arm': 0.21.5 + '@esbuild/linux-arm64': 0.21.5 + '@esbuild/linux-ia32': 0.21.5 + '@esbuild/linux-loong64': 0.21.5 + '@esbuild/linux-mips64el': 0.21.5 + '@esbuild/linux-ppc64': 0.21.5 + '@esbuild/linux-riscv64': 0.21.5 + '@esbuild/linux-s390x': 0.21.5 + '@esbuild/linux-x64': 0.21.5 + '@esbuild/netbsd-x64': 0.21.5 + '@esbuild/openbsd-x64': 0.21.5 + '@esbuild/sunos-x64': 0.21.5 + '@esbuild/win32-arm64': 0.21.5 + '@esbuild/win32-ia32': 0.21.5 + '@esbuild/win32-x64': 0.21.5 + + esbuild@0.28.2: + optionalDependencies: + '@esbuild/aix-ppc64': 0.28.2 + '@esbuild/android-arm': 0.28.2 + '@esbuild/android-arm64': 0.28.2 + '@esbuild/android-x64': 0.28.2 + '@esbuild/darwin-arm64': 0.28.2 + '@esbuild/darwin-x64': 0.28.2 + '@esbuild/freebsd-arm64': 0.28.2 + '@esbuild/freebsd-x64': 0.28.2 + '@esbuild/linux-arm': 0.28.2 + '@esbuild/linux-arm64': 0.28.2 + '@esbuild/linux-ia32': 0.28.2 + '@esbuild/linux-loong64': 0.28.2 + '@esbuild/linux-mips64el': 0.28.2 + '@esbuild/linux-ppc64': 0.28.2 + '@esbuild/linux-riscv64': 0.28.2 + '@esbuild/linux-s390x': 0.28.2 + '@esbuild/linux-x64': 0.28.2 + '@esbuild/netbsd-arm64': 0.28.2 + '@esbuild/netbsd-x64': 0.28.2 + '@esbuild/openbsd-arm64': 0.28.2 + '@esbuild/openbsd-x64': 0.28.2 + '@esbuild/openharmony-arm64': 0.28.2 + '@esbuild/sunos-x64': 0.28.2 + '@esbuild/win32-arm64': 0.28.2 + '@esbuild/win32-ia32': 0.28.2 + '@esbuild/win32-x64': 0.28.2 + + escalade@3.2.0: {} + + eslint-formatter-pretty@4.1.0: + dependencies: + '@types/eslint': 7.29.0 + ansi-escapes: 4.3.2 + chalk: 4.1.2 + eslint-rule-docs: 1.1.235 + log-symbols: 4.1.0 + plur: 4.0.0 + string-width: 4.2.3 + supports-hyperlinks: 2.3.0 + + eslint-rule-docs@1.1.235: {} + + estree-walker@2.0.2: {} + + fast-glob@3.3.3: + dependencies: + '@nodelib/fs.stat': 2.0.5 + '@nodelib/fs.walk': 1.2.8 + glob-parent: 5.1.2 + merge2: 1.4.1 + micromatch: 4.0.8 + + fastdom@1.0.12: + dependencies: + strictdom: 1.0.1 + + fastq@1.20.3: + dependencies: + reusify: 1.1.0 + + fill-range@7.1.1: + dependencies: + to-regex-range: 5.0.1 + + find-up@4.1.0: + dependencies: + locate-path: 5.0.0 + path-exists: 4.0.0 + + find-up@5.0.0: + dependencies: + locate-path: 6.0.0 + path-exists: 4.0.0 + + focus-trap@7.8.0: + dependencies: + tabbable: 6.5.0 + + foreground-child@3.3.1: + dependencies: + cross-spawn: 7.0.6 + signal-exit: 4.1.0 + + fsevents@2.3.3: + optional: true + + function-bind@1.1.2: {} + + get-caller-file@2.0.5: {} + + glob-parent@5.1.2: + dependencies: + is-glob: 4.0.3 + + glob@10.5.0: + dependencies: + foreground-child: 3.3.1 + jackspeak: 3.4.3 + minimatch: 9.0.9 + minipass: 7.1.3 + package-json-from-dist: 1.0.1 + path-scurry: 1.11.1 + + globby@11.1.0: + dependencies: + array-union: 2.1.0 + dir-glob: 3.0.1 + fast-glob: 3.3.3 + ignore: 5.3.2 + merge2: 1.4.1 + slash: 3.0.0 + + hachure-fill@0.5.2: {} + + hard-rejection@2.1.0: {} + + has-flag@4.0.0: {} + + hasown@2.0.4: + dependencies: + function-bind: 1.1.2 + + hast-util-to-html@9.0.5: + dependencies: + '@types/hast': 3.0.5 + '@types/unist': 3.0.3 + ccount: 2.0.1 + comma-separated-tokens: 2.0.3 + hast-util-whitespace: 3.0.0 + html-void-elements: 3.0.0 + mdast-util-to-hast: 13.2.1 + property-information: 7.2.0 + space-separated-tokens: 2.0.2 + stringify-entities: 4.0.4 + zwitch: 2.0.4 + + hast-util-whitespace@3.0.0: + dependencies: + '@types/hast': 3.0.5 + + hookable@5.5.3: {} + + hosted-git-info@2.8.9: {} + + hosted-git-info@4.1.0: + dependencies: + lru-cache: 6.0.0 + + html-escaper@2.0.2: {} + + html-void-elements@3.0.0: {} + + iconv-lite@0.6.3: + dependencies: + safer-buffer: 2.1.2 + + ignore@5.3.2: {} + + import-meta-resolve@4.2.0: {} + + indent-string@4.0.0: {} + + internmap@1.0.1: {} + + internmap@2.0.3: {} + + irregular-plurals@3.5.0: {} + + is-arrayish@0.2.1: {} + + is-core-module@2.16.2: + dependencies: + hasown: 2.0.4 + + is-extglob@2.1.1: {} + + is-fullwidth-code-point@3.0.0: {} + + is-glob@4.0.3: + dependencies: + is-extglob: 2.1.1 + + is-number@7.0.0: {} + + is-plain-obj@1.1.0: {} + + is-unicode-supported@0.1.0: {} + + isexe@2.0.0: {} + + istanbul-lib-coverage@3.2.2: {} + + istanbul-lib-report@3.0.1: + dependencies: + istanbul-lib-coverage: 3.2.2 + make-dir: 4.0.0 + supports-color: 7.2.0 + + istanbul-reports@3.2.0: + dependencies: + html-escaper: 2.0.2 + istanbul-lib-report: 3.0.1 + + jackspeak@3.4.3: + dependencies: + '@isaacs/cliui': 8.0.2 + optionalDependencies: + '@pkgjs/parseargs': 0.11.0 + + jest-diff@29.7.0: + dependencies: + chalk: 4.1.2 + diff-sequences: 29.6.3 + jest-get-type: 29.6.3 + pretty-format: 29.7.0 + + jest-get-type@29.6.3: {} + + js-tokens@4.0.0: {} + + json-parse-even-better-errors@2.3.1: {} + + katex@0.16.47: + dependencies: + commander: 8.3.0 + + khroma@2.1.0: {} + + kind-of@6.0.3: {} + + layout-base@1.0.2: {} + + layout-base@2.0.1: {} + + lines-and-columns@1.2.4: {} + + locate-path@5.0.0: + dependencies: + p-locate: 4.1.0 + + locate-path@6.0.0: + dependencies: + p-locate: 5.0.0 + + lodash-es@4.18.1: {} + + log-symbols@4.1.0: + dependencies: + chalk: 4.1.2 + is-unicode-supported: 0.1.0 + + lru-cache@10.4.3: {} + + lru-cache@6.0.0: + dependencies: + yallist: 4.0.0 + + magic-string@0.30.21: + dependencies: + '@jridgewell/sourcemap-codec': 1.6.0 + + make-dir@4.0.0: + dependencies: + semver: 7.8.5 + + map-obj@1.0.1: {} + + map-obj@4.3.0: {} + + mark.js@8.11.1: {} + + marked@16.4.2: {} + + mdast-util-to-hast@13.2.1: + dependencies: + '@types/hast': 3.0.5 + '@types/mdast': 4.0.4 + '@ungap/structured-clone': 1.4.0 + devlop: 1.1.0 + micromark-util-sanitize-uri: 2.0.1 + trim-lines: 3.0.1 + unist-util-position: 5.0.0 + unist-util-visit: 5.1.0 + vfile: 6.0.3 + + meow@9.0.0: + dependencies: + '@types/minimist': 1.2.5 + camelcase-keys: 6.2.2 + decamelize: 1.2.0 + decamelize-keys: 1.1.1 + hard-rejection: 2.1.0 + minimist-options: 4.1.0 + normalize-package-data: 3.0.3 + read-pkg-up: 7.0.1 + redent: 3.0.0 + trim-newlines: 3.0.1 + type-fest: 0.18.1 + yargs-parser: 20.2.9 + + merge2@1.4.1: {} + + mermaid@11.17.2: + dependencies: + '@braintree/sanitize-url': 7.1.2 + '@iconify/utils': 3.1.5 + '@mermaid-js/parser': 1.2.1 + '@types/d3': 7.4.3 + '@upsetjs/venn.js': 2.0.0 + cytoscape: 3.34.2 + cytoscape-cose-bilkent: 4.1.0(cytoscape@3.34.2) + cytoscape-fcose: 2.2.0(cytoscape@3.34.2) + d3: 7.9.0 + d3-sankey: 0.12.3 + dagre-d3-es: 7.0.14 + dayjs: 1.11.23 + dompurify: 3.4.14 + es-toolkit: 1.52.0 + fastdom: 1.0.12 + katex: 0.16.47 + khroma: 2.1.0 + marked: 16.4.2 + roughjs: 4.6.6 + stylis: 4.4.0 + ts-dedent: 2.3.0 + uuid: 14.0.2 + + micromark-util-character@2.1.1: + dependencies: + micromark-util-symbol: 2.0.1 + micromark-util-types: 2.0.2 + + micromark-util-encode@2.0.1: {} + + micromark-util-sanitize-uri@2.0.1: + dependencies: + micromark-util-character: 2.1.1 + micromark-util-encode: 2.0.1 + micromark-util-symbol: 2.0.1 + + micromark-util-symbol@2.0.1: {} + + micromark-util-types@2.0.2: {} + + micromatch@4.0.8: + dependencies: + braces: 3.0.3 + picomatch: 2.3.2 + + min-indent@1.0.1: {} + + minimatch@10.2.6: + dependencies: + brace-expansion: 5.0.9 + + minimatch@9.0.9: + dependencies: + brace-expansion: 2.1.4 + + minimist-options@4.1.0: + dependencies: + arrify: 1.0.1 + is-plain-obj: 1.1.0 + kind-of: 6.0.3 + + minipass@7.1.3: {} + + minisearch@7.2.0: {} + + mitt@3.0.1: {} + + nanoid@3.3.18: {} + + non-layered-tidy-tree-layout@2.0.2: + optional: true + + normalize-package-data@2.5.0: + dependencies: + hosted-git-info: 2.8.9 + resolve: 1.22.12 + semver: 5.7.2 + validate-npm-package-license: 3.0.4 + + normalize-package-data@3.0.3: + dependencies: + hosted-git-info: 4.1.0 + is-core-module: 2.16.2 + semver: 7.8.5 + validate-npm-package-license: 3.0.4 + + on-exit-leak-free@2.1.2: {} + + oniguruma-to-es@3.1.1: + dependencies: + emoji-regex-xs: 1.0.0 + regex: 6.1.0 + regex-recursion: 6.0.2 + + oxfmt@0.58.0: + dependencies: + tinypool: 2.1.0 + optionalDependencies: + '@oxfmt/binding-android-arm-eabi': 0.58.0 + '@oxfmt/binding-android-arm64': 0.58.0 + '@oxfmt/binding-darwin-arm64': 0.58.0 + '@oxfmt/binding-darwin-x64': 0.58.0 + '@oxfmt/binding-freebsd-x64': 0.58.0 + '@oxfmt/binding-linux-arm-gnueabihf': 0.58.0 + '@oxfmt/binding-linux-arm-musleabihf': 0.58.0 + '@oxfmt/binding-linux-arm64-gnu': 0.58.0 + '@oxfmt/binding-linux-arm64-musl': 0.58.0 + '@oxfmt/binding-linux-ppc64-gnu': 0.58.0 + '@oxfmt/binding-linux-riscv64-gnu': 0.58.0 + '@oxfmt/binding-linux-riscv64-musl': 0.58.0 + '@oxfmt/binding-linux-s390x-gnu': 0.58.0 + '@oxfmt/binding-linux-x64-gnu': 0.58.0 + '@oxfmt/binding-linux-x64-musl': 0.58.0 + '@oxfmt/binding-openharmony-arm64': 0.58.0 + '@oxfmt/binding-win32-arm64-msvc': 0.58.0 + '@oxfmt/binding-win32-ia32-msvc': 0.58.0 + '@oxfmt/binding-win32-x64-msvc': 0.58.0 + + oxlint@1.81.0: + optionalDependencies: + '@oxlint/binding-android-arm-eabi': 1.81.0 + '@oxlint/binding-android-arm64': 1.81.0 + '@oxlint/binding-darwin-arm64': 1.81.0 + '@oxlint/binding-darwin-x64': 1.81.0 + '@oxlint/binding-freebsd-x64': 1.81.0 + '@oxlint/binding-linux-arm-gnueabihf': 1.81.0 + '@oxlint/binding-linux-arm-musleabihf': 1.81.0 + '@oxlint/binding-linux-arm64-gnu': 1.81.0 + '@oxlint/binding-linux-arm64-musl': 1.81.0 + '@oxlint/binding-linux-ppc64-gnu': 1.81.0 + '@oxlint/binding-linux-riscv64-gnu': 1.81.0 + '@oxlint/binding-linux-riscv64-musl': 1.81.0 + '@oxlint/binding-linux-s390x-gnu': 1.81.0 + '@oxlint/binding-linux-x64-gnu': 1.81.0 + '@oxlint/binding-linux-x64-musl': 1.81.0 + '@oxlint/binding-openharmony-arm64': 1.81.0 + '@oxlint/binding-win32-arm64-msvc': 1.81.0 + '@oxlint/binding-win32-ia32-msvc': 1.81.0 + '@oxlint/binding-win32-x64-msvc': 1.81.0 + + p-limit@2.3.0: + dependencies: + p-try: 2.2.0 + + p-limit@3.1.0: + dependencies: + yocto-queue: 0.1.0 + + p-locate@4.1.0: + dependencies: + p-limit: 2.3.0 + + p-locate@5.0.0: + dependencies: + p-limit: 3.1.0 + + p-try@2.2.0: {} + + package-json-from-dist@1.0.1: {} + + package-manager-detector@1.8.0: {} + + parse-json@5.2.0: + dependencies: + '@babel/code-frame': 7.29.7 + error-ex: 1.3.4 + json-parse-even-better-errors: 2.3.1 + lines-and-columns: 1.2.4 + + path-data-parser@0.1.0: {} + + path-exists@4.0.0: {} + + path-key@3.1.1: {} + + path-parse@1.0.7: {} + + path-scurry@1.11.1: + dependencies: + lru-cache: 10.4.3 + minipass: 7.1.3 + + path-type@4.0.0: {} + + perfect-debounce@1.0.0: {} + + picocolors@1.1.1: {} + + picomatch@2.3.2: {} + + pino-abstract-transport@3.0.0: + dependencies: + split2: 4.2.0 + + pino-std-serializers@7.1.0: {} + + pino@10.3.1: + dependencies: + '@pinojs/redact': 0.4.0 + atomic-sleep: 1.0.0 + on-exit-leak-free: 2.1.2 + pino-abstract-transport: 3.0.0 + pino-std-serializers: 7.1.0 + process-warning: 5.1.0 + quick-format-unescaped: 4.0.4 + real-require: 0.2.0 + safe-stable-stringify: 2.5.0 + sonic-boom: 4.2.1 + thread-stream: 4.2.0 + + plur@4.0.0: + dependencies: + irregular-plurals: 3.5.0 + + points-on-curve@0.2.0: {} + + points-on-path@0.2.1: + dependencies: + path-data-parser: 0.1.0 + points-on-curve: 0.2.0 + + postcss@8.5.28: + dependencies: + nanoid: 3.3.18 + picocolors: 1.1.1 + source-map-js: 1.2.1 + + preact@10.29.8: {} + + pretty-format@29.7.0: + dependencies: + '@jest/schemas': 29.6.3 + ansi-styles: 5.2.0 + react-is: 18.3.1 + + process-warning@5.1.0: {} + + property-information@7.2.0: {} + + queue-microtask@1.2.3: {} + + quick-format-unescaped@4.0.4: {} + + quick-lru@4.0.1: {} + + react-is@18.3.1: {} + + read-pkg-up@7.0.1: + dependencies: + find-up: 4.1.0 + read-pkg: 5.2.0 + type-fest: 0.8.1 + + read-pkg@5.2.0: + dependencies: + '@types/normalize-package-data': 2.4.4 + normalize-package-data: 2.5.0 + parse-json: 5.2.0 + type-fest: 0.6.0 + + real-require@0.2.0: {} + + real-require@1.0.0: {} + + redent@3.0.0: + dependencies: + indent-string: 4.0.0 + strip-indent: 3.0.0 + + regex-recursion@6.0.2: + dependencies: + regex-utilities: 2.3.0 + + regex-utilities@2.3.0: {} + + regex@6.1.0: + dependencies: + regex-utilities: 2.3.0 + + require-directory@2.1.1: {} + + resolve@1.22.12: + dependencies: + es-errors: 1.3.0 + is-core-module: 2.16.2 + path-parse: 1.0.7 + supports-preserve-symlinks-flag: 1.0.0 + + reusify@1.1.0: {} + + rfdc@1.4.1: {} + + robust-predicates@3.0.3: {} + + rollup@4.63.1: + dependencies: + '@types/estree': 1.0.9 + optionalDependencies: + '@napi-rs/lzma-linux-x64-gnu': 1.5.1 + '@rollup/rollup-android-arm-eabi': 4.63.1 + '@rollup/rollup-android-arm64': 4.63.1 + '@rollup/rollup-darwin-arm64': 4.63.1 + '@rollup/rollup-darwin-x64': 4.63.1 + '@rollup/rollup-freebsd-arm64': 4.63.1 + '@rollup/rollup-freebsd-x64': 4.63.1 + '@rollup/rollup-linux-arm-gnueabihf': 4.63.1 + '@rollup/rollup-linux-arm-musleabihf': 4.63.1 + '@rollup/rollup-linux-arm64-gnu': 4.63.1 + '@rollup/rollup-linux-arm64-musl': 4.63.1 + '@rollup/rollup-linux-loong64-gnu': 4.63.1 + '@rollup/rollup-linux-loong64-musl': 4.63.1 + '@rollup/rollup-linux-ppc64-gnu': 4.63.1 + '@rollup/rollup-linux-ppc64-musl': 4.63.1 + '@rollup/rollup-linux-riscv64-gnu': 4.63.1 + '@rollup/rollup-linux-riscv64-musl': 4.63.1 + '@rollup/rollup-linux-s390x-gnu': 4.63.1 + '@rollup/rollup-linux-x64-gnu': 4.63.1 + '@rollup/rollup-linux-x64-musl': 4.63.1 + '@rollup/rollup-openbsd-x64': 4.63.1 + '@rollup/rollup-openharmony-arm64': 4.63.1 + '@rollup/rollup-win32-arm64-msvc': 4.63.1 + '@rollup/rollup-win32-ia32-msvc': 4.63.1 + '@rollup/rollup-win32-x64-gnu': 4.63.1 + '@rollup/rollup-win32-x64-msvc': 4.63.1 + fsevents: 2.3.3 + + roughjs@4.6.6: + dependencies: + hachure-fill: 0.5.2 + path-data-parser: 0.1.0 + points-on-curve: 0.2.0 + points-on-path: 0.2.1 + + run-parallel@1.2.0: + dependencies: + queue-microtask: 1.2.3 + + rw@1.3.3: {} + + safe-stable-stringify@2.5.0: {} + + safer-buffer@2.1.2: {} + + search-insights@2.17.3: {} + + semver@5.7.2: {} + + semver@7.8.5: {} + + shebang-command@2.0.0: + dependencies: + shebang-regex: 3.0.0 + + shebang-regex@3.0.0: {} + + shiki@2.5.0: + dependencies: + '@shikijs/core': 2.5.0 + '@shikijs/engine-javascript': 2.5.0 + '@shikijs/engine-oniguruma': 2.5.0 + '@shikijs/langs': 2.5.0 + '@shikijs/themes': 2.5.0 + '@shikijs/types': 2.5.0 + '@shikijs/vscode-textmate': 10.0.2 + '@types/hast': 3.0.5 + + signal-exit@4.1.0: {} + + slash@3.0.0: {} + + sonic-boom@4.2.1: + dependencies: + atomic-sleep: 1.0.0 + + source-map-js@1.2.1: {} + + space-separated-tokens@2.0.2: {} + + spdx-correct@3.2.0: + dependencies: + spdx-expression-parse: 3.0.1 + spdx-license-ids: 3.0.23 + + spdx-exceptions@2.5.0: {} + + spdx-expression-parse@3.0.1: + dependencies: + spdx-exceptions: 2.5.0 + spdx-license-ids: 3.0.23 + + spdx-license-ids@3.0.23: {} + + speakingurl@14.0.1: {} + + split2@4.2.0: {} + + strictdom@1.0.1: {} + + string-width@4.2.3: + dependencies: + emoji-regex: 8.0.0 + is-fullwidth-code-point: 3.0.0 + strip-ansi: 6.0.1 + + string-width@5.1.2: + dependencies: + eastasianwidth: 0.2.0 + emoji-regex: 9.2.2 + strip-ansi: 7.2.0 + + stringify-entities@4.0.4: + dependencies: + character-entities-html4: 2.1.0 + character-entities-legacy: 3.0.0 + + strip-ansi@6.0.1: + dependencies: + ansi-regex: 5.0.1 + + strip-ansi@7.2.0: + dependencies: + ansi-regex: 6.3.0 + + strip-indent@3.0.0: + dependencies: + min-indent: 1.0.1 + + stylis@4.4.0: {} + + superjson@2.2.6: + dependencies: + copy-anything: 4.1.0 + + supports-color@7.2.0: + dependencies: + has-flag: 4.0.0 + + supports-hyperlinks@2.3.0: + dependencies: + has-flag: 4.0.0 + supports-color: 7.2.0 + + supports-preserve-symlinks-flag@1.0.0: {} + + tabbable@6.5.0: {} + + test-exclude@7.0.2: + dependencies: + '@istanbuljs/schema': 0.1.6 + glob: 10.5.0 + minimatch: 10.2.6 + + thread-stream@4.2.0: + dependencies: + real-require: 1.0.0 + + tinyexec@1.3.1: {} + + tinypool@2.1.0: {} + + to-regex-range@5.0.1: + dependencies: + is-number: 7.0.0 + + trim-lines@3.0.1: {} + + trim-newlines@3.0.1: {} + + ts-dedent@2.3.0: {} + + tsd@0.31.2: + dependencies: + '@tsd/typescript': 5.4.5 + eslint-formatter-pretty: 4.1.0 + globby: 11.1.0 + jest-diff: 29.7.0 + meow: 9.0.0 + path-exists: 4.0.0 + read-pkg-up: 7.0.1 + + type-fest@0.18.1: {} + + type-fest@0.21.3: {} + + type-fest@0.6.0: {} + + type-fest@0.8.1: {} + + typescript@7.0.2: + optionalDependencies: + '@typescript/typescript-aix-ppc64': 7.0.2 + '@typescript/typescript-darwin-arm64': 7.0.2 + '@typescript/typescript-darwin-x64': 7.0.2 + '@typescript/typescript-freebsd-arm64': 7.0.2 + '@typescript/typescript-freebsd-x64': 7.0.2 + '@typescript/typescript-linux-arm': 7.0.2 + '@typescript/typescript-linux-arm64': 7.0.2 + '@typescript/typescript-linux-loong64': 7.0.2 + '@typescript/typescript-linux-mips64el': 7.0.2 + '@typescript/typescript-linux-ppc64': 7.0.2 + '@typescript/typescript-linux-riscv64': 7.0.2 + '@typescript/typescript-linux-s390x': 7.0.2 + '@typescript/typescript-linux-x64': 7.0.2 + '@typescript/typescript-netbsd-arm64': 7.0.2 + '@typescript/typescript-netbsd-x64': 7.0.2 + '@typescript/typescript-openbsd-arm64': 7.0.2 + '@typescript/typescript-openbsd-x64': 7.0.2 + '@typescript/typescript-sunos-x64': 7.0.2 + '@typescript/typescript-win32-arm64': 7.0.2 + '@typescript/typescript-win32-x64': 7.0.2 + + unist-util-is@6.0.1: + dependencies: + '@types/unist': 3.0.3 + + unist-util-position@5.0.0: + dependencies: + '@types/unist': 3.0.3 + + unist-util-stringify-position@4.0.0: + dependencies: + '@types/unist': 3.0.3 + + unist-util-visit-parents@6.0.2: + dependencies: + '@types/unist': 3.0.3 + unist-util-is: 6.0.1 + + unist-util-visit@5.1.0: + dependencies: + '@types/unist': 3.0.3 + unist-util-is: 6.0.1 + unist-util-visit-parents: 6.0.2 + + uuid@14.0.2: {} + + v8-to-istanbul@9.3.0: + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@types/istanbul-lib-coverage': 2.0.6 + convert-source-map: 2.0.0 + + validate-npm-package-license@3.0.4: + dependencies: + spdx-correct: 3.2.0 + spdx-expression-parse: 3.0.1 + + vfile-message@4.0.3: + dependencies: + '@types/unist': 3.0.3 + unist-util-stringify-position: 4.0.0 + + vfile@6.0.3: + dependencies: + '@types/unist': 3.0.3 + vfile-message: 4.0.3 + + vite@5.4.21: + dependencies: + esbuild: 0.21.5 + postcss: 8.5.28 + rollup: 4.63.1 + optionalDependencies: + fsevents: 2.3.3 + + vitepress-plugin-mermaid@2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)): + dependencies: + mermaid: 11.17.2 + vitepress: 1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) + optionalDependencies: + '@mermaid-js/mermaid-mindmap': 9.3.0 + + vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2): + dependencies: + '@docsearch/css': 3.8.2 + '@docsearch/js': 3.8.2(@algolia/client-search@5.57.0)(search-insights@2.17.3) + '@iconify-json/simple-icons': 1.2.94 + '@shikijs/core': 2.5.0 + '@shikijs/transformers': 2.5.0 + '@shikijs/types': 2.5.0 + '@types/markdown-it': 14.2.0 + '@vitejs/plugin-vue': 5.2.4(vite@5.4.21)(vue@3.5.42(typescript@7.0.2)) + '@vue/devtools-api': 7.7.10 + '@vue/shared': 3.5.42 + '@vueuse/core': 12.8.2(typescript@7.0.2) + '@vueuse/integrations': 12.8.2(focus-trap@7.8.0)(typescript@7.0.2) + focus-trap: 7.8.0 + mark.js: 8.11.1 + minisearch: 7.2.0 + shiki: 2.5.0 + vite: 5.4.21 + vue: 3.5.42(typescript@7.0.2) + optionalDependencies: + postcss: 8.5.28 + transitivePeerDependencies: + - '@algolia/client-search' + - '@types/node' + - '@types/react' + - async-validator + - axios + - change-case + - drauu + - fuse.js + - idb-keyval + - jwt-decode + - less + - lightningcss + - nprogress + - preact-render-to-string + - qrcode + - react + - react-dom + - sass + - sass-embedded + - search-insights + - sortablejs + - stylus + - sugarss + - terser + - typescript + - universal-cookie + + vue@3.5.42(typescript@7.0.2): + dependencies: + '@vue/compiler-dom': 3.5.42 + '@vue/compiler-sfc': 3.5.42 + '@vue/runtime-dom': 3.5.42 + '@vue/server-renderer': 3.5.42 + '@vue/shared': 3.5.42 + optionalDependencies: + typescript: 7.0.2 + + which@2.0.2: + dependencies: + isexe: 2.0.0 + + wrap-ansi@7.0.0: + dependencies: + ansi-styles: 4.3.0 + string-width: 4.2.3 + strip-ansi: 6.0.1 + + wrap-ansi@8.1.0: + dependencies: + ansi-styles: 6.2.3 + string-width: 5.1.2 + strip-ansi: 7.2.0 + + y18n@5.0.8: {} + + yallist@4.0.0: {} + + yargs-parser@20.2.9: {} + + yargs-parser@21.1.1: {} + + yargs@17.7.3: + dependencies: + cliui: 8.0.1 + escalade: 3.2.0 + get-caller-file: 2.0.5 + require-directory: 2.1.1 + string-width: 4.2.3 + y18n: 5.0.8 + yargs-parser: 21.1.1 + + yocto-queue@0.1.0: {} + + zwitch@2.0.4: {} diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml new file mode 100644 index 0000000..5ed0b5a --- /dev/null +++ b/pnpm-workspace.yaml @@ -0,0 +1,2 @@ +allowBuilds: + esbuild: true From 6e7890a4c64f426f734596e6b2ce6afa3de66f63 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Fri, 4 Sep 2026 18:19:27 +0300 Subject: [PATCH 03/67] fix: bring syncom infra up to par after globalstorage migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The migration from metarhia/globalstorage copied files but adapted nothing: require('@alexify/syncom') threw on a missing metautil dependency, browser.js was an 810-line hand-duplicated bundle importing a nonexistent ./metautil.js, npm test matched zero files, and bench/docs/scripts were empty directories already wired into six npm scripts. - Vendor the 5 helpers syncom's src/ needs (Emitter, generateUUID, between, fileExists, directoryExists) into src/utils.js / utils.browser.js, keeping the zero-runtime-dependency promise honest - Replace browser.js with a 2-line shim over a real src/index.browser.js barrel; rename browser modules to the *.browser.js pattern - Fix two real bugs surfaced while wiring this up: Storage.validate() threw on no arguments instead of validating the whole chain, and Emitter.emit silently dropped every argument after the first (Record's `update` event never delivered its delta) - Rename tests/*.js to *.test.js and fix stale ../lib/ paths so `pnpm test` actually runs (0 -> 130 passing tests, 97%+ coverage) - Add scripts/size.js (bundle-size budget + self-contained bundle check), bench/ (harness + 5 benchmarks), and tests/package/consistency.test.js (10 checks tying package.json/exports/files/docs together) - Add a VitePress docs site, rewritten README, CHANGELOG, SECURITY, CONTRIBUTING, CLAUDE.md, and a 3-job GitHub Actions CI workflow - Remove emulation/ and the broken root example.js (legacy dead code) No new features โ€” this only makes the copied code actually run and gives it the same tooling wrpc has. --- .github/workflows/ci.yml | 60 ++ .oxlintrc.json | 2 +- CHANGELOG.md | 72 ++ CLAUDE.md | 73 ++ CONTRIBUTING.md | 105 +++ README.md | 214 +++-- SECURITY.md | 50 ++ bench/chain.js | 67 ++ bench/crdt.js | 89 ++ bench/emitter.js | 82 ++ bench/hash.js | 50 ++ bench/run-all.js | 23 + bench/storage.js | 72 ++ bench/support/harness.js | 52 ++ browser.d.ts | 59 ++ browser.js | 811 +----------------- docs/.vitepress/config.mts | 198 +++++ docs/.vitepress/theme/custom.css | 64 ++ docs/.vitepress/theme/index.ts | 35 + docs/guide/browser.md | 69 ++ docs/guide/chain.md | 73 ++ docs/guide/contracts.md | 65 ++ docs/guide/crdt.md | 98 +++ docs/guide/getting-started.md | 90 ++ docs/guide/storage.md | 124 +++ docs/guide/sync.md | 57 ++ docs/index.md | 51 ++ docs/public/llms.txt | 32 + docs/public/logo-mark.svg | 14 + docs/public/robots.txt | 4 + docs/reference/api.md | 137 +++ emulation/data.js | 392 --------- emulation/example.js | 147 ---- emulation/storage.js | 360 -------- example.js | 91 -- examples/main.js | 4 +- index.d.ts | 29 +- index.js | 8 +- package.json | 49 +- pnpm-lock.yaml | 218 ++--- scripts/size.js | 112 +++ src/contract.js | 4 + src/crdt.js | 13 +- src/{crypto-browser.js => crypto.browser.js} | 2 +- src/{fsi-browser.js => fsi.browser.js} | 0 src/fsi.js | 2 +- src/index.browser.js | 15 + src/index.js | 11 + src/storage.js | 14 +- src/{sync-browser.js => sync.browser.js} | 8 +- src/sync.js | 12 +- src/utils.browser.js | 123 +++ src/utils.js | 141 +++ tests/browser.test-d.ts | 20 + tests/{chain.js => chain.test.js} | 16 +- tests/{contract.js => contract.test.js} | 40 +- tests/{crdt.js => crdt.test.js} | 71 +- ...ypto-browser.js => crypto.browser.test.js} | 7 +- tests/{fsi-browser.js => fsi.browser.test.js} | 7 +- tests/{fsi.js => fsi.test.js} | 2 +- tests/index.browser.test.js | 64 ++ tests/index.test-d.ts | 80 ++ tests/integration.js | 51 -- tests/integration.test.js | 44 + tests/{keys.js => keys.test.js} | 2 +- tests/package/consistency.test.js | 138 +++ tests/{storage.js => storage.test.js} | 71 +- tests/suites/crypto-keys.js | 15 +- tests/suites/emitter.js | 112 +++ tests/suites/sync-manager.js | 10 +- .../{sync-browser.js => sync.browser.test.js} | 2 +- tests/{sync.js => sync.test.js} | 4 +- tests/utils.browser.test.js | 43 + tests/utils.test.js | 41 + tests/utils/mock-opfs.js | 5 +- vercel.json | 7 + 76 files changed, 3372 insertions(+), 2227 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 CHANGELOG.md create mode 100644 CLAUDE.md create mode 100644 CONTRIBUTING.md create mode 100644 SECURITY.md create mode 100644 bench/chain.js create mode 100644 bench/crdt.js create mode 100644 bench/emitter.js create mode 100644 bench/hash.js create mode 100644 bench/run-all.js create mode 100644 bench/storage.js create mode 100644 bench/support/harness.js create mode 100644 browser.d.ts create mode 100644 docs/.vitepress/config.mts create mode 100644 docs/.vitepress/theme/custom.css create mode 100644 docs/.vitepress/theme/index.ts create mode 100644 docs/guide/browser.md create mode 100644 docs/guide/chain.md create mode 100644 docs/guide/contracts.md create mode 100644 docs/guide/crdt.md create mode 100644 docs/guide/getting-started.md create mode 100644 docs/guide/storage.md create mode 100644 docs/guide/sync.md create mode 100644 docs/index.md create mode 100644 docs/public/llms.txt create mode 100644 docs/public/logo-mark.svg create mode 100644 docs/public/robots.txt create mode 100644 docs/reference/api.md delete mode 100644 emulation/data.js delete mode 100644 emulation/example.js delete mode 100644 emulation/storage.js delete mode 100644 example.js create mode 100644 scripts/size.js rename src/{crypto-browser.js => crypto.browser.js} (98%) rename src/{fsi-browser.js => fsi.browser.js} (100%) create mode 100644 src/index.browser.js rename src/{sync-browser.js => sync.browser.js} (83%) create mode 100644 src/utils.browser.js create mode 100644 src/utils.js create mode 100644 tests/browser.test-d.ts rename tests/{chain.js => chain.test.js} (84%) rename tests/{contract.js => contract.test.js} (76%) rename tests/{crdt.js => crdt.test.js} (71%) rename tests/{crypto-browser.js => crypto.browser.test.js} (83%) rename tests/{fsi-browser.js => fsi.browser.test.js} (62%) rename tests/{fsi.js => fsi.test.js} (88%) create mode 100644 tests/index.browser.test.js create mode 100644 tests/index.test-d.ts delete mode 100644 tests/integration.js create mode 100644 tests/integration.test.js rename tests/{keys.js => keys.test.js} (90%) create mode 100644 tests/package/consistency.test.js rename tests/{storage.js => storage.test.js} (83%) create mode 100644 tests/suites/emitter.js rename tests/{sync-browser.js => sync.browser.test.js} (82%) rename tests/{sync.js => sync.test.js} (81%) create mode 100644 tests/utils.browser.test.js create mode 100644 tests/utils.test.js create mode 100644 vercel.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..52eff18 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,60 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm lint + - run: pnpm format:check + # Smoke: keeps the bundle-size script working and its numbers visible in + # CI logs. It also enforces the browser budget and rejects any bare + # import that reaches a browser bundle. + - run: pnpm size + + test: + runs-on: ubuntu-latest + # RSA-2048 key generation runs once per Storage and the chain benchmarks + # are filesystem-bound, so a wedged test costs real time โ€” cap the job + # rather than let a hang burn the six-hour default. + timeout-minutes: 20 + strategy: + matrix: + node-version: [22, 24] + steps: + - uses: actions/checkout@v4 + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node-version }} + cache: pnpm + - run: pnpm install --frozen-lockfile + - run: pnpm test:coverage + - run: pnpm test:types + + # Node 22 only: the docs build exercises VitePress, not the package runtime, + # so repeating it across the test matrix would just repeat the same work. + docs: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + - uses: pnpm/action-setup@v4 + - uses: actions/setup-node@v4 + with: + node-version: 22 + cache: pnpm + - run: pnpm install --frozen-lockfile + # A broken docs build (including dead links) should fail the PR, not the deploy. + - run: pnpm run docs:build diff --git a/.oxlintrc.json b/.oxlintrc.json index 1200964..fa75f73 100644 --- a/.oxlintrc.json +++ b/.oxlintrc.json @@ -10,6 +10,7 @@ }, "env": { "builtin": true, + "browser": true, "es2026": true, "node": true }, @@ -203,7 +204,6 @@ "no-useless-escape": "error", "no-useless-rename": "error", "no-useless-return": "error", - "no-void": "error", "no-with": "error", "prefer-const": [ "error", diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f69b40e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,72 @@ +# Changelog + +All notable changes to **`@alexify/syncom`** are documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +This package has not been published to npm yet โ€” there is no released version +section until the first publish; everything below lands there on release day. + +### Added + +- `src/utils.js` and `src/utils.browser.js` โ€” the `Emitter`, `generateUUID`, + `between`, `fileExists` and `directoryExists` helpers the source needs, + vendored in so the package keeps its zero-runtime-dependency promise. +- `browser.d.ts` โ€” the browser half of the type surface, declaring + `calculateHash`, `encrypt` and `decrypt` as promise-returning (Web Crypto is + async where `node:crypto` is not) and re-exporting everything else from + `index.d.ts`. +- `scripts/size.js` โ€” bundle-size report per `exports` entry, with a + min+gzip budget on the browser entry that fails the run when exceeded, and a + resolve-time check that rejects any bare import reaching a browser bundle. +- `bench/` โ€” `crdt`, `hash`, `chain`, `storage` and `emitter` benchmarks over a + shared harness, plus `run-all.js` so `pnpm bench` covers the whole folder. +- `tests/package/consistency.test.js` โ€” asserts that `exports`, `files`, the + `browser` map, the `.d.ts` files, the `scripts/size.js` rows and the docs + keywords/version all agree with `package.json`. +- `tests/utils.test.js`, `tests/utils.browser.test.js`, + `tests/index.browser.test.js` and `tests/{index,browser}.test-d.ts`. +- A VitePress documentation site under `docs/`, and `.github/workflows/ci.yml` + running lint + format + size, the test matrix, and the docs build. +- `CHANGELOG.md`, `SECURITY.md`, `CONTRIBUTING.md`, `CLAUDE.md`, `vercel.json`. + +### Fixed + +- `Storage.validate()` threw `Cannot use 'in' operator` when called with no + arguments, and silently returned `false` for `validate({})`. It discriminated + on `'last' in idOrOptions`; it now keys off whether the first argument is a + record id (a string), which is what actually separates the two call shapes. +- `Emitter.emit` dropped every argument after the first, so the `update` event + a `Record` emits delivered `data` but never `delta` โ€” despite `index.d.ts` + typing the listener as `(data, delta) => void`. +- `browser.js` was a hand-inlined 810-line duplicate of `src/`, importing a + `./metautil.js` that did not exist and calling an `fsi` binding that was + never defined. It is now a two-line shim over `src/index.browser.js`. +- `index.js` did not re-export `SyncManager`, which `index.d.ts` declared. +- `package.json#browser` mapped `./src/runtime/node.js` to + `./src/runtime/browser.js`; neither path existed. +- The `test` script globbed `tests/*.test.js` while every test file was named + `tests/.js`, so it matched nothing. Five tests additionally required a + `../lib/` directory that does not exist in this layout. +- `examples/main.js` required a `globalstorage.js` that does not exist. + +### Changed (breaking) + +- **Node.js โ‰ฅ 22 is required** (was `>=18`). +- Browser modules follow the `*.browser.js` pattern: + `src/{crypto,fsi,sync}-browser.js` โ†’ `src/{crypto,fsi,sync}.browser.js`. +- The default storage directory is `syncom` (was `gs`). +- Type names `GsRecord` / `GsCollection` โ†’ `SyncomRecord` / `SyncomCollection`. +- `exports` gained a `browser` condition resolving to `browser.js` / + `browser.d.ts`, and a `./package.json` entry. +- The `bench:compare` and `size:compare` scripts were removed; they pointed at + files that were never written. + +### Removed + +- `emulation/` โ€” a legacy prototype duplicating `Storage` and `SyncManager`. +- The root `example.js` โ€” it required two modules that do not exist. + `examples/` covers the same ground. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b4c6f69 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,73 @@ +# CLAUDE.md + +Repo-specific notes for agents working on `@alexify/syncom`. Read +[CONTRIBUTING.md](./CONTRIBUTING.md) too โ€” this file only adds what is easy to +get wrong. + +## What this package is + +A local-first data engine: records on disk (or OPFS), every write committed to +a hash-linked log, CRDT deltas for merging. **Zero runtime dependencies**, and +that is a load-bearing constraint, not a preference. + +## Layout + +``` +index.js / browser.js two-line shims over src/index.js / src/index.browser.js +index.d.ts the Node type surface (hand-written) +browser.d.ts re-exports index.d.ts, overrides the 3 async crypto fns +src/ ships as-is โ€” no build step, CommonJS, 'use strict' +tests/ *.test.js are tests; anything else is a helper +bench/ one file per subject + support/harness.js +scripts/size.js bundle-size report with a browser budget +docs/ VitePress site +``` + +## Rules that bite + +- **CommonJS everywhere in `src/`.** No `import`, no `type: module`, no build. +- **No runtime dependencies.** Helpers go in `src/utils.js`. `scripts/size.js` + fails at resolve time on any bare import reaching a browser bundle, so a + stray `require('some-pkg')` in shared code breaks `pnpm size`, not just the + dependency count. +- **Browser halves are `*.browser.js`** (dot, not dash). Adding one means + three edits: the file, `package.json#browser`, and `src/index.browser.js`. + `tests/package/consistency.test.js` catches a missed one. +- **Async asymmetry is real.** `calculateHash`, `encrypt` and `decrypt` are + synchronous in `src/crypto.js` and promise-returning in + `src/crypto.browser.js`. Shared code must `await` them; `browser.d.ts` + exists to declare that difference. Do not "fix" it by making the Node half + async. +- **A `.d.ts` change and its `tsd` test land together** (`tests/*.test-d.ts`). +- **`tests/package/consistency.test.js` guards the metadata.** If it goes red + after you add an entry point, update `exports`, `files`, the `browser` map, + `scripts/size.js` `ENTRIES` and the docs keywords โ€” the test is right. +- **Constructors that return promises.** `Storage`, `Blockchain` and + `SyncManager` all `return this.#init()` from the constructor, so they are + used as `await new Blockchain(path)`. The types model this; keep it that way. +- **`merge` adds numbers.** `merge({ hits: 10 }, { hits: 1 })` is `11`. This is + the counter CRDT and it is deliberate โ€” but it means a `diff` of numeric + fields is not a plain patch. Do not "correct" it, and do not write docs or + examples that assume replacement. + +## Testing + +`node:test` + `node:assert`. Browser halves run under Node with the OPFS mock +(`tests/utils/mock-opfs.js`) and `node:crypto`'s `webcrypto` โ€” see +`tests/index.browser.test.js`. Cross-platform behaviour goes in `tests/suites/` +and is run against both halves. + +Coverage thresholds (95/95/90/95) are enforced in CI. They currently pass with +headroom; do not lower them to land a change. + +## Documentation + +Document what the code does today. `sync.start()` is a no-op and every page +that mentions it says so. When adding docs, verify each example by running it +before writing it down โ€” several examples inherited from the legacy repo were +wrong in exactly that way. + +## Formatting + +`oxlint` + `oxfmt`, not eslint/prettier. `printWidth` is 120. Run +`pnpm format` before committing; CI runs `format:check`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..42aead0 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,105 @@ +# Contributing + +Thanks for looking. This file covers how to work on `@alexify/syncom`, the +house rules that are not obvious from the code, and the release checklist. + +## Setup + +```bash +pnpm install +pnpm test +``` + +Requires **Node.js โ‰ฅ 22** and **pnpm** (`packageManager` pins the version). +There is no build step: `src/` ships as-is. + +## Commands + +```bash +pnpm test # node --test, recursive over tests/ +pnpm test:coverage # c8 over src/ โ€” thresholds 95 lines / 95 statements / 90 branches / 95 functions +pnpm test:types # tsd against the .d.ts files +pnpm lint # oxlint +pnpm format # oxfmt (format:check in CI) +pnpm size # bundle-size report; the browser budget fails the run +pnpm bench # every script in bench/ +pnpm docs:dev # the VitePress site (docs:build / docs:preview too) +``` + +Run a single file with `node --test tests/storage.test.js`, or filter by name +with `node --test --test-name-pattern 'swap'`. + +## House rules + +- **CommonJS, `'use strict'`.** No build step, no transpilation. `src/` is what + gets published. +- **Zero runtime dependencies.** Everything in `package.json` is a + devDependency, and `scripts/size.js` fails the run if a bare import reaches + a browser bundle. If you need a helper, write it in `src/utils.js`. +- **Browser halves are named `*.browser.js`** and wired up through + `package.json#browser` plus `src/index.browser.js`. A new split module needs + an entry in both. +- **A `.d.ts` change and its `tsd` test land together.** `index.d.ts` is the + Node surface; `browser.d.ts` re-exports it and overrides only the three + crypto functions that are async in the browser. +- **Anything in `tests/` not named `*.test.js` is a helper** and is skipped by + `node --test` discovery โ€” that is why `tests/suites/`, `tests/utils/` and + `tests/package/` work the way they do. +- **Document what the code does, not what it will do.** `sync.start()` is a + no-op today and the docs say so. + +## Tests + +`node:test` with `node:assert`. One top-level `test(name, async (t) => ...)` +per subject, nested `await t.test(...)` per case, teardown registered with +`t.after(...)` immediately after setup. + +Behaviour shared between the Node and browser halves lives in `tests/suites/` +and is run twice โ€” see `tests/suites/emitter.js` and `tests/suites/fsi.js`. +The browser halves are exercised under Node using the OPFS mock in +`tests/utils/mock-opfs.js` and `node:crypto`'s `webcrypto`. + +`tests/package/consistency.test.js` is the machine-checked half of the release +checklist: it fails if `exports`, `files`, the `browser` map, the `.d.ts` +files, the `scripts/size.js` rows or the docs keywords/version drift from +`package.json`. When it goes red, the fix is usually the metadata, not the test. + +## CI + +Three jobs (`.github/workflows/ci.yml`): + +1. **lint** โ€” `lint`, `format:check`, `size` (the last both smoke-tests the + script and puts the numbers in the log). +2. **test** โ€” matrix over Node 22 and 24: `test:coverage`, then `test:types`. +3. **docs** โ€” `docs:build`, so a broken page or dead link fails the PR rather + than the deploy. + +Benchmarks are not run in CI: they measure a machine, not a change. + +## Bundle size + +`pnpm size` prints one row per `exports` entry and enforces a min+gzip budget +on the browser entry. The budget is a **ratchet, not a target**: raise it +deliberately, in the same change that earns the bytes, and say why in a comment +next to the number. + +## Release checklist + +Releases are a person running these commands by hand, on purpose: no release +workflow, no publish automation. + +1. Everything green: `pnpm lint && pnpm format:check && pnpm test:coverage && + pnpm test:types && pnpm size && pnpm run docs:build`. +2. Move `[Unreleased]` in `CHANGELOG.md` to the new version with today's date. +3. Bump `version` in `package.json`, and the matching nav label in + `docs/.vitepress/config.mts` (the consistency test checks this). +4. Refresh the bundle-size table in `README.md` from `pnpm size`. +5. `npm publish --dry-run` โ€” check the file list against `files`. +6. `npm pack`, then smoke the tarball: `require()` it from a scratch CJS file, + and typecheck it with + `npx tsc --moduleResolution bundler --module esnext --noEmit`. +7. Tag and push: `git tag v && git push --tags`. +8. `npm publish`. +9. Write the GitHub release notes from the changelog section. +10. Verify the docs deploy: + `curl -o /dev/null -sw '%{http_code}' https://syncomjs.vercel.app/`. diff --git a/README.md b/README.md index 72aec75..11f0315 100644 --- a/README.md +++ b/README.md @@ -1,72 +1,176 @@ -# syncom -Fast, zero-dependency distributed data sync engine ๐ŸŒ +
+ syncom +

syncom

+
+ +Fast, zero-dependency distributed data sync engine ๐ŸŒ [![npm](https://img.shields.io/npm/v/%40alexify%2Fsyncom)](https://www.npmjs.com/package/@alexify/syncom) [![CI](https://github.com/Alexis-Technologies/syncom/actions/workflows/ci.yml/badge.svg)](https://github.com/Alexis-Technologies/syncom/actions/workflows/ci.yml) [![node](https://img.shields.io/node/v/%40alexify%2Fsyncom)](#installation) [![dependencies](https://img.shields.io/badge/runtime_dependencies-0-brightgreen)](#bundle-size) -[![docs](https://img.shields.io/badge/docs-online-blue)](https://syncom.vercel.app/) +[![docs](https://img.shields.io/badge/docs-online-blue)](https://syncomjs.vercel.app/) [![license](https://img.shields.io/npm/l/%40alexify%2Fsyncom)](./LICENSE) -## The Concept +`syncom` is a local-first data engine for JavaScript. It stores records on +disk โ€” or in the browser's origin-private filesystem โ€” commits every write to +a hash-linked log so tampering is detectable, and produces CRDT deltas that +peers can merge without coordination. It has no runtime dependencies. + +```js +'use strict'; + +const { open } = require('@alexify/syncom'); -๐Ÿ“ฆ `syncom` is a modular distributed data sync engine for JavaScript and the Webโ€”featuring blockchain integrity, local-first sync, CRDTs, and smart contracts. +const main = async () => { + const storage = await open({ path: 'data' }); -### Key Features + const id = await storage.insert({ name: 'Ada', hits: 0 }); + const record = await storage.record(id); -- **Distributed architecture** with global unique IDs and deterministic replication -- **Blockchain log** for tamper-proof operation history -- **JavaScript Smart Contracts** executed within your app -- **CRDT synchronization** for offline-first and P2P support -- **Local-first storage**: supports `fs`, `IndexedDB`, and `OPFS` -- **Metaschema-driven domain modeling** for validation and sync -- **CAS operations** for consistency and concurrency safety -- **Web-ready**: works in Web browsers and Node.js runtime -- **Secure transport** via `metacom` protocol and WebSocket -- **Crypto support**: RSA, SHA-256, and more -- **Built-in integration** with [Impress Application Server](https://github.com/metarhia/impress) + record.on('update', (data, delta) => console.log('changed', delta)); -### Install + record.hits = 1; + await record.save(); + + console.log(await storage.validate({ last: 10 })); // true +}; + +main(); +``` -Requires **Node.js 18+**. +๐Ÿ“š **[Documentation](https://syncomjs.vercel.app/)** ยท +[Getting started](https://syncomjs.vercel.app/guide/getting-started) ยท +[API reference](https://syncomjs.vercel.app/reference/api) + +## Table of Contents + +- [Installation](#installation) +- [Bundle size](#bundle-size) +- [Quick Start](#quick-start) +- [Features](#features) +- [Browser usage](#browser-usage) +- [Status](#status) +- [Documentation](#documentation) +- [Contributing](#contributing) +- [Changelog](#changelog) +- [Security](#security) +- [License](#license) + +## Installation + +Requires **Node.js โ‰ฅ 22**, or a modern browser with OPFS and Web Crypto. ```bash npm install @alexify/syncom ``` -### Usage - -- **Node.js** loads the CommonJS entry `syncom.js`. -- **Browser** exports `syncom.mjs` (ESM). -- **TypeScript** types for development are maintained in [`syncom.d.ts`](./syncom.d.ts). - -### Ideal For - -- Local-first apps -- Real-time collaboration tools -- P2P distributed systems -- Full-stack JavaScript projects -- Blockchain-based data pipelines - -### Additional features - -- Built-in database engine will minimize interprocess communication to access data -- Minimize data transformations: all data should be stored, transmitted, handled and placed in RAM in the same format -- Maximum memory usage, read-ahead and lazy-write, minimizing data conversion -- Using metadata everywhere, special declarative format for subject domein representation (including fields, relations, and indices), memory structures, GUI, API, etc. -- The same API for client-side runtime and server-side runtime: - - server-side storage engine - - client-side storage engine (multiple implementations for different platforms including mobile, desktop and browser) - - sharding for distributed storage of large data amounts, geo-distribution, save backup copies, access load balancing - - allows user to exchange data in P2P mode -- Synchronization between client and server in realtime (close to realtime) and in lazy mode -- Offline-first approach: applications can work in online and offline (with locally stored data) -- Bidirectional data sync with CRDT and hierarchical versioning like git have -- Global data structures unification for applications working with [Metarhia](https://github.com/metarhia/Metarhia) technological stack and [Impress](https://github.com/metarhia/impress) application server -- Ability to work with non-unified data structures (custom schemas), specific to certain subject domain -- Data structures have global distributed identification system, so data can be inserted anywhere and will not bring ID conflicts -- Data reliability is provided by distributed storage facilities, so each data structure should have master server and multiple backup and cache servers; using those servers globalstorage supports addressing, versioning and branching - -## Contributors - -See GitHub for full [contributors list](https://github.com/metarhia/globalstorage/graphs/contributors) +### Bundle size + +Measured by `pnpm size` on every CI run. The browser entry carries a budget +that fails the build when exceeded. + +| Entry | raw | min | min+gzip | budget | +| ----- | ---:| ---:| --------:| ------:| +| main entry โ€” browser (@alexify/syncom) | 30.9 KB | 13.0 KB | 4.7 KB | 5.0 KB | +| main entry โ€” node (@alexify/syncom) | 27.7 KB | 12.6 KB | 4.5 KB | โ€” | + +## Quick Start + +```js +const { open, diff, merge } = require('@alexify/syncom'); + +// A schema turns entity names into collections. +const storage = await open({ + path: 'data', + schema: { entities: { Post: {}, Author: {} } }, +}); + +const author = await storage.Author.insert({ name: 'Ada' }); +const post = await storage.Post.insert({ title: 'Draft', author: author.id }); + +// Records expose live properties and track their own delta. +post.title = 'Published'; +console.log(post.delta()); // { title: 'Published' } +await post.save(); + +// Compare-and-swap, for writers that must not clobber each other. +await storage.swap(post.id, { title: 'Final' }, { title: 'Published' }); + +// The CRDT helpers work standalone, on plain objects. Numbers ADD, which is +// what lets two peers each increment a counter and converge. +merge({ hits: 10 }, { hits: 1 }); // { hits: 11 } +diff({ tags: ['a'] }, { tags: ['a', 'b'] }); // { tags: ['b'] } +``` + +## Features + +| Area | What you get | +| --- | --- | +| **Storage** | Records with live properties and per-record deltas, schema-driven collections, compare-and-swap writes, optional RSA encryption at rest | +| **Integrity** | Every write appends a hash-linked block; reads re-verify the record against the block that committed it and throw on a mismatch | +| **CRDT** | `diff` / `merge` / `apply` over plain objects โ€” nested maps merge recursively, arrays are grow-only sets, numbers add so counters converge | +| **Contracts** | Store a procedure in the chain by hash, load it anywhere, run it against a read-only view of storage; failures are recorded as blocks | +| **Portability** | One API over `node:fs` + `node:crypto` on the server and OPFS + Web Crypto in the browser, chosen by the `browser` export condition | +| **Dependencies** | None | + +## Browser usage + +The `browser` export condition selects `browser.js` automatically โ€” no bundler +configuration needed. + +```js +import { open, diff, merge } from '@alexify/syncom'; +``` + +Crypto is asynchronous there (`calculateHash`, `encrypt` and `decrypt` return +promises), and storage lives in the origin-private filesystem. `browser.d.ts` +declares those signatures separately. See +[Browser](https://syncomjs.vercel.app/guide/browser). + +## Status + +Pre-1.0, and one part of the surface is a stub worth calling out: `storage.sync` +registers, persists and lists peer nodes, but **no transport is implemented** โ€” +`sync.start()` is a no-op. Everything else on this page works today. See +[Sync](https://syncomjs.vercel.app/guide/sync). + +## Documentation + +- [Getting started](https://syncomjs.vercel.app/guide/getting-started) +- [Storage](https://syncomjs.vercel.app/guide/storage) ยท + [CRDT deltas](https://syncomjs.vercel.app/guide/crdt) ยท + [Blockchain log](https://syncomjs.vercel.app/guide/chain) ยท + [Smart contracts](https://syncomjs.vercel.app/guide/contracts) +- [Sync](https://syncomjs.vercel.app/guide/sync) ยท + [Browser](https://syncomjs.vercel.app/guide/browser) +- [API reference](https://syncomjs.vercel.app/reference/api) + +## Contributing + +See [CONTRIBUTING.md](./CONTRIBUTING.md). + +```bash +pnpm install +pnpm test # node --test, recursive over tests/ +pnpm test:coverage # c8 over src/ +pnpm test:types # tsd against the .d.ts files +pnpm lint # oxlint +pnpm format # oxfmt (format:check in CI) +pnpm size # bundle-size report; the browser budget fails the run +pnpm bench # every script in bench/ +pnpm docs:dev # the VitePress site +``` + +## Changelog + +See [CHANGELOG.md](./CHANGELOG.md). + +## Security + +See [SECURITY.md](./SECURITY.md). Report vulnerabilities privately โ€” do not +open a public issue. + +## License + +[MIT](./LICENSE) ยฉ Alexis Technologies diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..59cce8e --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,50 @@ +# Security Policy + +## Reporting a vulnerability + +Report suspected vulnerabilities **privately** via +[GitHub Security Advisories](https://github.com/Alexis-Technologies/syncom/security/advisories/new) +or by emailing the author (see `package.json`). Do not open public issues or +pull requests for security reports โ€” a fix should ship before the details do. + +What to include: the affected surface (see below), a minimal reproduction, and +the impact as you understand it. Reports are acknowledged within **7 days**; a +confirmed vulnerability gets a fix or a mitigation plan within **30 days**, and +coordinated disclosure (an advisory + a patched release) once the fix is out. +Credit is given unless you ask otherwise. + +## Scope + +These are the areas where a report matters most, roughly in order: + +- **Smart contracts** (`src/contract.js`). `SmartContract.load` reconstructs a + stored procedure with `new Function` and runs it in your process. Anything + that can append a `smart contract` block to a chain you load from can + therefore execute arbitrary code. This is the sharpest surface in the + library; treat a chain as trusted input, and report anything that widens the + gap between "can write a block" and "can run code". +- **Chain integrity** (`src/chain.js`, `calculateHash`). The whole tamper- + evidence claim rests on blocks hashing to what they are filed under and on + `validate` walking `prev` honestly. A way to mutate a record or block and + still pass `validate` is a vulnerability. +- **Crypto** (`src/crypto.js`, `src/crypto.browser.js`). RSA-2048 key + generation, PEM handling, encrypt/decrypt of records stored with + `{ encrypted: true }`, and the base64url encoding in the browser half. Note + the two halves are deliberately not ciphertext-compatible (default padding + vs RSA-OAEP/SHA-256); a *silent* mismatch that produces wrong plaintext + rather than an error is in scope. +- **Filesystem access** (`src/fsi.js`, `src/fsi.browser.js`). Paths are built + from record ids and caller-supplied directory names. Traversal out of the + storage directory via a crafted id or path is in scope. + +## Not in scope + +- `src/sync.js` moves no data โ€” `start()` is a no-op and there is no transport + to attack. Node URLs and tokens are stored as given, in plaintext, under + `/remote/`; that is the documented behaviour today, not a finding. +- Anything requiring an attacker to already have write access to the storage + directory, except where it defeats `validate` (see Chain integrity above). + +## Supported versions + +Pre-1.0: only the latest release receives fixes. diff --git a/bench/chain.js b/bench/chain.js new file mode 100644 index 0000000..8a7df32 --- /dev/null +++ b/bench/chain.js @@ -0,0 +1,67 @@ +/** + * `Blockchain` โ€” the integrity log every write goes through. `addBlock` is on + * the write path of every `Storage.saveData`, and it costs one hash plus TWO + * file writes (the block, then the rewritten tail pointer), so it is + * filesystem-bound rather than crypto-bound. + * + * `validate` is the read side: it walks the chain backwards re-hashing each + * block, so its cost is linear in how far back you ask it to go. The `last: N` + * rows are there to make that slope visible โ€” an unbounded `validate()` over a + * long chain is the shape to watch out for. + * + * Writes go to a temp directory that is removed on exit. + * + * Run with `node bench/chain.js` or `pnpm bench`. + */ + +'use strict'; + +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { bench } = require('./support/harness.js'); +const { Blockchain } = require('../src/chain.js'); + +const makeTemp = () => fs.mkdtemp(path.join(os.tmpdir(), 'syncom-bench-chain-')); + +const main = async () => { + console.log(`Blockchain benchmark โ€” Node ${process.version}\n`); + + const roots = []; + const newChain = async () => { + const root = await makeTemp(); + roots.push(root); + return new Blockchain(root); + }; + + try { + { + const chain = await newChain(); + let i = 0; + await bench('addBlock โ€” small payload', () => chain.addBlock({ id: `r${i++}`, hash: 'x' })); + } + { + const chain = await newChain(); + const data = { blob: 'x'.repeat(4096) }; + await bench('addBlock โ€” 4 KB payload', () => chain.addBlock(data)); + } + { + const chain = await newChain(); + const hash = await chain.addBlock({ id: 'r0', hash: 'x' }); + await bench('readBlock โ€” single block', () => chain.readBlock(hash)); + } + + // One chain, deep enough that the `last` rows below differ meaningfully. + const chain = await newChain(); + for (let i = 0; i < 200; i++) await chain.addBlock({ id: `r${i}`, hash: 'x' }); + + await bench('validate โ€” last 1 block', () => chain.validate({ last: 1 })); + await bench('validate โ€” last 10 blocks', () => chain.validate({ last: 10 })); + await bench('validate โ€” last 100 blocks', () => chain.validate({ last: 100 })); + await bench('validate โ€” full 200-block chain', () => chain.validate(), { warmup: 5, measureMs: 500 }); + } finally { + for (const root of roots) await fs.rm(root, { recursive: true, force: true }); + } +}; + +void main(); diff --git a/bench/crdt.js b/bench/crdt.js new file mode 100644 index 0000000..84c1bb1 --- /dev/null +++ b/bench/crdt.js @@ -0,0 +1,89 @@ +/** + * `diff` / `merge` / `apply` โ€” the pure-JS core of syncom's replication. Every + * `Record.save()` produces a delta through `diff`, every inbound update lands + * through `merge`, and a replaying peer walks its whole history through + * `apply`. None of them touch the filesystem or crypto, so this file is the + * one place where a change to src/crdt.js shows up as a number instead of + * being hidden behind I/O. + * + * The nested rows matter most: `diff` and `merge` recurse per key, so cost + * grows with shape, not with byte count. + * + * Run with `node bench/crdt.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const { diff, merge, apply } = require('../src/crdt.js'); + +const flat = (n, offset = 0) => { + const object = {}; + for (let i = 0; i < n; i++) object[`k${i}`] = i + offset; + return object; +}; + +const nested = (depth, width) => { + if (depth === 0) return flat(width); + const object = {}; + for (let i = 0; i < width; i++) object[`k${i}`] = nested(depth - 1, width); + return object; +}; + +const main = async () => { + console.log(`CRDT benchmark โ€” Node ${process.version}\n`); + + { + const source = flat(20); + const target = flat(20, 1); + await bench('diff โ€” 20 flat keys, all changed', () => diff(source, target)); + } + { + const source = flat(20); + const target = { ...source, k0: 999 }; + await bench('diff โ€” 20 flat keys, one changed', () => diff(source, target)); + } + { + const source = nested(3, 4); + const target = nested(3, 4); + target.k0.k0.k0.k0 = 999; + await bench('diff โ€” depth 3, width 4', () => diff(source, target)); + } + { + const source = { tags: ['a', 'b', 'c'] }; + const target = { tags: ['c', 'd', 'e'] }; + await bench('diff โ€” array as a grow-only set', () => diff(source, target)); + } + + { + const source = flat(20); + const delta = flat(20, 1); + await bench('merge โ€” 20 flat keys', () => merge(source, delta)); + } + { + const source = nested(3, 4); + const delta = { k0: { k0: { k0: { k0: 1 } } } }; + await bench('merge โ€” depth 3, width 4', () => merge(source, delta)); + } + { + const source = { hits: 10 }; + const delta = { hits: 1 }; + // Numbers merge by addition โ€” the counter CRDT, and the hottest single case. + await bench('merge โ€” numeric counter increment', () => merge(source, delta)); + } + + { + const history = []; + for (let i = 0; i < 100; i++) history.push({ type: 'inc', key: 'hits', value: 1 }); + const source = { hits: 0 }; + await bench('apply โ€” 100 inc records', () => apply(source, history)); + } + { + const history = []; + for (let i = 0; i < 100; i++) history.push({ type: 'delta', data: { [`k${i}`]: i } }); + const source = {}; + await bench('apply โ€” 100 delta records', () => apply(source, history)); + } +}; + +void main(); diff --git a/bench/emitter.js b/bench/emitter.js new file mode 100644 index 0000000..7f346a4 --- /dev/null +++ b/bench/emitter.js @@ -0,0 +1,82 @@ +/** + * `Emitter.emit` โ€” the notification path for every mutation that goes through + * a live `Record`: `Storage.set`, `Storage.update` and `Storage.swap` all end + * in `#emitUpdate`, and a record property setter emits on every assignment. + * Real call sites have exactly one synchronous listener, so that row is the + * number that matters; the rest exist to show what the generic path costs by + * comparison. + * + * The awaited rows measure dispatch + settle. The fire-and-forget row measures + * what `void emitter.emit(...)` costs where nobody awaits โ€” the allocation, + * not the microtask drain โ€” so it is batched inside a single iteration with + * opsPerIteration, because the harness awaits fn(). + * + * Run with `node bench/emitter.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const { Emitter } = require('../src/utils.js'); + +const BATCH = 1000; + +let sink = 0; +const sync = (value) => { + sink += value; +}; + +const main = async () => { + console.log(`Emitter benchmark โ€” Node ${process.version}\n`); + + { + const emitter = new Emitter(); + let i = 0; + await bench('emit awaited โ€” 0 listeners', () => emitter.emit('update', i++)); + } + { + const emitter = new Emitter(); + emitter.on('update', sync); + let i = 0; + await bench('emit awaited โ€” 1 sync listener', () => emitter.emit('update', i++)); + } + { + // What Storage actually emits: (data, delta). + const emitter = new Emitter(); + emitter.on('update', (data, delta) => void (sink += data.v + delta.v)); + const data = { v: 1 }; + const delta = { v: 1 }; + await bench('emit awaited โ€” 1 listener, (data, delta)', () => emitter.emit('update', data, delta)); + } + { + const emitter = new Emitter(); + emitter.on('update', async (value) => { + sink += value; + }); + let i = 0; + await bench('emit awaited โ€” 1 async listener', () => emitter.emit('update', i++)); + } + { + const emitter = new Emitter(); + emitter.on('update', sync); + emitter.on('update', (value) => void (sink -= value)); + let i = 0; + await bench('emit awaited โ€” 2 sync listeners', () => emitter.emit('update', i++)); + } + { + const emitter = new Emitter(); + emitter.on('update', sync); + let i = 0; + await bench( + 'emit fire-and-forget โ€” 1 sync listener', + async () => { + for (let n = 0; n < BATCH; n++) void emitter.emit('update', i++); + }, + { opsPerIteration: BATCH }, + ); + } + + if (sink === Number.MIN_SAFE_INTEGER) console.log('unreachable'); +}; + +void main(); diff --git a/bench/hash.js b/bench/hash.js new file mode 100644 index 0000000..67377e8 --- /dev/null +++ b/bench/hash.js @@ -0,0 +1,50 @@ +/** + * `calculateHash` โ€” on the write path twice per record: once over the stored + * entry, once over the block that commits it (src/storage.js `saveData`), and + * once per block again on every `Blockchain.validate()` walk. It is + * `JSON.stringify` followed by SHA-256, so the stringify half dominates at + * small sizes and the digest half takes over as records grow โ€” which is why + * the rows below scale the payload rather than repeating one shape. + * + * Run with `node bench/hash.js` or `pnpm bench`. + */ + +'use strict'; + +const { bench } = require('./support/harness.js'); +const { calculateHash } = require('../src/crypto.js'); + +const record = (fields, valueLength) => { + const object = {}; + const value = 'x'.repeat(valueLength); + for (let i = 0; i < fields; i++) object[`field${i}`] = value; + return object; +}; + +const main = async () => { + console.log(`Hash benchmark โ€” Node ${process.version}\n`); + + { + const block = { prev: '0', timestamp: Date.now(), data: {} }; + await bench('calculateHash โ€” empty genesis block', () => calculateHash(block)); + } + { + const data = record(5, 16); + await bench('calculateHash โ€” small record (5 x 16 B)', () => calculateHash(data)); + } + { + const data = record(50, 64); + await bench('calculateHash โ€” medium record (50 x 64 B)', () => calculateHash(data)); + } + { + const data = record(200, 256); + await bench('calculateHash โ€” large record (200 x 256 B)', () => calculateHash(data)); + } + { + // Deep nesting costs the stringify half, not the digest half. + const data = { a: { b: { c: { d: { e: record(10, 32) } } } } }; + await bench('calculateHash โ€” deeply nested record', () => calculateHash(data)); + } +}; + +void main(); diff --git a/bench/run-all.js b/bench/run-all.js new file mode 100644 index 0000000..0a63111 --- /dev/null +++ b/bench/run-all.js @@ -0,0 +1,23 @@ +/** + * Runs every benchmark script in this directory, one child process at a + * time, so `pnpm bench` always covers the full `bench/` folder instead of + * a single hardcoded entry point. + * + * Run with `pnpm bench`. + */ + +const { readdirSync } = require('node:fs'); +const path = require('node:path'); +const { spawnSync } = require('node:child_process'); + +const SELF = path.basename(__filename); + +const files = readdirSync(__dirname) + .filter((file) => file.endsWith('.js') && file !== SELF) + .sort(); + +for (const file of files) { + console.log(`\n=== ${file} ===\n`); + const result = spawnSync(process.execPath, [path.join(__dirname, file)], { stdio: 'inherit' }); + if (result.status !== 0) process.exitCode = result.status ?? 1; +} diff --git a/bench/storage.js b/bench/storage.js new file mode 100644 index 0000000..c0badf7 --- /dev/null +++ b/bench/storage.js @@ -0,0 +1,72 @@ +/** + * `Storage` โ€” the full write path end to end: hash the record, append a block + * (two more file writes, see bench/chain.js), write the entry, populate the + * cache. That makes `insert` the most expensive operation in the library, and + * the gap between it and a cached `get` is the number worth watching. + * + * `get` has two very different costs depending on whether the id is in the + * in-memory cache: a cold read re-hashes the entry and re-reads its block to + * verify integrity, so it is several file reads, not one. + * + * `swap` is the CAS path โ€” it reads, compares, and writes only on a match; the + * mismatch row costs a read and nothing else. + * + * Writes go to a temp directory that is removed on exit. + * + * Run with `node bench/storage.js` or `pnpm bench`. + */ + +'use strict'; + +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { bench } = require('./support/harness.js'); +const { open } = require('../src/storage.js'); + +const main = async () => { + console.log(`Storage benchmark โ€” Node ${process.version}\n`); + + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'syncom-bench-storage-')); + + try { + // Key generation is RSA-2048 and happens once per Storage; keeping a + // single instance keeps that cost out of every row below. + const storage = await open({ path: root }); + + const record = { name: 'Example', email: 'example@example.com', hits: 0 }; + + await bench('insert โ€” small record', () => storage.insert(record), { warmup: 20 }); + + const id = await storage.insert(record); + + await bench('get โ€” cached', () => storage.get(id)); + await bench( + 'get โ€” cold (re-verifies the block)', + async () => { + const cold = await open({ path: root }); + return cold.get(id); + }, + { warmup: 3, measureMs: 500 }, + ); + + await bench('update โ€” one field', () => storage.update(id, { hits: 1 }), { warmup: 20 }); + await bench('set โ€” whole record', () => storage.set(id, record), { warmup: 20 }); + + await bench( + 'swap โ€” match (writes)', + async () => { + await storage.swap(id, { name: 'Example' }, { name: 'Example' }); + }, + { warmup: 20 }, + ); + await bench('swap โ€” mismatch (reads only)', () => storage.swap(id, { name: 'x' }, { name: 'nope' })); + + await bench('has โ€” existing id', () => storage.has(id)); + await bench('record โ€” cached Record instance', () => storage.record(id)); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } +}; + +void main(); diff --git a/bench/support/harness.js b/bench/support/harness.js new file mode 100644 index 0000000..f211764 --- /dev/null +++ b/bench/support/harness.js @@ -0,0 +1,52 @@ +'use strict'; + +const { performance } = require('node:perf_hooks'); + +const WARMUP_ITERATIONS = 200; +const MEASURE_MS = 1_000; +const FIRST_CALL_MS = 10_000; + +// A bench whose fn awaits something that never settles โ€” a renamed method, an +// event that stopped going on the wire โ€” wedges the whole run with no output +// at all, which is far harder to diagnose than a thrown error. The first +// warmup call is the canary: racing ONLY that one keeps the guard out of the +// measured loop, where a per-iteration timer would allocate and skew the very +// numbers this file exists to report. +async function firstCall(name, fn) { + let timer; + const stalled = new Promise((_resolve, reject) => { + timer = setTimeout( + () => reject(new Error(`bench "${name}" did not settle within ${FIRST_CALL_MS} ms`)), + FIRST_CALL_MS, + ); + }); + try { + await Promise.race([fn(), stalled]); + } finally { + clearTimeout(timer); + } +} + +async function bench(name, fn, options = {}) { + const warmup = options.warmup ?? WARMUP_ITERATIONS; + const measureMs = options.measureMs ?? MEASURE_MS; + // A pipelined fn issues several calls per iteration, so ops and iterations + // stop being the same number. + const opsPerIteration = options.opsPerIteration ?? 1; + + if (warmup > 0) await firstCall(name, fn); + for (let i = 1; i < warmup; i++) await fn(); + + let iterations = 0; + const start = performance.now(); + while (performance.now() - start < measureMs) { + await fn(); + iterations += 1; + } + const elapsed = performance.now() - start; + const opsPerSec = Math.round(((iterations * opsPerIteration) / elapsed) * 1000); + console.log(`${name.padEnd(60)} ${opsPerSec.toLocaleString('en-US').padStart(12)} ops/sec`); + return { name, opsPerSec }; +} + +module.exports = { bench, WARMUP_ITERATIONS, MEASURE_MS }; diff --git a/browser.d.ts b/browser.d.ts new file mode 100644 index 0000000..ac0dbea --- /dev/null +++ b/browser.d.ts @@ -0,0 +1,59 @@ +// The browser half of the package, reached through the `browser` export +// condition (see package.json#exports and #browser). +// +// Everything that is platform-agnostic โ€” CRDT, contracts, chain, storage and +// their types โ€” is re-exported from ./index.js unchanged. What differs is the +// crypto and OPFS layer: node:crypto is synchronous, the Web Crypto API is +// not, so `calculateHash`, `encrypt` and `decrypt` return promises here and +// are declared below rather than re-exported. + +import type { JSONSafe as JSONSafeValue } from './index.js'; + +export type { + JSONSafe, + KeyPair, + CrdtDelta, + CrdtHistoryRecord, + Block, + ChainDataEntry, + BlockchainConstructor, + ContractContext, + ContractBlock, + ErrorBlock, + ContractExecuteArgs, + ContractProcedure, + SchemaEntity, + SchemaEntities, + Schema, + StorageOptions, + StorageDataOptions, + StorageEntry, + Node, + NodeData, + SyncManagerConstructor, + StorageUpdate, + StorageConstructor, +} from './index.js'; + +export { + generateKeys, + loadKeys, + read, + write, + exists, + remove, + diff, + merge, + apply, + Blockchain, + DataReader, + SmartContract, + SyncManager, + Storage, + open, +} from './index.js'; + +// Web Crypto: async where node:crypto is sync. +export function calculateHash(data: JSONSafeValue): Promise; +export function encrypt(data: unknown, publicKey: string): Promise; +export function decrypt(data: string, privateKey: string): Promise; diff --git a/browser.js b/browser.js index 4e6d47f..49220df 100644 --- a/browser.js +++ b/browser.js @@ -1,810 +1,3 @@ -import { Emitter, generateUUID } from './metautil.js'; +'use strict'; -// crdt.js - -const diff = (source, target) => { - if (typeof source !== 'object') source = {}; - if (typeof target !== 'object') target = {}; - const delta = {}; - const sourceKeys = new Set(Object.keys(source)); - const targetKeys = new Set(Object.keys(target)); - - for (const key of targetKeys) { - if (!sourceKeys.has(key)) { - delta[key] = target[key]; - } else if (Array.isArray(target[key]) || target[key] instanceof Set) { - const sourceArr = Array.isArray(source[key]) ? source[key] : source[key]; - const targetArr = Array.isArray(target[key]) ? target[key] : target[key]; - const sourceSet = new Set(sourceArr); - const targetSet = new Set(targetArr); - const diffSet = new Set([...targetSet].filter((x) => !sourceSet.has(x))); - if (diffSet.size > 0) { - delta[key] = Array.from(diffSet); - } - } else if (typeof target[key] === 'object' && target[key] !== null) { - const nestedDiff = diff(source[key], target[key]); - if (Object.keys(nestedDiff).length > 0) { - delta[key] = nestedDiff; - } - } else if (source[key] !== target[key]) { - delta[key] = target[key]; - } - } - - for (const key of sourceKeys) { - if (!targetKeys.has(key)) { - delta[key] = undefined; - } - } - - return delta; -}; - -const merge = (source, delta) => { - if (!delta || typeof delta !== 'object') return source; - if (source === null || typeof source !== 'object') return delta; - - const result = Array.isArray(source) ? [...source] : { ...source }; - - for (const key in delta) { - if (delta[key] === undefined) { - delete result[key]; - continue; - } - - if (Array.isArray(delta[key]) || delta[key] instanceof Set) { - const sourceSet = new Set( - Array.isArray(result[key]) ? result[key] : result[key] || [], - ); - const deltaSet = new Set( - Array.isArray(delta[key]) ? delta[key] : delta[key], - ); - result[key] = Array.from(new Set([...sourceSet, ...deltaSet])); - } else if (typeof delta[key] === 'object' && delta[key] !== null) { - if (typeof result[key] === 'object' && result[key] !== null) { - result[key] = merge(result[key], delta[key]); - } else { - result[key] = delta[key]; - } - } else if ( - typeof delta[key] === 'number' && - typeof result[key] === 'number' - ) { - result[key] += delta[key]; - } else { - result[key] = delta[key]; - } - } - - return result; -}; - -const apply = (source, history) => { - if (!Array.isArray(history)) return source; - let result = { ...source }; - - const handlers = { - write: (record, state) => ({ ...state }), - delta: (record, state) => merge(state, record.data), - delete: (record, state) => ({ ...state, [record.key]: undefined }), - inc: (record, state) => merge(state, { [record.key]: record.value }), - dec: (record, state) => merge(state, { [record.key]: -record.value }), - }; - - for (const record of history) { - if (!record || typeof record !== 'object' || !record.type) continue; - const handler = handlers[record.type]; - if (handler) { - result = handler(record, result); - } - } - - return result; -}; - -// contract.js - -const deserializeFunction = (source) => { - const arrowIndex = source.indexOf('=>'); - const header = source.slice(0, arrowIndex).trim(); - let body = source.slice(arrowIndex + 2).trim(); - const argsStart = header.indexOf('('); - const argsEnd = header.lastIndexOf(')'); - const args = header.slice(argsStart + 1, argsEnd); - if (body.startsWith('{')) body = body.slice(1, -1); - else body = `return ${body};`; - body = `return async (${args}) => {${body}}`; - return new Function('reader', 'args', body)(); -}; - -class DataReader { - #storage; - - constructor(storage) { - this.#storage = storage; - } - - async get(id) { - return this.#storage.loadData(id); - } -} - -class SmartContract { - #storage; - #chain; - #proc; - - constructor(name, proc, { storage, chain }) { - this.name = name; - this.#storage = storage; - this.#chain = chain; - this.#proc = proc; - } - - async execute(args) { - const reader = new DataReader(this.#storage); - try { - const result = await this.#proc(reader, args); - await this.#storage.saveData(args.id, result); - return result; - } catch (error) { - const contract = this.name; - const timestamp = Date.now(); - const block = { contract, args, error: error.message, timestamp }; - await this.#chain.addBlock(block); - throw error; - } - } - - static async save(name, chain, proc) { - const source = proc.toString(); - const block = { type: 'smart contract', name, source }; - const hash = await chain.addBlock(block); - return hash; - } - - static async load(hash, { storage, chain }) { - const block = await chain.readBlock(hash); - if (!block) throw new Error('Block not found'); - if (block.data.type !== 'smart contract') { - throw new Error('Not a smart contract block'); - } - const { name, source } = block.data; - const proc = deserializeFunction(source); - const contract = new SmartContract(name, proc, { storage, chain }); - return contract; - } -} - -// fsi-browser.js - -const resolveDir = async (dirPath) => { - const root = await navigator.storage.getDirectory(); - const parts = dirPath.split('/').filter(Boolean); - let dir = root; - for (const part of parts) { - dir = await dir.getDirectoryHandle(part, { create: true }); - } - return dir; -}; - -const read = async (dir, filename) => { - const dirHandle = await resolveDir(dir); - const fileHandle = await dirHandle.getFileHandle(filename); - const file = await fileHandle.getFile(); - return file.text(); -}; - -const write = async (dir, filename, content) => { - const dirHandle = await resolveDir(dir); - const fileHandle = await dirHandle.getFileHandle(filename, { create: true }); - const writable = await fileHandle.createWritable(); - await writable.write(content); - await writable.close(); -}; - -const exists = async (dir, filename) => { - const dirHandle = await resolveDir(dir); - try { - await dirHandle.getFileHandle(filename); - return true; - } catch { - return false; - } -}; - -const remove = async (dir, filename) => { - const dirHandle = await resolveDir(dir); - try { - await dirHandle.removeEntry(filename); - } catch { - // File may not exist - } -}; - -// crypto-browser.js - -const PUBLIC_KEY_HEADER = '-----BEGIN PUBLIC KEY-----'; -const PUBLIC_KEY_FOOTER = '-----END PUBLIC KEY-----'; -const PRIVATE_KEY_HEADER = '-----BEGIN PRIVATE KEY-----'; -const PRIVATE_KEY_FOOTER = '-----END PRIVATE KEY-----'; - -const ALG = { name: 'RSA-OAEP', hash: 'SHA-256' }; - -const calculateHash = async (data) => { - const block = JSON.stringify(data); - const dataBuffer = new TextEncoder().encode(block); - const hashBuffer = await crypto.subtle.digest('SHA-256', dataBuffer); - const hashArray = Array.from(new Uint8Array(hashBuffer)); - return hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); -}; - -const toBase64UrlSafe = (buffer) => { - const base64 = btoa(String.fromCharCode(...new Uint8Array(buffer))); - return base64.replaceAll('+', '-').replaceAll('/', '_').replaceAll('=', ''); -}; - -const base64ToBytes = (base64) => { - const normalized = base64.replaceAll('-', '+').replaceAll('_', '/'); - const binary = atob(normalized); - const bytes = new Uint8Array(binary.length); - for (let i = 0; i < binary.length; i++) { - bytes[i] = binary.charCodeAt(i); - } - return bytes; -}; - -const cleanPemKey = (pemKey, header, footer) => { - const s = pemKey.replace(header, '').replace(footer, ''); - return s.replaceAll(/\s/g, ''); -}; - -const generateKeys = async () => { - const keyPair = await crypto.subtle.generateKey( - { ...ALG, modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]) }, - true, - ['encrypt', 'decrypt'], - ); - const pubBuf = await crypto.subtle.exportKey('spki', keyPair.publicKey); - const privBuf = await crypto.subtle.exportKey('pkcs8', keyPair.privateKey); - const pub = toBase64UrlSafe(pubBuf); - const priv = toBase64UrlSafe(privBuf); - return { - publicKey: [PUBLIC_KEY_HEADER, pub, PUBLIC_KEY_FOOTER].join('\n'), - privateKey: [PRIVATE_KEY_HEADER, priv, PRIVATE_KEY_FOOTER].join('\n'), - }; -}; - -const importPublicKey = async (pemKey) => { - const base64 = cleanPemKey(pemKey, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER); - const bytes = base64ToBytes(base64); - return crypto.subtle.importKey('spki', bytes, ALG, false, ['encrypt']); -}; - -const importPrivateKey = async (pemKey) => { - const base64 = cleanPemKey(pemKey, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER); - const bytes = base64ToBytes(base64); - return crypto.subtle.importKey('pkcs8', bytes, ALG, false, ['decrypt']); -}; - -const encrypt = async (data, publicKey) => { - const key = await importPublicKey(publicKey); - const dataBuffer = new TextEncoder().encode(JSON.stringify(data)); - const encryptedBuffer = await crypto.subtle.encrypt(ALG, key, dataBuffer); - return toBase64UrlSafe(encryptedBuffer); -}; - -const decrypt = async (data, privateKey) => { - const key = await importPrivateKey(privateKey); - const bytes = base64ToBytes(data); - const buffer = await crypto.subtle.decrypt(ALG, key, bytes); - return JSON.parse(new TextDecoder().decode(buffer)); -}; - -const loadKeys = async (basePath = 'keys') => { - let publicKey, privateKey; - if (await fsi.exists(basePath, 'public.pem')) { - publicKey = await fsi.read(basePath, 'public.pem'); - privateKey = await fsi.read(basePath, 'private.pem'); - } else { - const keys = await generateKeys(); - ({ publicKey, privateKey } = keys); - await fsi.write(basePath, 'public.pem', publicKey); - await fsi.write(basePath, 'private.pem', privateKey); - } - return { publicKey, privateKey }; -}; - -// sync-browser.js - -class SyncManager { - #nodes = new Set(); - - constructor() { - return this.#init(); - } - - async #init() { - await this.start(); - return this; - } - - async start() { - // TODO: Implement metacom-based sync (WebSocket client) - // Future: WebRTC transport for peer-to-peer sync - const uplinks = Array.from(this.#nodes).filter( - (node) => node.type === 'uplink' && node.enabled, - ); - if (uplinks.length === 0) return; - for (const uplink of uplinks) { - try { - // Sync here - } catch (error) { - console.error(`Sync failed with ${uplink.url}:`, error.message); - } - } - } - - async addNode(nodeData) { - const { url, type, token } = nodeData; - if (!url) throw new Error('Node URL is required'); - if (type !== 'uplink' && type !== 'downlink') { - throw new Error('Node type must be uplink or downlink'); - } - const existing = Array.from(this.#nodes).find((n) => n.url === url); - if (existing) throw new Error(`Node with URL ${url} already exists`); - const now = Date.now(); - const node = { type, url, token, enabled: true, lastSync: now }; - this.#nodes.add(node); - await this.start(); - } - - async removeNode(url) { - const node = Array.from(this.#nodes).find((n) => n.url === url); - if (!node) throw new Error(`Node with URL ${url} not found`); - this.#nodes.delete(node); - } - - listNodes() { - return Array.from(this.#nodes); - } -} - -// chain.js - -const BLOCKCHAIN = '.blockchain.json'; - -class Blockchain { - constructor(basePath) { - this.path = basePath; - this.tailHash = null; - return this.#init(); - } - - async #init() { - const exists = await fsi.exists(this.path, BLOCKCHAIN); - if (!exists) { - const block = { prev: '0', timestamp: Date.now(), data: {} }; - this.tailHash = await this.writeBlock(block); - await this.writeChain(); - } else { - const raw = await fsi.read(this.path, BLOCKCHAIN); - const chain = JSON.parse(raw); - this.tailHash = chain.tailHash; - } - return this; - } - - async writeChain() { - const chain = { tailHash: this.tailHash }; - await fsi.write(this.path, BLOCKCHAIN, JSON.stringify(chain)); - } - - async addBlock(data) { - const prev = this.tailHash; - const timestamp = Date.now(); - const block = { prev, timestamp, data }; - const hash = await this.writeBlock(block); - this.tailHash = hash; - await this.writeChain(); - return hash; - } - - async writeBlock(block) { - const hash = await calculateHash(block); - await fsi.write(this.path, `${hash}.json`, JSON.stringify(block)); - return hash; - } - - async readBlock(hash) { - const exists = await fsi.exists(this.path, `${hash}.json`); - if (!exists) throw new Error(`Block not found: ${hash}`); - const raw = await fsi.read(this.path, `${hash}.json`); - return JSON.parse(raw); - } - - async validate({ last, from } = {}) { - let currentHash = from || this.tailHash; - if (!currentHash) return false; - let count = 0; - while (currentHash) { - const block = await this.readBlock(currentHash); - const expectedHash = await calculateHash(block); - const valid = currentHash === expectedHash; - if (!valid) return false; - count++; - if (last && count >= last) break; - if (block.prev === '0') break; - currentHash = block.prev; - } - return true; - } -} - -// storage.js - -const calculateDelta = (oldData, newData) => { - const delta = {}; - for (const key in newData) { - if (oldData[key] !== newData[key]) { - delta[key] = newData[key]; - } - } - return delta; -}; - -class Reference { - #id; - #storage = null; - - constructor(id, storage) { - this.#id = id; - this.#storage = storage; - } - - async record() { - return await this.#storage.record(this.#id); - } -} - -class Record extends Emitter { - #id; - #storage = null; - #data = null; - #changes = {}; - - constructor(id, storage) { - super(); - this.#id = id; - this.#storage = storage; - return this.#load(); - } - - async #load() { - if (this.#data !== null) return this; - const rawData = await this.#storage.get(this.#id); - if (rawData) { - this.#data = JSON.parse(JSON.stringify(rawData)); - this.#convertToReferences(this.#data); - this.#defineProperties(); - } - return this; - } - - #convertToReferences(data) { - if (!data || typeof data !== 'object') return; - for (const key in data) { - const value = data[key]; - if (typeof value === 'string') { - if (this.#storage.hasRecord(value)) { - data[key] = new Reference(value, this.#storage); - } - } else if (value && typeof value === 'object') { - const isReference = value instanceof Reference; - if (!isReference) { - this.#convertToReferences(value); - } - } - } - } - - get id() { - return this.#id; - } - - #defineProperties() { - for (const key in this.#data) { - if (key === 'id') continue; - if (Object.prototype.hasOwnProperty.call(this, key)) continue; - Object.defineProperty(this, key, { - get() { - const value = this.#data?.[key]; - if (value instanceof Reference) { - return value; - } - return value; - }, - set(newValue) { - const oldValue = this.#data?.[key]; - if (oldValue === newValue) return; - this.#data[key] = newValue; - this.#changes[key] = newValue; - const delta = { [key]: newValue }; - this.emit('update', this.#data, delta); - }, - enumerable: true, - configurable: true, - }); - } - } - - async data() { - return this.#data; - } - - delta() { - return { ...this.#changes }; - } - - async save() { - if (Object.keys(this.#changes).length === 0) return; - const delta = this.delta(); - const data = this.#data; - await this.#storage.saveData(this.#id, data); - this.#changes = {}; - this.#storage.addUpdate({ - type: 'delta', - id: this.#id, - timestamp: Date.now(), - data: delta, - }); - } - - async delete() { - await this.#storage.delete(this.#id); - this.#data = null; - this.#changes = {}; - } -} - -class Collection { - #storage; - #name; - - constructor(storage, name) { - this.#storage = storage; - this.#name = name; - } - - get name() { - return this.#name; - } - - async insert(data) { - const id = generateUUID(); - const record = { ...data, $type: this.#name }; - await this.#storage.saveData(id, record); - return this.#storage.record(id); - } - - async get(id) { - return this.#storage.get(id); - } - - async delete(id) { - return this.#storage.delete(id); - } - - async update(id, delta) { - return this.#storage.update(id, delta); - } - - async record(id) { - return this.#storage.record(id); - } -} - -class Storage { - #path; - #keys = null; - #chain = null; - #records = null; - #sync = null; - #updates = []; - #cache = new Map(); - #schema = null; - - constructor(options = {}) { - this.#path = options.path || options.name || 'gs'; - this.#records = new Map(); - if (options.schema) this.#schema = options.schema; - return this.#init(); - } - - async #init() { - this.#keys = await loadKeys(`${this.#path}/keys`); - this.#chain = await new Blockchain(`${this.#path}/blockchain`); - this.#sync = await new SyncManager(this.#path, this, this.#chain); - this.#initCollections(); - return this; - } - - #initCollections() { - const entities = this.#schema?.entities; - if (!entities) return; - for (const [name] of entities.entries?.() ?? Object.entries(entities)) { - this[name] = new Collection(this, name); - } - } - - get schema() { - return this.#schema; - } - - #emitUpdate(id, data, delta) { - const emitter = this.#records.get(id); - if (emitter) { - emitter.emit('update', data, delta); - } - } - - addUpdate(update) { - this.#updates.push(update); - } - - async saveData(id, data, options = {}) { - const { encrypted = false } = options; - const record = encrypted ? encrypt(data, this.#keys.publicKey) : data; - const timestamp = Date.now(); - - const hash = await calculateHash(record); - const blockHash = await this.#chain.addBlock({ id, hash }); - - const entry = { data: record, encrypted, timestamp, block: blockHash }; - await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); - this.#cache.set(id, data); - - this.#updates.push({ type: 'write', id, timestamp, data }); - } - - async loadData(id) { - if (this.#cache.has(id)) return this.#cache.get(id); - const dataDir = `${this.#path}/data`; - const exists = await fsi.exists(dataDir, `${id}.json`); - if (!exists) { - this.#cache.set(id, null); - return null; - } - const raw = await fsi.read(dataDir, `${id}.json`); - const { data, encrypted, block } = JSON.parse(raw); - const isValid = await this.validate(id, data, block); - if (!isValid) throw new Error(`Storage record ${id} is invalid`); - const decrypted = encrypted ? decrypt(data, this.#keys.privateKey) : data; - this.#cache.set(id, decrypted); - return decrypted; - } - - async validate(idOrOptions, data, blockHash) { - const isChainValidation = - !data && !blockHash && ('last' in idOrOptions || 'from' in idOrOptions); - if (isChainValidation) { - return await this.#chain.validate(idOrOptions); - } - const id = idOrOptions; - try { - const block = await this.#chain.readBlock(blockHash); - const expectedHash = await calculateHash(data); - if (!block) return false; - return block.data.hash === expectedHash && block.data.id === id; - } catch { - return false; - } - } - - async insert(data) { - const id = generateUUID(); - await this.saveData(id, data); - return id; - } - - async has(id) { - return fsi.exists(`${this.#path}/data`, `${id}.json`); - } - - hasRecord(id) { - return this.#cache.has(id) || this.#records.has(id); - } - - async get(id) { - return await this.loadData(id); - } - - async set(id, data) { - const oldData = await this.get(id); - await this.saveData(id, data); - if (oldData) { - const delta = calculateDelta(oldData, data); - this.#emitUpdate(id, data, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); - } - } - - async delete(id) { - await fsi.remove(`${this.#path}/data`, `${id}.json`); - this.#records.delete(id); - this.#cache.delete(id); - this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); - } - - async update(id, delta) { - const data = await this.get(id); - if (!data) throw new Error(`Record ${id} not found`); - const updated = { ...data, ...delta }; - await this.saveData(id, updated); - this.#emitUpdate(id, updated, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); - } - - async swap(id, changes, prev) { - const data = await this.get(id); - if (!data) throw new Error(`Record ${id} not found`); - const matches = Object.keys(prev).every((key) => data[key] === prev[key]); - if (!matches) return false; - const updated = { ...data, ...changes }; - await this.saveData(id, updated); - this.#emitUpdate(id, updated, changes); - return true; - } - - async record(id) { - if (!this.#records.has(id)) { - const record = await new Record(id, this); - this.#records.set(id, record); - } - return this.#records.get(id); - } - - getCachedData(id) { - return this.#cache.has(id) ? this.#cache.get(id) : null; - } - - get sync() { - return this.#sync; - } -} - -Storage.Collection = Collection; -Storage.Record = Record; - -const open = (options) => new Storage(options); - -export { - diff, - merge, - apply, - DataReader, - SmartContract, - read, - write, - exists, - remove, - calculateHash, - generateKeys, - encrypt, - decrypt, - loadKeys, - SyncManager, - Blockchain, - Storage, - open, -}; +module.exports = require('./src/index.browser.js'); diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts new file mode 100644 index 0000000..a0ba138 --- /dev/null +++ b/docs/.vitepress/config.mts @@ -0,0 +1,198 @@ +import { defineConfig } from 'vitepress'; +import { withMermaid } from 'vitepress-plugin-mermaid'; + +const ogTitle = 'syncom โ€” distributed data sync engine for Node.js and the browser'; +const ogDescription = + 'Fast, zero-dependency distributed data sync engine: local-first storage with a hash-linked ' + + 'integrity log, CRDT deltas, JavaScript smart contracts, and one API across Node.js and the Web.'; +const repo = 'https://github.com/Alexis-Technologies/syncom'; +const base = '/'; +const hostname = 'https://syncomjs.vercel.app/'; +const ogImage = `${hostname}logo-mark.svg`; + +// Mirrors package.json "keywords" โ€” kept as a one-term-per-line list so the +// two stay easy to diff. tests/package/consistency.test.js fails if they drift. +const keywords = [ + 'syncom', + 'distributed', + 'data', + 'sync', + 'engine', + 'real-time', + 'collaboration', + 'p2p', + 'peer-to-peer', + 'replication', + 'conflict-resolution', + 'CRDT', + 'event-sourcing', + 'blockchain', + 'smart-contracts', + 'local-first', + 'offline-first', + 'storage', + 'database', + 'zero-dependency', + 'javascript', + 'nodejs', + 'browser', +].join(', '); + +// schema.org structured data โ€” helps search and AI engines understand the +// package as a software entity, not just text on a page. +const jsonLd = { + '@context': 'https://schema.org', + '@type': 'SoftwareApplication', + name: '@alexify/syncom', + alternateName: 'syncom', + description: ogDescription, + applicationCategory: 'DeveloperApplication', + operatingSystem: 'Node.js >= 22, modern browsers', + url: hostname, + image: ogImage, + downloadUrl: 'https://www.npmjs.com/package/@alexify/syncom', + codeRepository: repo, + license: 'https://opensource.org/licenses/MIT', + keywords, + author: { '@type': 'Organization', name: 'Alexis Technologies' }, + offers: { '@type': 'Offer', price: '0', priceCurrency: 'USD' }, +}; + +// https://vitepress.dev/reference/site-config +export default withMermaid({ + ...defineConfig({ + title: '@alexify/syncom', + titleTemplate: ':title โ€” syncom', + description: ogDescription, + lang: 'en-US', + base, + cleanUrls: true, + lastUpdated: true, + sitemap: { hostname }, + + head: [ + ['link', { rel: 'icon', type: 'image/svg+xml', href: `${base}logo-mark.svg` }], + ['meta', { name: 'theme-color', content: '#3f7fd0' }], + ['meta', { name: 'author', content: 'Alexis Technologies' }], + ['meta', { name: 'keywords', content: keywords }], + ['meta', { name: 'robots', content: 'index, follow' }], + ['meta', { property: 'og:type', content: 'website' }], + ['meta', { property: 'og:site_name', content: '@alexify/syncom' }], + ['meta', { property: 'og:title', content: ogTitle }], + ['meta', { property: 'og:description', content: ogDescription }], + ['meta', { property: 'og:image', content: ogImage }], + ['meta', { name: 'twitter:card', content: 'summary_large_image' }], + ['meta', { name: 'twitter:title', content: ogTitle }], + ['meta', { name: 'twitter:description', content: ogDescription }], + ['meta', { name: 'twitter:image', content: ogImage }], + ['script', { type: 'application/ld+json' }, JSON.stringify(jsonLd)], + ], + + // Mermaid pulls in a few CJS-only transitive deps (dayjs among them). Vite's + // dev server hands them to the browser unbundled, where `import dayjs from` has + // no default export and the whole app fails to boot. Pre-bundling them fixes dev; + // noExternal keeps mermaid inlined for the SSR build. + vite: { + optimizeDeps: { include: ['mermaid', 'dayjs'] }, + ssr: { noExternal: ['mermaid'] }, + }, + + // Per-page canonical + og:url for clean SEO indexing. + transformPageData(pageData) { + const path = pageData.relativePath.replace(/index\.md$/, '').replace(/\.md$/, ''); + const canonical = `${hostname}${path}`; + pageData.frontmatter.head ??= []; + pageData.frontmatter.head.push( + ['link', { rel: 'canonical', href: canonical }], + ['meta', { property: 'og:url', content: canonical }], + ); + }, + + themeConfig: { + logo: '/logo-mark.svg', + + // โ”€โ”€โ”€ Top navigation โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + nav: [ + { text: 'Guide', link: '/guide/getting-started', activeMatch: '/guide/' }, + { text: 'Reference', link: '/reference/api', activeMatch: '/reference/' }, + { + // Hand-synced with package.json "version" โ€” part of the release + // checklist, and asserted by tests/package/consistency.test.js. + text: 'v0.0.1', + items: [ + { text: 'Changelog', link: `${repo}/blob/main/CHANGELOG.md` }, + { text: 'npm', link: 'https://www.npmjs.com/package/@alexify/syncom' }, + { text: 'Releases', link: `${repo}/releases` }, + ], + }, + ], + + // โ”€โ”€โ”€ Sidebar โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + sidebar: { + '/guide/': [ + { + text: 'Introduction', + items: [{ text: 'Getting Started', link: '/guide/getting-started' }], + }, + { + text: 'Core', + items: [ + { text: 'Storage', link: '/guide/storage' }, + { text: 'CRDT deltas', link: '/guide/crdt' }, + { text: 'Blockchain log', link: '/guide/chain' }, + { text: 'Smart contracts', link: '/guide/contracts' }, + ], + }, + { + text: 'Platforms', + items: [ + { text: 'Sync', link: '/guide/sync' }, + { text: 'Browser', link: '/guide/browser' }, + ], + }, + ], + '/reference/': [ + { + text: 'Reference', + items: [{ text: 'API', link: '/reference/api' }], + }, + ], + }, + + // โ”€โ”€โ”€ Local, zero-config full-text search โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + search: { provider: 'local' }, + + socialLinks: [{ icon: 'github', link: repo }], + + editLink: { + pattern: `${repo}/edit/main/docs/:path`, + text: 'Edit this page on GitHub', + }, + + footer: { + message: 'Released under the MIT License.', + copyright: 'Copyright ยฉ 2026 Alexis Technologies', + }, + + docFooter: { + prev: 'Previous page', + next: 'Next page', + }, + }, + }), + + // โ”€โ”€โ”€ Mermaid โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ + // Deliberately no `theme` here: vitepress-plugin-mermaid picks 'default' or + // 'dark' from VitePress' own isDark and re-renders on toggle, and anything we + // set would win over that and freeze the diagrams in one theme. + mermaid: { + fontFamily: 'var(--vp-font-family-base)', + flowchart: { useMaxWidth: true, htmlLabels: true, padding: 12 }, + sequence: { useMaxWidth: true, actorMargin: 40, wrap: true }, + }, + + mermaidPlugin: { + // `vp-raw` keeps VitePress' own markdown styling off the rendered SVG. + class: 'mermaid vp-raw', + }, +}); diff --git a/docs/.vitepress/theme/custom.css b/docs/.vitepress/theme/custom.css new file mode 100644 index 0000000..33bfcc9 --- /dev/null +++ b/docs/.vitepress/theme/custom.css @@ -0,0 +1,64 @@ +/** + * Brand theme for syncom โ€” the same approach Pinia/Vite/Vue use: override + * VitePress CSS variables to restyle the default theme rather than fork its + * components. + * + * Palette: a replication blue (#3F7FD0 / #7FB0EC) on ink. syncom is about data + * moving between peers and staying consistent, so it wears a cool, calm blue + * rather than a platform colour. + * + * Note on contrast: #3F7FD0 as link text on white is 3.9:1, which fails WCAG + * AA for body text. So in the light theme brand-1/-2 are darkened blues (5.9:1 + * and 4.6:1) and the brighter blue is reserved for filled surfaces, where the + * text on top is white. + */ + +:root { + /* โ”€โ”€โ”€ Brand colors (buttons, links, accents) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ */ + --vp-c-brand-1: #245a9e; + --vp-c-brand-2: #2f6cb8; + --vp-c-brand-3: #3f7fd0; + --vp-c-brand-soft: rgba(63, 127, 208, 0.16); + + /* Default theme alias mappings (kept in sync with brand) */ + --vp-c-default-1: var(--vp-c-gray-1); + --vp-c-default-2: var(--vp-c-gray-2); + --vp-c-default-3: var(--vp-c-gray-3); + --vp-c-default-soft: var(--vp-c-gray-soft); + + --vp-c-tip-1: var(--vp-c-brand-1); + --vp-c-tip-2: var(--vp-c-brand-2); + --vp-c-tip-3: var(--vp-c-brand-3); + --vp-c-tip-soft: var(--vp-c-brand-soft); +} + +.dark { + --vp-c-brand-1: #7fb0ec; + --vp-c-brand-2: #a3c7f3; + --vp-c-brand-3: #2f6cb8; + --vp-c-brand-soft: rgba(63, 127, 208, 0.18); +} + +/* โ”€โ”€โ”€ Home hero: a soft brand wash behind the name โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ */ +:root { + --vp-home-hero-name-color: transparent; + --vp-home-hero-name-background: -webkit-linear-gradient(120deg, #3f7fd0, #6ad0c8); + --vp-home-hero-image-background-image: linear-gradient(-45deg, #3f7fd0 40%, #6ad0c8 60%); + --vp-home-hero-image-filter: blur(48px); +} + +/* โ”€โ”€โ”€ Buttons: white on the brighter blue, ink text on hover โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ */ +:root { + --vp-button-brand-bg: var(--vp-c-brand-3); + --vp-button-brand-text: #ffffff; + --vp-button-brand-border: transparent; + --vp-button-brand-hover-bg: var(--vp-c-brand-2); + --vp-button-brand-hover-text: #ffffff; + --vp-button-brand-active-bg: var(--vp-c-brand-1); + --vp-button-brand-active-text: #ffffff; +} + +/* Inline code in prose reads as data, not as a link. */ +.vp-doc :not(pre) > code { + --vp-code-color: var(--vp-c-text-1); +} diff --git a/docs/.vitepress/theme/index.ts b/docs/.vitepress/theme/index.ts new file mode 100644 index 0000000..847494c --- /dev/null +++ b/docs/.vitepress/theme/index.ts @@ -0,0 +1,35 @@ +import DefaultTheme from 'vitepress/theme'; +import type { Theme } from 'vitepress'; +import './custom.css'; + +// Extends the default VitePress theme with the syncom brand styling (see custom.css). +// This is exactly the pattern Pinia / Vite / Vue use to brand their docs โ€” the +// layout and components stay the default theme, the look comes from CSS variables. +export default { + extends: DefaultTheme, + enhanceApp({ router }) { + // Vercel Web Analytics + Speed Insights. The framework-agnostic `inject` + // entrypoints are used rather than the `/vue` components: those require + // `vue-router`, which VitePress does not use (it ships its own router). + // Both are browser-only, so they are imported dynamically to stay out of + // the SSR build that `pnpm docs:build` runs. + if (import.meta.env.SSR) return; + void import('@vercel/analytics').then(({ inject }) => { + // `mode: 'auto'` logs to the console on localhost and only sends data + // from the deployed site; the injected script picks up VitePress' + // pushState navigations on its own, so no per-route call is needed. + inject({ mode: 'auto', framework: 'vitepress' }); + }); + void import('@vercel/speed-insights').then(({ injectSpeedInsights }) => { + const insights = injectSpeedInsights({ framework: 'vitepress' }); + if (!insights) return; + // Speed Insights does need the route pushed to it by hand. Chain onto any + // existing handler instead of replacing it. + const previous = router.onAfterRouteChange; + router.onAfterRouteChange = (to) => { + insights.setRoute(to); + return previous?.call(router, to); + }; + }); + }, +} satisfies Theme; diff --git a/docs/guide/browser.md b/docs/guide/browser.md new file mode 100644 index 0000000..d566109 --- /dev/null +++ b/docs/guide/browser.md @@ -0,0 +1,69 @@ +# Browser + +The browser build is selected by the `browser` export condition, so any +bundler that honours `package.json#exports` and `#browser` picks it up with no +configuration: + +```js +import { open, diff, merge } from '@alexify/syncom'; +``` + +Under the hood the entry resolves to `browser.js`, and four modules are +swapped for their browser twins: + +| Node | Browser | Backed by | +| --- | --- | --- | +| `src/fsi.js` | `src/fsi.browser.js` | OPFS (`navigator.storage.getDirectory()`) | +| `src/crypto.js` | `src/crypto.browser.js` | Web Crypto (`crypto.subtle`) | +| `src/sync.js` | `src/sync.browser.js` | in-memory node registry | +| `src/utils.js` | `src/utils.browser.js` | `crypto.randomUUID()` | + +CRDT, contracts, the chain and storage are the same code in both halves. + +## What differs + +**Crypto is asynchronous.** `node:crypto` hashes and encrypts synchronously; +the Web Crypto API does not. So in the browser: + +```js +await calculateHash({ a: 1 }); // Promise โ€” sync in Node.js +await encrypt(data, publicKey); +await decrypt(data, privateKey); +``` + +`browser.d.ts` declares these signatures separately from `index.d.ts` for +exactly this reason. + +The cipher differs too: Node.js uses RSA with the default padding via +`publicEncrypt`; the browser uses RSA-OAEP with SHA-256. Keys generated on one +platform are PEM-compatible, but ciphertext is not portable between them. + +**Storage is origin-private.** `fsi.browser.js` resolves paths against +`navigator.storage.getDirectory()`, so `open({ path: 'data' })` creates a +`data/` directory inside the origin's OPFS rather than on disk. It is +per-origin and the browser may evict it. + +**Sync nodes are not persisted.** See [Sync](./sync). + +## Requirements + +OPFS and Web Crypto โ€” available in current Chrome, Edge, Firefox and Safari. +Both APIs require a **secure context** (HTTPS or `localhost`). + +## Bundle size + +The browser entry is measured on every CI run with a budget that fails the +build when exceeded: + +```bash +pnpm size +``` + +The self-contained check in `scripts/size.js` also rejects any bare import +that sneaks into the browser bundle, so it cannot silently grow a dependency. + +## Testing without a browser + +The suite exercises the browser halves under Node.js using a mock OPFS +(`tests/utils/mock-opfs.js`) and `node:crypto`'s `webcrypto`. See +`tests/index.browser.test.js` for the pattern. diff --git a/docs/guide/chain.md b/docs/guide/chain.md new file mode 100644 index 0000000..4d31b52 --- /dev/null +++ b/docs/guide/chain.md @@ -0,0 +1,73 @@ +# Blockchain log + +Every write goes through an append-only, hash-linked log. Each block names its +predecessor by hash, so changing any block invalidates every block after it. + +```mermaid +flowchart LR + G["genesis
prev: 0"] --> B1["block 1
prev: hash(genesis)"] + B1 --> B2["block 2
prev: hash(block 1)"] + B2 --> T["tail
.blockchain.json"] +``` + +## Using it directly + +```js +const { Blockchain } = require('@alexify/syncom'); + +const chain = await new Blockchain('data/blockchain'); +``` + +::: tip The constructor returns a promise +`Blockchain` initialises asynchronously โ€” it reads or writes the genesis block +before it is usable โ€” so it is always `await new Blockchain(...)`. The same is +true of `Storage` and `SyncManager`. The types model this. +::: + +```js +const hash = await chain.addBlock({ id: 'r1', hash: 'abc' }); +const block = await chain.readBlock(hash); +// { prev: '', timestamp: 1757000000000, data: { id: 'r1', hash: 'abc' } } +``` + +A block is `{ prev, timestamp, data }`. Its own hash is the SHA-256 of that +object, and it is stored as `.json`. The tail hash lives in +`.blockchain.json` so the chain can be reopened without scanning the directory. + +`readBlock` throws on an unknown hash. + +## Validating + +```js +await chain.validate({ last: 10 }); // the 10 most recent blocks +await chain.validate({ from: someHash }); // backwards from a given block +await chain.validate(); // the whole chain +``` + +Validation re-hashes each block and compares it to the hash it is filed under. +It walks backwards until it reaches the genesis block (`prev === '0'`) or the +`last` limit. + +Cost is linear in the number of blocks visited, and each visit is a file read โ€” +`validate()` over a long chain is slow by construction. Bound it with `last` +on hot paths. See `bench/chain.js` for the numbers. + +## Through storage + +`Storage` keeps its own chain and adds a block on every `saveData`, recording +the record id and the hash of the stored entry. + +```js +await storage.validate({ last: 10 }); // delegates to the chain +``` + +The per-record form checks one entry against its committing block: + +```js +const entry = JSON.parse(await fsi.read('data/data', `${id}.json`)); +await storage.validate(id, entry.data, entry.block); // true if intact +``` + +This is exactly what `loadData` does on every read โ€” a record whose bytes no +longer hash to what its block recorded raises +`Storage record is invalid` rather than returning the modified data. diff --git a/docs/guide/contracts.md b/docs/guide/contracts.md new file mode 100644 index 0000000..ecfd165 --- /dev/null +++ b/docs/guide/contracts.md @@ -0,0 +1,65 @@ +# Smart contracts + +A smart contract is a JavaScript procedure stored **in the chain** by hash, +loaded back by that hash, and run against a read-only view of storage. It is a +way to make the code that produced a record as auditable as the record itself. + +```js +const { SmartContract, DataReader } = require('@alexify/syncom'); +``` + +## Saving + +`SmartContract.save` serialises the procedure's source into a block and +returns its hash. + +```js +const hash = await SmartContract.save('publish', chain, async (reader, args) => { + const post = await reader.get(args.id); + return { ...post, status: 'published' }; +}); +``` + +The block is `{ type: 'smart contract', name, source }`. + +## Loading and executing + +```js +const contract = await SmartContract.load(hash, { storage, chain }); +const result = await contract.execute({ id: postId, ...args }); +``` + +`execute` calls the procedure with a `DataReader` and your arguments, then +saves the returned value back to storage under `args.id`. If the procedure +throws, the failure is appended to the chain as +`{ contract, args, error, timestamp }` and the error is re-thrown โ€” so a +rejected execution leaves an audit trail rather than vanishing. + +`DataReader` exposes exactly one method, `get(id)`, backed by +`storage.loadData`. A contract cannot write except through its return value. + +## The serialisation format + +::: danger Contracts are `new Function` over stored source +`SmartContract.load` reconstructs the procedure with `new Function`. Anything +that can append a `smart contract` block to your chain can therefore run +arbitrary code in your process. Only load contracts from a chain you control. +::: + +The parser is deliberately narrow. It handles a **single arrow function**, +splitting on the first `=>`: + +```js +// both of these round-trip +async (reader, args) => { /* body */ } +(reader, args) => expression +``` + +It does not handle `function` declarations, default parameter values +containing `=>`, or destructured parameters spanning the arrow. A procedure +outside that shape will not survive the round trip, so keep the stored +procedure to a plain arrow function and put anything complex behind data it +reads. + +Loading a hash whose block is not a contract throws +`Not a smart contract block`. diff --git a/docs/guide/crdt.md b/docs/guide/crdt.md new file mode 100644 index 0000000..ec351b9 --- /dev/null +++ b/docs/guide/crdt.md @@ -0,0 +1,98 @@ +# CRDT deltas + +`diff`, `merge` and `apply` are pure functions over plain objects. They have +no dependency on storage, the chain or the filesystem โ€” you can use them on +their own. + +```js +const { diff, merge, apply } = require('@alexify/syncom'); +``` + +## `diff(source, target)` + +Returns the change that turns `source` into `target`. + +```js +diff({ a: 1, b: 2 }, { a: 1, b: 3 }); // { b: 3 } +diff({ a: 1 }, { a: 1, b: 2 }); // { b: 2 } +diff({ a: 1, b: 2 }, { a: 1 }); // { b: undefined } +``` + +A key that disappeared is reported as `undefined` โ€” that is the deletion +marker `merge` understands. + +Nested objects recurse, and only the changed leaves come back: + +```js +diff({ user: { name: 'Ada', age: 36 } }, { user: { name: 'Ada', age: 37 } }); +// { user: { age: 37 } } +``` + +Arrays and `Set`s are treated as **grow-only sets** โ€” the diff is what the +target added, never what it removed: + +```js +diff({ tags: ['a', 'b'] }, { tags: ['b', 'c'] }); // { tags: ['c'] } +``` + +## `merge(source, delta)` + +Applies a delta. The rules are what make concurrent merges converge: + +| Value in delta | Behaviour | +| --- | --- | +| `undefined` | deletes the key | +| array or `Set` | unions with the existing collection | +| object | merges recursively | +| number, where the target is also a number | **adds** โ€” this is the counter CRDT | +| anything else | replaces | + +```js +merge({ a: 1 }, { b: 2 }); // { a: 1, b: 2 } +merge({ a: 1, b: 2 }, { b: undefined }); // { a: 1 } +merge({ tags: ['a'] }, { tags: ['b'] }); // { tags: ['a', 'b'] } +merge({ hits: 10 }, { hits: 1 }); // { hits: 11 } โ† added, not replaced +merge({ user: { a: 1 } }, { user: { b: 2 } }); // { user: { a: 1, b: 2 } } +``` + +::: warning Numbers add, they do not replace +`merge({ hits: 10 }, { hits: 1 })` is `11`, not `1`. This is deliberate โ€” it is +what lets two peers each increment a counter and converge โ€” but it means a +delta produced by `diff` is **not** interchangeable with a plain object patch +when numeric fields are involved. `diff({ hits: 10 }, { hits: 3 })` yields +`{ hits: 3 }`, and merging that gives `13`. + +For absolute numeric values, write them through `Storage.set` rather than +routing them through `merge`. +::: + +`merge` never mutates its arguments; it returns a new object (or array). + +## `apply(source, history)` + +Replays an operation log onto a starting state. Records that are not objects, +or whose `type` is unrecognised, are skipped. + +| `type` | Effect | +| --- | --- | +| `write` | copies the state forward | +| `delta` | `merge(state, record.data)` | +| `delete` | removes `record.key` | +| `inc` | adds `record.value` to `record.key` | +| `dec` | subtracts `record.value` from `record.key` | + +```js +apply({ hits: 0 }, [ + { type: 'inc', key: 'hits', value: 5 }, + { type: 'inc', key: 'hits', value: 3 }, + { type: 'dec', key: 'hits', value: 2 }, +]); +// { hits: 6 } +``` + +## Where storage uses them + +`Record.save()` sends a `delta` record into the storage update log, and +`Storage.set` / `update` / `swap` do the same for their own changes. The log +is what a future sync transport would exchange with peers โ€” see +[Sync](./sync) for what is and is not implemented today. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md new file mode 100644 index 0000000..244da58 --- /dev/null +++ b/docs/guide/getting-started.md @@ -0,0 +1,90 @@ +# Getting Started + +`syncom` is a local-first data engine. It stores records on disk (or in the +browser's origin-private filesystem), commits every write to a hash-linked log +so tampering is detectable, and produces CRDT deltas that peers can merge +without coordination. + +It has **no runtime dependencies**. + +## Requirements + +- **Node.js โ‰ฅ 22**, or any modern browser with OPFS and Web Crypto. + +## Installation + +```bash +npm install @alexify/syncom +``` + +## Opening a storage + +`open()` is asynchronous: it generates or loads an RSA key pair, initialises +the chain, and returns once the storage is ready to use. + +```js +'use strict'; + +const { open } = require('@alexify/syncom'); + +const main = async () => { + const storage = await open({ path: 'data' }); + + const id = await storage.insert({ name: 'Ada', email: 'ada@example.com' }); + console.log(await storage.get(id)); // { name: 'Ada', email: 'ada@example.com' } + + await storage.update(id, { email: 'ada@lovelace.dev' }); + console.log(await storage.get(id)); // { name: 'Ada', email: 'ada@lovelace.dev' } +}; + +main(); +``` + +`path` (or `name`) selects the directory; it defaults to `syncom`. Under it +syncom creates three subdirectories: + +| Directory | Contents | +| --- | --- | +| `keys/` | `public.pem` / `private.pem`, generated on first open | +| `blockchain/` | one JSON file per block, plus `.blockchain.json` holding the tail hash | +| `data/` | one JSON file per record, each naming the block that committed it | +| `remote/` | one JSON file per registered sync node | + +## Records + +A `Record` is a live view of one stored object. Its properties are real +getters and setters: assigning to one stages a change and emits an `update` +event, and `save()` writes the accumulated delta. + +```js +const record = await storage.record(id); + +record.on('update', (data, delta) => { + console.log('changed', delta); +}); + +record.email = 'ada@lovelace.dev'; +console.log(record.delta()); // { email: 'ada@lovelace.dev' } +await record.save(); +``` + +See [Storage](./storage) for collections, CAS writes and the explicit API. + +## Verifying integrity + +Every record carries the hash of the block that committed it. `validate()` +walks the chain backwards, re-hashing each block: + +```js +await storage.validate({ last: 10 }); // true โ€” the last 10 blocks are intact +await storage.validate(); // the whole chain +``` + +Reading a record that no longer matches its block throws rather than returning +the modified data. See [Blockchain log](./chain). + +## What is not implemented yet + +`storage.sync` registers and persists peer nodes, but **no transport is +implemented** โ€” `sync.start()` is a no-op. See [Sync](./sync) for what exists +today. diff --git a/docs/guide/storage.md b/docs/guide/storage.md new file mode 100644 index 0000000..1e888ba --- /dev/null +++ b/docs/guide/storage.md @@ -0,0 +1,124 @@ +# Storage + +`Storage` is the entry point. `open(options)` returns a promise โ€” key +generation and chain initialisation happen before it resolves. + +```js +const { open } = require('@alexify/syncom'); + +const storage = await open({ path: 'data' }); +``` + +| Option | Default | Meaning | +| --- | --- | --- | +| `path` | `'syncom'` | Directory (or OPFS path) holding keys, chain, data and peers | +| `name` | โ€” | Alias for `path` | +| `schema` | โ€” | Entity definitions; each name becomes a collection (see below) | + +## The explicit API + +Every method addresses a record by its id. Ids are UUIDs generated by +`insert()`. + +```js +const id = await storage.insert({ title: 'Draft', hits: 0 }); + +await storage.has(id); // true +await storage.get(id); // { title: 'Draft', hits: 0 } +await storage.update(id, { hits: 1 }); // merge a partial change +await storage.set(id, { title: 'Final', hits: 1 }); // replace wholesale +await storage.delete(id); +``` + +`set()` and `update()` both emit an `update` event on any live `Record` for +that id, and append the change to the pending update log. + +### Compare-and-swap + +`swap()` writes only if the fields you name still hold the values you expect, +which is how two writers avoid clobbering each other: + +```js +const ok = await storage.swap(id, { hits: 2 }, { hits: 1 }); +// ok === false if hits was no longer 1 โ€” nothing was written +``` + +### Encryption at rest + +`saveData()` takes an `encrypted` flag. The record is encrypted with the +storage's own RSA public key and decrypted on read: + +```js +await storage.saveData(id, { ssn: '...' }, { encrypted: true }); +``` + +RSA-OAEP has a hard payload ceiling (a couple of hundred bytes at 2048-bit +keys), so this is for small secrets, not for whole documents. + +## Records + +`storage.record(id)` returns a live `Record`. It is cached per id, so two +calls for the same record return the same object โ€” and therefore the same +event listeners. + +```js +const record = await storage.record(id); + +record.on('update', (data, delta) => { + console.log(delta); // { title: 'Final' } +}); + +record.title = 'Final'; // stages the change and emits +record.delta(); // { title: 'Final' } +await record.save(); // writes, then clears the delta +await record.data(); // the full current object +await record.delete(); +``` + +Properties are defined from whatever keys the record had when it was loaded; +`id` is reserved and never becomes a settable property. + +### References + +When a stored string happens to be the id of another record that storage +already knows about, reading it back gives you a `Reference` rather than the +raw string. `reference.record()` resolves it: + +```js +const post = await storage.record(postId); +const author = await post.author.record(); +``` + +## Collections + +Pass a `schema` and every entity name becomes a collection on the storage +object. Records inserted through a collection carry a `$type` field naming it. + +```js +const storage = await open({ + path: 'data', + schema: { entities: { Post: {}, Author: {} } }, +}); + +const post = await storage.Post.insert({ title: 'Hello' }); +post.title; // 'Hello' +await storage.Post.get(post.id); // { title: 'Hello', $type: 'Post' } +await storage.Post.update(post.id, { title: 'Hi' }); +await storage.Post.delete(post.id); +``` + +`entities` may be a plain object or a `Map`. Without a schema the storage has +no collections and you address records by id only. + +## On disk + +``` +data/ +โ”œโ”€โ”€ keys/ public.pem, private.pem +โ”œโ”€โ”€ blockchain/ .json per block, .blockchain.json holds the tail +โ”œโ”€โ”€ data/ .json per record: { data, encrypted, timestamp, block } +โ””โ”€โ”€ remote/ .json per registered sync node +``` + +Every record names the block that committed it, which is what makes +[validation](./chain) possible. diff --git a/docs/guide/sync.md b/docs/guide/sync.md new file mode 100644 index 0000000..584e58b --- /dev/null +++ b/docs/guide/sync.md @@ -0,0 +1,57 @@ +# Sync + +::: warning No transport is implemented yet +`SyncManager` registers, persists and lists peer nodes. It does **not** move +any data: `start()` walks the enabled uplinks and does nothing with them. +Treat this page as a description of the node registry, not of working +replication. +::: + +## The node registry + +```js +const storage = await open({ path: 'data' }); + +await storage.sync.addNode({ + type: 'uplink', + url: 'ws://example.com:8001', + token: 'secret', +}); + +storage.sync.listNodes(); +// [{ type: 'uplink', url: 'ws://example.com:8001', token: 'secret', +// enabled: true, lastSync: 1757000000000 }] + +await storage.sync.removeNode('ws://example.com:8001'); +``` + +| Field | Meaning | +| --- | --- | +| `type` | `'uplink'` (a peer this node pulls from) or `'downlink'` (a peer that pulls from this one) | +| `url` | Required, and unique โ€” adding a duplicate throws | +| `token` | Optional shared secret, stored as given | +| `enabled` | Set to `true` on registration | +| `lastSync` | Timestamp, set on registration | + +`addNode` throws on a missing `url`, on a `type` outside those two values, and +on a `url` already registered. `removeNode` throws for an unknown url. + +## Persistence + +On Node.js each node is written to `/remote/.json`, where the host +is taken from the url, and the registry is reloaded from that directory when +the storage is reopened. The browser `SyncManager` keeps its nodes in memory +only โ€” nothing is written to OPFS. + +## What a transport would need + +The pieces the registry is built around, for when a transport lands: + +- `Storage` accumulates an update log (`{ type, id, timestamp, data }` + entries) from every `set`, `update`, `swap` and `Record.save()`. +- [`apply`](./crdt#apply-source-history) replays such a log onto a state, and + [`merge`](./crdt#merge-source-delta) reconciles concurrent deltas. +- The [chain](./chain) gives each peer a verifiable ordering to compare + against. + +None of that is wired to a socket today. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..1a7e68e --- /dev/null +++ b/docs/index.md @@ -0,0 +1,51 @@ +--- +layout: home + +hero: + name: syncom + text: Distributed data sync engine + tagline: Local-first storage with a tamper-evident log, CRDT deltas and JavaScript smart contracts โ€” zero runtime dependencies, one API for Node.js and the browser. + image: + src: /logo-mark.svg + alt: syncom + actions: + - theme: brand + text: Get started + link: /guide/getting-started + - theme: alt + text: API reference + link: /reference/api + - theme: alt + text: GitHub + link: https://github.com/Alexis-Technologies/syncom + +features: + - icon: ๐Ÿ”— + title: Tamper-evident by construction + details: Every write appends a hash-linked block. Reading a record re-derives its hash and checks it against the block that committed it, so silent corruption surfaces as an error instead of bad data. + link: /guide/chain + linkText: Blockchain log + - icon: ๐Ÿงฉ + title: CRDT deltas + details: diff, merge and apply operate on plain objects โ€” nested maps merge recursively, arrays behave as grow-only sets, and numbers merge by addition, so counters converge without coordination. + link: /guide/crdt + linkText: CRDT deltas + - icon: ๐Ÿ“‡ + title: Records and collections + details: A schema turns entity names into collections. Records expose live properties that track their own delta, emit update events, and persist with a single save(). + link: /guide/storage + linkText: Storage + - icon: ๐Ÿ“œ + title: JavaScript smart contracts + details: Store a procedure in the chain by hash, load it back anywhere, and run it against a read-only view of storage. Failures are recorded as blocks. + link: /guide/contracts + linkText: Smart contracts + - icon: ๐ŸŒ + title: Node.js and the browser + details: The same API over node:fs and node:crypto on the server, over OPFS and Web Crypto in the browser โ€” selected by the browser export condition, not by a runtime check. + link: /guide/browser + linkText: Browser + - icon: ๐Ÿ“ฆ + title: Zero runtime dependencies + details: Nothing is installed alongside it. The browser bundle is under 5 KB min+gzip, and CI fails the build if that budget is exceeded. +--- diff --git a/docs/public/llms.txt b/docs/public/llms.txt new file mode 100644 index 0000000..f295d37 --- /dev/null +++ b/docs/public/llms.txt @@ -0,0 +1,32 @@ +# @alexify/syncom + +> Fast, zero-dependency distributed data sync engine for JavaScript. Local-first +> storage with a hash-linked integrity log, CRDT deltas, and JavaScript smart +> contracts โ€” one API across Node.js and the browser. + +## Key facts + +- Package: `@alexify/syncom` (npm), MIT, source at https://github.com/Alexis-Technologies/syncom +- Requires Node.js >= 22, or a browser with OPFS and Web Crypto (secure context) +- **Zero runtime dependencies.** Browser bundle is ~4.7 KB min+gzip +- CommonJS, no build step; `src/` ships as published +- Entry points: `.` (Node `index.js` / browser `browser.js` via the `browser` export condition) +- Exports: `open`, `Storage`, `Blockchain`, `SmartContract`, `DataReader`, `SyncManager`, + `diff`, `merge`, `apply`, `calculateHash`, `generateKeys`, `encrypt`, `decrypt`, + `loadKeys`, `read`, `write`, `exists`, `remove` +- `Storage`, `Blockchain` and `SyncManager` initialise asynchronously: `await new Blockchain(path)` +- In the browser `calculateHash`, `encrypt` and `decrypt` return promises; in Node.js they are synchronous +- `merge` ADDS numbers rather than replacing them โ€” this is the counter CRDT and is deliberate +- **Not implemented:** `storage.sync` registers and persists peer nodes, but no + transport exists; `sync.start()` is a no-op + +## Documentation + +- [Getting Started](https://syncomjs.vercel.app/guide/getting-started): install, open a storage, records, validation. +- [Storage](https://syncomjs.vercel.app/guide/storage): explicit API, compare-and-swap, records, references, schema collections, on-disk layout. +- [CRDT deltas](https://syncomjs.vercel.app/guide/crdt): diff/merge/apply semantics, grow-only sets, the numeric-addition rule. +- [Blockchain log](https://syncomjs.vercel.app/guide/chain): block structure, hash linking, validate cost, per-record verification. +- [Smart contracts](https://syncomjs.vercel.app/guide/contracts): storing procedures in the chain, DataReader, the `new Function` risk, serialisation limits. +- [Sync](https://syncomjs.vercel.app/guide/sync): the node registry and what a transport would still need. +- [Browser](https://syncomjs.vercel.app/guide/browser): OPFS and Web Crypto, the async signature differences, bundle size. +- [API reference](https://syncomjs.vercel.app/reference/api): every export with signatures and return types. diff --git a/docs/public/logo-mark.svg b/docs/public/logo-mark.svg new file mode 100644 index 0000000..222374f --- /dev/null +++ b/docs/public/logo-mark.svg @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/docs/public/robots.txt b/docs/public/robots.txt new file mode 100644 index 0000000..4bd684b --- /dev/null +++ b/docs/public/robots.txt @@ -0,0 +1,4 @@ +User-agent: * +Allow: / + +Sitemap: https://syncomjs.vercel.app/sitemap.xml diff --git a/docs/reference/api.md b/docs/reference/api.md new file mode 100644 index 0000000..64e2a4d --- /dev/null +++ b/docs/reference/api.md @@ -0,0 +1,137 @@ +# API reference + +Everything below is exported from the package root. The browser build exports +the same names โ€” see [Browser](../guide/browser) for the three signatures that +differ. + +```js +const syncom = require('@alexify/syncom'); +``` + +## Storage + +### `open(options?)` + +Returns `Promise`. Options: `path` (default `'syncom'`), `name` +(alias for `path`), `schema`. + +### `Storage` + +`await new Storage(options)` โ€” `open()` is the shorthand. + +| Method | Returns | Notes | +| --- | --- | --- | +| `insert(data)` | `Promise` | Generates a UUID and saves | +| `get(id)` | `Promise` | `null` when absent; verifies against the chain | +| `has(id)` | `Promise` | Filesystem check, no read | +| `set(id, data)` | `Promise` | Replaces; emits `update` with the computed delta | +| `update(id, delta)` | `Promise` | Shallow merge; throws if the record is absent | +| `swap(id, changes, prev)` | `Promise` | CAS โ€” `false`, and no write, on mismatch | +| `delete(id)` | `Promise` | | +| `record(id)` | `Promise` | Cached per id | +| `saveData(id, data, options?)` | `Promise` | `{ encrypted: true }` for RSA at rest | +| `loadData(id)` | `Promise` | Throws if the record no longer matches its block | +| `validate(options?)` | `Promise` | `{ last, from }`; no argument walks the whole chain | +| `validate(id, data, blockHash)` | `Promise` | One record against its block | +| `getCachedData(id)` | `unknown \| null` | Synchronous cache peek | +| `hasRecord(id)` | `boolean` | Whether the id is known to the cache | +| `addUpdate(update)` | `void` | Appends to the pending update log | + +Properties: `schema`, `sync`. Statics: `Storage.Collection`, `Storage.Record`. + +### `Record` + +Extends `Emitter`. Emits `update` with `(data, delta)`. + +`id`, `data()`, `delta()`, `save()`, `delete()`, plus a live property per +field of the loaded object. + +### `Collection` + +Created per schema entity and exposed as `storage.`. + +`name`, `insert(data)`, `get(id)`, `update(id, delta)`, `delete(id)`, +`record(id)`. + +## CRDT + +| Function | Returns | +| --- | --- | +| `diff(source, target)` | The delta turning `source` into `target` | +| `merge(source, delta)` | A new object with the delta applied | +| `apply(source, history)` | The state after replaying an operation log | + +See [CRDT deltas](../guide/crdt) for the merge rules โ€” numbers add rather than +replace. + +## Chain + +### `Blockchain` + +`await new Blockchain(basePath)`. + +| Method | Returns | +| --- | --- | +| `addBlock(data)` | `Promise` โ€” the new tail hash | +| `readBlock(hash)` | `Promise` โ€” throws if unknown | +| `writeBlock(block)` | `Promise` | +| `writeChain()` | `Promise` โ€” persists the tail pointer | +| `validate(options?)` | `Promise` โ€” `{ last, from }` | + +Properties: `path`, `tailHash`. A block is `{ prev, timestamp, data }`. + +### `calculateHash(data)` + +SHA-256 over `JSON.stringify(data)`, hex-encoded. Synchronous in Node.js, +`Promise` in the browser. + +## Contracts + +### `SmartContract` + +`new SmartContract(name, proc, { storage, chain })`. + +- `execute(args)` โ€” runs `proc(reader, args)`, saves the result under + `args.id`, records failures as chain blocks and re-throws. +- `SmartContract.save(name, chain, proc)` โ€” `Promise` (the block hash). +- `SmartContract.load(hash, { storage, chain })` โ€” `Promise`. + +### `DataReader` + +`new DataReader(storage)`; `get(id)` returns `Promise`. The only +storage access a contract has. + +## Crypto + +| Function | Node.js | Browser | +| --- | --- | --- | +| `generateKeys()` | `Promise` | `Promise` | +| `loadKeys(basePath)` | `Promise` | `Promise` | +| `encrypt(data, publicKey)` | `string` | `Promise` | +| `decrypt(data, privateKey)` | `unknown` | `Promise` | +| `calculateHash(data)` | `string` | `Promise` | + +`KeyPair` is `{ publicKey, privateKey }`, PEM-encoded RSA-2048. +`loadKeys` reads `public.pem` / `private.pem` from `basePath`, generating and +writing them if absent. + +## Filesystem interface + +Backed by `node:fs/promises` on the server and OPFS in the browser. All four +take a directory and a filename. + +`read(dir, file)` โ†’ `Promise`, `write(dir, file, content)`, +`exists(dir, file)` โ†’ `Promise`, `remove(dir, file)`. + +`write` creates the directory; `remove` is a no-op when the file is absent. + +## Sync + +### `SyncManager` + +`await new SyncManager(basePath, storage, chain)` โ€” `storage.sync` is the one +you want. + +`addNode(nodeData)`, `removeNode(url)`, `listNodes()`, `start()`. + +See [Sync](../guide/sync) โ€” `start()` is a no-op today. diff --git a/emulation/data.js b/emulation/data.js deleted file mode 100644 index f15f759..0000000 --- a/emulation/data.js +++ /dev/null @@ -1,392 +0,0 @@ -'use strict'; - -const { generateUUID } = require('metautil'); - -const createInitialData = () => { - const now = new Date().toISOString(); - const author1Id = generateUUID(); - const author2Id = generateUUID(); - const author3Id = generateUUID(); - const author4Id = generateUUID(); - const feed1Id = generateUUID(); - const feed2Id = generateUUID(); - const chat1Id = generateUUID(); - const chat2Id = generateUUID(); - const file1Id = generateUUID(); - const file2Id = generateUUID(); - const file3Id = generateUUID(); - const folder1Id = generateUUID(); - const folder2Id = generateUUID(); - const message1Id = generateUUID(); - const message2Id = generateUUID(); - const message3Id = generateUUID(); - const message4Id = generateUUID(); - const post1Id = generateUUID(); - const post2Id = generateUUID(); - const post3Id = generateUUID(); - const post4Id = generateUUID(); - const node1Id = generateUUID(); - const node2Id = generateUUID(); - const peer1Id = generateUUID(); - const peer2Id = generateUUID(); - - const data = {}; - - data[author1Id] = { - nick: 'marcus_aurelius', - name: 'Marcus Aurelius Antoninus', - bio: 'Roman Emperor, Stoic philosopher, author of Meditations', - icon: 'โš”๏ธ', - photo: file1Id, - status: 'online', - created: now, - updated: now, - settings: { theme: 'dark', notifications: true }, - nodes: [node1Id, node2Id], - }; - - data[author2Id] = { - nick: 'faustina', - name: 'Faustina the Younger', - bio: 'Empress of Rome, wife of Marcus Aurelius', - icon: '๐Ÿ‘‘', - photo: file2Id, - status: 'away', - created: now, - updated: now, - settings: { theme: 'light', notifications: true }, - nodes: [node1Id], - }; - - data[author3Id] = { - nick: 'lucius_verus', - name: 'Lucius Verus', - bio: 'Roman co-emperor, brother of Marcus Aurelius', - icon: '๐Ÿ›ก๏ธ', - photo: null, - status: 'online', - created: now, - updated: now, - settings: { theme: 'dark', notifications: false }, - nodes: [node2Id], - }; - - data[author4Id] = { - nick: 'commodus', - name: 'Commodus Antoninus', - bio: 'Roman Emperor, son of Marcus Aurelius', - icon: '๐Ÿ—ก๏ธ', - photo: file3Id, - status: 'dnd', - created: now, - updated: now, - settings: { theme: 'dark', notifications: true }, - nodes: [node1Id], - }; - - data[folder1Id] = { - name: 'Imperial Documents', - owner: author1Id, - icon: '๐Ÿ“œ', - description: 'Official imperial decrees and philosophical works', - created: now, - }; - - data[folder2Id] = { - name: 'Personal Correspondence', - owner: author2Id, - icon: '๐Ÿ“ฌ', - description: 'Private letters and messages', - created: now, - }; - - data[file1Id] = { - name: 'marcus_aurelius_portrait.jpg', - owner: author1Id, - folder: folder1Id, - icon: '๐Ÿ–ผ๏ธ', - description: 'Official imperial portrait', - mime: 'image/jpeg', - size: 245678, - checksum: - 'a1b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef123456', - created: now, - modified: now, - visibility: 'public', - }; - - data[file2Id] = { - name: 'faustina_bust.sculpture', - owner: author2Id, - folder: folder2Id, - icon: '๐Ÿ—ฟ', - description: 'Marble bust sculpture', - mime: 'image/sculpture', - size: 1024000, - checksum: - 'b2c3d4e5f6789012345678901234567890abcdef1234567890abcdef12345678', - created: now, - modified: now, - visibility: 'unlisted', - }; - - data[file3Id] = { - name: 'meditations_book.pdf', - owner: author1Id, - folder: folder1Id, - icon: '๐Ÿ“–', - description: 'Meditations by Marcus Aurelius', - mime: 'application/pdf', - size: 512000, - checksum: - 'c3d4e5f6789012345678901234567890abcdef1234567890abcdef1234567890', - created: now, - modified: now, - visibility: 'public', - }; - - data[feed1Id] = { - name: 'Senatus Populusque Romanus', - owner: author1Id, - icon: '๐Ÿ›๏ธ', - description: 'Official announcements from the Senate and People of Rome', - created: now, - visibility: 'public', - joinPolicy: 'open', - pinned: [post1Id], - }; - - data[feed2Id] = { - name: 'Stoic Philosophy', - owner: author1Id, - icon: '๐Ÿ“š', - description: 'Discussions on Stoic philosophy and wisdom', - created: now, - visibility: 'public', - joinPolicy: 'open', - pinned: [post2Id], - }; - - data[post1Id] = { - title: 'On the Duty of an Emperor', - subtitle: 'Meditations Book I', - content: - 'From my grandfather Verus: I learned good morals and the government ' + - 'of my temper. From the reputation and remembrance of my father: ' + - 'modesty and a manly character. From my mother: piety and ' + - 'beneficence, and abstinence, not only from evil deeds, but even ' + - 'from evil thoughts; and further, simplicity in my way of living, ' + - 'far removed from the habits of the rich.', - feed: feed1Id, - author: author1Id, - created: now, - edited: null, - published: now, - deleted: null, - status: 'published', - reactions: { 'โš”๏ธ': [author2Id, author3Id], '๐Ÿ“š': [author4Id] }, - pinned: true, - attachments: [file3Id], - }; - - data[post2Id] = { - title: 'The Universe is Change', - subtitle: 'Meditations Book IV', - content: - 'The universe is transformation: life is opinion. If a man will take ' + - 'to heart that all things which happen have been from eternity, and ' + - 'in like manner will be to all eternity, he will more easily bear ' + - 'himself contentedly.', - feed: feed2Id, - author: author1Id, - created: now, - edited: null, - published: now, - deleted: null, - status: 'published', - reactions: { '๐Ÿ“š': [author2Id, author3Id], '๐Ÿ’ญ': [author4Id] }, - pinned: true, - attachments: [], - }; - - data[post3Id] = { - title: 'Preparations for the Parthian Campaign', - subtitle: 'Military Strategy', - content: - 'Lucius Verus and I shall lead the legions east. The Parthians have ' + - 'crossed our borders. We must act with haste but maintain discipline. ' + - 'The soldiers trust in our leadership.', - feed: feed1Id, - author: author1Id, - created: now, - edited: null, - published: now, - deleted: null, - status: 'published', - reactions: { 'โš”๏ธ': [author3Id], '๐Ÿ›ก๏ธ': [author2Id] }, - pinned: false, - attachments: [], - }; - - data[post4Id] = { - title: 'The Duties of an Empress', - subtitle: 'Personal Reflections', - content: - 'As empress, my role extends beyond the palace walls. I must support ' + - 'my husband in his duties while maintaining the dignity of the ' + - 'imperial household. The people look to us as examples.', - feed: feed2Id, - author: author2Id, - created: now, - edited: null, - published: null, - deleted: null, - status: 'draft', - reactions: {}, - pinned: false, - attachments: [file2Id], - }; - - data[chat1Id] = { - name: 'Imperial Council', - owner: author1Id, - icon: '๐Ÿ‘‘', - description: 'Private discussions of the imperial family and advisors', - created: now, - lastActivity: now, - members: [author1Id, author2Id, author3Id, author4Id], - admins: [author1Id], - moderators: [author2Id], - mute: [], - ban: [], - pinned: [message1Id], - }; - - data[chat2Id] = { - name: 'Military Command', - owner: author1Id, - icon: 'โš”๏ธ', - description: 'Strategic discussions and military planning', - created: now, - lastActivity: now, - members: [author1Id, author3Id], - admins: [author1Id], - moderators: [], - mute: [], - ban: [], - pinned: [message3Id], - }; - - data[message1Id] = { - chat: chat1Id, - author: author1Id, - content: - 'Greetings to all. We must discuss the upcoming campaigns and the ' + - 'administration of the provinces. Your counsel is valued.', - created: now, - edited: null, - deleted: null, - replyTo: null, - forwarded: null, - reactions: { '๐Ÿ‘': [author2Id, author3Id] }, - pinned: true, - attachments: [], - }; - - data[message2Id] = { - chat: chat1Id, - author: author2Id, - content: - 'The preparations for the victory celebrations are underway. The ' + - 'people expect great spectacle. We must not disappoint them.', - created: now, - edited: null, - deleted: null, - replyTo: message1Id, - forwarded: null, - reactions: { '๐Ÿ‘‘': [author1Id] }, - pinned: false, - attachments: [], - }; - - data[message3Id] = { - chat: chat2Id, - author: author3Id, - content: - 'The legions are ready. We have sufficient supplies for the march. ' + - 'When do we depart for the eastern provinces?', - created: now, - edited: null, - deleted: null, - replyTo: null, - forwarded: null, - reactions: { 'โš”๏ธ': [author1Id] }, - pinned: true, - attachments: [], - }; - - data[message4Id] = { - chat: chat2Id, - author: author1Id, - content: - 'We march at dawn in three days. Ensure the cavalry is prepared. ' + - 'The Parthians favor mounted combat - we must match their mobility.', - created: now, - edited: null, - deleted: null, - replyTo: message3Id, - forwarded: null, - reactions: { '๐Ÿ›ก๏ธ': [author3Id] }, - pinned: false, - attachments: [], - }; - - data[node1Id] = { - name: 'Roma Capital', - domain: 'roma.imperium', - ip: '192.168.1.100', - ports: [8001, 8002], - }; - - data[node2Id] = { - name: 'Antioch Province', - domain: 'antioch.imperium', - ip: '192.168.1.101', - ports: [8001], - }; - - data[peer1Id] = { - name: 'Athens Academy', - domain: 'athens.imperium', - ip: '192.168.1.102', - ports: [8001], - }; - - data[peer2Id] = { - name: 'Alexandria Library', - domain: 'alexandria.imperium', - ip: '192.168.1.103', - ports: [8001, 8002], - }; - - return data; -}; - -let cachedData = null; -let cachedIds = null; - -const getInitialDataWithIds = () => { - if (!cachedData) { - cachedData = createInitialData(); - const ids = Object.keys(cachedData); - cachedIds = { - authors: ids.filter((id) => cachedData[id].nick), - posts: ids.filter((id) => cachedData[id].title), - chats: ids.filter((id) => cachedData[id].members), - messages: ids.filter((id) => cachedData[id].chat), - }; - } - return { data: cachedData, ids: cachedIds }; -}; - -module.exports = { createInitialData, getInitialDataWithIds }; diff --git a/emulation/example.js b/emulation/example.js deleted file mode 100644 index c876443..0000000 --- a/emulation/example.js +++ /dev/null @@ -1,147 +0,0 @@ -'use strict'; - -const globalStorage = require('./storage.js'); -const { getInitialDataWithIds } = require('./data.js'); - -const formatAuthor = (author) => - `${author.icon} ${author.name} (@${author.nick}) - ${author.bio}`; - -const formatMessage = (message, authors) => { - const author = authors.get(message.author); - const authorName = author ? author.name : 'Unknown'; - const reactions = Object.entries(message.reactions || {}) - .map(([emoji, authorIds]) => { - const names = authorIds - .map((id) => { - const a = authors.get(id); - return a ? a.name : 'Unknown'; - }) - .join(', '); - return `${emoji} (${names})`; - }) - .join(' | '); - const reactionsText = reactions ? `\n Reactions: ${reactions}` : ''; - return ` ${authorName}: ${message.content}${reactionsText}`; -}; - -const formatPost = (post, authors) => { - const author = authors.get(post.author); - const authorName = author ? author.name : 'Unknown'; - const status = post.status === 'published' ? 'โœ“' : 'โ—‹'; - const reactions = Object.entries(post.reactions || {}) - .map(([emoji, authorIds]) => { - const count = authorIds ? authorIds.length : 0; - return `${emoji} ${count}`; - }) - .join(' '); - const reactionsText = reactions ? ` [${reactions}]` : ''; - const preview = post.content.substring(0, 100); - const dots = post.content.length > 100 ? '...' : ''; - const byLine = `\n by ${authorName}\n ${preview}${dots}`; - return `[${status}] ${post.title}${reactionsText}${byLine}`; -}; - -const main = async () => { - const storage = await globalStorage.open(); - - const authors = new Map(); - const posts = []; - const messages = []; - const chats = new Map(); - - const { ids } = getInitialDataWithIds(); - - for (const authorId of ids.authors) { - const author = await storage.get(authorId); - if (author) authors.set(authorId, author); - } - - for (const postId of ids.posts) { - const post = await storage.get(postId); - if (post) posts.push(post); - } - - for (const chatId of ids.chats) { - const chat = await storage.get(chatId); - if (chat) chats.set(chatId, chat); - } - - for (const messageId of ids.messages) { - const message = await storage.get(messageId); - if (message) messages.push(message); - } - - console.log('\nIMPERIAL AUTHOR DIRECTORY\n'); - - authors.forEach((author) => { - console.log(formatAuthor(author)); - }); - - console.log('\nRECENT POSTS\n'); - - posts.forEach((post) => { - console.log(formatPost(post, authors)); - console.log(''); - }); - - console.log('\nMESSAGES\n'); - - chats.forEach((chat, chatId) => { - const desc = chat.description || ''; - console.log(` conversation: ${chat.name} (${desc})`); - const chatMessages = messages.filter((m) => m.chat === chatId); - chatMessages.forEach((message) => { - console.log(formatMessage(message, authors)); - }); - console.log(''); - }); - - console.log('\nADDING REACTIONS\n'); - - const { ids: initialIds } = getInitialDataWithIds(); - const firstPostId = initialIds.posts[0]; - const firstPost = await storage.get(firstPostId); - if (firstPost) { - const currentReactions = firstPost.reactions || {}; - const marcusId = initialIds.authors[0]; - if (!currentReactions['๐Ÿ‘']) { - currentReactions['๐Ÿ‘'] = []; - } - if (!currentReactions['๐Ÿ‘'].includes(marcusId)) { - currentReactions['๐Ÿ‘'].push(marcusId); - } - await storage.update(firstPostId, { reactions: currentReactions }); - console.log('โœ… Added ๐Ÿ‘ reaction to post:', firstPost.title); - } - - const firstMessageId = initialIds.messages[0]; - const firstMessage = await storage.get(firstMessageId); - if (firstMessage) { - const currentReactions = firstMessage.reactions || {}; - const faustinaId = initialIds.authors[1]; - if (!currentReactions['๐Ÿ’ฌ']) { - currentReactions['๐Ÿ’ฌ'] = []; - } - if (!currentReactions['๐Ÿ’ฌ'].includes(faustinaId)) { - currentReactions['๐Ÿ’ฌ'].push(faustinaId); - } - await storage.update(firstMessageId, { reactions: currentReactions }); - console.log('โœ… Added ๐Ÿ’ฌ reaction to message'); - } - - console.log('\nUPDATED DATA\n'); - - const updatedPost = await storage.get(firstPostId); - if (updatedPost) { - console.log(formatPost(updatedPost, authors)); - } - - const updatedMessage = await storage.get(firstMessageId); - if (updatedMessage) { - console.log('\n' + formatMessage(updatedMessage, authors)); - } - - console.log('\n'); -}; - -main().catch(console.error); diff --git a/emulation/storage.js b/emulation/storage.js deleted file mode 100644 index c73c611..0000000 --- a/emulation/storage.js +++ /dev/null @@ -1,360 +0,0 @@ -'use strict'; - -const { Emitter, generateUUID } = require('metautil'); -const { getInitialDataWithIds } = require('./data.js'); - -const calculateDelta = (oldData, newData) => { - const delta = {}; - for (const key in newData) { - if (oldData[key] !== newData[key]) { - delta[key] = newData[key]; - } - } - return delta; -}; - -class Reference { - #id; - #storage = null; - - constructor(id, storage) { - this.#id = id; - this.#storage = storage; - } - - async record() { - return await this.#storage.record(this.#id); - } -} - -class Record extends Emitter { - #id; - #storage = null; - #data = null; - #changes = {}; - - constructor(id, storage) { - super(); - this.#id = id; - this.#storage = storage; - return this.#load(); - } - - async #load() { - if (this.#data !== null) return this; - const rawData = await this.#storage.get(this.#id); - if (rawData) { - this.#data = JSON.parse(JSON.stringify(rawData)); - this.#convertToReferences(this.#data); - this.#defineProperties(); - } - return this; - } - - #convertToReferences(data) { - if (!data || typeof data !== 'object') return; - for (const key in data) { - const value = data[key]; - if (typeof value === 'string') { - if (this.#storage.hasRecord(value)) { - data[key] = new Reference(value, this.#storage); - } - } else if (value && typeof value === 'object') { - const isReference = value instanceof Reference; - if (!isReference) { - this.#convertToReferences(value); - } - } - } - } - - get id() { - return this.#id; - } - - #defineProperties() { - for (const key in this.#data) { - if (key === 'id') continue; - if (Object.prototype.hasOwnProperty.call(this, key)) continue; - Object.defineProperty(this, key, { - get() { - const value = this.#data?.[key]; - if (value instanceof Reference) { - return value; - } - return value; - }, - set(newValue) { - const oldValue = this.#data?.[key]; - if (oldValue === newValue) return; - this.#data[key] = newValue; - this.#changes[key] = newValue; - const delta = { [key]: newValue }; - this.emit('update', this.#data, delta); - }, - enumerable: true, - configurable: true, - }); - } - } - - async data() { - return this.#data; - } - - delta() { - return { ...this.#changes }; - } - - async save() { - if (Object.keys(this.#changes).length === 0) return; - const delta = this.delta(); - const data = this.#data; - await this.#storage.saveData(this.#id, data); - this.#changes = {}; - this.#storage.addUpdate({ - type: 'delta', - id: this.#id, - timestamp: Date.now(), - data: delta, - }); - } - - async delete() { - await this.#storage.delete(this.#id); - this.#data = null; - this.#changes = {}; - } -} - -class SyncManager { - #nodes = new Set(); - - async addNode(nodeData) { - const { url, type, token } = nodeData; - if (!url) throw new Error('Node URL is required'); - if (type !== 'uplink' && type !== 'downlink') { - throw new Error('Node type must be uplink or downlink'); - } - const now = Date.now(); - const node = { type, url, token, enabled: true, lastSync: now }; - this.#nodes.add(node); - } - - async removeNode(url) { - const node = Array.from(this.#nodes).find((n) => n.url === url); - if (!node) throw new Error(`Node with URL ${url} not found`); - this.#nodes.delete(node); - } - - listNodes() { - return Array.from(this.#nodes); - } - - // eslint-disable-next-line class-methods-use-this - async start() { - return Promise.resolve(); - } -} - -class Collection { - #storage; - #name; - - constructor(storage, name) { - this.#storage = storage; - this.#name = name; - } - - get name() { - return this.#name; - } - - async insert(data) { - const id = generateUUID(); - const record = { ...data, $type: this.#name }; - await this.#storage.saveData(id, record); - return this.#storage.record(id); - } - - async get(id) { - return this.#storage.get(id); - } - - async delete(id) { - return this.#storage.delete(id); - } - - async update(id, delta) { - return this.#storage.update(id, delta); - } - - async record(id) { - return this.#storage.record(id); - } -} - -class Storage { - #data = new Map(); - #records = new Map(); - #sync = null; - #cache = new Map(); - #updates = []; - #schema = null; - - constructor(options = {}) { - if (options.schema) this.#schema = options.schema; - this.#sync = new SyncManager(); - return this.#init(); - } - - async #init() { - const { data } = getInitialDataWithIds(); - for (const [id, item] of Object.entries(data)) { - this.#data.set(id, item); - this.#cache.set(id, item); - } - this.#initCollections(); - return this; - } - - #initCollections() { - const entities = this.#schema?.entities; - if (!entities) return; - for (const [name] of entities.entries?.() ?? Object.entries(entities)) { - this[name] = new Collection(this, name); - } - } - - get schema() { - return this.#schema; - } - - #emitUpdate(id, data, delta) { - const emitter = this.#records.get(id); - if (emitter) { - emitter.emit('update', data, delta); - } - } - - // eslint-disable-next-line no-unused-vars - async saveData(id, data, options = {}) { - this.#data.set(id, { ...data }); - this.#cache.set(id, data); - this.#updates.push({ type: 'write', id, timestamp: Date.now(), data }); - } - - async insert(data) { - const id = generateUUID(); - await this.saveData(id, data); - return id; - } - - async has(id) { - return this.#data.has(id); - } - - async loadData(id) { - if (this.#cache.has(id)) { - return this.#cache.get(id); - } - const data = this.#data.get(id); - if (data) { - this.#cache.set(id, data); - return { ...data }; - } - this.#cache.set(id, null); - return null; - } - - async get(id) { - return this.loadData(id); - } - - getCachedData(id) { - return this.#cache.has(id) ? this.#cache.get(id) : null; - } - - hasRecord(id) { - return this.#cache.has(id) || this.#records.has(id); - } - - addUpdate(update) { - this.#updates.push(update); - } - - async set(id, data) { - const oldData = this.#data.get(id); - await this.saveData(id, data); - if (oldData) { - const delta = calculateDelta(oldData, data); - this.#emitUpdate(id, data, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); - } - } - - async delete(id) { - this.#data.delete(id); - this.#records.delete(id); - this.#cache.delete(id); - this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); - } - - async update(id, delta) { - const data = this.#data.get(id); - if (!data) throw new Error(`Record ${id} not found`); - const updated = { ...data, ...delta }; - await this.saveData(id, updated); - this.#emitUpdate(id, updated, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); - } - - async swap(id, changes, prev) { - const data = this.#data.get(id); - if (!data) throw new Error(`Record ${id} not found`); - const matches = Object.keys(prev).every((key) => data[key] === prev[key]); - if (!matches) return false; - const updated = { ...data, ...changes }; - this.#data.set(id, updated); - this.#emitUpdate(id, updated, changes); - return true; - } - - async record(id) { - if (!this.#records.has(id)) { - const record = await new Record(id, this); - this.#records.set(id, record); - } - return this.#records.get(id); - } - - // eslint-disable-next-line no-unused-vars - async validate(idOrOptions, _data, _blockHash) { - if (typeof idOrOptions === 'object' && idOrOptions !== null) { - return true; - } - const id = idOrOptions; - return this.#data.has(id); - } - - get sync() { - return this.#sync; - } -} - -Storage.Collection = Collection; -Storage.Record = Record; - -const open = async (options = {}) => new Storage(options); - -module.exports = { Storage, open, SyncManager }; diff --git a/example.js b/example.js deleted file mode 100644 index ac571f5..0000000 --- a/example.js +++ /dev/null @@ -1,91 +0,0 @@ -'use strict'; - -const globalStorage = require('./globalstorage.js'); -const xxii = require('xxii-schema'); - -const main = async () => { - const xxiiSchema = await xxii.load(); - console.log(xxiiSchema); - const storage = await globalStorage.open({ schema: xxiiSchema }); - - await storage.sync - .addNode({ - type: 'uplink', - url: 'ws://example.com:8001', - name: 'Main Server', - token: 'secret-token-123', - }) - .then(() => { - console.log('Added uplink node'); - }) - .catch((error) => { - console.error('Error adding uplink node:', error.message); - }); - - await storage.sync - .addNode({ - type: 'downlink', - url: 'ws://client.example.com:8002', - name: 'Client Node', - token: 'secret-token-456', - }) - .then(() => { - console.log('Added downlink node'); - }) - .catch((error) => { - console.error('Error adding downlink node:', error.message); - }); - - const nodes = storage.sync.listNodes(); - console.log('List nodes', nodes); - - // Implicit API - Record with dynamic getters/setters - - const authorData = { name: 'Timur', email: 'timur@example.com' }; - const authorId = await storage.insert(authorData); - const authorRecord = await storage.record(authorId); - authorRecord.email = 'timur.shemsedinov@gmail.com'; - await authorRecord.save(); - - const post = { title: 'Example', content: 'Text', author: authorId }; - const postRecord = await storage.Post.insert(post); - - postRecord.on('update', (data, delta) => { - console.log('Record updated:', { id: postRecord.id, data, delta }); - }); - - console.log('Title:', postRecord.title); - console.log('Content:', postRecord.content); - console.log('Author:', postRecord.author); - - postRecord.content += 'example'; - postRecord.title = 'Updated Example'; - - const delta = postRecord.delta(); - console.log('Delta before save:', delta); - - await postRecord.save(); - - console.log('Author name:', postRecord.author.name); - - // Explicit API (still available) - - const exists = await storage.has(postRecord.id); - if (exists) { - const data = await storage.get(postRecord.id); - data.content += ' more changes'; - await storage.set(postRecord.id, data); - } - - await storage.sync.start(); - console.log('Synced'); - - const valid = await storage.validate({ last: 10 }); - console.log('Storage valid after adding:', valid); - - await storage.sync.removeNode('ws://client.example.com:8002'); - console.log('Removed downlink node'); - console.log('Remaining nodes:', storage.sync.listNodes()); -}; - -main(); diff --git a/examples/main.js b/examples/main.js index 25ebf24..efed032 100644 --- a/examples/main.js +++ b/examples/main.js @@ -1,10 +1,10 @@ 'use strict'; -const globalStorage = require('../globalstorage.js'); +const syncom = require('..'); const domain = require('./domain.js'); const main = async () => { - const storage = await globalStorage.open(); + const storage = await syncom.open(); domain.init(storage); const authorId = await storage.insert({ diff --git a/index.d.ts b/index.d.ts index f06bb9f..8260dec 100644 --- a/index.d.ts +++ b/index.d.ts @@ -16,6 +16,11 @@ export function encrypt(data: unknown, publicKey: string): string; export function decrypt(data: string, privateKey: string): unknown; export function loadKeys(basePath: string): Promise; +export function read(dir: string, filename: string): Promise; +export function write(dir: string, filename: string, content: string): Promise; +export function exists(dir: string, filename: string): Promise; +export function remove(dir: string, filename: string): Promise; + export type CrdtDelta = Record; export type CrdtHistoryRecord = @@ -183,7 +188,7 @@ export interface Storage { options?: StorageDataOptions, ): Promise; loadData(id: string): Promise; - validate(options: { last?: number; from?: string }): Promise; + validate(options?: { last?: number; from?: string }): Promise; validate(id: string, data: unknown, blockHash: string): Promise; insert(data: unknown): Promise; has(id: string): Promise; @@ -198,7 +203,7 @@ export interface Storage { addUpdate(update: StorageUpdate): void; } -export interface GsRecord { +export interface SyncomRecord { readonly id: string; on(event: 'update', listener: (data: unknown, delta: unknown) => void): void; data(): Promise; @@ -207,33 +212,33 @@ export interface GsRecord { delete(): Promise; } -declare class GsCollection { +declare class SyncomCollection { constructor(storage: Storage, name: string); readonly name: string; - insert(data: unknown): Promise; + insert(data: unknown): Promise; get(id: string): Promise; delete(id: string): Promise; update(id: string, delta: unknown): Promise; - record(id: string): Promise; + record(id: string): Promise; } -declare const GsRecordCtor: { - readonly prototype: GsRecord; - new (id: string, storage: Storage): Promise; +declare const SyncomRecordCtor: { + readonly prototype: SyncomRecord; + new (id: string, storage: Storage): Promise; }; export interface StorageConstructor { readonly prototype: Storage; new (options?: StorageOptions): Promise; - Collection: typeof GsCollection; - Record: typeof GsRecordCtor; + Collection: typeof SyncomCollection; + Record: typeof SyncomRecordCtor; } export const Storage: StorageConstructor; export namespace Storage { - export type StorageRecord = GsRecord; - export type StorageCollection = GsCollection; + export type StorageRecord = SyncomRecord; + export type StorageCollection = SyncomCollection; } export function open(options?: StorageOptions): Promise; diff --git a/index.js b/index.js index 0401c05..e33990d 100644 --- a/index.js +++ b/index.js @@ -1,9 +1,3 @@ 'use strict'; -const chain = require('./src/chain.js'); -const crypto = require('./src/crypto.js'); -const crdt = require('./src/crdt.js'); -const contract = require('./src/contract.js'); -const storage = require('./src/storage.js'); - -module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...storage }; +module.exports = require('./src/index.js'); diff --git a/package.json b/package.json index bb4a264..f618a04 100644 --- a/package.json +++ b/package.json @@ -5,33 +5,39 @@ "main": "index.js", "browser": { "./index.js": "./browser.js", - "./src/runtime/node.js": "./src/runtime/browser.js" + "./src/index.js": "./src/index.browser.js", + "./src/crypto.js": "./src/crypto.browser.js", + "./src/fsi.js": "./src/fsi.browser.js", + "./src/sync.js": "./src/sync.browser.js", + "./src/utils.js": "./src/utils.browser.js" }, "types": "index.d.ts", "exports": { ".": { + "browser": { + "types": "./browser.d.ts", + "default": "./browser.js" + }, "types": "./index.d.ts", - "browser": "./browser.js", "default": "./index.js" - } + }, + "./package.json": "./package.json" }, "directories": { "test": "tests" }, "scripts": { - "test": "node --test tests/*.test.js", - "test:coverage": "c8 --all --include 'src/**' --check-coverage --lines 95 --statements 95 --branches 90 --functions 95 --reporter text --reporter lcov node --test tests/*.test.js", + "test": "node --test", + "test:coverage": "c8 --all --include 'src/**' --check-coverage --lines 95 --statements 95 --branches 90 --functions 95 --reporter text --reporter lcov node --test", "test:types": "tsd", - "bench": "node bench/bench.js", + "bench": "node bench/run-all.js", "lint": "oxlint src tests scripts bench", "format": "oxfmt src tests scripts bench", "format:check": "oxfmt --check src tests scripts bench", "size": "node scripts/size.js", "docs:dev": "vitepress dev docs", "docs:build": "vitepress build docs", - "docs:preview": "vitepress preview docs", - "bench:compare": "node bench/compare.js", - "size:compare": "node scripts/size-compare.js" + "docs:preview": "vitepress preview docs" }, "tsd": { "directory": "tests" @@ -54,13 +60,16 @@ "conflict-resolution", "CRDT", "event-sourcing", - "database", - "globalstorage", - "localstorage", + "blockchain", + "smart-contracts", + "local-first", + "offline-first", "storage", - "global", - "warehouse", - "distributed" + "database", + "zero-dependency", + "javascript", + "nodejs", + "browser" ], "author": "Alex Dolid ", "license": "MIT", @@ -69,21 +78,22 @@ }, "homepage": "https://syncomjs.vercel.app/", "devDependencies": { + "@types/node": "^22.10.0", "@vercel/analytics": "^2.0.1", "@vercel/speed-insights": "^2.0.0", "c8": "^10.1.3", "esbuild": "^0.28.1", "mermaid": "^11.16.0", - "oxfmt": "^0.58.0", - "oxlint": "^1.73.0", + "oxfmt": "^0.62.0", + "oxlint": "^1.77.0", "pino": "^10.3.1", - "tsd": "^0.31.2", + "tsd": "^0.33.0", "typescript": "^7.0.2", "vitepress": "^1.6.4", "vitepress-plugin-mermaid": "^2.0.17" }, "engines": { - "node": ">=18.0.0" + "node": ">=22" }, "publishConfig": { "access": "public" @@ -93,6 +103,7 @@ "index.js", "index.d.ts", "browser.js", + "browser.d.ts", "src/", "LICENSE", "README.md", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 451a01c..cbcd381 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -8,6 +8,9 @@ importers: .: devDependencies: + '@types/node': + specifier: ^22.10.0 + version: 22.20.1 '@vercel/analytics': specifier: ^2.0.1 version: 2.0.1(vue@3.5.42(typescript@7.0.2)) @@ -24,26 +27,26 @@ importers: specifier: ^11.16.0 version: 11.17.2 oxfmt: - specifier: ^0.58.0 - version: 0.58.0 + specifier: ^0.62.0 + version: 0.62.0 oxlint: - specifier: ^1.73.0 + specifier: ^1.77.0 version: 1.81.0 pino: specifier: ^10.3.1 version: 10.3.1 tsd: - specifier: ^0.31.2 - version: 0.31.2 + specifier: ^0.33.0 + version: 0.33.0 typescript: specifier: ^7.0.2 version: 7.0.2 vitepress: specifier: ^1.6.4 - version: 1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) + version: 1.6.4(@algolia/client-search@5.57.0)(@types/node@22.20.1)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) vitepress-plugin-mermaid: specifier: ^2.0.17 - version: 2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)) + version: 2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(@types/node@22.20.1)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)) packages: @@ -533,124 +536,124 @@ packages: resolution: {integrity: sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==} engines: {node: '>= 8'} - '@oxfmt/binding-android-arm-eabi@0.58.0': - resolution: {integrity: sha512-Uz62sHduGGPftXtILGyxdSW4PX82rUg+rfdNqhsgxe881g4rIoXlIqmZQ6HVKcF4f+F8qMhdD03Bx5u7gmeTdg==} + '@oxfmt/binding-android-arm-eabi@0.62.0': + resolution: {integrity: sha512-pdsv0C4gPjJ8H1+sd8u0BDx+yLACTL+rgeMIOL1ln4ihSnhw8CWXtYWgvcSkyTfgGBIzFKab+d8rx9Xl4en/Kw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [android] - '@oxfmt/binding-android-arm64@0.58.0': - resolution: {integrity: sha512-rD0lRaJp1b+9vw6X4A2dJWKukd6X8yxiicN4JxXcXayolmUypRZxk+lKR+fVOu5q/iYc0fh5fR4bgmfOfVlbaA==} + '@oxfmt/binding-android-arm64@0.62.0': + resolution: {integrity: sha512-WC3YQ7uS/KtDrjmqwBviwFKe9qeoi+eXx8aX1z/ffG23Md75myjrJaQqTuJvdOLPoa4EYTjDWH0dHXfwulCVog==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@oxfmt/binding-darwin-arm64@0.58.0': - resolution: {integrity: sha512-uzbPPk7O6M+w2K65vcQ1woga3wgP8zghjL1KOG5b6qJ8dvYHZJ1VShaslg2KOK6yQIwCQtcMCXqLBM6sqXUNTg==} + '@oxfmt/binding-darwin-arm64@0.62.0': + resolution: {integrity: sha512-GM8Yf3LjjaR1I8PD0SfeoIlwhsh9GvSF+cQ8sf624Yxnjsyumn95aFzYfKJVefblfDIiOAnZ7QVm2sa21Er/0Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@oxfmt/binding-darwin-x64@0.58.0': - resolution: {integrity: sha512-L0nKYDxU32oxeQqJj21W9SlIMnf81VZEhyah6iDvFhf5q0oynq498Fopth7blErUJVBpVtxQ98RMCfMPqpJX6w==} + '@oxfmt/binding-darwin-x64@0.62.0': + resolution: {integrity: sha512-d5THp7F8bCxLqNogEXDORRsQD6dosf3EyFtnXfBer6v+8tGdcWIjoDX9WaXrrF/26zOmL8qHpPTKCEvpBDmZkQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@oxfmt/binding-freebsd-x64@0.58.0': - resolution: {integrity: sha512-woNwfD58dC5PGS9LSLSD5JYfo/EFK5iG9vhDWkcCg3q78ag7KC8bpDqgvPHrMoXpx83OLXxoSOhu6z8FsVTHlg==} + '@oxfmt/binding-freebsd-x64@0.62.0': + resolution: {integrity: sha512-1DnrtXGZooOZ0fHgAXZUaDQzBVh1CM2MNW4oBXyQ2aWKvCHjyljvT9fgBkOM0fEOb96X5eqtcfJ0YUVt9jj66g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@oxfmt/binding-linux-arm-gnueabihf@0.58.0': - resolution: {integrity: sha512-Sqs8nMLxuQpY21NKJ1u4stPDmO5hskBCNNh2E3AdCfI1QqWtf4m+Qn4mGEIUO4KGmuq3SWc/SZ80uy5IiwTCDw==} + '@oxfmt/binding-linux-arm-gnueabihf@0.62.0': + resolution: {integrity: sha512-4pQDHOYRH+Huqe0StIaWyvk2CVl/aTaqSrbZpA3/pLS2xH24ME7lBgYprhQF2fRkHBzhGGGKliwxFsDdHwx59g==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxfmt/binding-linux-arm-musleabihf@0.58.0': - resolution: {integrity: sha512-Vd4exzBI5B5hB9m22JiTQzIL23WvHo/Pe+sNXPNeBLXSP9swCBPKCEBRwKpmpQzYhlgYaCgfPcGXPKAJBRIiZQ==} + '@oxfmt/binding-linux-arm-musleabihf@0.62.0': + resolution: {integrity: sha512-X0jAaZJFMCVKhB6YyWVTQ/wN2DLsBcZKSMqTS76bF6riT+XZdtg2FPEdjDvdVbunO9cG+tWiVaEs4Zs38lxYog==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@oxfmt/binding-linux-arm64-gnu@0.58.0': - resolution: {integrity: sha512-bUWi5mHV+4Vi56RLHE1h6q/HHfwAIT3XoB9vJAVeRzfu5NriXM8y6eeJu0vlKa0C9kq2rq1sOWRClhdLHPocrg==} + '@oxfmt/binding-linux-arm64-gnu@0.62.0': + resolution: {integrity: sha512-682Z8T5s8T5ATArYtsejKvbIfd8LEAXyyDkKkoZVq8HND7Vx8TYLlrDjDSeYfodMeVwHOgkj13lJYR8cj6vUSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-arm64-musl@0.58.0': - resolution: {integrity: sha512-2ZHxemzgHcjtktAuVUwSoyXmGo/t+aF5tS1ciPpPei4rhSyrz3JOqDosXXrmhN/yLUSzJjtuW7ToTWqfQpCj2w==} + '@oxfmt/binding-linux-arm64-musl@0.62.0': + resolution: {integrity: sha512-lk25fAl7KWaLWVJcW0CHEXB7QlQZtx5eDkjpaGMK0hzXTjUe0Wmlu8IKuFHoviSOcEJedRTs4VE/506VqGxGew==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@oxfmt/binding-linux-ppc64-gnu@0.58.0': - resolution: {integrity: sha512-AwKkVwjVmFQ3bcO7j0McGYAqCKH2a326fswfofng/E8VewCT/raeeGQr4huVhY704deK8AWASSTlxzMj0eZc6Q==} + '@oxfmt/binding-linux-ppc64-gnu@0.62.0': + resolution: {integrity: sha512-SFyNqHQLwySceWNLhiSldx7wPXRAzP0L0WcW9GegP3uWrpZGJiZlQO85NbHAFPEfxR9PhZ9qSnZryEh7+v+4Gw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-riscv64-gnu@0.58.0': - resolution: {integrity: sha512-xsRpTxfUnJF8D3AUKko/qyWdjw4GZVHlCVFuGlzSCTeewLmykKINW8em1+wx+axsDVtJJcMtvsiaXggXxrlHgw==} + '@oxfmt/binding-linux-riscv64-gnu@0.62.0': + resolution: {integrity: sha512-KYj55C1ywJfHo6+aKDuEmUtVEdJALsC5GwayDGsI6FGz2GxFqNr/mA8nxVsNbJzm7sE5MRqTQ9ziImSzhYXysA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-riscv64-musl@0.58.0': - resolution: {integrity: sha512-Z4AYOTcy7nYEIiXwD62PlerimyYRcfJOgUbQAEBjXz098kxKuERBlRntofGy69HHhe9E0TLVNMl1yspVNu+efw==} + '@oxfmt/binding-linux-riscv64-musl@0.62.0': + resolution: {integrity: sha512-BhZDNo5GOU5nC378RhD0/XpvaEBHsH3HLgJp8YZX3A0InC7oivzA63HsRmiXFLtLSHAstEVrDf6fbC7Rs8Jh/A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [riscv64] os: [linux] libc: [musl] - '@oxfmt/binding-linux-s390x-gnu@0.58.0': - resolution: {integrity: sha512-A3nhhtZPC/TKVWOPj9q/H3p2znJDCcHWYlJBhWL8hGq/bFmBaNBHC8Np6E581yVq1w9Mi3rMDNzDalWvtUfJtQ==} + '@oxfmt/binding-linux-s390x-gnu@0.62.0': + resolution: {integrity: sha512-UyAFmyHkgSgUJ/wOM4p3U8AC2yAFvRH5PNBs7TnK0fObTT/XSWcdr/lAzPSWaekHaZFaMeFZyk9n93Joq3J93A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-x64-gnu@0.58.0': - resolution: {integrity: sha512-2g+tVkgwqphw8R4hgo+kF4oz8+P5RwVOtr9+irsC7uwEp0e9j7Crw8kDGKL20uYlLPD7g02DqA61mC/UNYx98A==} + '@oxfmt/binding-linux-x64-gnu@0.62.0': + resolution: {integrity: sha512-1iYMP0leytWazFubD/WnINJuIrzRPuoL1aWEJdlGezEzDbTxcd29R4r8IUzP2oWeKst5V02uMJgR2NILlPlG6w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@oxfmt/binding-linux-x64-musl@0.58.0': - resolution: {integrity: sha512-rc15P6AbyyB7426aN8AakLd02Trb3a6ML/mmfAQeVHJEfVofWLcWIrBdy6zDEY+DIaL/s8E4GGPboVw+oP3+EA==} + '@oxfmt/binding-linux-x64-musl@0.62.0': + resolution: {integrity: sha512-4rA/URtJSTVNVAQz6Q8wf7SaRvOXVy+TizriT9hs/Y1XhLR/R+92uWKRQG8yFWRAIEBbFHJ6WevQcl/G9SXEfw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@oxfmt/binding-openharmony-arm64@0.58.0': - resolution: {integrity: sha512-ZWoTM27/HYPOh9iq86DAbhPu9nXb8qKvvGU/h8OfliyVUFAMMNTLDkGsWDKKnDqIkqvZ9+dXlgUOsH1LYO3O7g==} + '@oxfmt/binding-openharmony-arm64@0.62.0': + resolution: {integrity: sha512-mSZuFHU2ar1KLUjXpI2QBQcJ1VsOB3mOCgQXuXCpKs19dgh4u+OaovNfrWDfiJb+ihJ2+f7YFcaO9bS2dlTCXA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@oxfmt/binding-win32-arm64-msvc@0.58.0': - resolution: {integrity: sha512-LHZnqFXe2dEfkRI4XdZS/57nEOT/I4UCRX5IyM9v4GYW9XwQCjGe1IUK59SuKw3POwvcgWQ4pme2cYXmNqTNPg==} + '@oxfmt/binding-win32-arm64-msvc@0.62.0': + resolution: {integrity: sha512-OfwuhkcjDlqC4EgDojtiV9mzpLqeB9KqTOWPOjLEYBVdDCVSxqW3qzp/xcIxsbtI0UgGCnKvAqYKyY25kf5JZw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@oxfmt/binding-win32-ia32-msvc@0.58.0': - resolution: {integrity: sha512-mZKpg20TpheCJym1rarcZCUJeW1sSruw8zAAaCYWvuVfwIUDN1CXdrPU/JgCWReXTCTrEfCB8Wyo3hh9jSZ2EA==} + '@oxfmt/binding-win32-ia32-msvc@0.62.0': + resolution: {integrity: sha512-P9uDDNFRzghO3X8QAzhkjKhK7JvtABsVn8UYtFX7uor12IAnwNt8nNIctvfWj1JkQU/kE+fmLRPiw7XlrIHsZw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ia32] os: [win32] - '@oxfmt/binding-win32-x64-msvc@0.58.0': - resolution: {integrity: sha512-N/wUU4N5PZ2orBtI+Ko7MnMfYLfE7K91UrGMY/c/pYyHR3lA9kwst1XugkZx+92YcRh/Eo+iv2eTESSWXfiZPA==} + '@oxfmt/binding-win32-x64-msvc@0.62.0': + resolution: {integrity: sha512-dlI5SY7XYQCiCBafntWagCR6HcAJB/NpsLtdlPx8x08+Osz8Ok1HHz1GZuusegCe/VoJ6pAnF5a4pd5OZAq7qQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -949,8 +952,8 @@ packages: '@sinclair/typebox@0.27.12': resolution: {integrity: sha512-hhyNJ+nbR6ZR7pToHvllEFun9TL0sbL+tk/ON75lo+Xas054uez98qRbsuNt7MBCyZKK4+8Yli/OAGZhmfBZ/g==} - '@tsd/typescript@5.4.5': - resolution: {integrity: sha512-saiCxzHRhUrRxQV2JhH580aQUZiKQUXI38FcAcikcfOomAil4G4lxT0RfrrKywoAYP/rqAdYXYmNRLppcd+hQQ==} + '@tsd/typescript@5.9.3': + resolution: {integrity: sha512-JSSdNiS0wgd8GHhBwnMAI18Y8XPhLVN+dNelPfZCXFhy9Lb3NbnFyp9JKxxr54jSUkEJPk3cidvCoHducSaRMQ==} engines: {node: '>=14.17'} '@types/d3-array@3.2.2': @@ -1079,6 +1082,9 @@ packages: '@types/minimist@1.2.5': resolution: {integrity: sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag==} + '@types/node@22.20.1': + resolution: {integrity: sha512-EANqOCF9QFyra+4pfxUcX9STKJpCLjMbObVzljIJomAWSnuSIEAvyzEU53GaajbXJEgdh0iEcPL+DGvpUd4k1Q==} + '@types/normalize-package-data@2.4.4': resolution: {integrity: sha512-37i+OaWTh9qeK4LSHPsyRC7NahnGotNuZvjLSgcPzblpHB3rrCJxAOgI5gCdKm7coonsaX1Of0ILiTcnZjbfxA==} @@ -2063,8 +2069,8 @@ packages: oniguruma-to-es@3.1.1: resolution: {integrity: sha512-bUH8SDvPkH3ho3dvwJwfonjlQ4R80vjyvrU8YpxuROddv55vAEJrTuCuCVUhhsHbtlD9tGGbaNApGQckXhS8iQ==} - oxfmt@0.58.0: - resolution: {integrity: sha512-8feG/7NVEHDVwc1OUpP6Pks+TnaDFUw2jLLFIMi5bcmmwxAX2wBQvjSzj62RRTYBf2Op1Wt8xbkmagmPTR5ETg==} + oxfmt@0.62.0: + resolution: {integrity: sha512-vxgGHTmnDU9j4CX7dDBLzxgmHxfda/yPcgJkGCMUSCwRmz+euo/V08xXLNgXTeqAB9Fhf3Pe2nO1RNKLCVgphQ==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: @@ -2414,8 +2420,8 @@ packages: resolution: {integrity: sha512-JfJeIHke7y2egdGGgRAvpCwYFUsHlM2gPcrVOxFkznt/4uzQ7HFmvE63iFHVLBJNDuyDOQgijDK/tXH/f6Msjg==} engines: {node: '>=6.10'} - tsd@0.31.2: - resolution: {integrity: sha512-VplBAQwvYrHzVihtzXiUVXu5bGcr7uH1juQZ1lmKgkuGNGT+FechUCqmx9/zk7wibcqR2xaNEwCkDyKh+VVZnQ==} + tsd@0.33.0: + resolution: {integrity: sha512-/PQtykJFVw90QICG7zyPDMIyueOXKL7jOJVoX5pILnb3Ux+7QqynOxfVvarE+K+yi7BZyOSY4r+OZNWSWRiEwQ==} engines: {node: '>=14.16'} hasBin: true @@ -2440,6 +2446,9 @@ packages: engines: {node: '>=16.20.0'} hasBin: true + undici-types@6.21.0: + resolution: {integrity: sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==} + unist-util-is@6.0.1: resolution: {integrity: sha512-LsiILbtBETkDz8I9p1dQ0uyRUWuaQzd/cuEeS1hoRSyW5E5XGmTzlwY1OrNzzakGowI9Dr/I8HVaw4hTtnxy8g==} @@ -2953,61 +2962,61 @@ snapshots: '@nodelib/fs.scandir': 2.1.5 fastq: 1.20.3 - '@oxfmt/binding-android-arm-eabi@0.58.0': + '@oxfmt/binding-android-arm-eabi@0.62.0': optional: true - '@oxfmt/binding-android-arm64@0.58.0': + '@oxfmt/binding-android-arm64@0.62.0': optional: true - '@oxfmt/binding-darwin-arm64@0.58.0': + '@oxfmt/binding-darwin-arm64@0.62.0': optional: true - '@oxfmt/binding-darwin-x64@0.58.0': + '@oxfmt/binding-darwin-x64@0.62.0': optional: true - '@oxfmt/binding-freebsd-x64@0.58.0': + '@oxfmt/binding-freebsd-x64@0.62.0': optional: true - '@oxfmt/binding-linux-arm-gnueabihf@0.58.0': + '@oxfmt/binding-linux-arm-gnueabihf@0.62.0': optional: true - '@oxfmt/binding-linux-arm-musleabihf@0.58.0': + '@oxfmt/binding-linux-arm-musleabihf@0.62.0': optional: true - '@oxfmt/binding-linux-arm64-gnu@0.58.0': + '@oxfmt/binding-linux-arm64-gnu@0.62.0': optional: true - '@oxfmt/binding-linux-arm64-musl@0.58.0': + '@oxfmt/binding-linux-arm64-musl@0.62.0': optional: true - '@oxfmt/binding-linux-ppc64-gnu@0.58.0': + '@oxfmt/binding-linux-ppc64-gnu@0.62.0': optional: true - '@oxfmt/binding-linux-riscv64-gnu@0.58.0': + '@oxfmt/binding-linux-riscv64-gnu@0.62.0': optional: true - '@oxfmt/binding-linux-riscv64-musl@0.58.0': + '@oxfmt/binding-linux-riscv64-musl@0.62.0': optional: true - '@oxfmt/binding-linux-s390x-gnu@0.58.0': + '@oxfmt/binding-linux-s390x-gnu@0.62.0': optional: true - '@oxfmt/binding-linux-x64-gnu@0.58.0': + '@oxfmt/binding-linux-x64-gnu@0.62.0': optional: true - '@oxfmt/binding-linux-x64-musl@0.58.0': + '@oxfmt/binding-linux-x64-musl@0.62.0': optional: true - '@oxfmt/binding-openharmony-arm64@0.58.0': + '@oxfmt/binding-openharmony-arm64@0.62.0': optional: true - '@oxfmt/binding-win32-arm64-msvc@0.58.0': + '@oxfmt/binding-win32-arm64-msvc@0.62.0': optional: true - '@oxfmt/binding-win32-ia32-msvc@0.58.0': + '@oxfmt/binding-win32-ia32-msvc@0.62.0': optional: true - '@oxfmt/binding-win32-x64-msvc@0.58.0': + '@oxfmt/binding-win32-x64-msvc@0.62.0': optional: true '@oxlint/binding-android-arm-eabi@1.81.0': @@ -3189,7 +3198,7 @@ snapshots: '@sinclair/typebox@0.27.12': {} - '@tsd/typescript@5.4.5': {} + '@tsd/typescript@5.9.3': {} '@types/d3-array@3.2.2': {} @@ -3340,6 +3349,10 @@ snapshots: '@types/minimist@1.2.5': {} + '@types/node@22.20.1': + dependencies: + undici-types: 6.21.0 + '@types/normalize-package-data@2.4.4': {} '@types/trusted-types@2.0.7': @@ -3424,9 +3437,9 @@ snapshots: optionalDependencies: vue: 3.5.42(typescript@7.0.2) - '@vitejs/plugin-vue@5.2.4(vite@5.4.21)(vue@3.5.42(typescript@7.0.2))': + '@vitejs/plugin-vue@5.2.4(vite@5.4.21(@types/node@22.20.1))(vue@3.5.42(typescript@7.0.2))': dependencies: - vite: 5.4.21 + vite: 5.4.21(@types/node@22.20.1) vue: 3.5.42(typescript@7.0.2) '@vue/compiler-core@3.5.42': @@ -4313,29 +4326,29 @@ snapshots: regex: 6.1.0 regex-recursion: 6.0.2 - oxfmt@0.58.0: + oxfmt@0.62.0: dependencies: tinypool: 2.1.0 optionalDependencies: - '@oxfmt/binding-android-arm-eabi': 0.58.0 - '@oxfmt/binding-android-arm64': 0.58.0 - '@oxfmt/binding-darwin-arm64': 0.58.0 - '@oxfmt/binding-darwin-x64': 0.58.0 - '@oxfmt/binding-freebsd-x64': 0.58.0 - '@oxfmt/binding-linux-arm-gnueabihf': 0.58.0 - '@oxfmt/binding-linux-arm-musleabihf': 0.58.0 - '@oxfmt/binding-linux-arm64-gnu': 0.58.0 - '@oxfmt/binding-linux-arm64-musl': 0.58.0 - '@oxfmt/binding-linux-ppc64-gnu': 0.58.0 - '@oxfmt/binding-linux-riscv64-gnu': 0.58.0 - '@oxfmt/binding-linux-riscv64-musl': 0.58.0 - '@oxfmt/binding-linux-s390x-gnu': 0.58.0 - '@oxfmt/binding-linux-x64-gnu': 0.58.0 - '@oxfmt/binding-linux-x64-musl': 0.58.0 - '@oxfmt/binding-openharmony-arm64': 0.58.0 - '@oxfmt/binding-win32-arm64-msvc': 0.58.0 - '@oxfmt/binding-win32-ia32-msvc': 0.58.0 - '@oxfmt/binding-win32-x64-msvc': 0.58.0 + '@oxfmt/binding-android-arm-eabi': 0.62.0 + '@oxfmt/binding-android-arm64': 0.62.0 + '@oxfmt/binding-darwin-arm64': 0.62.0 + '@oxfmt/binding-darwin-x64': 0.62.0 + '@oxfmt/binding-freebsd-x64': 0.62.0 + '@oxfmt/binding-linux-arm-gnueabihf': 0.62.0 + '@oxfmt/binding-linux-arm-musleabihf': 0.62.0 + '@oxfmt/binding-linux-arm64-gnu': 0.62.0 + '@oxfmt/binding-linux-arm64-musl': 0.62.0 + '@oxfmt/binding-linux-ppc64-gnu': 0.62.0 + '@oxfmt/binding-linux-riscv64-gnu': 0.62.0 + '@oxfmt/binding-linux-riscv64-musl': 0.62.0 + '@oxfmt/binding-linux-s390x-gnu': 0.62.0 + '@oxfmt/binding-linux-x64-gnu': 0.62.0 + '@oxfmt/binding-linux-x64-musl': 0.62.0 + '@oxfmt/binding-openharmony-arm64': 0.62.0 + '@oxfmt/binding-win32-arm64-msvc': 0.62.0 + '@oxfmt/binding-win32-ia32-msvc': 0.62.0 + '@oxfmt/binding-win32-x64-msvc': 0.62.0 oxlint@1.81.0: optionalDependencies: @@ -4689,9 +4702,9 @@ snapshots: ts-dedent@2.3.0: {} - tsd@0.31.2: + tsd@0.33.0: dependencies: - '@tsd/typescript': 5.4.5 + '@tsd/typescript': 5.9.3 eslint-formatter-pretty: 4.1.0 globby: 11.1.0 jest-diff: 29.7.0 @@ -4730,6 +4743,8 @@ snapshots: '@typescript/typescript-win32-arm64': 7.0.2 '@typescript/typescript-win32-x64': 7.0.2 + undici-types@6.21.0: {} + unist-util-is@6.0.1: dependencies: '@types/unist': 3.0.3 @@ -4776,22 +4791,23 @@ snapshots: '@types/unist': 3.0.3 vfile-message: 4.0.3 - vite@5.4.21: + vite@5.4.21(@types/node@22.20.1): dependencies: esbuild: 0.21.5 postcss: 8.5.28 rollup: 4.63.1 optionalDependencies: + '@types/node': 22.20.1 fsevents: 2.3.3 - vitepress-plugin-mermaid@2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)): + vitepress-plugin-mermaid@2.0.17(mermaid@11.17.2)(vitepress@1.6.4(@algolia/client-search@5.57.0)(@types/node@22.20.1)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2)): dependencies: mermaid: 11.17.2 - vitepress: 1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) + vitepress: 1.6.4(@algolia/client-search@5.57.0)(@types/node@22.20.1)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2) optionalDependencies: '@mermaid-js/mermaid-mindmap': 9.3.0 - vitepress@1.6.4(@algolia/client-search@5.57.0)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2): + vitepress@1.6.4(@algolia/client-search@5.57.0)(@types/node@22.20.1)(postcss@8.5.28)(search-insights@2.17.3)(typescript@7.0.2): dependencies: '@docsearch/css': 3.8.2 '@docsearch/js': 3.8.2(@algolia/client-search@5.57.0)(search-insights@2.17.3) @@ -4800,7 +4816,7 @@ snapshots: '@shikijs/transformers': 2.5.0 '@shikijs/types': 2.5.0 '@types/markdown-it': 14.2.0 - '@vitejs/plugin-vue': 5.2.4(vite@5.4.21)(vue@3.5.42(typescript@7.0.2)) + '@vitejs/plugin-vue': 5.2.4(vite@5.4.21(@types/node@22.20.1))(vue@3.5.42(typescript@7.0.2)) '@vue/devtools-api': 7.7.10 '@vue/shared': 3.5.42 '@vueuse/core': 12.8.2(typescript@7.0.2) @@ -4809,7 +4825,7 @@ snapshots: mark.js: 8.11.1 minisearch: 7.2.0 shiki: 2.5.0 - vite: 5.4.21 + vite: 5.4.21(@types/node@22.20.1) vue: 3.5.42(typescript@7.0.2) optionalDependencies: postcss: 8.5.28 diff --git a/scripts/size.js b/scripts/size.js new file mode 100644 index 0000000..0b60bef --- /dev/null +++ b/scripts/size.js @@ -0,0 +1,112 @@ +/** + * Honest bundle-size measurement for the README "Bundle size" table. + * + * Every entry from package.json#exports is bundled the way a consumer's + * bundler would (browser condition for the browser half, plain Node + * resolution for the node half), then reports: + * - raw: bundled, un-minified + * - min: fully minified (whitespace + identifier mangling + syntax, + * comments stripped) + * - min+gzip: the number that matters for shipping to browsers + * + * Only the entry that DOES reach a browser carries a `budget` (min+gzip KB), + * and exceeding it fails the run, which is what keeps CI honest about bundle + * growth. The node entry is reported for visibility, not as a shipping cost. + * + * Run with `pnpm size`. + */ + +'use strict'; + +const { gzipSync } = require('node:zlib'); +const path = require('node:path'); +const esbuild = require('esbuild'); + +const ROOT = path.join(__dirname, '..'); + +// One row per package.json#exports entry โ€” tests/package/consistency.test.js +// fails if the two ever drift apart. +const ENTRIES = [ + // Budget set from the first measured run and treated as a ratchet, not a + // target: raise it deliberately, in the same change that earns the bytes. + { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 5 }, + { label: 'main entry โ€” node (@alexify/syncom)', entry: 'index.js', platform: 'node' }, +]; + +// A browser entry has to be self-contained: no node builtins, and no packages +// either โ€” esbuild resolves devDependencies happily, so a stray +// `require('metautil')` would be inlined into the bundle and only show up as a +// few extra KB. Failing at RESOLVE time names the offending import and its +// importer, which a substring search over the output cannot. +const selfContained = { + name: 'self-contained', + setup(build) { + build.onResolve({ filter: /.*/ }, (args) => { + if (args.kind === 'entry-point') return null; + if (args.path.startsWith('.') || path.isAbsolute(args.path)) return null; + const text = `browser bundle must not import '${args.path}' (from ${path.relative(ROOT, args.importer)})`; + return { errors: [{ text }] }; + }); + }, +}; + +async function bundle(entry, platform, minify) { + const result = await esbuild.build({ + entryPoints: [path.join(ROOT, entry)], + bundle: true, + minify, + platform, + conditions: platform === 'browser' ? ['browser'] : [], + plugins: platform === 'browser' ? [selfContained] : [], + write: false, + logLevel: 'silent', + }); + return Buffer.from(result.outputFiles[0].contents); +} + +function kb(bytes) { + return `${(bytes / 1024).toFixed(1)} KB`; +} + +async function main() { + const rows = []; + const over = []; + for (const { label, entry, platform, budget } of ENTRIES) { + // The browser entry is bundled through the self-contained plugin, so an + // import that does not belong fails the build here rather than inflating a + // number nobody reads. + const raw = await bundle(entry, platform, false); + const min = await bundle(entry, platform, true); + const gzip = gzipSync(min, { level: 9 }).length; + if (budget !== undefined && gzip > budget * 1024) { + over.push(`${entry}: ${kb(gzip)} min+gzip is over its ${budget} KB budget`); + } + rows.push({ + label, + raw: kb(raw.length), + min: kb(min.length), + gzip: kb(gzip), + budget: budget === undefined ? 'โ€”' : `${budget}.0 KB`, + }); + } + + console.log('| Entry | raw | min | min+gzip | budget |'); + console.log('| ----- | ---:| ---:| --------:| ------:|'); + for (const row of rows) { + console.log(`| ${row.label} | ${row.raw} | ${row.min} | ${row.gzip} | ${row.budget} |`); + } + + // Reported after the table: seeing every number is what makes a budget + // failure actionable. + if (over.length > 0) throw new Error(`bundle size budget exceeded\n ${over.join('\n ')}`); +} + +main().catch((error) => { + // esbuild rejects with an object whose useful part is `errors[].text`; the + // object itself prints as a wall of getters, which is how a clear "you + // imported a package into the browser bundle" turns into noise. + const reported = Array.isArray(error?.errors) ? error.errors : null; + if (reported) for (const { text } of reported) console.error(text); + else console.error(error); + process.exitCode = 1; +}); diff --git a/src/contract.js b/src/contract.js index fde0b67..ad65798 100644 --- a/src/contract.js +++ b/src/contract.js @@ -10,6 +10,10 @@ const deserializeFunction = (source) => { if (body.startsWith('{')) body = body.slice(1, -1); else body = `return ${body};`; body = `return async (${args}) => {${body}}`; + // This IS the feature: a smart contract is source stored in the chain and + // reconstituted here. See SECURITY.md โ€” a chain you load contracts from is + // trusted input, because anything that can append a block can run code. + // oxlint-disable-next-line no-new-func return new Function('reader', 'args', body)(); }; diff --git a/src/crdt.js b/src/crdt.js index 11f8f06..e7ab411 100644 --- a/src/crdt.js +++ b/src/crdt.js @@ -51,12 +51,8 @@ const merge = (source, delta) => { } if (Array.isArray(delta[key]) || delta[key] instanceof Set) { - const sourceSet = new Set( - Array.isArray(result[key]) ? result[key] : result[key] || [], - ); - const deltaSet = new Set( - Array.isArray(delta[key]) ? delta[key] : delta[key], - ); + const sourceSet = new Set(Array.isArray(result[key]) ? result[key] : result[key] || []); + const deltaSet = new Set(Array.isArray(delta[key]) ? delta[key] : delta[key]); result[key] = Array.from(new Set([...sourceSet, ...deltaSet])); } else if (typeof delta[key] === 'object' && delta[key] !== null) { if (typeof result[key] === 'object' && result[key] !== null) { @@ -64,10 +60,7 @@ const merge = (source, delta) => { } else { result[key] = delta[key]; } - } else if ( - typeof delta[key] === 'number' && - typeof result[key] === 'number' - ) { + } else if (typeof delta[key] === 'number' && typeof result[key] === 'number') { result[key] += delta[key]; } else { result[key] = delta[key]; diff --git a/src/crypto-browser.js b/src/crypto.browser.js similarity index 98% rename from src/crypto-browser.js rename to src/crypto.browser.js index 574588a..d625e94 100644 --- a/src/crypto-browser.js +++ b/src/crypto.browser.js @@ -1,6 +1,6 @@ 'use strict'; -const fsi = require('./fsi-browser.js'); +const fsi = require('./fsi.browser.js'); const PUBLIC_KEY_HEADER = '-----BEGIN PUBLIC KEY-----'; const PUBLIC_KEY_FOOTER = '-----END PUBLIC KEY-----'; diff --git a/src/fsi-browser.js b/src/fsi.browser.js similarity index 100% rename from src/fsi-browser.js rename to src/fsi.browser.js diff --git a/src/fsi.js b/src/fsi.js index 5600986..190da5a 100644 --- a/src/fsi.js +++ b/src/fsi.js @@ -2,7 +2,7 @@ const fs = require('node:fs/promises'); const path = require('node:path'); -const { fileExists } = require('metautil'); +const { fileExists } = require('./utils.js'); const read = (dir, filename) => fs.readFile(path.join(dir, filename), 'utf8'); diff --git a/src/index.browser.js b/src/index.browser.js new file mode 100644 index 0000000..7468af5 --- /dev/null +++ b/src/index.browser.js @@ -0,0 +1,15 @@ +'use strict'; + +// The browser twin of ./index.js: it names the browser variant of every split +// module directly. The requires NESTED inside chain.js / storage.js (./fsi.js, +// ./crypto.js, ./sync.js, ./utils.js) are redirected by package.json#browser, +// so a bundler that ignores that field would still pull the Node halves in. +const chain = require('./chain.js'); +const contract = require('./contract.js'); +const crdt = require('./crdt.js'); +const crypto = require('./crypto.browser.js'); +const fsi = require('./fsi.browser.js'); +const storage = require('./storage.js'); +const sync = require('./sync.browser.js'); + +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync }; diff --git a/src/index.js b/src/index.js index e69de29..c60a42f 100644 --- a/src/index.js +++ b/src/index.js @@ -0,0 +1,11 @@ +'use strict'; + +const chain = require('./chain.js'); +const contract = require('./contract.js'); +const crdt = require('./crdt.js'); +const crypto = require('./crypto.js'); +const fsi = require('./fsi.js'); +const storage = require('./storage.js'); +const sync = require('./sync.js'); + +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync }; diff --git a/src/storage.js b/src/storage.js index 9e583b2..134782e 100644 --- a/src/storage.js +++ b/src/storage.js @@ -1,6 +1,6 @@ 'use strict'; -const { Emitter, generateUUID } = require('metautil'); +const { Emitter, generateUUID } = require('./utils.js'); const { encrypt, decrypt, loadKeys, calculateHash } = require('./crypto.js'); const { Blockchain } = require('./chain.js'); const { SyncManager } = require('./sync.js'); @@ -178,7 +178,7 @@ class Storage { #schema = null; constructor(options = {}) { - this.#path = options.path || options.name || 'gs'; + this.#path = options.path || options.name || 'syncom'; this.#records = new Map(); if (options.schema) this.#schema = options.schema; return this.#init(); @@ -247,9 +247,13 @@ class Storage { return decrypted; } - async validate(idOrOptions, data, blockHash) { - const isChainValidation = - !data && !blockHash && ('last' in idOrOptions || 'from' in idOrOptions); + async validate(idOrOptions = {}, data, blockHash) { + // Two shapes share one method: `validate(id, data, blockHash)` checks a + // single record against its block, anything else walks the chain. A record + // id is always a string, so that is the discriminator โ€” keying off `'last' + // in idOrOptions` instead made the no-argument and `{}` calls fall through + // to the record branch, where they threw or silently returned false. + const isChainValidation = typeof idOrOptions !== 'string' && !data && !blockHash; if (isChainValidation) { return await this.#chain.validate(idOrOptions); } diff --git a/src/sync-browser.js b/src/sync.browser.js similarity index 83% rename from src/sync-browser.js rename to src/sync.browser.js index 12cb086..c345956 100644 --- a/src/sync-browser.js +++ b/src/sync.browser.js @@ -13,11 +13,9 @@ class SyncManager { } async start() { - // TODO: Implement metacom-based sync (WebSocket client) - // Future: WebRTC transport for peer-to-peer sync - const uplinks = Array.from(this.#nodes).filter( - (node) => node.type === 'uplink' && node.enabled, - ); + // TODO: no transport yet โ€” see the note in ./sync.js. A browser + // WebSocket client goes here; WebRTC would follow for peer-to-peer. + const uplinks = Array.from(this.#nodes).filter((node) => node.type === 'uplink' && node.enabled); if (uplinks.length === 0) return; for (const uplink of uplinks) { try { diff --git a/src/sync.js b/src/sync.js index a1c7724..dfc7f97 100644 --- a/src/sync.js +++ b/src/sync.js @@ -2,7 +2,7 @@ const fs = require('node:fs/promises'); const path = require('node:path'); -const { between } = require('metautil'); +const { between } = require('./utils.js'); class SyncManager { #path; @@ -28,12 +28,10 @@ class SyncManager { } async start() { - // TODO: Implement metacom-based sync - // This would connect to the node using metacom protocol - // and exchange CRDT delta logs for sync - const uplinks = Array.from(this.#nodes).filter( - (node) => node.type === 'uplink' && node.enabled, - ); + // TODO: no transport yet โ€” start() walks the enabled uplinks and does + // nothing with them. A WebSocket client that exchanges CRDT delta logs + // goes here. + const uplinks = Array.from(this.#nodes).filter((node) => node.type === 'uplink' && node.enabled); if (uplinks.length === 0) return; for (const uplink of uplinks) { try { diff --git a/src/utils.browser.js b/src/utils.browser.js new file mode 100644 index 0000000..4640daa --- /dev/null +++ b/src/utils.browser.js @@ -0,0 +1,123 @@ +'use strict'; + +// Browser twin of ./utils.js, wired in through package.json#browser. The +// filesystem predicates (fileExists / directoryExists) have no counterpart +// here: the browser storage layer goes through OPFS in ./fsi.browser.js. +// `emit` takes `...args` rather than metautil's single `value`: Record and +// Storage both emit `('update', data, delta)`, and index.d.ts types the +// listener as `(data, delta) => void`. A single-value emit silently drops the +// delta, which is exactly the contract consumers read from the types. +class Emitter { + #events = new Map(); + #maxListeners = 10; + + constructor(options = {}) { + this.#maxListeners = options.maxListeners ?? 10; + } + + emit(eventName, ...args) { + const event = this.#events.get(eventName); + if (!event) { + if (eventName !== 'error') return Promise.resolve(); + throw new Error('Unhandled error'); + } + const listeners = event.on.slice(); + const promises = listeners.map(async (fn) => fn(...args)); + if (event.once.size > 0) { + const len = event.on.length; + const remaining = new Array(len); + let index = 0; + for (let i = 0; i < len; i++) { + const listener = event.on[i]; + if (!event.once.has(listener)) remaining[index++] = listener; + } + if (index === 0) { + this.#events.delete(eventName); + } else { + remaining.length = index; + this.#events.set(eventName, { on: remaining, once: new Set() }); + } + } + return Promise.all(promises).then(() => undefined); + } + + #addListener(eventName, listener, once) { + let event = this.#events.get(eventName); + if (!event) { + const on = [listener]; + event = { on, once: once ? new Set(on) : new Set() }; + this.#events.set(eventName, event); + } else { + if (event.on.includes(listener)) { + throw new Error('Duplicate listeners detected'); + } + event.on.push(listener); + if (once) event.once.add(listener); + } + if (event.on.length > this.#maxListeners) { + throw new Error( + `MaxListenersExceededWarning: Possible memory leak. ` + `Current maxListeners is ${this.#maxListeners}.`, + ); + } + } + + on(eventName, listener) { + this.#addListener(eventName, listener, false); + } + + once(eventName, listener) { + this.#addListener(eventName, listener, true); + } + + off(eventName, listener) { + if (!listener) return void this.#events.delete(eventName); + const event = this.#events.get(eventName); + if (!event) return; + const index = event.on.indexOf(listener); + if (index > -1) event.on.splice(index, 1); + event.once.delete(listener); + } + + toPromise(eventName) { + return new Promise((resolve) => { + this.once(eventName, resolve); + }); + } + + clear(eventName) { + if (!eventName) return void this.#events.clear(); + this.#events.delete(eventName); + } + + listeners(eventName) { + if (!eventName) throw new Error('Expected eventName'); + const event = this.#events.get(eventName); + return event ? event.on : []; + } + + listenerCount(eventName) { + if (!eventName) throw new Error('Expected eventName'); + const event = this.#events.get(eventName); + return event ? event.on.length : 0; + } + + eventNames() { + return Array.from(this.#events.keys()); + } +} + +const generateUUID = () => crypto.randomUUID(); + +const between = (s, prefix, suffix) => { + let i = s.indexOf(prefix); + if (i === -1) return ''; + s = s.substring(i + prefix.length); + if (suffix) { + i = s.indexOf(suffix); + if (i === -1) return ''; + s = s.substring(0, i); + } + return s; +}; + +module.exports = { Emitter, generateUUID, between }; diff --git a/src/utils.js b/src/utils.js new file mode 100644 index 0000000..fc3930f --- /dev/null +++ b/src/utils.js @@ -0,0 +1,141 @@ +'use strict'; + +const fsp = require('node:fs/promises'); +const crypto = require('node:crypto'); + +// `emit` takes `...args` rather than metautil's single `value`: Record and +// Storage both emit `('update', data, delta)`, and index.d.ts types the +// listener as `(data, delta) => void`. A single-value emit silently drops the +// delta, which is exactly the contract consumers read from the types. +class Emitter { + #events = new Map(); + #maxListeners = 10; + + constructor(options = {}) { + this.#maxListeners = options.maxListeners ?? 10; + } + + emit(eventName, ...args) { + const event = this.#events.get(eventName); + if (!event) { + if (eventName !== 'error') return Promise.resolve(); + throw new Error('Unhandled error'); + } + const listeners = event.on.slice(); + const promises = listeners.map(async (fn) => fn(...args)); + if (event.once.size > 0) { + const len = event.on.length; + const remaining = new Array(len); + let index = 0; + for (let i = 0; i < len; i++) { + const listener = event.on[i]; + if (!event.once.has(listener)) remaining[index++] = listener; + } + if (index === 0) { + this.#events.delete(eventName); + } else { + remaining.length = index; + this.#events.set(eventName, { on: remaining, once: new Set() }); + } + } + return Promise.all(promises).then(() => undefined); + } + + #addListener(eventName, listener, once) { + let event = this.#events.get(eventName); + if (!event) { + const on = [listener]; + event = { on, once: once ? new Set(on) : new Set() }; + this.#events.set(eventName, event); + } else { + if (event.on.includes(listener)) { + throw new Error('Duplicate listeners detected'); + } + event.on.push(listener); + if (once) event.once.add(listener); + } + if (event.on.length > this.#maxListeners) { + throw new Error( + `MaxListenersExceededWarning: Possible memory leak. ` + `Current maxListeners is ${this.#maxListeners}.`, + ); + } + } + + on(eventName, listener) { + this.#addListener(eventName, listener, false); + } + + once(eventName, listener) { + this.#addListener(eventName, listener, true); + } + + off(eventName, listener) { + if (!listener) return void this.#events.delete(eventName); + const event = this.#events.get(eventName); + if (!event) return; + const index = event.on.indexOf(listener); + if (index > -1) event.on.splice(index, 1); + event.once.delete(listener); + } + + toPromise(eventName) { + return new Promise((resolve) => { + this.once(eventName, resolve); + }); + } + + clear(eventName) { + if (!eventName) return void this.#events.clear(); + this.#events.delete(eventName); + } + + listeners(eventName) { + if (!eventName) throw new Error('Expected eventName'); + const event = this.#events.get(eventName); + return event ? event.on : []; + } + + listenerCount(eventName) { + if (!eventName) throw new Error('Expected eventName'); + const event = this.#events.get(eventName); + return event ? event.on.length : 0; + } + + eventNames() { + return Array.from(this.#events.keys()); + } +} + +const generateUUID = crypto.randomUUID; + +const between = (s, prefix, suffix) => { + let i = s.indexOf(prefix); + if (i === -1) return ''; + s = s.substring(i + prefix.length); + if (suffix) { + i = s.indexOf(suffix); + if (i === -1) return ''; + s = s.substring(0, i); + } + return s; +}; + +const fileExists = async (path) => { + const stats = await fsp.stat(path).catch(() => null); + if (!stats) return false; + return !stats.isDirectory(); +}; + +const directoryExists = async (path) => { + const stats = await fsp.stat(path).catch(() => null); + if (!stats) return false; + return stats.isDirectory(); +}; + +module.exports = { + Emitter, + generateUUID, + between, + fileExists, + directoryExists, +}; diff --git a/tests/browser.test-d.ts b/tests/browser.test-d.ts new file mode 100644 index 0000000..bf2ad85 --- /dev/null +++ b/tests/browser.test-d.ts @@ -0,0 +1,20 @@ +import { expectType } from 'tsd'; +import * as browser from '../browser.js'; +import type { Blockchain, CrdtDelta, KeyPair, Storage, SyncManager } from '../browser.js'; + +// The whole point of the split: Web Crypto is async where node:crypto is not. +expectType>(browser.calculateHash({ a: 1 })); +expectType>(browser.encrypt({ a: 1 }, 'pub')); +expectType>(browser.decrypt('cipher', 'priv')); + +// Everything else keeps the node signatures. +expectType>(browser.generateKeys()); +expectType>(browser.loadKeys('keys')); +expectType(browser.diff({ a: 1 }, { a: 2 })); +expectType(browser.merge({ a: 1 }, { a: 2 })); +expectType>(new browser.Blockchain('base')); +expectType>(browser.open({ path: 'data' })); +expectType>(browser.read('dir', 'file.json')); + +declare const storage: Storage; +expectType(storage.sync); diff --git a/tests/chain.js b/tests/chain.test.js similarity index 84% rename from tests/chain.js rename to tests/chain.test.js index dda10b6..e5218d1 100644 --- a/tests/chain.js +++ b/tests/chain.test.js @@ -3,15 +3,15 @@ const test = require('node:test'); const assert = require('node:assert'); const path = require('node:path'); -const { fileExists } = require('metautil'); -const globalStorage = require('..'); +const { fileExists } = require('../src/utils.js'); +const syncom = require('..'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); test('Chain module', async (t) => { await t.test('calculateHash function', () => { const data = { test: 'data' }; - const hash1 = globalStorage.calculateHash(data); - const hash2 = globalStorage.calculateHash(data); + const hash1 = syncom.calculateHash(data); + const hash2 = syncom.calculateHash(data); assert.strictEqual(typeof hash1, 'string'); assert.strictEqual(hash1.length, 64); @@ -21,7 +21,7 @@ test('Chain module', async (t) => { await t.test('Blockchain constructor and initialization', async () => { const tempDir = await createTemp(); try { - const blockchain = await new globalStorage.Blockchain(tempDir); + const blockchain = await new syncom.Blockchain(tempDir); assert.strictEqual(blockchain.path, tempDir); assert.strictEqual(typeof blockchain.tailHash, 'string'); @@ -36,7 +36,7 @@ test('Chain module', async (t) => { await t.test('Blockchain addBlock and readBlock', async () => { const tempDir = await createTemp(); try { - const blockchain = await new globalStorage.Blockchain(tempDir); + const blockchain = await new syncom.Blockchain(tempDir); const testData = { message: 'Test block data' }; const hash = await blockchain.addBlock(testData); @@ -55,7 +55,7 @@ test('Chain module', async (t) => { await t.test('Blockchain chain validation', async () => { const tempDir = await createTemp(); try { - const blockchain = await new globalStorage.Blockchain(tempDir); + const blockchain = await new syncom.Blockchain(tempDir); await blockchain.addBlock({ data: 'block1' }); await blockchain.addBlock({ data: 'block2' }); @@ -74,7 +74,7 @@ test('Chain module', async (t) => { await t.test('Blockchain writeBlock', async () => { const tempDir = await createTemp(); try { - const blockchain = await new globalStorage.Blockchain(tempDir); + const blockchain = await new syncom.Blockchain(tempDir); const timestamp = Date.now(); const data = { test: 'data' }; const testBlock = { prev: '0', timestamp, data }; diff --git a/tests/contract.js b/tests/contract.test.js similarity index 76% rename from tests/contract.js rename to tests/contract.test.js index af212c5..09175e6 100644 --- a/tests/contract.js +++ b/tests/contract.test.js @@ -3,15 +3,15 @@ const path = require('node:path'); const test = require('node:test'); const assert = require('node:assert'); -const globalStorage = require('..'); -const { SmartContract, DataReader, Blockchain } = globalStorage; +const syncom = require('..'); +const { SmartContract, DataReader, Blockchain } = syncom; const { createTemp, cleanupTemp } = require('./utils/temp.js'); test('Contract module', async (t) => { await t.test('DataReader get reads storage', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const reader = new DataReader(storage); assert.strictEqual(await reader.get('missing'), null); @@ -26,9 +26,9 @@ test('Contract module', async (t) => { await t.test('SmartContract execute method', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const chainPath = path.join(tempDir, 'blockchain'); - const blockchain = await new globalStorage.Blockchain(chainPath); + const blockchain = await new syncom.Blockchain(chainPath); const context = { storage, chain: blockchain }; const testProc = async (reader, args) => ({ @@ -55,9 +55,9 @@ test('Contract module', async (t) => { await t.test('SmartContract execute method with error', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const chainPath = path.join(tempDir, 'blockchain'); - const blockchain = await new globalStorage.Blockchain(chainPath); + const blockchain = await new syncom.Blockchain(chainPath); const context = { storage, chain: blockchain }; const testProc = async () => { @@ -67,10 +67,7 @@ test('Contract module', async (t) => { const contract = new SmartContract('test-contract', testProc, context); const args = { id: 'test-1', input: 42 }; - await assert.rejects( - async () => await contract.execute(args), - /Test error/, - ); + await assert.rejects(async () => await contract.execute(args), /Test error/); const blocks = []; let currentHash = blockchain.tailHash; @@ -81,9 +78,7 @@ test('Contract module', async (t) => { } const errorBlock = blocks.find( - (block) => - block.data.contract === 'test-contract' && - block.data.error === 'Test error', + (block) => block.data.contract === 'test-contract' && block.data.error === 'Test error', ); assert.ok(errorBlock); } finally { @@ -94,20 +89,16 @@ test('Contract module', async (t) => { await t.test('SmartContract save and load static methods', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const chainPath = path.join(tempDir, 'blockchain'); - const blockchain = await new globalStorage.Blockchain(chainPath); + const blockchain = await new syncom.Blockchain(chainPath); const context = { storage, chain: blockchain }; const testProc = async (reader, args) => ({ result: args.input * 2, }); - const hash = await globalStorage.SmartContract.save( - 'test-contract', - blockchain, - testProc, - ); + const hash = await syncom.SmartContract.save('test-contract', blockchain, testProc); assert.strictEqual(typeof hash, 'string'); assert.strictEqual(hash.length, 64); @@ -128,17 +119,14 @@ test('Contract module', async (t) => { await t.test('SmartContract.load rejects non-contract block', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const chainPath = path.join(tempDir, 'extra-chain'); const blockchain = await new Blockchain(chainPath); const context = { storage, chain: blockchain }; const plainHash = await blockchain.addBlock({ kind: 'not-a-contract' }); - await assert.rejects( - () => SmartContract.load(plainHash, context), - /Not a smart contract block/, - ); + await assert.rejects(() => SmartContract.load(plainHash, context), /Not a smart contract block/); } finally { await cleanupTemp(tempDir); } diff --git a/tests/crdt.js b/tests/crdt.test.js similarity index 71% rename from tests/crdt.js rename to tests/crdt.test.js index 2d008b3..2439541 100644 --- a/tests/crdt.js +++ b/tests/crdt.test.js @@ -2,15 +2,15 @@ const test = require('node:test'); const assert = require('node:assert'); -const globalStorage = require('..'); +const syncom = require('..'); test('CRDT module', async (t) => { await t.test('diff function - basic object diff', () => { - assert.deepStrictEqual(globalStorage.diff({}, {}), {}); + assert.deepStrictEqual(syncom.diff({}, {}), {}); const source = { a: 1, b: 2, c: 3 }; const target = { a: 1, b: 20, d: 4 }; - const result = globalStorage.diff(source, target); + const result = syncom.diff(source, target); assert.deepStrictEqual(result, { b: 20, c: undefined, d: 4 }); }); @@ -18,7 +18,7 @@ test('CRDT module', async (t) => { await t.test('diff function - new keys added', () => { const source = { a: 1 }; const target = { a: 1, b: 2, c: 3 }; - const result = globalStorage.diff(source, target); + const result = syncom.diff(source, target); assert.deepStrictEqual(result, { b: 2, c: 3 }); }); @@ -26,29 +26,23 @@ test('CRDT module', async (t) => { await t.test('diff function - keys removed', () => { const source = { a: 1, b: 2, c: 3 }; const target = { a: 1 }; - const result = globalStorage.diff(source, target); + const result = syncom.diff(source, target); assert.deepStrictEqual(result, { b: undefined, c: undefined }); }); await t.test('diff function - arrays and sets', () => { - const arrDiff = globalStorage.diff( - { items: [1, 2, 3] }, - { items: [1, 2, 3, 4, 5] }, - ); + const arrDiff = syncom.diff({ items: [1, 2, 3] }, { items: [1, 2, 3, 4, 5] }); assert.deepStrictEqual(arrDiff, { items: [4, 5] }); - const setDiff = globalStorage.diff( - { tags: new Set(['a', 'b']) }, - { tags: new Set(['a', 'b', 'c']) }, - ); + const setDiff = syncom.diff({ tags: new Set(['a', 'b']) }, { tags: new Set(['a', 'b', 'c']) }); assert.deepStrictEqual(setDiff, { tags: ['c'] }); }); await t.test('diff function - nested objects', () => { const source = { user: { name: 'John', age: 30 } }; const target = { user: { name: 'Jane', age: 30, city: 'NYC' } }; - const result = globalStorage.diff(source, target); + const result = syncom.diff(source, target); assert.deepStrictEqual(result, { user: { name: 'Jane', city: 'NYC' }, @@ -58,7 +52,7 @@ test('CRDT module', async (t) => { await t.test('merge function - basic merge', () => { const source = { a: 1, b: 2 }; const delta = { b: 20, c: 3 }; - const result = globalStorage.merge(source, delta); + const result = syncom.merge(source, delta); assert.deepStrictEqual(result, { a: 1, b: 22, c: 3 }); }); @@ -66,7 +60,7 @@ test('CRDT module', async (t) => { await t.test('merge function - delete keys', () => { const source = { a: 1, b: 2, c: 3 }; const delta = { b: undefined }; - const result = globalStorage.merge(source, delta); + const result = syncom.merge(source, delta); assert.deepStrictEqual(result, { a: 1, c: 3 }); }); @@ -74,7 +68,7 @@ test('CRDT module', async (t) => { await t.test('merge function - arrays/sets merge', () => { const source = { items: [1, 2, 3] }; const delta = { items: [3, 4, 5] }; - const result = globalStorage.merge(source, delta); + const result = syncom.merge(source, delta); assert.deepStrictEqual(result, { items: [1, 2, 3, 4, 5] }); }); @@ -82,7 +76,7 @@ test('CRDT module', async (t) => { await t.test('merge function - counter merge', () => { const source = { count: 10 }; const delta = { count: 5 }; - const result = globalStorage.merge(source, delta); + const result = syncom.merge(source, delta); assert.strictEqual(result.count, 15); }); @@ -90,7 +84,7 @@ test('CRDT module', async (t) => { await t.test('merge function - nested objects', () => { const source = { user: { name: 'John', age: 30 } }; const delta = { user: { age: 31, city: 'NYC' } }; - const result = globalStorage.merge(source, delta); + const result = syncom.merge(source, delta); assert.deepStrictEqual(result, { user: { name: 'John', age: 61, city: 'NYC' }, @@ -99,8 +93,8 @@ test('CRDT module', async (t) => { await t.test('merge function - null/undefined delta', () => { const source = { a: 1, b: 2 }; - const result1 = globalStorage.merge(source, null); - const result2 = globalStorage.merge(source, undefined); + const result1 = syncom.merge(source, null); + const result2 = syncom.merge(source, undefined); assert.deepStrictEqual(result1, source); assert.deepStrictEqual(result2, source); @@ -108,7 +102,7 @@ test('CRDT module', async (t) => { await t.test('merge function - null source', () => { const delta = { a: 1, b: 2 }; - const result = globalStorage.merge(null, delta); + const result = syncom.merge(null, delta); assert.deepStrictEqual(result, delta); }); @@ -116,7 +110,7 @@ test('CRDT module', async (t) => { await t.test('apply function - write record', () => { const source = { a: 1, b: 2 }; const history = [{ type: 'write', data: { c: 3, d: 4 } }]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 1, b: 2 }); }); @@ -124,20 +118,18 @@ test('CRDT module', async (t) => { await t.test('apply function - delta record', () => { const source = { a: 1, b: 2 }; const history = [{ type: 'delta', data: { b: 20, c: 3 } }]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 1, b: 22, c: 3 }); - const fromEmpty = globalStorage.apply({}, [ - { type: 'delta', data: { a: 1, b: 2 } }, - ]); + const fromEmpty = syncom.apply({}, [{ type: 'delta', data: { a: 1, b: 2 } }]); assert.deepStrictEqual(fromEmpty, { a: 1, b: 2 }); }); await t.test('apply function - delete record', () => { const source = { a: 1, b: 2, c: 3 }; const history = [{ type: 'delete', key: 'b' }]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 1, b: undefined, c: 3 }); }); @@ -145,7 +137,7 @@ test('CRDT module', async (t) => { await t.test('apply function - inc record', () => { const source = { count: 10 }; const history = [{ type: 'inc', key: 'count', value: 5 }]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.strictEqual(result.count, 15); }); @@ -153,7 +145,7 @@ test('CRDT module', async (t) => { await t.test('apply function - dec record', () => { const source = { count: 10 }; const history = [{ type: 'dec', key: 'count', value: 3 }]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.strictEqual(result.count, 7); }); @@ -165,7 +157,7 @@ test('CRDT module', async (t) => { { type: 'delta', data: { c: 3 } }, { type: 'inc', key: 'a', value: 5 }, ]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 6, b: 22, c: 3 }); }); @@ -176,16 +168,16 @@ test('CRDT module', async (t) => { { type: 'delta', data: { d: 4 } }, { type: 'write', data: { x: 10 } }, ]; - const result = globalStorage.apply(source, history); + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 1, b: 2, c: 3, d: 4 }); }); await t.test('apply function - invalid history', () => { const source = { a: 1 }; - const result1 = globalStorage.apply(source, null); - const result2 = globalStorage.apply(source, 'not-array'); - const result3 = globalStorage.apply(source, []); + const result1 = syncom.apply(source, null); + const result2 = syncom.apply(source, 'not-array'); + const result3 = syncom.apply(source, []); assert.deepStrictEqual(result1, source); assert.deepStrictEqual(result2, source); @@ -194,13 +186,8 @@ test('CRDT module', async (t) => { await t.test('apply function - invalid records', () => { const source = { a: 1 }; - const history = [ - null, - { type: 'unknown' }, - { type: 'delta' }, - { type: 'delta', data: { b: 2 } }, - ]; - const result = globalStorage.apply(source, history); + const history = [null, { type: 'unknown' }, { type: 'delta' }, { type: 'delta', data: { b: 2 } }]; + const result = syncom.apply(source, history); assert.deepStrictEqual(result, { a: 1, b: 2 }); }); diff --git a/tests/crypto-browser.js b/tests/crypto.browser.test.js similarity index 83% rename from tests/crypto-browser.js rename to tests/crypto.browser.test.js index b94110b..ee80a35 100644 --- a/tests/crypto-browser.js +++ b/tests/crypto.browser.test.js @@ -6,7 +6,7 @@ const { webcrypto } = require('node:crypto'); const { installMockOpfs } = require('./utils/mock-opfs.js'); const { runCryptoKeysTests } = require('./suites/crypto-keys.js'); -test('crypto-browser module', async (t) => { +test('crypto.browser module', async (t) => { const opfs = installMockOpfs(); t.after(() => opfs.uninstall()); @@ -29,10 +29,9 @@ test('crypto-browser module', async (t) => { globalThis.atob = (s) => Buffer.from(s, 'base64').toString('latin1'); } - const cryptoBrowser = require('../lib/crypto-browser.js'); + const cryptoBrowser = require('../src/crypto.browser.js'); await runCryptoKeysTests(t, assert, cryptoBrowser, { - getKeysBase: async () => - `keys-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, + getKeysBase: async () => `keys-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, }); }); diff --git a/tests/fsi-browser.js b/tests/fsi.browser.test.js similarity index 62% rename from tests/fsi-browser.js rename to tests/fsi.browser.test.js index 026a84e..3303fb8 100644 --- a/tests/fsi-browser.js +++ b/tests/fsi.browser.test.js @@ -4,15 +4,14 @@ const test = require('node:test'); const { installMockOpfs } = require('./utils/mock-opfs.js'); const { runFsiTests } = require('./suites/fsi.js'); -test('fsi-browser module (OPFS mock)', async (t) => { +test('fsi.browser module (OPFS mock)', async (t) => { const opfs = installMockOpfs(); t.after(() => opfs.uninstall()); - const fsiBrowser = require('../lib/fsi-browser.js'); + const fsiBrowser = require('../src/fsi.browser.js'); await runFsiTests(t, fsiBrowser, { - getRoot: async () => - `vfs-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, + getRoot: async () => `vfs-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, joinPath: (root, ...parts) => [root, ...parts].join('/'), }); }); diff --git a/tests/fsi.js b/tests/fsi.test.js similarity index 88% rename from tests/fsi.js rename to tests/fsi.test.js index cdc07ae..7235b9d 100644 --- a/tests/fsi.js +++ b/tests/fsi.test.js @@ -1,7 +1,7 @@ 'use strict'; const test = require('node:test'); -const fsi = require('../lib/fsi.js'); +const fsi = require('../src/fsi.js'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); const { runFsiTests } = require('./suites/fsi.js'); diff --git a/tests/index.browser.test.js b/tests/index.browser.test.js new file mode 100644 index 0000000..8a78c21 --- /dev/null +++ b/tests/index.browser.test.js @@ -0,0 +1,64 @@ +'use strict'; + +// The browser barrel is what `browser.js` (and therefore every bundler that +// honours the `browser` condition) resolves to. Loading it under the OPFS + +// WebCrypto mocks proves the browser halves are wired together at all โ€” the +// node barrel passing says nothing about this one. + +const test = require('node:test'); +const assert = require('node:assert'); +const { webcrypto } = require('node:crypto'); +const { installMockOpfs } = require('./utils/mock-opfs.js'); + +test('browser barrel', async (t) => { + const opfs = installMockOpfs(); + t.after(() => opfs.uninstall()); + + const prevCrypto = Object.getOwnPropertyDescriptor(globalThis, 'crypto'); + Object.defineProperty(globalThis, 'crypto', { + value: webcrypto, + configurable: true, + writable: true, + enumerable: true, + }); + t.after(() => { + if (prevCrypto) Object.defineProperty(globalThis, 'crypto', prevCrypto); + else delete globalThis.crypto; + }); + + if (typeof globalThis.btoa !== 'function') { + globalThis.btoa = (s) => Buffer.from(s, 'latin1').toString('base64'); + } + if (typeof globalThis.atob !== 'function') { + globalThis.atob = (s) => Buffer.from(s, 'base64').toString('latin1'); + } + + const browser = require('../src/index.browser.js'); + const node = require('../src/index.js'); + + await t.test('exports the same names as the node barrel', () => { + assert.deepStrictEqual(Object.keys(browser).sort(), Object.keys(node).sort()); + }); + + await t.test('picks the browser half of every split module', () => { + // The node calculateHash is synchronous; the WebCrypto one returns a + // promise. That difference is the cheapest proof the swap happened. + const hash = browser.calculateHash({ a: 1 }); + assert.ok(hash instanceof Promise); + return hash.then((value) => assert.match(value, /^[0-9a-f]{64}$/)); + }); + + await t.test('crdt and contract halves are shared, not duplicated', () => { + assert.strictEqual(browser.diff, node.diff); + assert.strictEqual(browser.merge, node.merge); + assert.strictEqual(browser.apply, node.apply); + assert.strictEqual(browser.SmartContract, node.SmartContract); + }); + + await t.test('SyncManager is the in-memory browser one', async () => { + const sync = await new browser.SyncManager(); + assert.notStrictEqual(browser.SyncManager, node.SyncManager); + await sync.addNode({ url: 'ws://peer.example.com:9000', type: 'uplink' }); + assert.strictEqual(sync.listNodes().length, 1); + }); +}); diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts new file mode 100644 index 0000000..196a99b --- /dev/null +++ b/tests/index.test-d.ts @@ -0,0 +1,80 @@ +import { expectAssignable, expectError, expectType } from 'tsd'; +import * as syncom from '../index.js'; +import type { + Block, + Blockchain, + CrdtDelta, + KeyPair, + Node, + Schema, + Storage, + StorageOptions, + SyncManager, + SyncomCollection, + SyncomRecord, +} from '../index.js'; + +// crypto โ€” node:crypto is synchronous, unlike the browser half. +expectType(syncom.calculateHash({ a: 1 })); +expectType>(syncom.generateKeys()); +expectType(syncom.encrypt({ a: 1 }, 'pub')); +expectType(syncom.decrypt('cipher', 'priv')); +expectType>(syncom.loadKeys('keys')); +expectError(syncom.calculateHash(() => 1)); + +// fsi +expectType>(syncom.read('dir', 'file.json')); +expectType>(syncom.write('dir', 'file.json', '{}')); +expectType>(syncom.exists('dir', 'file.json')); +expectType>(syncom.remove('dir', 'file.json')); + +// crdt +expectType(syncom.diff({ a: 1 }, { a: 2 })); +expectType(syncom.merge({ a: 1 }, { a: 2 })); +expectType(syncom.apply({}, [{ type: 'inc', key: 'hits', value: 1 }])); +expectError(syncom.apply({}, [{ type: 'nope' }])); + +// chain โ€” the constructor returns a promise, the class is async-initialised. +expectType>(new syncom.Blockchain('base')); +declare const chain: Blockchain; +expectType>(chain.addBlock({ id: 'r1', hash: 'h' })); +expectType>(chain.readBlock('hash')); +expectType>(chain.validate({ last: 10 })); +expectType>(chain.validate()); + +// contracts +declare const storage: Storage; +const reader = new syncom.DataReader(storage); +expectType>(reader.get('id')); +expectType>(syncom.SmartContract.save('name', chain, async () => 1)); +expectType>(syncom.SmartContract.load('hash', { storage, chain })); + +// storage +expectType>(syncom.open()); +expectType>(syncom.open({ path: 'data' })); +expectAssignable({ path: 'data', schema: {} as Schema }); +expectError(syncom.open({ path: 1 })); +expectType>(storage.insert({ a: 1 })); +expectType>(storage.get('id')); +expectType>(storage.has('id')); +expectType>(storage.swap('id', { a: 2 }, { a: 1 })); +expectType>(storage.record('id')); +expectType(storage.hasRecord('id')); + +// The record/collection pair reached through the Storage namespace. +expectAssignable(syncom.Storage.Collection); +declare const record: SyncomRecord; +expectType(record.id); +expectType>(record.data()); +expectType>(record.delta()); +record.on('update', (data, delta) => { + expectType(data); + expectType(delta); +}); + +// sync +expectType(storage.sync); +declare const sync: SyncManager; +expectType(sync.listNodes()); +expectType>(sync.addNode({ url: 'ws://a', type: 'uplink' })); +expectError(sync.addNode({ url: 'ws://a', type: 'sideways' })); diff --git a/tests/integration.js b/tests/integration.js deleted file mode 100644 index d0c06dd..0000000 --- a/tests/integration.js +++ /dev/null @@ -1,51 +0,0 @@ -'use strict'; - -const path = require('node:path'); -const test = require('node:test'); -const assert = require('node:assert'); -const globalStorage = require('..'); -const { createTemp, cleanupTemp } = require('./utils/temp.js'); - -test('Integration tests', async (t) => { - await t.test( - 'Full workflow: keys -> storage -> contract -> chain', - async () => { - const tempDir = await createTemp(); - try { - const storage = await globalStorage.open({ path: tempDir }); - const chainPath = path.join(tempDir, 'blockchain'); - const blockchain = await new globalStorage.Blockchain(chainPath); - - const contractProc = async (reader, args) => { - const data = await reader.get(args.id); - const existingData = data || { count: 0 }; - return { - ...existingData, - count: existingData.count + 1, - lastUpdate: Date.now(), - }; - }; - - const contract = new globalStorage.SmartContract( - 'counter-contract', - contractProc, - { storage, chain: blockchain }, - ); - - const result1 = await contract.execute({ id: 'counter-1' }); - assert.strictEqual(result1.count, 1); - - const result2 = await contract.execute({ id: 'counter-1' }); - assert.strictEqual(result2.count, 2); - - const loadedData = await storage.loadData('counter-1'); - assert.strictEqual(loadedData.count, 2); - - const isValid = await blockchain.validate(); - assert.strictEqual(isValid, true); - } finally { - await cleanupTemp(tempDir); - } - }, - ); -}); diff --git a/tests/integration.test.js b/tests/integration.test.js new file mode 100644 index 0000000..495c955 --- /dev/null +++ b/tests/integration.test.js @@ -0,0 +1,44 @@ +'use strict'; + +const path = require('node:path'); +const test = require('node:test'); +const assert = require('node:assert'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +test('Integration tests', async (t) => { + await t.test('Full workflow: keys -> storage -> contract -> chain', async () => { + const tempDir = await createTemp(); + try { + const storage = await syncom.open({ path: tempDir }); + const chainPath = path.join(tempDir, 'blockchain'); + const blockchain = await new syncom.Blockchain(chainPath); + + const contractProc = async (reader, args) => { + const data = await reader.get(args.id); + const existingData = data || { count: 0 }; + return { + ...existingData, + count: existingData.count + 1, + lastUpdate: Date.now(), + }; + }; + + const contract = new syncom.SmartContract('counter-contract', contractProc, { storage, chain: blockchain }); + + const result1 = await contract.execute({ id: 'counter-1' }); + assert.strictEqual(result1.count, 1); + + const result2 = await contract.execute({ id: 'counter-1' }); + assert.strictEqual(result2.count, 2); + + const loadedData = await storage.loadData('counter-1'); + assert.strictEqual(loadedData.count, 2); + + const isValid = await blockchain.validate(); + assert.strictEqual(isValid, true); + } finally { + await cleanupTemp(tempDir); + } + }); +}); diff --git a/tests/keys.js b/tests/keys.test.js similarity index 90% rename from tests/keys.js rename to tests/keys.test.js index 30cc562..71e1cf8 100644 --- a/tests/keys.js +++ b/tests/keys.test.js @@ -2,7 +2,7 @@ const test = require('node:test'); const assert = require('node:assert'); -const cryptoMod = require('../lib/crypto.js'); +const cryptoMod = require('../src/crypto.js'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); const { runCryptoKeysTests } = require('./suites/crypto-keys.js'); diff --git a/tests/package/consistency.test.js b/tests/package/consistency.test.js new file mode 100644 index 0000000..d74fb5b --- /dev/null +++ b/tests/package/consistency.test.js @@ -0,0 +1,138 @@ +'use strict'; + +// The machine-checked version of the release checklist. `files` is an explicit +// allowlist, so a new root shim or .d.ts that was not added to it silently +// disappears from the tarball; an exports entry whose file does not exist +// fails only at a consumer's first import. This suite makes every one of +// those a red test instead. + +const path = require('node:path'); +const { existsSync, readFileSync } = require('node:fs'); +const { test } = require('node:test'); +const assert = require('node:assert'); + +const ROOT = path.join(__dirname, '..', '..'); +const pkg = require('../../package.json'); + +// Every file an exports condition can resolve to, flattened. +const exportTargets = (value, out = []) => { + if (typeof value === 'string') out.push(value); + else if (value && typeof value === 'object') { + for (const nested of Object.values(value)) exportTargets(nested, out); + } + return out; +}; + +const shipped = (relative) => { + // The allowlist ships whole directories (src/) and named root files. + const clean = relative.replace(/^\.\//, ''); + return pkg.files.some((entry) => (entry.endsWith('/') ? clean.startsWith(entry) : clean === entry)); +}; + +test('every exports target exists and is in the files allowlist', () => { + for (const [key, value] of Object.entries(pkg.exports)) { + for (const target of exportTargets(value)) { + assert.ok(existsSync(path.join(ROOT, target)), `${key} -> ${target} does not exist`); + assert.ok(shipped(target), `${key} -> ${target} is not in the files allowlist`); + } + } +}); + +test('every subpath has a types condition wherever it has a runtime one', () => { + for (const [key, value] of Object.entries(pkg.exports)) { + if (key === './package.json') continue; + const check = (conditions, label) => { + assert.ok(typeof conditions === 'object', `${label} must be a conditions object`); + assert.ok(conditions.types, `${label} has no types condition`); + assert.ok(conditions.default, `${label} has no default condition`); + // exports conditions match top-down: a `types` AFTER `default` is dead. + const keys = Object.keys(conditions); + assert.ok(keys.indexOf('types') < keys.indexOf('default'), `${label}: types must come before default`); + if (conditions.browser && typeof conditions.browser === 'object') check(conditions.browser, `${label}.browser`); + }; + check(value, key); + } +}); + +test('every root shim requires a file that exists', () => { + for (const [key, value] of Object.entries(pkg.exports)) { + for (const target of exportTargets(value)) { + if (!target.endsWith('.js')) continue; + const source = readFileSync(path.join(ROOT, target), 'utf8'); + const match = source.match(/require\('([^']+)'\)/); + if (!match) continue; // not a shim + const required = path.join(ROOT, path.dirname(target), match[1]); + assert.ok(existsSync(required), `${key} shim ${target} requires missing ${match[1]}`); + } + } +}); + +test('every published entry point has a tsd file', () => { + for (const [key, value] of Object.entries(pkg.exports)) { + if (key === './package.json') continue; + for (const target of exportTargets(value)) { + if (!target.endsWith('.d.ts')) continue; + const name = path.basename(target, '.d.ts'); + const testFile = path.join(ROOT, 'tests', `${name}.test-d.ts`); + assert.ok(existsSync(testFile), `${target} has no tests/${name}.test-d.ts`); + } + } +}); + +test('the browser field maps to files that ship', () => { + for (const [from, to] of Object.entries(pkg.browser)) { + for (const target of [from, to]) { + assert.ok(existsSync(path.join(ROOT, target)), `browser field references missing ${target}`); + assert.ok(shipped(target), `browser field references unshipped ${target}`); + } + } +}); + +test('every files entry exists', () => { + for (const entry of pkg.files) { + assert.ok(existsSync(path.join(ROOT, entry)), `files entry ${entry} does not exist`); + } +}); + +test('the docs keywords list mirrors package.json', () => { + const config = readFileSync(path.join(ROOT, 'docs', '.vitepress', 'config.mts'), 'utf8'); + const block = config.match(/const keywords = \[([\s\S]*?)\]\.join/); + assert.ok(block, 'docs config has no `const keywords = [...]` block'); + const listed = [...block[1].matchAll(/'([^']+)'/g)].map(([, term]) => term); + assert.deepStrictEqual(listed, pkg.keywords, 'docs keywords have drifted from package.json#keywords'); +}); + +test('the docs nav version label matches package.json', () => { + const config = readFileSync(path.join(ROOT, 'docs', '.vitepress', 'config.mts'), 'utf8'); + assert.ok(config.includes(`'v${pkg.version}'`), `docs nav does not carry v${pkg.version}`); +}); + +test('every exports entry point has a row in scripts/size.js', () => { + const source = readFileSync(path.join(ROOT, 'scripts', 'size.js'), 'utf8'); + const measured = new Set([...source.matchAll(/entry: '([^']+)'/g)].map(([, entry]) => entry)); + for (const [key, value] of Object.entries(pkg.exports)) { + if (key === './package.json') continue; + for (const target of exportTargets(value)) { + if (!target.endsWith('.js')) continue; + const clean = target.replace(/^\.\//, ''); + assert.ok(measured.has(clean), `${clean} has no ENTRIES row in scripts/size.js`); + } + } +}); + +test('every runtime export is declared in the types', () => { + const check = (entry, types) => { + const runtime = Object.keys(require(path.join(ROOT, entry))); + const source = readFileSync(path.join(ROOT, types), 'utf8'); + for (const name of runtime) { + const declared = + new RegExp(`export (?:declare )?(?:function|const|class) ${name}\\b`).test(source) || + new RegExp(`^\\s*${name},?$`, 'm').test(source); + assert.ok(declared, `${entry} exports ${name}, but ${types} does not declare it`); + } + }; + check('index.js', 'index.d.ts'); + // browser.js is only loadable under OPFS/WebCrypto globals, so compare its + // barrel's names against the node one instead of requiring it here. + check('index.js', 'browser.d.ts'); +}); diff --git a/tests/storage.js b/tests/storage.test.js similarity index 83% rename from tests/storage.js rename to tests/storage.test.js index 1c36222..a16adba 100644 --- a/tests/storage.js +++ b/tests/storage.test.js @@ -4,8 +4,8 @@ const test = require('node:test'); const assert = require('node:assert'); const fs = require('node:fs/promises'); const path = require('node:path'); -const { directoryExists } = require('metautil'); -const globalStorage = require('..'); +const { directoryExists } = require('../src/utils.js'); +const syncom = require('..'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); const mockSchema = { @@ -20,9 +20,9 @@ test('Storage module', async (t) => { await t.test('Storage constructor and initialization', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); - assert.ok(storage instanceof globalStorage.Storage); + assert.ok(storage instanceof syncom.Storage); assert.strictEqual(await directoryExists(tempDir), true); } finally { await cleanupTemp(tempDir); @@ -32,7 +32,7 @@ test('Storage module', async (t) => { await t.test('Storage saveData and loadData without encryption', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const testData = { message: 'Hello, Storage!', number: 123 }; await storage.saveData('test-1', testData); @@ -47,7 +47,7 @@ test('Storage module', async (t) => { await t.test('Storage saveData and loadData with encryption', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const testData = { secret: 'encrypted data', value: 456 }; await storage.saveData('test-2', testData, { encrypted: true }); @@ -62,7 +62,7 @@ test('Storage module', async (t) => { await t.test('Storage loadData with non-existent data', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const result = await storage.loadData('nonexistent'); assert.strictEqual(result, null); @@ -74,7 +74,7 @@ test('Storage module', async (t) => { await t.test('Storage validate method', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const testData = { message: 'Test validation' }; await storage.saveData('test-3', testData); @@ -86,14 +86,12 @@ test('Storage module', async (t) => { const isValid = await storage.validate('test-3', entry.data, entry.block); assert.strictEqual(isValid, true); - const isInvalid = await storage.validate( - 'test-3', - { modified: 'data' }, - entry.block, - ); + const isInvalid = await storage.validate('test-3', { modified: 'data' }, entry.block); assert.strictEqual(isInvalid, false); const chainOk = await storage.validate({ last: 2 }); + assert.strictEqual(await storage.validate(), true); + assert.strictEqual(await storage.validate({}), true); assert.strictEqual(chainOk, true); } finally { await cleanupTemp(tempDir); @@ -103,7 +101,7 @@ test('Storage module', async (t) => { await t.test('Storage loadData throws if tampered', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); await storage.saveData('tampered', { ok: true }); const filePath = path.join(tempDir, 'data', 'tampered.json'); @@ -111,7 +109,7 @@ test('Storage module', async (t) => { entry.data = { ok: false }; await fs.writeFile(filePath, JSON.stringify(entry), 'utf8'); - const storage2 = await globalStorage.open({ path: tempDir }); + const storage2 = await syncom.open({ path: tempDir }); await assert.rejects(() => storage2.loadData('tampered'), /invalid/); } finally { await cleanupTemp(tempDir); @@ -121,7 +119,7 @@ test('Storage module', async (t) => { await t.test('Storage insert set swap and update errors', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); const id = await storage.insert({ n: 1 }); assert.strictEqual(typeof id, 'string'); @@ -137,10 +135,7 @@ test('Storage module', async (t) => { const notSwapped = await storage.swap(id, { v: 9 }, { n: 999 }); assert.strictEqual(notSwapped, false); - await assert.rejects( - () => storage.update('missing-id', { x: 1 }), - /not found/, - ); + await assert.rejects(() => storage.update('missing-id', { x: 1 }), /not found/); } finally { await cleanupTemp(tempDir); } @@ -150,12 +145,12 @@ test('Storage module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); assert.strictEqual(storage.schema, mockSchema); - assert.ok(storage.Post instanceof globalStorage.Storage.Collection); - assert.ok(storage.Author instanceof globalStorage.Storage.Collection); - assert.ok(storage.Comment instanceof globalStorage.Storage.Collection); + assert.ok(storage.Post instanceof syncom.Storage.Collection); + assert.ok(storage.Author instanceof syncom.Storage.Collection); + assert.ok(storage.Comment instanceof syncom.Storage.Collection); assert.strictEqual(storage.Post.name, 'Post'); assert.strictEqual(storage.Author.name, 'Author'); } finally { @@ -173,10 +168,10 @@ test('Storage module', async (t) => { ]), }; const options = { path: tempDir, schema: mapSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); - assert.ok(storage.User instanceof globalStorage.Storage.Collection); - assert.ok(storage.Article instanceof globalStorage.Storage.Collection); + assert.ok(storage.User instanceof syncom.Storage.Collection); + assert.ok(storage.Article instanceof syncom.Storage.Collection); assert.strictEqual(storage.User.name, 'User'); assert.strictEqual(storage.Article.name, 'Article'); @@ -192,7 +187,7 @@ test('Storage module', async (t) => { await t.test('Storage without schema has no collections', async () => { const tempDir = await createTemp(); try { - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); assert.strictEqual(storage.schema, null); assert.strictEqual(storage.Post, undefined); @@ -208,12 +203,12 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const authorData = { name: 'John', email: 'john@example.com' }; const authorRecord = await storage.Author.insert(authorData); - assert.ok(authorRecord instanceof globalStorage.Storage.Record); + assert.ok(authorRecord instanceof syncom.Storage.Record); assert.ok(authorRecord.id); assert.strictEqual(authorRecord.name, 'John'); assert.strictEqual(authorRecord.email, 'john@example.com'); @@ -226,7 +221,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const postData = { title: 'Hello', content: 'World' }; const postRecord = await storage.Post.insert(postData); @@ -244,7 +239,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const authorRecord = await storage.Author.insert({ name: 'Jane', @@ -264,7 +259,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const postRecord = await storage.Post.insert({ title: 'Original', @@ -285,7 +280,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const commentRecord = await storage.Comment.insert({ text: 'Nice post!', @@ -308,7 +303,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const authorRecord = await storage.Author.insert({ name: 'Bob', @@ -317,7 +312,7 @@ test('Collection module', async (t) => { const id = authorRecord.id; const record = await storage.Author.record(id); - assert.ok(record instanceof globalStorage.Storage.Record); + assert.ok(record instanceof syncom.Storage.Record); assert.strictEqual(record.id, id); assert.strictEqual(record.name, 'Bob'); } finally { @@ -329,7 +324,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const author = await storage.Author.insert({ name: 'Alice', @@ -366,7 +361,7 @@ test('Collection module', async (t) => { const tempDir = await createTemp(); try { const options = { path: tempDir, schema: mockSchema }; - const storage = await globalStorage.open(options); + const storage = await syncom.open(options); const post = await storage.Post.insert({ title: 'Draft', diff --git a/tests/suites/crypto-keys.js b/tests/suites/crypto-keys.js index a452723..d543730 100644 --- a/tests/suites/crypto-keys.js +++ b/tests/suites/crypto-keys.js @@ -1,11 +1,6 @@ 'use strict'; -const runCryptoKeysTests = async ( - t, - assert, - cryptoMod, - { getKeysBase } = {}, -) => { +const runCryptoKeysTests = async (t, assert, cryptoMod, { getKeysBase } = {}) => { if (typeof getKeysBase !== 'function') { throw new TypeError('runCryptoKeysTests requires getKeysBase'); } @@ -22,15 +17,11 @@ const runCryptoKeysTests = async ( const keys = await Promise.resolve(cryptoMod.generateKeys()); const testData = { message: 'Hello, World!', number: 42 }; - const encrypted = await Promise.resolve( - cryptoMod.encrypt(testData, keys.publicKey), - ); + const encrypted = await Promise.resolve(cryptoMod.encrypt(testData, keys.publicKey)); assert.strictEqual(typeof encrypted, 'string'); assert.notStrictEqual(encrypted, JSON.stringify(testData)); - const decrypted = await Promise.resolve( - cryptoMod.decrypt(encrypted, keys.privateKey), - ); + const decrypted = await Promise.resolve(cryptoMod.decrypt(encrypted, keys.privateKey)); assert.deepStrictEqual(decrypted, testData); }); diff --git a/tests/suites/emitter.js b/tests/suites/emitter.js new file mode 100644 index 0000000..62dbf23 --- /dev/null +++ b/tests/suites/emitter.js @@ -0,0 +1,112 @@ +'use strict'; + +// Shared by tests/utils.test.js and tests/utils.browser.test.js โ€” the Emitter +// is byte-identical in both halves, so the contract is asserted once. +const runEmitterTests = async (t, assert, Emitter) => { + await t.test('emit forwards every argument to every listener', async () => { + const emitter = new Emitter(); + const seen = []; + emitter.on('update', (data, delta) => void seen.push([data, delta])); + emitter.on('update', (data, delta) => void seen.push([delta, data])); + await emitter.emit('update', { a: 1 }, { b: 2 }); + assert.deepStrictEqual(seen, [ + [{ a: 1 }, { b: 2 }], + [{ b: 2 }, { a: 1 }], + ]); + }); + + await t.test('emit awaits async listeners', async () => { + const emitter = new Emitter(); + let done = false; + emitter.on('x', async () => { + await new Promise((resolve) => setTimeout(resolve, 5)); + done = true; + }); + await emitter.emit('x'); + assert.strictEqual(done, true); + }); + + await t.test('emit on an unknown event resolves', async () => { + const emitter = new Emitter(); + assert.strictEqual(await emitter.emit('nobody'), undefined); + }); + + await t.test('emit of unhandled error throws', () => { + const emitter = new Emitter(); + assert.throws(() => emitter.emit('error', new Error('boom')), /Unhandled error/); + }); + + await t.test('once fires a single time and is swept', async () => { + const emitter = new Emitter(); + let calls = 0; + emitter.once('tick', () => void calls++); + await emitter.emit('tick'); + await emitter.emit('tick'); + assert.strictEqual(calls, 1); + assert.deepStrictEqual(emitter.eventNames(), []); + }); + + await t.test('once alongside on keeps the persistent listener', async () => { + const emitter = new Emitter(); + const calls = []; + emitter.on('tick', () => void calls.push('on')); + emitter.once('tick', () => void calls.push('once')); + await emitter.emit('tick'); + await emitter.emit('tick'); + assert.deepStrictEqual(calls, ['on', 'once', 'on']); + assert.strictEqual(emitter.listenerCount('tick'), 1); + }); + + await t.test('off removes one listener, or the whole event', async () => { + const emitter = new Emitter(); + const listener = () => {}; + emitter.on('a', listener); + emitter.off('a', listener); + assert.strictEqual(emitter.listenerCount('a'), 0); + emitter.off('a', () => {}); + emitter.on('b', listener); + emitter.off('b'); + assert.deepStrictEqual(emitter.listeners('b'), []); + emitter.off('missing'); + }); + + await t.test('clear drops one event or all of them', () => { + const emitter = new Emitter(); + emitter.on('a', () => {}); + emitter.on('b', () => {}); + emitter.clear('a'); + assert.deepStrictEqual(emitter.eventNames(), ['b']); + emitter.clear(); + assert.deepStrictEqual(emitter.eventNames(), []); + }); + + await t.test('duplicate listeners are rejected', () => { + const emitter = new Emitter(); + const listener = () => {}; + emitter.on('a', listener); + assert.throws(() => emitter.on('a', listener), /Duplicate listeners/); + }); + + await t.test('maxListeners is enforced', () => { + const emitter = new Emitter({ maxListeners: 2 }); + emitter.on('a', () => {}); + emitter.on('a', () => {}); + assert.throws(() => emitter.on('a', () => {}), /MaxListenersExceededWarning/); + }); + + await t.test('listeners and listenerCount require an event name', () => { + const emitter = new Emitter(); + assert.throws(() => emitter.listeners(), /Expected eventName/); + assert.throws(() => emitter.listenerCount(), /Expected eventName/); + assert.strictEqual(emitter.listenerCount('nothing'), 0); + }); + + await t.test('toPromise resolves on the next emit', async () => { + const emitter = new Emitter(); + const promise = emitter.toPromise('ready'); + await emitter.emit('ready', 42); + assert.strictEqual(await promise, 42); + }); +}; + +module.exports = { runEmitterTests }; diff --git a/tests/suites/sync-manager.js b/tests/suites/sync-manager.js index be660ab..40705ad 100644 --- a/tests/suites/sync-manager.js +++ b/tests/suites/sync-manager.js @@ -22,10 +22,7 @@ const runSyncManagerCoreTests = async (t, assert, getSync) => { await t.test('addNode rejects bad input', async (t2) => { const sync = await getSync(t2); - await assert.rejects( - () => sync.addNode({ type: 'uplink' }), - /URL is required/, - ); + await assert.rejects(() => sync.addNode({ type: 'uplink' }), /URL is required/); await assert.rejects( () => @@ -53,10 +50,7 @@ const runSyncManagerCoreTests = async (t, assert, getSync) => { await t.test('removeNode throws when unknown', async (t2) => { const sync = await getSync(t2); - await assert.rejects( - () => sync.removeNode('http://nope.example.com:1'), - /not found/, - ); + await assert.rejects(() => sync.removeNode('http://nope.example.com:1'), /not found/); }); }; diff --git a/tests/sync-browser.js b/tests/sync.browser.test.js similarity index 82% rename from tests/sync-browser.js rename to tests/sync.browser.test.js index f66b2cc..a2a1ede 100644 --- a/tests/sync-browser.js +++ b/tests/sync.browser.test.js @@ -2,7 +2,7 @@ const test = require('node:test'); const assert = require('node:assert'); -const { SyncManager } = require('../lib/sync-browser.js'); +const { SyncManager } = require('../src/sync.browser.js'); const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); test('SyncManager (browser)', async (t) => { diff --git a/tests/sync.js b/tests/sync.test.js similarity index 81% rename from tests/sync.js rename to tests/sync.test.js index 8728276..9432b31 100644 --- a/tests/sync.js +++ b/tests/sync.test.js @@ -2,7 +2,7 @@ const test = require('node:test'); const assert = require('node:assert'); -const globalStorage = require('..'); +const syncom = require('..'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); @@ -10,7 +10,7 @@ test('SyncManager', async (t) => { await runSyncManagerCoreTests(t, assert, async (t2) => { const tempDir = await createTemp(); t2.after(() => cleanupTemp(tempDir)); - const storage = await globalStorage.open({ path: tempDir }); + const storage = await syncom.open({ path: tempDir }); return storage.sync; }); }); diff --git a/tests/utils.browser.test.js b/tests/utils.browser.test.js new file mode 100644 index 0000000..f0ff3ae --- /dev/null +++ b/tests/utils.browser.test.js @@ -0,0 +1,43 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { webcrypto } = require('node:crypto'); +const { runEmitterTests } = require('./suites/emitter.js'); + +test('utils.browser module', async (t) => { + const prevCrypto = Object.getOwnPropertyDescriptor(globalThis, 'crypto'); + Object.defineProperty(globalThis, 'crypto', { + value: webcrypto, + configurable: true, + writable: true, + enumerable: true, + }); + t.after(() => { + if (prevCrypto) Object.defineProperty(globalThis, 'crypto', prevCrypto); + else delete globalThis.crypto; + }); + + const utils = require('../src/utils.browser.js'); + + await t.test('Emitter matches the node half', async (t2) => { + await runEmitterTests(t2, assert, utils.Emitter); + }); + + await t.test('generateUUID delegates to the Web Crypto API', () => { + const id = utils.generateUUID(); + assert.match(id, /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/); + assert.notStrictEqual(id, utils.generateUUID()); + }); + + await t.test('between matches the node half', () => { + assert.strictEqual(utils.between('ws://peer.example.com:9000', '://', ':'), 'peer.example.com'); + assert.strictEqual(utils.between('abc', 'z'), ''); + assert.strictEqual(utils.between('abc', 'a', 'z'), ''); + }); + + await t.test('no filesystem predicates are exported', () => { + assert.strictEqual(utils.fileExists, undefined); + assert.strictEqual(utils.directoryExists, undefined); + }); +}); diff --git a/tests/utils.test.js b/tests/utils.test.js new file mode 100644 index 0000000..9cc3c91 --- /dev/null +++ b/tests/utils.test.js @@ -0,0 +1,41 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const path = require('node:path'); +const fs = require('node:fs/promises'); +const { Emitter, generateUUID, between, fileExists, directoryExists } = require('../src/utils.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runEmitterTests } = require('./suites/emitter.js'); + +test('Emitter', async (t) => { + await runEmitterTests(t, assert, Emitter); +}); + +test('generateUUID', () => { + const id = generateUUID(); + assert.match(id, /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/); + assert.notStrictEqual(id, generateUUID()); +}); + +test('between', () => { + assert.strictEqual(between('ws://peer.example.com:9000', '://', ':'), 'peer.example.com'); + assert.strictEqual(between('abc', 'a'), 'bc'); + assert.strictEqual(between('abc', 'z'), ''); + assert.strictEqual(between('abc', 'a', 'z'), ''); +}); + +test('fileExists and directoryExists', async (t) => { + const tempDir = await createTemp(); + t.after(() => cleanupTemp(tempDir)); + const file = path.join(tempDir, 'file.txt'); + await fs.writeFile(file, 'content'); + + assert.strictEqual(await fileExists(file), true); + assert.strictEqual(await fileExists(tempDir), false); + assert.strictEqual(await fileExists(path.join(tempDir, 'missing')), false); + + assert.strictEqual(await directoryExists(tempDir), true); + assert.strictEqual(await directoryExists(file), false); + assert.strictEqual(await directoryExists(path.join(tempDir, 'missing')), false); +}); diff --git a/tests/utils/mock-opfs.js b/tests/utils/mock-opfs.js index 1c8daa0..7e7e91e 100644 --- a/tests/utils/mock-opfs.js +++ b/tests/utils/mock-opfs.js @@ -73,10 +73,7 @@ class MockDirectoryHandle { const installMockOpfs = () => { const rootMap = new Map(); - const prevNavigator = Object.getOwnPropertyDescriptor( - globalThis, - 'navigator', - ); + const prevNavigator = Object.getOwnPropertyDescriptor(globalThis, 'navigator'); Object.defineProperty(globalThis, 'navigator', { configurable: true, diff --git a/vercel.json b/vercel.json new file mode 100644 index 0000000..2a10a3f --- /dev/null +++ b/vercel.json @@ -0,0 +1,7 @@ +{ + "$schema": "https://openapi.vercel.sh/vercel.json", + "buildCommand": "pnpm run docs:build", + "outputDirectory": "docs/.vitepress/dist", + "installCommand": "pnpm install", + "cleanUrls": true +} From 7bbe2455eeb3c67c3cbeb64c157e7f56f197f629 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 08:19:25 +0300 Subject: [PATCH 04/67] feat: support custom id generation via getId option Mirrors kerberos's getCallId pattern: Storage.generateId() is the public default (crypto.randomUUID), overridable per instance with a sync or async getId function. Lets callers use ULID, CUID2, nanoid or server-issued ids without adding a runtime dependency to the package. Co-Authored-By: Claude Sonnet 5 --- docs/guide/browser.md | 2 +- docs/guide/storage.md | 38 ++++++++++++++++- docs/reference/api.md | 7 +-- index.d.ts | 3 ++ src/storage.js | 18 ++++++-- tests/browser.test-d.ts | 3 ++ tests/index.test-d.ts | 4 ++ tests/storage.test.js | 94 +++++++++++++++++++++++++++++++++++++++++ 8 files changed, 160 insertions(+), 9 deletions(-) diff --git a/docs/guide/browser.md b/docs/guide/browser.md index d566109..5358501 100644 --- a/docs/guide/browser.md +++ b/docs/guide/browser.md @@ -16,7 +16,7 @@ swapped for their browser twins: | `src/fsi.js` | `src/fsi.browser.js` | OPFS (`navigator.storage.getDirectory()`) | | `src/crypto.js` | `src/crypto.browser.js` | Web Crypto (`crypto.subtle`) | | `src/sync.js` | `src/sync.browser.js` | in-memory node registry | -| `src/utils.js` | `src/utils.browser.js` | `crypto.randomUUID()` | +| `src/utils.js` | `src/utils.browser.js` | `crypto.randomUUID()` โ€” the default `getId` | CRDT, contracts, the chain and storage are the same code in both halves. diff --git a/docs/guide/storage.md b/docs/guide/storage.md index 1e888ba..3d7df0f 100644 --- a/docs/guide/storage.md +++ b/docs/guide/storage.md @@ -14,11 +14,12 @@ const storage = await open({ path: 'data' }); | `path` | `'syncom'` | Directory (or OPFS path) holding keys, chain, data and peers | | `name` | โ€” | Alias for `path` | | `schema` | โ€” | Entity definitions; each name becomes a collection (see below) | +| `getId` | `crypto.randomUUID()` | Id generator for `insert()`; sync or async ([see below](#custom-identifiers)) | ## The explicit API -Every method addresses a record by its id. Ids are UUIDs generated by -`insert()`. +Every method addresses a record by its id. `insert()` generates one โ€” a UUID by +default, or whatever `getId` returns. ```js const id = await storage.insert({ title: 'Draft', hits: 0 }); @@ -43,6 +44,39 @@ const ok = await storage.swap(id, { hits: 2 }, { hits: 1 }); // ok === false if hits was no longer 1 โ€” nothing was written ``` +### Custom identifiers + +Pass `getId` to replace UUIDs with ULIDs, CUID2s, nanoids or ids minted by your +own server. The package has no runtime dependencies, so the generator is yours +to install and supply: + +```js +const { ulid } = require('ulid'); + +const storage = await open({ path: 'data', getId: () => ulid() }); +const id = await storage.insert({ title: 'Draft' }); +// 01ARZ3NDEKTSV4RRFFQ69G5FAV โ€” lexicographically sortable by creation time +``` + +The generator takes no arguments and may be asynchronous โ€” `insert()` awaits +it either way: + +```js +await open({ path: 'data', getId: async () => (await fetch('/next-id')).text() }); +``` + +`Storage.generateId()` is the default generator, should you want to fall back to +it. A `getId` that is not a function is ignored and the default stands. + +Two things the engine does not check for you: + +- **Ids become filenames.** A record is written to `data/${id}.json` with no + sanitising, so a generator that can emit `/`, `\` or `..` will break the + write. Keep ids to filename-safe characters. +- **Ids are matched against string fields.** A `Record` turns any string field + whose value is a known id into a [reference](#references). Short, guessable + ids raise the odds of an accidental match; UUIDs, ULIDs and CUID2s do not. + ### Encryption at rest `saveData()` takes an `encrypted` flag. The record is encrypted with the diff --git a/docs/reference/api.md b/docs/reference/api.md index 64e2a4d..e79e586 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -13,7 +13,7 @@ const syncom = require('@alexify/syncom'); ### `open(options?)` Returns `Promise`. Options: `path` (default `'syncom'`), `name` -(alias for `path`), `schema`. +(alias for `path`), `schema`, `getId`. ### `Storage` @@ -21,7 +21,7 @@ Returns `Promise`. Options: `path` (default `'syncom'`), `name` | Method | Returns | Notes | | --- | --- | --- | -| `insert(data)` | `Promise` | Generates a UUID and saves | +| `insert(data)` | `Promise` | Generates an id via `getId` (a UUID by default) and saves | | `get(id)` | `Promise` | `null` when absent; verifies against the chain | | `has(id)` | `Promise` | Filesystem check, no read | | `set(id, data)` | `Promise` | Replaces; emits `update` with the computed delta | @@ -37,7 +37,8 @@ Returns `Promise`. Options: `path` (default `'syncom'`), `name` | `hasRecord(id)` | `boolean` | Whether the id is known to the cache | | `addUpdate(update)` | `void` | Appends to the pending update log | -Properties: `schema`, `sync`. Statics: `Storage.Collection`, `Storage.Record`. +Properties: `schema`, `sync`. Statics: `Storage.Collection`, `Storage.Record`, +`Storage.generateId()` โ€” the default `getId`, synchronous in both halves. ### `Record` diff --git a/index.d.ts b/index.d.ts index 8260dec..21504d7 100644 --- a/index.d.ts +++ b/index.d.ts @@ -126,6 +126,7 @@ export interface StorageOptions { path?: string; name?: string; schema?: Schema; + getId?: () => string | Promise; } export interface StorageDataOptions { @@ -232,6 +233,8 @@ export interface StorageConstructor { new (options?: StorageOptions): Promise; Collection: typeof SyncomCollection; Record: typeof SyncomRecordCtor; + // The default `getId` โ€” synchronous in both halves, unlike the crypto three. + generateId(): string; } export const Storage: StorageConstructor; diff --git a/src/storage.js b/src/storage.js index 134782e..bd27874 100644 --- a/src/storage.js +++ b/src/storage.js @@ -144,9 +144,8 @@ class Collection { } async insert(data) { - const id = generateUUID(); const record = { ...data, $type: this.#name }; - await this.#storage.saveData(id, record); + const id = await this.#storage.insert(record); return this.#storage.record(id); } @@ -176,11 +175,23 @@ class Storage { #updates = []; #cache = new Map(); #schema = null; + #getId = null; + + // The default id generator, exposed so it stays reachable and stubbable. + // `generateUUID` never touches `this`, so the reference is safe to store + // unbound in `#getId`. + static generateId() { + return generateUUID(); + } constructor(options = {}) { + this.#getId = Storage.generateId; this.#path = options.path || options.name || 'syncom'; this.#records = new Map(); if (options.schema) this.#schema = options.schema; + // A non-function is ignored rather than rejected: the id generator is a + // convenience seam, not a security boundary, and the default still holds. + if (typeof options.getId === 'function') this.#getId = options.getId; return this.#init(); } @@ -269,7 +280,8 @@ class Storage { } async insert(data) { - const id = generateUUID(); + // `getId` may be sync or async โ€” both call sites already await this. + const id = await this.#getId(); await this.saveData(id, data); return id; } diff --git a/tests/browser.test-d.ts b/tests/browser.test-d.ts index bf2ad85..e17e6de 100644 --- a/tests/browser.test-d.ts +++ b/tests/browser.test-d.ts @@ -14,6 +14,9 @@ expectType(browser.diff({ a: 1 }, { a: 2 })); expectType(browser.merge({ a: 1 }, { a: 2 })); expectType>(new browser.Blockchain('base')); expectType>(browser.open({ path: 'data' })); +expectType>(browser.open({ path: 'data', getId: () => 'id' })); +// The id generator is not part of the async asymmetry: sync in both halves. +expectType(browser.Storage.generateId()); expectType>(browser.read('dir', 'file.json')); declare const storage: Storage; diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 196a99b..3e01373 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -53,7 +53,11 @@ expectType>(syncom.SmartContract.load('hash', { st expectType>(syncom.open()); expectType>(syncom.open({ path: 'data' })); expectAssignable({ path: 'data', schema: {} as Schema }); +expectAssignable({ getId: () => 'id' }); +expectAssignable({ getId: async () => 'id' }); +expectType(syncom.Storage.generateId()); expectError(syncom.open({ path: 1 })); +expectError(syncom.open({ getId: 'nope' })); expectType>(storage.insert({ a: 1 })); expectType>(storage.get('id')); expectType>(storage.has('id')); diff --git a/tests/storage.test.js b/tests/storage.test.js index a16adba..746730d 100644 --- a/tests/storage.test.js +++ b/tests/storage.test.js @@ -377,3 +377,97 @@ test('Collection module', async (t) => { } }); }); + +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +test('Storage id generation', async (t) => { + await t.test('Storage.generateId returns unique UUIDs', () => { + const id = syncom.Storage.generateId(); + assert.match(id, UUID); + assert.notStrictEqual(id, syncom.Storage.generateId()); + }); + + await t.test('custom getId is used once per insert', async () => { + const tempDir = await createTemp(); + try { + let calls = 0; + const getId = () => `custom-${++calls}`; + const storage = await syncom.open({ path: tempDir, getId }); + + assert.strictEqual(await storage.insert({ a: 1 }), 'custom-1'); + assert.strictEqual(await storage.insert({ a: 2 }), 'custom-2'); + assert.strictEqual(calls, 2); + assert.deepStrictEqual(await storage.get('custom-1'), { a: 1 }); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('custom getId may be async', async () => { + const tempDir = await createTemp(); + try { + const getId = async () => 'async-id'; + const storage = await syncom.open({ path: tempDir, getId }); + + const id = await storage.insert({ a: 1 }); + assert.strictEqual(id, 'async-id'); + assert.deepStrictEqual(await storage.get(id), { a: 1 }); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('custom getId reaches Collection insert', async () => { + const tempDir = await createTemp(); + try { + let calls = 0; + const getId = () => `rec_${++calls}`; + const options = { path: tempDir, schema: mockSchema, getId }; + const storage = await syncom.open(options); + + const author = await storage.Author.insert({ name: 'John' }); + assert.strictEqual(author.id, 'rec_1'); + + const data = await storage.get(author.id); + assert.strictEqual(data.$type, 'Author'); + assert.strictEqual(data.name, 'John'); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('a non-function getId falls back to the default', async () => { + const tempDir = await createTemp(); + try { + const storage = await syncom.open({ path: tempDir, getId: 'nope' }); + assert.match(await storage.insert({ a: 1 }), UUID); + } finally { + await cleanupTemp(tempDir); + } + }); + + await t.test('non-UUID ids survive the whole record lifecycle', async () => { + const tempDir = await createTemp(); + try { + let calls = 0; + const getId = () => `01ARZ3NDEKTSV4RRFFQ69G5FA${calls++}`; + const storage = await syncom.open({ path: tempDir, getId }); + + const id = await storage.insert({ title: 'Draft', hits: 0 }); + assert.deepStrictEqual(await storage.get(id), { title: 'Draft', hits: 0 }); + assert.strictEqual(await storage.has(id), true); + + await storage.update(id, { hits: 1 }); + assert.strictEqual(await storage.swap(id, { hits: 2 }, { hits: 1 }), true); + + const record = await storage.record(id); + assert.strictEqual(record.id, id); + assert.strictEqual(await storage.validate(), true); + + await storage.delete(id); + assert.strictEqual(await storage.has(id), false); + } finally { + await cleanupTemp(tempDir); + } + }); +}); From e2e0988dc5ce12b807659a7c92dd2ef6a7e41c8f Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 16:39:33 +0300 Subject: [PATCH 05/67] feat: replicate storage between nodes over an injected transport MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turns storage.sync from a registry-only stub into working replication. Every write becomes an operation (write/set/inc/delete) appended to a persisted, WAL-ordered log; a field is a last-writer-wins register or a per-origin counter, never both, so ops converge in any order and any number of times. SyncManager dials registered peers, pushes what a host lacks, subscribes to live changes, relays across a mesh de-duplicated by version vector, and bootstraps a late joiner from a streamed snapshot. The transport is injected, never a dependency: @alexify/syncom/wrpc adapts a WrpcClient to the small SyncLink contract, leaving reconnect, backoff and subscription resume to wrpc itself. @alexify/wrpc is a devDependency for one skip-guarded integration test only. Also: encryption at rest is now a hybrid AES-256-GCM + RSA-OAEP/SHA-256 envelope, identical (and portable) on both platform halves, replacing the broken un-awaited browser path and the ~200-byte RSA ceiling. Fixed along the way: apply()'s write/delete handling, Blockchain.validate on an unknown hash, loadKeys with a missing private half, and the node registry's host-collision bug. sync.browser.js is gone โ€” one sync.js over fsi serves both platforms. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 45 +++ CLAUDE.md | 77 +++- CONTRIBUTING.md | 22 +- README.md | 60 ++- SECURITY.md | 31 +- bench/log.js | 70 ++++ browser.d.ts | 26 +- docs/.vitepress/config.mts | 10 +- docs/guide/browser.md | 21 +- docs/guide/chain.md | 11 +- docs/guide/crdt.md | 14 +- docs/guide/getting-started.md | 27 +- docs/guide/storage.md | 65 ++- docs/guide/sync.md | 321 +++++++++++++-- docs/guide/wrpc.md | 136 +++++++ docs/index.md | 12 +- docs/public/llms.txt | 34 +- docs/reference/api.md | 126 ++++-- index.d.ts | 220 +++++++++- package.json | 10 +- pnpm-lock.yaml | 10 + scripts/size.js | 12 +- src/chain.js | 4 + src/crdt.js | 8 +- src/crypto.browser.js | 93 +++-- src/crypto.js | 41 +- src/fsi.browser.js | 13 +- src/fsi.js | 15 +- src/index.browser.js | 14 +- src/index.js | 5 +- src/link.js | 15 + src/log.js | 332 +++++++++++++++ src/ops.js | 311 ++++++++++++++ src/storage.js | 558 ++++++++++++++++++++----- src/sync.browser.js | 54 --- src/sync.js | 652 +++++++++++++++++++++++++++--- src/utils.browser.js | 12 +- src/utils.js | 11 + src/vector.js | 119 ++++++ src/wrpc.js | 92 +++++ tests/browser.test-d.ts | 3 + tests/chain.test.js | 2 + tests/crdt.test.js | 20 +- tests/crypto.browser.test.js | 2 + tests/crypto.cross.test.js | 37 ++ tests/index.browser.test.js | 8 +- tests/index.test-d.ts | 54 ++- tests/keys.test.js | 2 + tests/log.test.js | 198 +++++++++ tests/ops.test.js | 187 +++++++++ tests/storage.browser.test.js | 41 ++ tests/storage.encrypted.test.js | 87 ++++ tests/storage.ops.test.js | 256 ++++++++++++ tests/storage.test.js | 3 +- tests/suites/crypto-keys.js | 31 +- tests/suites/fsi.js | 14 + tests/suites/sync-engine.js | 333 +++++++++++++++ tests/suites/sync-manager.js | 62 ++- tests/sync.browser.test.js | 16 +- tests/sync.engine.browser.test.js | 22 + tests/sync.engine.test.js | 24 ++ tests/sync.manager.test.js | 213 ++++++++++ tests/sync.test.js | 12 +- tests/utils/browser-env.js | 65 +++ tests/utils/memory-link.js | 123 ++++++ tests/utils/mesh.js | 79 ++++ tests/utils/mock-opfs.js | 13 + tests/utils/temp.js | 2 +- tests/vector.test.js | 53 +++ tests/wrpc.integration.test.js | 152 +++++++ tests/wrpc.test-d.ts | 18 + tests/wrpc.test.js | 145 +++++++ wrpc.d.ts | 40 ++ wrpc.js | 3 + 74 files changed, 5551 insertions(+), 478 deletions(-) create mode 100644 bench/log.js create mode 100644 docs/guide/wrpc.md create mode 100644 src/link.js create mode 100644 src/log.js create mode 100644 src/ops.js delete mode 100644 src/sync.browser.js create mode 100644 src/vector.js create mode 100644 src/wrpc.js create mode 100644 tests/crypto.cross.test.js create mode 100644 tests/log.test.js create mode 100644 tests/ops.test.js create mode 100644 tests/storage.browser.test.js create mode 100644 tests/storage.encrypted.test.js create mode 100644 tests/storage.ops.test.js create mode 100644 tests/suites/sync-engine.js create mode 100644 tests/sync.engine.browser.test.js create mode 100644 tests/sync.engine.test.js create mode 100644 tests/sync.manager.test.js create mode 100644 tests/utils/browser-env.js create mode 100644 tests/utils/memory-link.js create mode 100644 tests/utils/mesh.js create mode 100644 tests/vector.test.js create mode 100644 tests/wrpc.integration.test.js create mode 100644 tests/wrpc.test-d.ts create mode 100644 tests/wrpc.test.js create mode 100644 wrpc.d.ts create mode 100644 wrpc.js diff --git a/CHANGELOG.md b/CHANGELOG.md index f69b40e..f787b96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,29 @@ section until the first publish; everything below lands there on release day. ### Added +- **Replication.** Every write is now an operation โ€” `write` (`insert`/`set`), + `set` (`update`/`swap`/`record.save`), `inc`, `delete` โ€” appended to a + persisted log (`src/log.js`) and applied through a record-level model + (`src/ops.js`) that converges in any order: last-writer-wins registers, + per-origin counters, tombstones with revival. `SyncManager` dials the + registered nodes through an injected `sync.connect(node)` factory, pushes + what a peer lacks, subscribes to what it learns, relays across a mesh, + resumes from the last event id and bootstraps from a snapshot when it + cannot; `storage.sync.unit()` is the router unit a host mounts. +- `@alexify/syncom/wrpc` โ€” `createWrpcLink(client)` adapts a `WrpcClient` + (injected, never a dependency) to the link contract; `isSyncLink`, + `SYNC_INTROSPECTION`. +- `Storage`: `inc()`, `list()`, `close()`, `applyRemote()`, `snapshot()`, + `mergeSnapshot()`, `adoptVector()`, `nodeId`, `vector`, `log`, the `change` + and `error` events; `Record.inc()`; live records follow remote changes and + keep unsaved edits on top. Options `logger`, `encrypted`, `sync`. +- `SyncManager`: `stop()`, `sync()`, `peers()`, `enableNode()`, `unit()`, the + `peer`/`sync`/`bootstrap`/`error` events; scopes (`addNode({ scopes })`, + `sync.scope`) for partial replication. +- Encryption at rest is a hybrid envelope (AES-256-GCM under an RSA-OAEP + SHA-256 wrapped key), identical in both halves, with no payload ceiling; + `open({ encrypted: true })` and `insert/set(โ€ฆ, { encrypted })`. +- `fsi.list(dir)`; `OpLog`, `applyOp`, `mergeEntry`, `isAlive` exports. - `src/utils.js` and `src/utils.browser.js` โ€” the `Emitter`, `generateUUID`, `between`, `fileExists` and `directoryExists` helpers the source needs, vendored in so the package keeps its zero-runtime-dependency promise. @@ -35,6 +58,17 @@ section until the first publish; everything below lands there on release day. ### Fixed +- `Storage.saveData()` did not await `encrypt`/`decrypt`, so encryption at + rest in the browser stored a serialized promise (`{}`) instead of ciphertext. +- `apply()` ignored the data of a `write` record and only understood `delete` + by `key`; a full write now replaces the state and Storage's own + `{ type: 'delete', id }` shape empties it. +- `Blockchain.validate({ from })` threw on an unknown hash instead of + answering `false`. +- `loadKeys()` threw `ENOENT` when `public.pem` existed without `private.pem`; + the pair is regenerated. +- `SyncManager` persisted nodes under `.json`, so two peers on one host, + or a url without a port, overwrote each other. - `Storage.validate()` threw `Cannot use 'in' operator` when called with no arguments, and silently returned `false` for `validate({})`. It discriminated on `'last' in idOrOptions`; it now keys off whether the first argument is a @@ -55,6 +89,17 @@ section until the first publish; everything below lands there on release day. ### Changed (breaking) +- `SyncManager` is constructed as `new SyncManager(storage, options)`; nodes + no longer carry `type: 'uplink' | 'downlink'` โ€” every registered node is + dialed, and any node can host. `src/sync.browser.js` is gone: one + `src/sync.js` over `fsi` serves both halves, and the browser registry is + persisted in OPFS. +- The in-memory `Storage.#updates` array and `addUpdate()` are gone; the + operation log replaces them. `set()` on an existing record is a whole-record + write; `update()` on a counter field, and `inc()` on a register field, throw. +- `delete()` leaves a tombstone on disk; `list()`, `get()` and `has()` hide it. +- The browser bundle budget was raised from 5 KB to 14 KB min+gzip for the + replication engine. - **Node.js โ‰ฅ 22 is required** (was `>=18`). - Browser modules follow the `*.browser.js` pattern: `src/{crypto,fsi,sync}-browser.js` โ†’ `src/{crypto,fsi,sync}.browser.js`. diff --git a/CLAUDE.md b/CLAUDE.md index b4c6f69..c7fcb9d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,19 +7,30 @@ get wrong. ## What this package is A local-first data engine: records on disk (or OPFS), every write committed to -a hash-linked log, CRDT deltas for merging. **Zero runtime dependencies**, and -that is a load-bearing constraint, not a preference. +a hash-linked log AND appended to an operation log that peers replicate over +an injected transport (wrpc through `@alexify/syncom/wrpc`). **Zero runtime +dependencies**, and that is a load-bearing constraint, not a preference โ€” wrpc +is a devDependency for one skip-guarded integration test and is otherwise an +object the caller hands in. ## Layout ``` index.js / browser.js two-line shims over src/index.js / src/index.browser.js +wrpc.js / wrpc.d.ts the @alexify/syncom/wrpc subpath (src/wrpc.js) index.d.ts the Node type surface (hand-written) browser.d.ts re-exports index.d.ts, overrides the 3 async crypto fns +src/storage.js Storage, Record, Collection โ€” every write is an op +src/ops.js the record-level model: applyOp / mergeEntry / isAlive, packOps +src/vector.js clocks, version vectors, event ids +src/log.js OpLog โ€” the persisted operation log over fsi +src/sync.js SyncManager (dialing side) + unit() (hosting side) +src/link.js isSyncLink โ€” the link contract check +src/wrpc.js createWrpcLink โ€” a WrpcClient as a SyncLink (no wrpc import) src/ ships as-is โ€” no build step, CommonJS, 'use strict' tests/ *.test.js are tests; anything else is a helper bench/ one file per subject + support/harness.js -scripts/size.js bundle-size report with a browser budget +scripts/size.js bundle-size report with browser budgets docs/ VitePress site ``` @@ -30,9 +41,36 @@ docs/ VitePress site fails at resolve time on any bare import reaching a browser bundle, so a stray `require('some-pkg')` in shared code breaks `pnpm size`, not just the dependency count. -- **Browser halves are `*.browser.js`** (dot, not dash). Adding one means - three edits: the file, `package.json#browser`, and `src/index.browser.js`. - `tests/package/consistency.test.js` catches a missed one. +- **Browser halves are `*.browser.js`** (dot, not dash): `fsi`, `crypto`, + `utils`. Everything else (storage, chain, log, ops, sync, wrpc adapter) is + shared. Adding a split means three edits: the file, `package.json#browser`, + and `src/index.browser.js`. `tests/package/consistency.test.js` catches a + missed one. To run shared code against the browser halves under Node, load + it through `tests/utils/browser-env.js` โ€” requiring `src/index.browser.js` + alone does NOT redirect the nested requires. +- **wrpc is injected, never required.** Nothing under `src/` may + `require('@alexify/wrpc')`; `scripts/size.js` fails on any bare import in a + browser bundle anyway. `src/wrpc.js` duck-types the client it is handed, and + `unit()` builds plain `{ access, handler }` objects that `defineRouter` + accepts without wrpc's `procedure()`. The `tracked` shape it emits is + `Symbol.for('wrpc.tracked')` โ€” part of wrpc's frozen wire protocol. +- **Every write is an op, and a field is a register OR a counter.** `insert`/ + `set` โ†’ `write`, `update`/`swap`/`record.save` โ†’ `set`, `inc` โ†’ `inc`, + `delete` โ†’ tombstone. Registers are last-writer-wins by `(ts, origin)`; + counters are per-origin sums. Mixing the two on one field throws locally and + is refused (with an `error` event) from a peer. Do not add a path that lets + them meet โ€” it cannot be made to commute. +- **Ops from one origin apply in seq order; seqs run per origin AND scope** + (`origin@scope` vector keys), and ops born from a snapshot (`merge`, + `advance`) run in their own `origin@~` stream. `applyRemote` skips + duplicates and gaps; resume from the last event id fills the gaps. +- **Log first, then data.** `OpLog.append` resolves before the record file is + written and before the op is announced; a reopen replays the unsealed tail. + A node's own seq must never repeat โ€” it is recovered from the log's vector. +- **Reconnect/backoff/resume belong to the link.** `SyncManager` re-runs only + `hello` + `push` on `reconnect`; a refused restored subscription (428 "say + hello first" โ€” wrpc gives a reconnect a NEW server-side client) triggers a + fresh handshake. Do not add a reconnect loop to syncom. - **Async asymmetry is real.** `calculateHash`, `encrypt` and `decrypt` are synchronous in `src/crypto.js` and promise-returning in `src/crypto.browser.js`. Shared code must `await` them; `browser.d.ts` @@ -48,24 +86,39 @@ docs/ VitePress site - **`merge` adds numbers.** `merge({ hits: 10 }, { hits: 1 })` is `11`. This is the counter CRDT and it is deliberate โ€” but it means a `diff` of numeric fields is not a plain patch. Do not "correct" it, and do not write docs or - examples that assume replacement. + examples that assume replacement. Storage does NOT replicate through + `merge`; the op model in `src/ops.js` is what converges. +- **Encrypted records are local.** `{ encrypted: true }` seals a record for + this node's key; it produces no op and is excluded from snapshots. Do not + route an encrypted entry through `applyOp`. ## Testing `node:test` + `node:assert`. Browser halves run under Node with the OPFS mock (`tests/utils/mock-opfs.js`) and `node:crypto`'s `webcrypto` โ€” see `tests/index.browser.test.js`. Cross-platform behaviour goes in `tests/suites/` -and is run against both halves. +and is run against both halves; `tests/suites/sync-engine.js` runs the +convergence properties over `tests/utils/memory-link.js` (an in-process link +that mimics a wrpc client: JSON across the boundary, a reconnect re-opens the +subscription from its last event id against a fresh server-side client) and +`tests/utils/mesh.js` (N storages from an adjacency list, `settle()`). +`tests/wrpc.integration.test.js` runs two real wrpc `Server`s and a dialer, and +skips when `@alexify/wrpc` is not installed. Two things bite there: two nodes' +clocks are independent, so "later" needs a `tick()` between writes on +different nodes; and `settle()` ignores `@~` vector keys, because a node's +bootstrap stream never reaches the node it bootstrapped from. Coverage thresholds (95/95/90/95) are enforced in CI. They currently pass with headroom; do not lower them to land a change. ## Documentation -Document what the code does today. `sync.start()` is a no-op and every page -that mentions it says so. When adding docs, verify each example by running it -before writing it down โ€” several examples inherited from the legacy repo were -wrong in exactly that way. +Document what the code does today. When adding docs, verify each example by +running it before writing it down โ€” several examples inherited from the legacy +repo were wrong in exactly that way. A new page needs a sidebar entry in +`docs/.vitepress/config.mts`; `docs/guide/sync.md` and `docs/guide/wrpc.md` +are the two pages that describe the protocol โ€” keep them in step with +`src/sync.js`. ## Formatting diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 42aead0..96b7bf9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,8 +45,15 @@ with `node --test --test-name-pattern 'swap'`. - **Anything in `tests/` not named `*.test.js` is a helper** and is skipped by `node --test` discovery โ€” that is why `tests/suites/`, `tests/utils/` and `tests/package/` work the way they do. -- **Document what the code does, not what it will do.** `sync.start()` is a - no-op today and the docs say so. +- **Document what the code does, not what it will do.** Run every example + before writing it down. +- **wrpc is injected, never required.** `@alexify/wrpc` is a devDependency for + `tests/wrpc.integration.test.js` only, which skips when it is missing; nothing + under `src/` may `require` it. The adapter in `src/wrpc.js` duck-types the + client it is handed. +- **A field is a register or a counter.** `set`/`update` write registers, + `inc` writes counters, and the two never meet on one field โ€” do not add a + path that lets them. ## Tests @@ -55,9 +62,14 @@ per subject, nested `await t.test(...)` per case, teardown registered with `t.after(...)` immediately after setup. Behaviour shared between the Node and browser halves lives in `tests/suites/` -and is run twice โ€” see `tests/suites/emitter.js` and `tests/suites/fsi.js`. -The browser halves are exercised under Node using the OPFS mock in -`tests/utils/mock-opfs.js` and `node:crypto`'s `webcrypto`. +and is run twice โ€” see `tests/suites/emitter.js`, `tests/suites/fsi.js` and +`tests/suites/sync-engine.js`. The browser halves are exercised under Node.js +using the OPFS mock in `tests/utils/mock-opfs.js` and `node:crypto`'s +`webcrypto`; `tests/utils/browser-env.js` loads the shared modules the way a +bundler honouring `package.json#browser` would. The sync engine is tested over +`tests/utils/memory-link.js`, an in-process link that behaves like a wrpc +client at the boundary (JSON across, resume from the last event id on +reconnect), and once more over real wrpc in `tests/wrpc.integration.test.js`. `tests/package/consistency.test.js` is the machine-checked half of the release checklist: it fails if `exports`, `files`, the `browser` map, the `.d.ts` diff --git a/README.md b/README.md index 11f0315..c957270 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,11 @@ Fast, zero-dependency distributed data sync engine ๐ŸŒ `syncom` is a local-first data engine for JavaScript. It stores records on disk โ€” or in the browser's origin-private filesystem โ€” commits every write to -a hash-linked log so tampering is detectable, and produces CRDT deltas that -peers can merge without coordination. It has no runtime dependencies. +a hash-linked log so tampering is detectable, and replicates every write to +peers as an operation that converges without coordination: last-writer-wins +registers, per-origin counters, tombstones. The transport is injected โ€” +[wrpc](https://wrpc.vercel.app) out of the box โ€” and the package has no +runtime dependencies. ```js 'use strict'; @@ -25,13 +28,14 @@ const { open } = require('@alexify/syncom'); const main = async () => { const storage = await open({ path: 'data' }); - const id = await storage.insert({ name: 'Ada', hits: 0 }); + const id = await storage.insert({ name: 'Ada' }); const record = await storage.record(id); record.on('update', (data, delta) => console.log('changed', delta)); - record.hits = 1; + record.title = 'Ada Lovelace'; await record.save(); + await storage.inc(id, 'hits', 1); // a counter: increments add up across nodes console.log(await storage.validate({ last: 10 })); // true }; @@ -72,8 +76,9 @@ that fails the build when exceeded. | Entry | raw | min | min+gzip | budget | | ----- | ---:| ---:| --------:| ------:| -| main entry โ€” browser (@alexify/syncom) | 30.9 KB | 13.0 KB | 4.7 KB | 5.0 KB | -| main entry โ€” node (@alexify/syncom) | 27.7 KB | 12.6 KB | 4.5 KB | โ€” | +| main entry โ€” browser (@alexify/syncom) | 96.1 KB | 39.4 KB | 13.7 KB | 14.0 KB | +| main entry โ€” node (@alexify/syncom) | 89.0 KB | 38.5 KB | 13.4 KB | โ€” | +| wrpc adapter โ€” browser (@alexify/syncom/wrpc) | 3.3 KB | 1.4 KB | 0.7 KB | 2.0 KB | ## Quick Start @@ -103,16 +108,43 @@ merge({ hits: 10 }, { hits: 1 }); // { hits: 11 } diff({ tags: ['a'] }, { tags: ['a', 'b'] }); // { tags: ['b'] } ``` +### Syncing between nodes + +Host one end on a [wrpc](https://wrpc.vercel.app) server, dial it from the +other. wrpc is yours to install โ€” syncom only takes the client you built: + +```js +// the hub +const { Server, defineRouter } = require('@alexify/wrpc'); +const router = defineRouter({ syncom: storage.sync.unit({ access: 'public' }) }); +await new Server({ router, host: '0.0.0.0', port: 8000, protocol: 'http' }).listen(); + +// a node โ€” a server, a CLI, a browser tab +const { WrpcClient } = require('@alexify/wrpc'); +const { createWrpcLink } = require('@alexify/syncom/wrpc'); +const storage = await open({ + path: 'data', + sync: { connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)) }, +}); +await storage.sync.addNode({ url: 'ws://hub.example.com:8000/api' }); +``` + +Every node can be both, and ops relay hop by hop, so any connected graph of +nodes converges. + ## Features | Area | What you get | | --- | --- | -| **Storage** | Records with live properties and per-record deltas, schema-driven collections, compare-and-swap writes, optional RSA encryption at rest | +| **Storage** | Records with live properties and per-record deltas, schema-driven collections, compare-and-swap writes, counters, encryption at rest (AES-256-GCM under an RSA-wrapped key) | +| **Replication** | Every write is an op in a persisted log; peers exchange ops over an injected link, resume from where they stopped, and bootstrap from a streamed snapshot when they cannot; scopes for partial replication | +| **Convergence** | Last-writer-wins registers, per-origin counters, tombstones with revival โ€” any order, any number of times, same result | +| **Transport** | `@alexify/syncom/wrpc` adapts a `WrpcClient`; `storage.sync.unit()` is the router unit a wrpc server mounts. Reconnects, resumable subscriptions, binary streams and rooms come from wrpc | | **Integrity** | Every write appends a hash-linked block; reads re-verify the record against the block that committed it and throw on a mismatch | | **CRDT** | `diff` / `merge` / `apply` over plain objects โ€” nested maps merge recursively, arrays are grow-only sets, numbers add so counters converge | | **Contracts** | Store a procedure in the chain by hash, load it anywhere, run it against a read-only view of storage; failures are recorded as blocks | | **Portability** | One API over `node:fs` + `node:crypto` on the server and OPFS + Web Crypto in the browser, chosen by the `browser` export condition | -| **Dependencies** | None | +| **Dependencies** | None. wrpc is a devDependency for one integration test and is injected at runtime | ## Browser usage @@ -125,15 +157,16 @@ import { open, diff, merge } from '@alexify/syncom'; Crypto is asynchronous there (`calculateHash`, `encrypt` and `decrypt` return promises), and storage lives in the origin-private filesystem. `browser.d.ts` -declares those signatures separately. See +declares those signatures separately. A browser node dials the mesh through +`@alexify/syncom/wrpc` like any other. See [Browser](https://syncomjs.vercel.app/guide/browser). ## Status -Pre-1.0, and one part of the surface is a stub worth calling out: `storage.sync` -registers, persists and lists peer nodes, but **no transport is implemented** โ€” -`sync.start()` is a no-op. Everything else on this page works today. See -[Sync](https://syncomjs.vercel.app/guide/sync). +Pre-1.0. Everything on this page works today and is covered by the test suite, +including replication over real wrpc WebSockets. Known limits โ€” a counter +straddling a concurrent reset, no op signatures, encrypted records staying +local โ€” are listed under [Guarantees and limits](https://syncomjs.vercel.app/guide/sync#guarantees-and-limits). ## Documentation @@ -143,6 +176,7 @@ registers, persists and lists peer nodes, but **no transport is implemented** [Blockchain log](https://syncomjs.vercel.app/guide/chain) ยท [Smart contracts](https://syncomjs.vercel.app/guide/contracts) - [Sync](https://syncomjs.vercel.app/guide/sync) ยท + [wrpc transport](https://syncomjs.vercel.app/guide/wrpc) ยท [Browser](https://syncomjs.vercel.app/guide/browser) - [API reference](https://syncomjs.vercel.app/reference/api) diff --git a/SECURITY.md b/SECURITY.md index 59cce8e..1c89298 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -27,21 +27,36 @@ These are the areas where a report matters most, roughly in order: evidence claim rests on blocks hashing to what they are filed under and on `validate` walking `prev` honestly. A way to mutate a record or block and still pass `validate` is a vulnerability. +- **Replication** (`src/sync.js`, `src/ops.js`, `src/log.js`). A peer is + trusted with everything its scopes cover: anything that passes `hello` + (the unit's `access`, your `authorize(context, hello)`, the transport's own + authentication) can write any record in those scopes. Ops are not signed; + `origin` is a claim. In scope: a way for a peer to write outside its + scopes, to make another node apply an op twice with a different result + (idempotence), to push a `merge`/`advance` op that grants itself state it + was not given, prototype pollution through op payloads, or a malformed op + that crashes a node rather than being logged and skipped. The token in + `addNode({ token })` and `hello` is stored and sent as given; use `wss://`. - **Crypto** (`src/crypto.js`, `src/crypto.browser.js`). RSA-2048 key - generation, PEM handling, encrypt/decrypt of records stored with - `{ encrypted: true }`, and the base64url encoding in the browser half. Note - the two halves are deliberately not ciphertext-compatible (default padding - vs RSA-OAEP/SHA-256); a *silent* mismatch that produces wrong plaintext - rather than an error is in scope. + generation, PEM handling, and the hybrid envelope (AES-256-GCM under an + RSA-OAEP/SHA-256 wrapped key) records stored with `{ encrypted: true }` use, + identical in both halves. A *silent* mismatch that produces wrong plaintext + rather than an error is in scope, as is any way an encrypted record's + plaintext leaves the node (they are deliberately excluded from replication). - **Filesystem access** (`src/fsi.js`, `src/fsi.browser.js`). Paths are built from record ids and caller-supplied directory names. Traversal out of the storage directory via a crafted id or path is in scope. ## Not in scope -- `src/sync.js` moves no data โ€” `start()` is a no-op and there is no transport - to attack. Node URLs and tokens are stored as given, in plaintext, under - `/remote/`; that is the documented behaviour today, not a finding. +- The transport itself. wrpc (or whatever `connect(node)` returns) owns the + connection, its authentication and its encryption in transit; report those + to the transport's maintainers. +- Node URLs and tokens are stored as given, in plaintext, under + `/remote/`; the RSA private key sits unprotected in `/keys/`. + Both are documented behaviour, not findings. +- Denial of service by a peer that is allowed to sync (a flood of ops, a huge + snapshot): the limits are the transport's and the operator's. - Anything requiring an attacker to already have write access to the storage directory, except where it defeats `validate` (see Chain integrity above). diff --git a/bench/log.js b/bench/log.js new file mode 100644 index 0000000..1931f0c --- /dev/null +++ b/bench/log.js @@ -0,0 +1,70 @@ +/** + * `OpLog` โ€” the operation log every write and every replicated op goes + * through. `append` is one small file write per batch; `since` is an + * in-memory slice by seq. A roll seals the open files into a segment and is + * the one operation that touches many files at once. + * + * Writes go to a temp directory that is removed on exit. + * + * Run with `node bench/log.js` or `pnpm bench`. + */ + +'use strict'; + +const fs = require('node:fs/promises'); +const os = require('node:os'); +const path = require('node:path'); +const { bench } = require('./support/harness.js'); +const { OpLog } = require('../src/log.js'); + +const op = (seq) => ({ origin: 'bench', seq, ts: seq, type: 'set', id: `r${seq % 100}`, data: { v: seq } }); + +const main = async () => { + console.log(`OpLog benchmark โ€” Node ${process.version}\n`); + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'syncom-bench-log-')); + + try { + // Default roll cadence, unbounded retention: a realistic log that seals + // as it grows, so the reopen row reads segments, not thousands of files. + const log = await new OpLog(root, { retention: { ops: 1_000_000 } }); + let seq = 0; + + await bench('append โ€” one op', () => log.append([op(++seq)]), { warmup: 20 }); + await bench( + 'append โ€” batch of 100', + () => { + const batch = []; + for (let i = 0; i < 100; i++) batch.push(op(++seq)); + return log.append(batch); + }, + { warmup: 5, measureMs: 300, opsPerIteration: 100 }, + ); + + const cursor = log.formatId(Math.max(1, log.next - 500)); + await bench('since โ€” last 500 entries', () => log.since(cursor)); + await bench('since โ€” everything (no cursor)', () => log.since(undefined), { measureMs: 500 }); + await bench('since โ€” 20 from the middle, scoped', () => + log.since(log.formatId(1000), { scopes: ['x'], limit: 20 })); + + const rolling = await new OpLog(path.join(root, 'rolling'), { rollAfter: 8, retention: { ops: 10_000 } }); + let n = 0; + await bench('append with a roll every 8 batches', () => rolling.append([op(++n)]), { warmup: 16, measureMs: 500 }); + + await log.flush(); + await bench( + 'reopen โ€” index + segments + tail', + async () => { + const reopened = await new OpLog(root, { retention: { ops: 1_000_000 } }); + return reopened.length; + }, + { warmup: 1, measureMs: 500 }, + ); + } finally { + await fs.rm(root, { recursive: true, force: true }); + } +}; + +main().catch((error) => { + console.error(error); + process.exitCode = 1; +}); diff --git a/browser.d.ts b/browser.d.ts index ac0dbea..51a08b8 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -26,13 +26,31 @@ export type { SchemaEntities, Schema, StorageOptions, + Collections, + Reference, + SyncomRecord, + SyncomCollection, StorageDataOptions, StorageEntry, Node, NodeData, SyncManagerConstructor, - StorageUpdate, + SyncOptions, + SyncLink, StorageConstructor, + ClockStamp, + VersionVector, + SyncOpType, + SyncOp, + EntryMeta, + Entry, + ApplyResult, + LogEntry, + OpLogOptions, + OpLogConstructor, + Logger, + SnapshotPage, + ApplyRemoteResult, } from './index.js'; export { @@ -42,6 +60,11 @@ export { write, exists, remove, + list, + applyOp, + mergeEntry, + isAlive, + OpLog, diff, merge, apply, @@ -49,6 +72,7 @@ export { DataReader, SmartContract, SyncManager, + isSyncLink, Storage, open, } from './index.js'; diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index a0ba138..d82e543 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -23,6 +23,8 @@ const keywords = [ 'p2p', 'peer-to-peer', 'replication', + 'wrpc', + 'websocket', 'conflict-resolution', 'CRDT', 'event-sourcing', @@ -144,12 +146,16 @@ export default withMermaid({ ], }, { - text: 'Platforms', + text: 'Replication', items: [ { text: 'Sync', link: '/guide/sync' }, - { text: 'Browser', link: '/guide/browser' }, + { text: 'wrpc transport', link: '/guide/wrpc' }, ], }, + { + text: 'Platforms', + items: [{ text: 'Browser', link: '/guide/browser' }], + }, ], '/reference/': [ { diff --git a/docs/guide/browser.md b/docs/guide/browser.md index 5358501..e2babc5 100644 --- a/docs/guide/browser.md +++ b/docs/guide/browser.md @@ -8,17 +8,17 @@ configuration: import { open, diff, merge } from '@alexify/syncom'; ``` -Under the hood the entry resolves to `browser.js`, and four modules are +Under the hood the entry resolves to `browser.js`, and three modules are swapped for their browser twins: | Node | Browser | Backed by | | --- | --- | --- | | `src/fsi.js` | `src/fsi.browser.js` | OPFS (`navigator.storage.getDirectory()`) | | `src/crypto.js` | `src/crypto.browser.js` | Web Crypto (`crypto.subtle`) | -| `src/sync.js` | `src/sync.browser.js` | in-memory node registry | | `src/utils.js` | `src/utils.browser.js` | `crypto.randomUUID()` โ€” the default `getId` | -CRDT, contracts, the chain and storage are the same code in both halves. +CRDT, contracts, the chain, storage, the operation log and sync are the same +code in both halves; so is the `@alexify/syncom/wrpc` adapter. ## What differs @@ -34,16 +34,17 @@ await decrypt(data, privateKey); `browser.d.ts` declares these signatures separately from `index.d.ts` for exactly this reason. -The cipher differs too: Node.js uses RSA with the default padding via -`publicEncrypt`; the browser uses RSA-OAEP with SHA-256. Keys generated on one -platform are PEM-compatible, but ciphertext is not portable between them. +The cipher is the same on both: AES-256-GCM under a key wrapped with +RSA-OAEP (SHA-256). Keys and ciphertext move between the platforms. **Storage is origin-private.** `fsi.browser.js` resolves paths against `navigator.storage.getDirectory()`, so `open({ path: 'data' })` creates a `data/` directory inside the origin's OPFS rather than on disk. It is per-origin and the browser may evict it. -**Sync nodes are not persisted.** See [Sync](./sync). +**A browser node dials, it cannot host.** Its registry, log and cursors live +in OPFS like everything else, so a reload resumes where it left off. See +[Sync](./sync) and [wrpc transport](./wrpc#in-the-browser). ## Requirements @@ -65,5 +66,7 @@ that sneaks into the browser bundle, so it cannot silently grow a dependency. ## Testing without a browser The suite exercises the browser halves under Node.js using a mock OPFS -(`tests/utils/mock-opfs.js`) and `node:crypto`'s `webcrypto`. See -`tests/index.browser.test.js` for the pattern. +(`tests/utils/mock-opfs.js`) and `node:crypto`'s `webcrypto`. +`tests/utils/browser-env.js` loads the shared modules the way a bundler +honouring `package.json#browser` would, so storage and sync run against OPFS +too โ€” see `tests/storage.browser.test.js` and `tests/sync.engine.browser.test.js`. diff --git a/docs/guide/chain.md b/docs/guide/chain.md index 4d31b52..90f98e7 100644 --- a/docs/guide/chain.md +++ b/docs/guide/chain.md @@ -54,8 +54,15 @@ on hot paths. See `bench/chain.js` for the numbers. ## Through storage -`Storage` keeps its own chain and adds a block on every `saveData`, recording -the record id and the hash of the stored entry. +`Storage` keeps its own chain and adds a block for every write it applies. A +local write commits `{ id, hash, from }` โ€” the record, the hash of its data, +and its position in the [log](./sync#the-log); a batch of ops applied from a +peer commits one block `{ records: { [id]: hash }, from, to }` for every record +it touched. `validate(id, data, block)` accepts both shapes. + +Each node's chain is its **own** audit log: the order in which *it* applied +things. Two nodes that hold the same records hold different chains โ€” that is +expected, and it is not a consensus mechanism. ```js await storage.validate({ last: 10 }); // delegates to the chain diff --git a/docs/guide/crdt.md b/docs/guide/crdt.md index ec351b9..1fd0942 100644 --- a/docs/guide/crdt.md +++ b/docs/guide/crdt.md @@ -75,9 +75,9 @@ or whose `type` is unrecognised, are skipped. | `type` | Effect | | --- | --- | -| `write` | copies the state forward | +| `write` | replaces the state with `record.data` โ€” a full record write; a bare `write` without data copies the state forward | | `delta` | `merge(state, record.data)` | -| `delete` | removes `record.key` | +| `delete` | removes `record.key`; without a `key` (Storage's `{ type: 'delete', id }`) the record is gone and the state becomes `{}` | | `inc` | adds `record.value` to `record.key` | | `dec` | subtracts `record.value` from `record.key` | @@ -92,7 +92,9 @@ apply({ hits: 0 }, [ ## Where storage uses them -`Record.save()` sends a `delta` record into the storage update log, and -`Storage.set` / `update` / `swap` do the same for their own changes. The log -is what a future sync transport would exchange with peers โ€” see -[Sync](./sync) for what is and is not implemented today. +Storage does **not** replicate through `merge`. Its writes are typed +operations โ€” `write`, `set`, `inc`, `delete` โ€” applied with last-writer-wins +registers and per-origin counters, because a delta that *adds* numbers cannot +tell `hits = 3` from `hits += 3`. See [Sync](./sync#operations) for that +model. `diff`, `merge` and `apply` remain what they are here: standalone +helpers for plain objects. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 244da58..5fb72ee 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -20,7 +20,7 @@ npm install @alexify/syncom ## Opening a storage `open()` is asynchronous: it generates or loads an RSA key pair, initialises -the chain, and returns once the storage is ready to use. +the chain and the operation log, and returns once the storage is ready to use. ```js 'use strict'; @@ -43,11 +43,13 @@ main(); `path` (or `name`) selects the directory; it defaults to `syncom`. Under it syncom creates three subdirectories: -| Directory | Contents | +| Entry | Contents | | --- | --- | +| `node.json` | this node's identity โ€” the `origin` of every op it makes | | `keys/` | `public.pem` / `private.pem`, generated on first open | | `blockchain/` | one JSON file per block, plus `.blockchain.json` holding the tail hash | | `data/` | one JSON file per record, each naming the block that committed it | +| `log/` | the operation log โ€” what peers replicate from | | `remote/` | one JSON file per registered sync node | ## Records @@ -83,8 +85,21 @@ await storage.validate(); // the whole chain Reading a record that no longer matches its block throws rather than returning the modified data. See [Blockchain log](./chain). -## What is not implemented yet +## Syncing with other nodes -`storage.sync` registers and persists peer nodes, but **no transport is -implemented** โ€” `sync.start()` is a no-op. See [Sync](./sync) for what exists -today. +Hand `open()` a way to reach a peer and register it; from then on writes flow +both ways and converge without coordination: + +```js +const { createWrpcLink } = require('@alexify/syncom/wrpc'); +const { WrpcClient } = require('@alexify/wrpc'); + +const storage = await open({ + path: 'data', + sync: { connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)) }, +}); +await storage.sync.addNode({ url: 'ws://hub.example.com:8000/api', token: 'secret' }); +``` + +See [Sync](./sync) for the model โ€” registers, counters, tombstones, the log โ€” +and [wrpc transport](./wrpc) for hosting the other end. diff --git a/docs/guide/storage.md b/docs/guide/storage.md index 3d7df0f..d97fa44 100644 --- a/docs/guide/storage.md +++ b/docs/guide/storage.md @@ -15,6 +15,9 @@ const storage = await open({ path: 'data' }); | `name` | โ€” | Alias for `path` | | `schema` | โ€” | Entity definitions; each name becomes a collection (see below) | | `getId` | `crypto.randomUUID()` | Id generator for `insert()`; sync or async ([see below](#custom-identifiers)) | +| `logger` | `console` | A Console or pino-shaped logger, or `false` for silence | +| `encrypted` | `false` | Encrypt every record at rest by default ([see below](#encryption-at-rest)) | +| `sync` | โ€” | Replication: `connect`, `scope`, `retention`, โ€ฆ โ€” see [Sync](./sync#options) | ## The explicit API @@ -22,17 +25,29 @@ Every method addresses a record by its id. `insert()` generates one โ€” a UUID b default, or whatever `getId` returns. ```js -const id = await storage.insert({ title: 'Draft', hits: 0 }); +const id = await storage.insert({ title: 'Draft' }); await storage.has(id); // true -await storage.get(id); // { title: 'Draft', hits: 0 } -await storage.update(id, { hits: 1 }); // merge a partial change -await storage.set(id, { title: 'Final', hits: 1 }); // replace wholesale +await storage.get(id); // { title: 'Draft' } +await storage.update(id, { status: 'review' }); // set some fields +await storage.inc(id, 'views', 1); // add to a counter field +await storage.set(id, { title: 'Final' }); // replace wholesale โ€” status and views are gone +await storage.list(); // every live id await storage.delete(id); +await storage.close(); // stop replication, seal the log ``` -`set()` and `update()` both emit an `update` event on any live `Record` for -that id, and append the change to the pending update log. +Every one of these is an **operation**: applied here, written to the +[log](./sync#the-log), announced as a `change` event, and replicated to peers +when a transport is configured. `set()`, `update()` and `inc()` also emit an +`update` event on any live `Record` for that id. + +::: tip Registers and counters +A field written with `insert()`, `set()` or `update()` is a *register* โ€” last +writer wins. A field created with `inc()` is a *counter* โ€” increments add up +across nodes. A field cannot be both: `update()` on a counter and `inc()` on a +register throw. See [Sync](./sync#registers-and-counters). +::: ### Compare-and-swap @@ -79,15 +94,27 @@ Two things the engine does not check for you: ### Encryption at rest -`saveData()` takes an `encrypted` flag. The record is encrypted with the -storage's own RSA public key and decrypted on read: +Open the storage with `encrypted: true`, or pass the flag per write: ```js -await storage.saveData(id, { ssn: '...' }, { encrypted: true }); +const vault = await open({ path: 'vault', encrypted: true }); +await vault.insert({ ssn: '...' }); // sealed on disk + +await storage.insert({ ssn: '...' }, { encrypted: true }); // one record, in a plain storage +await storage.saveData(id, data, { encrypted: true }); ``` -RSA-OAEP has a hard payload ceiling (a couple of hundred bytes at 2048-bit -keys), so this is for small secrets, not for whole documents. +A record is sealed with a fresh AES-256-GCM key, and that key is wrapped with +the storage's RSA-OAEP (SHA-256) public key โ€” the same envelope on Node.js and +in the browser, so ciphertext and keys move between the two. There is no size +ceiling. The key pair lives in `/keys/`, unprotected: whoever can read +the directory can read the records. + +Encrypted records are **local**. No peer holds this node's private key, so +they produce no operations, never appear in a snapshot, and `inc()` on one +throws โ€” `update()`, `set()` and `delete()` still work, they simply stay +here. A record never changes kind: writing an encrypted record over a +replicated one (or the reverse) is refused. ## Records @@ -105,6 +132,7 @@ record.on('update', (data, delta) => { record.title = 'Final'; // stages the change and emits record.delta(); // { title: 'Final' } await record.save(); // writes, then clears the delta +await record.inc('views', 1); // a counter, through the record await record.data(); // the full current object await record.delete(); ``` @@ -112,6 +140,11 @@ await record.delete(); Properties are defined from whatever keys the record had when it was loaded; `id` is reserved and never becomes a settable property. +A record also follows changes made **elsewhere** โ€” `storage.update()` in the +same process, or an op from a peer: its properties are refreshed, `update` +fires with the delta, and edits staged but not yet saved stay on top. A +record deleted elsewhere reports `update` with `null` data. + ### References When a stored string happens to be the id of another record that storage @@ -148,11 +181,15 @@ no collections and you address records by id only. ``` data/ +โ”œโ”€โ”€ node.json { id } โ€” this node's identity โ”œโ”€โ”€ keys/ public.pem, private.pem โ”œโ”€โ”€ blockchain/ .json per block, .blockchain.json holds the tail -โ”œโ”€โ”€ data/ .json per record: { data, encrypted, timestamp, block } -โ””โ”€โ”€ remote/ .json per registered sync node +โ”œโ”€โ”€ data/ .json per record: { data, meta, encrypted, timestamp, block } +โ”œโ”€โ”€ log/ the operation log (see Sync) +โ””โ”€โ”€ remote/ .json per registered sync node ``` Every record names the block that committed it, which is what makes -[validation](./chain) possible. +[validation](./chain) possible. `meta` is the replication bookkeeping โ€” per +field clocks, the reset watermark, the tombstone, the counters โ€” and is what +lets two nodes' copies merge; a deleted record keeps its file as a tombstone. diff --git a/docs/guide/sync.md b/docs/guide/sync.md index 584e58b..b8cae77 100644 --- a/docs/guide/sync.md +++ b/docs/guide/sync.md @@ -1,57 +1,302 @@ # Sync -::: warning No transport is implemented yet -`SyncManager` registers, persists and lists peer nodes. It does **not** move -any data: `start()` walks the enabled uplinks and does nothing with them. -Treat this page as a description of the node registry, not of working -replication. -::: +`storage.sync` replicates a storage with other nodes. Every node is a peer: +it applies its own writes, applies what its peers send, and forwards both to +everyone else it talks to. There is no coordinator and no consensus โ€” records +converge because every operation is designed to be applied in any order, any +number of times, with the same result. -## The node registry +The transport is **injected**. syncom describes the small contract a link has +to satisfy, ships an adapter for [wrpc](./wrpc), and depends on nothing. ```js -const storage = await open({ path: 'data' }); +const { open } = require('@alexify/syncom'); +const { createWrpcLink } = require('@alexify/syncom/wrpc'); +const { WrpcClient } = require('@alexify/wrpc'); -await storage.sync.addNode({ - type: 'uplink', - url: 'ws://example.com:8001', - token: 'secret', +const storage = await open({ + path: 'data', + sync: { + connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)), + }, }); +await storage.sync.addNode({ url: 'ws://hub.example.com:8000/api', token: 'secret' }); +// From here on, writes flow both ways. +``` + +## Operations + +Every write becomes an **op** โ€” the unit that travels between nodes and sits +in the log: + +``` +{ origin, seq, ts, type, id, scope?, data?, key?, value? } +``` + +| `type` | Produced by | Effect on the record | +| --- | --- | --- | +| `write` | `insert()`, `set()`, `saveData()` | Replaces the record: fields not written are gone | +| `set` | `update()`, `swap()`, `record.save()` | Replaces the listed fields; last writer wins per field | +| `inc` | `inc()`, `record.inc()` | Adds to a counter field | +| `delete` | `delete()`, `record.delete()` | Leaves a tombstone; a newer write revives the record | +| `merge` | a snapshot page being joined | Joins a whole entry (see [Bootstrap](#bootstrap)) | +| `advance` | a snapshot being joined | Records which ops the joined state already covers | + +`origin` is the node that made the op, `seq` its sequence number on that node, +`ts` a millisecond stamp that never runs backwards and never falls behind a +stamp the node has seen from a peer. + +### Registers and counters + +A field is either a **register** or a **counter**, never both. + +- A register holds whatever was written last: two nodes writing the same field + concurrently agree on the winner by `(ts, origin)`, so both end up with the + same value. +- A counter is a sum. Two nodes each adding 1 concurrently end up at +2 on + both, because the sum keeps one running total per origin and adds them up. + +```js +const id = await storage.insert({ title: 'Draft' }); // title is a register +await storage.inc(id, 'views', 1); // views is a counter +await storage.update(id, { title: 'Final' }); // fine +await storage.update(id, { views: 10 }); // throws: views is a counter; use inc() +await storage.inc(id, 'title', 1); // throws: title is a register; use update() +``` + +Do not initialise a counter in `insert()` โ€” `{ views: 0 }` makes `views` a +register. `inc()` creates the field. A peer's op that breaks the rule is +refused for that field and reported as an `error` event on the storage; the +rest of the op still applies. + +### Whole-record writes and deletes + +`set()` and `delete()` reset the record: a concurrent `update()` from another +node that is *older* than the reset loses the fields it wrote, one that is +*newer* keeps them โ€” whichever order the two arrive in. A `delete()` leaves a +tombstone that hides the record from `get()`, `has()` and `list()`; any newer +write brings it back. + +One case does not converge and is left that way on purpose: a counter +incremented by one origin both before and after a concurrent reset by another. +Resettable counters need causal metadata syncom does not carry. + +## The log + +Every op a node applies โ€” its own and its peers' โ€” is appended to +`/log/` before the record is written, so a crash between the two is +recovered by replaying the log on the next open: + +``` +log/ +โ”œโ”€โ”€ open/.json one file per appended batch, never rewritten +โ”œโ”€โ”€ seg-.json sealed segments, rolled from open/ every `rollAfter` batches +โ””โ”€โ”€ index.json { epoch, floor, vector, segments } +``` + +The log has two cursors, for two different questions. + +**Where am I in *your* log?** An **event id** โ€” `.` โ€” names a +position in one node's log. It is what a subscriber hands back to resume, and +it is exactly what wrpc carries as a subscription's `lastEventId`. The epoch is +minted when the log is created, so an id from a wiped or restored directory is +recognised as foreign rather than mistaken for a position. + +**What have I applied?** A **version vector** โ€” `{ [origin]: seq }` โ€” is the +highest sequence number applied per origin. Sequence numbers run per origin +*and scope*, so a node syncing one scope sees no holes; the key is then +`origin@scope`. Ops from one origin are applied in order; an op that skips +ahead is set aside until the ones before it arrive. + +Retention bounds the log: `retention.ops` (default 10 000) and +`retention.days` drop the oldest sealed segments, and `floor` moves up. A +peer whose cursor fell below the floor cannot be resumed โ€” it is told to +[bootstrap](#bootstrap) instead. Size retention for the longest outage a peer +should be able to sit out. + +## Protocol + +A node **dials** the nodes in its registry through `connect(node)`; the node +on the other end **hosts** a unit of procedures. Both roles are the same class; +which one a node plays on a given link is just who opened the connection. + +| Procedure | Direction | Purpose | +| --- | --- | --- | +| `hello` | dialer โ†’ host | Exchange node ids, vectors and log epochs; the host may `authorize` | +| `push` | dialer โ†’ host | Ops the host lacks, by its vector | +| `changes` (subscription) | host โ†’ dialer | Everything that lands in the host's log from the cursor on, live | +| `snapshot` | dialer โ†’ host | One page of the host's state, for bootstrap | + +```mermaid +sequenceDiagram + autonumber + participant D as dialer + participant H as host + D->>H: hello { node, vector, scopes, token } + H-->>D: { node, vector, epoch } + D->>H: push { ops the host lacks } + D->>H: subscribe changes { lastEventId } + H-->>D: data ร— n (replay from the cursor) + loop live + D->>H: push (local and relayed ops) + H-->>D: data (ops landing on the host) + end +``` + +A push carries the dialer's own writes **and** what it received from its other +peers, so ops travel across the mesh hop by hop. The version vector stops them +looping: an op a node already has is dropped, not re-sent. Each op reaches a +node at most as many times as that node has links. + +The subscription's cursor is remembered per registered node. Reconnecting, +backing off and re-opening the subscription from the last event id are the +**link's** job โ€” a wrpc client already does all three. After a reconnect the +manager only repeats `hello` + `push`, the outbound half a transport cannot know +about. + +### Bootstrap + +A dialer whose cursor the host cannot resume โ€” a brand-new node, or one that was +away longer than the host's retention โ€” receives `{ bootstrap: true }` on the +subscription, and switches to `snapshot`. The host pages through **every** +entry, tombstones included, with its merge metadata. On a binary-capable +transport a page is streamed as NDJSON with backpressure; otherwise it comes +inline. The dialer joins each record (`mergeEntry`), adopts the host's vector as +of the first page, and re-subscribes from that page's event id. Whatever landed +while paging replays and folds in as a no-op. + +The joined state enters the dialer's own log as `merge` and `advance` ops, so +a third node resuming from the dialer gets it too โ€” bootstrap is transitive. +Those ops never travel back to the node the snapshot came from. + +## Scopes + +By default a storage is one scope, `'*'`, and every peer gets everything. To +replicate a subset, assign records to scopes and register the peer with the +scopes it should carry: + +```js +const storage = await open({ + path: 'data', + sync: { + connect, + scope: (id, data) => data.$type ?? '*', // a record's scope, from its data + }, +}); + +await storage.sync.addNode({ url, scopes: ['Doc'] }); // Doc records, plus everything global +``` + +An unscoped record (scope `'*'`) reaches every peer. On a wrpc host the +subscriber joins the room `syncom:` for each scope it asked for, so +`server.rooms.count('syncom:Doc')` and the cluster's `fetchClients({ room })` +describe the sync topology without any code in syncom. Delivery itself stays +on the subscription โ€” a room broadcast is at-most-once and cannot be resumed. + +A node that carries one scope can only relay that scope: a peer wanting more +has to reach a fuller node. + +## The registry + +```js +await storage.sync.addNode({ url: 'wss://peer/api', token: 'secret', scopes: ['*'] }); +await storage.sync.enableNode('wss://peer/api', false); // keep, but do not dial +await storage.sync.removeNode('wss://peer/api'); storage.sync.listNodes(); -// [{ type: 'uplink', url: 'ws://example.com:8001', token: 'secret', -// enabled: true, lastSync: 1757000000000 }] +// [{ url, token, scopes, enabled, lastSync, cursor, remote: { id, epoch } }] +``` -await storage.sync.removeNode('ws://example.com:8001'); +Each node is a file under `/remote/`, named by its percent-encoded url, +on Node.js and in the browser alike. The `cursor` and the remote node's identity +are kept there too, which is why a restart resumes instead of bootstrapping. + +`url` is whatever `connect(node)` understands โ€” syncom never parses it. + +## Lifecycle and events + +```js +await storage.sync.start(); // dial every enabled node; automatic when `connect` is given +await storage.sync.stop(); // close every link +await storage.sync.sync(); // push what the peers lack right now +storage.sync.peers(); // [{ url, node, connected, subscribed, cursor, lastSync, pending }] + +storage.sync.on('peer', ({ url, connected, node, reconnected }) => {}); +storage.sync.on('sync', ({ url, pushed, received, applied, duplicates, gaps }) => {}); +storage.sync.on('bootstrap', ({ url, records }) => {}); +storage.sync.on('error', (error) => {}); // error.url names the peer + +storage.on('change', (op, via) => {}); // every op applied here; via is null for a local write +storage.on('error', (error) => {}); // a peer's op refused by the register/counter rule +``` + +`await storage.close()` stops sync and seals the log. + +## Options + +| `sync.โ€ฆ` | Default | Meaning | +| --- | --- | --- | +| `connect(node)` | โ€” | Returns a link (or a promise of one) for a registry node. Required to dial. | +| `autoStart` | `true` when `connect` is given | Dial on open | +| `scope(id, data)` | `() => '*'` | The scope of a record | +| `retention.ops` | `10000` | Ops kept in the log before the oldest segment is dropped | +| `retention.days` | โ€” | Age after which a segment is dropped | +| `log.rollAfter` | `64` | Appended batches per sealed segment | +| `pageSize` | `500` | Ops per push, records per snapshot page | + +## Hosting + +The unit a wrpc server (or anything shaped like one) mounts: + +```js +const unit = storage.sync.unit({ + access: 'public', // wrpc's default is 'session' + authorize: (context, hello) => hello.token === TOKEN, // the node's token, or context.sessionโ€ฆ +}); ``` -| Field | Meaning | -| --- | --- | -| `type` | `'uplink'` (a peer this node pulls from) or `'downlink'` (a peer that pulls from this one) | -| `url` | Required, and unique โ€” adding a duplicate throws | -| `token` | Optional shared secret, stored as given | -| `enabled` | Set to `true` on registration | -| `lastSync` | Timestamp, set on registration | +| Option | Default | Meaning | +| --- | --- | --- | +| `access` | `'session'` | Applied to every procedure, wrpc-style | +| `authorize(context, hello)` | โ€” | Gate the handshake; `hello.token` is the dialer's registered token | +| `tracked` | built in | wrpc's `tracked()`; the built-in produces the same shape | +| `pageSize` | `500` | Records per snapshot page | +| `pageBytes` | `4 MiB` | Above this a page is sent inline rather than streamed | -`addNode` throws on a missing `url`, on a `type` outside those two values, and -on a `url` already registered. `removeNode` throws for an unknown url. +See [wrpc transport](./wrpc) for mounting it. -## Persistence +## The link contract -On Node.js each node is written to `/remote/.json`, where the host -is taken from the url, and the registry is reloaded from that directory when -the storage is reopened. The browser `SyncManager` keeps its nodes in memory -only โ€” nothing is written to OPFS. +Anything with this shape can carry the protocol; `isSyncLink()` is the check: -## What a transport would need +```ts +interface SyncLink { + call(method: string, args: unknown): Promise; + subscribe(method, args, { lastEventId, onData(value, eventId), onError, onEnd }): { unsubscribe() }; + stream?(id: string): AsyncIterable; // binary-capable transports + on(event: 'open' | 'close' | 'reconnect', listener): void; + off(event, listener): void; + close(): void; +} +``` -The pieces the registry is built around, for when a transport lands: +`tests/utils/memory-link.js` in the repository is an in-process implementation +the engine's own tests run on. -- `Storage` accumulates an update log (`{ type, id, timestamp, data }` - entries) from every `set`, `update`, `swap` and `Record.save()`. -- [`apply`](./crdt#apply-source-history) replays such a log onto a state, and - [`merge`](./crdt#merge-source-delta) reconciles concurrent deltas. -- The [chain](./chain) gives each peer a verifiable ordering to compare - against. +## Guarantees and limits -None of that is wired to a socket today. +- **Convergence.** Two nodes that have applied the same set of ops hold the + same records, whatever the order โ€” except the documented counter/reset case. +- **At-least-once delivery, idempotent application.** An op may arrive more + than once; applying it again changes nothing. +- **A node's own sequence never repeats.** The log is written before the op is + announced, so a crash cannot mint two different ops with one number. +- **No fsync.** Files are written with the platform's default durability; on + OPFS there is no rename either, so a torn write can only affect the newest, + unannounced batch. +- **Encrypted records do not sync.** They are sealed for this node's key alone + โ€” see [Storage](./storage#encryption-at-rest). +- **Chains diverge.** Each node's [chain](./chain) is its own audit log of what + it applied, in the order it applied it. It is not a consensus. +- **No signatures.** Trust rests on `authorize`, the token and the transport + (use `wss://`). diff --git a/docs/guide/wrpc.md b/docs/guide/wrpc.md new file mode 100644 index 0000000..6ee7932 --- /dev/null +++ b/docs/guide/wrpc.md @@ -0,0 +1,136 @@ +# wrpc transport + +[wrpc](https://wrpc.vercel.app) is the transport syncom is built to run on: +WebSocket RPC with subscriptions that resume, reconnection with backoff, rooms +and binary streams โ€” everything the [sync protocol](./sync#protocol) leaves to +the link. The adapter lives in the `@alexify/syncom/wrpc` subpath and takes a +`WrpcClient` you construct; syncom never imports wrpc, so its dependency count +stays at zero and wrpc stays on your version. + +```bash +npm install @alexify/syncom @alexify/wrpc +``` + +## Hosting a node + +`storage.sync.unit()` returns a plain object of procedures shaped for +`defineRouter`. Mount it under `syncom`: + +```js +const { Server, defineRouter } = require('@alexify/wrpc'); +const { open } = require('@alexify/syncom'); + +const storage = await open({ path: 'data' }); + +const router = defineRouter({ + syncom: storage.sync.unit({ + access: 'public', + authorize: (context, hello) => hello.token === process.env.SYNC_TOKEN, + }), +}); + +const server = new Server({ router, host: '0.0.0.0', port: 8000, protocol: 'http' }); +await server.listen(); +``` + +`access` defaults to wrpc's own default, `'session'`, which refuses every peer +until one has a session. Either open it with `'public'` and gate with +`authorize`, or keep sessions and sign peers in through wrpc's +[`authenticate`](https://wrpc.vercel.app/guide/client#authenticating) hook โ€” +`context.session` is available inside `authorize` too. + +## Dialing a node + +```js +const { WrpcClient } = require('@alexify/wrpc'); +const { open } = require('@alexify/syncom'); +const { createWrpcLink } = require('@alexify/syncom/wrpc'); + +const storage = await open({ + path: 'data', + sync: { + connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)), + }, +}); + +await storage.sync.addNode({ url: 'ws://hub.example.com:8000/api', token: process.env.SYNC_TOKEN }); +``` + +`connect(node)` is called once per registered node and again whenever the node +is re-enabled, so every dial gets a fresh client. Keep wrpc's `reconnect` and +`heartbeat` defaults: reconnecting is the client's job, and after it the +subscription resumes from its `lastEventId` by itself โ€” syncom only repeats +its handshake to push what it wrote while offline. + +A node can be both. The same storage can mount its unit on a server **and** +dial other nodes; that is what makes a [mesh](./sync#protocol). + +## In the browser + +The subpath has no Node.js code, so the same two imports work in a bundle: + +```js +import { open } from '@alexify/syncom'; +import { createWrpcLink } from '@alexify/syncom/wrpc'; +import { WrpcClient } from '@alexify/wrpc'; + +const storage = await open({ + path: 'app', + sync: { connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)) }, +}); +await storage.sync.addNode({ url: 'wss://hub.example.com/api', token }); +``` + +A browser node dials; it cannot host. It joins the mesh through whichever +nodes it can reach, and a bootstrap arrives as a binary stream through +wrpc's `WrpcReadable`. + +## What the adapter does + +`createWrpcLink(client, { unit })`: + +- `call('syncom/x', args)` โ†’ `client.call('syncom/x', args)` โ€” no `load()` + needed. +- `subscribe('syncom/changes', โ€ฆ)` โ†’ `client.use(SYNC_INTROSPECTION)` (no + round trip) then `client.api.syncom.changes.subscribe(args, { lastEventId, + โ€ฆ })`. wrpc updates the handle's `lastEventId` from each packet before it + delivers the value, and the adapter passes that id along with it. +- `stream(id)` โ†’ `client.getStream(id)`, the async-iterable `WrpcReadable` a + streamed snapshot page arrives on. +- `open` / `close` / `reconnect` events and `close()` are the client's own. + +`unit` renames the router unit when you mounted it under something other than +`syncom`. `isSyncLink()` is exported too, and `SYNC_INTROSPECTION` if you want +to scaffold the unit yourself. + +## Rooms and presence + +On `hello` the host joins the peer to `syncom:` for every scope it asked +for, so wrpc's rooms and [cluster](https://wrpc.vercel.app/guide/cluster) +describe the sync topology for free: + +```js +server.rooms.count('syncom:*'); // peers on this instance +await server.cluster.fetchClients({ room: 'syncom:Doc' }); // across instances +server.cluster.disconnect({ room: 'syncom:*' }); // drop every peer +``` + +Delivery does **not** go through rooms: a room broadcast is at-most-once and +cannot be resumed, which is exactly what the subscription provides. + +## Why no codec + +wrpc's [wire codec](https://wrpc.vercel.app/guide/codec) is a connection-wide, +text-only setting your application owns. syncom's ops already travel packed +(one positional array per op, no repeated keys), permessage-deflate compresses +the rest, and the storage is JSON on both ends โ€” there is no binary +representation to preserve. Set a codec if your application wants one; syncom +neither needs nor notices it. + +## Testing + +The pattern from wrpc's own [testing guide](https://wrpc.vercel.app/guide/testing) +applies unchanged โ€” `port: 0`, `logger: false`, teardown registered with +`t.after` the moment the server is up. syncom's +`tests/wrpc.integration.test.js` boots two servers and a dialer that way and +is skipped, never failed, when `@alexify/wrpc` is not installed. diff --git a/docs/index.md b/docs/index.md index 1a7e68e..0259787 100644 --- a/docs/index.md +++ b/docs/index.md @@ -25,6 +25,16 @@ features: details: Every write appends a hash-linked block. Reading a record re-derives its hash and checks it against the block that committed it, so silent corruption surfaces as an error instead of bad data. link: /guide/chain linkText: Blockchain log + - icon: ๐Ÿ”„ + title: Replication without a coordinator + details: Every write is an operation โ€” last-writer-wins registers, per-origin counters, tombstones โ€” logged locally and exchanged with peers over an injected transport. Any order, any number of times, same result. + link: /guide/sync + linkText: Sync + - icon: ๐Ÿ”Œ + title: wrpc as the transport + details: Mount storage.sync.unit() on a wrpc server and dial it with a WrpcClient. Reconnects, resumable subscriptions and binary snapshot streams come from wrpc; syncom depends on nothing. + link: /guide/wrpc + linkText: wrpc transport - icon: ๐Ÿงฉ title: CRDT deltas details: diff, merge and apply operate on plain objects โ€” nested maps merge recursively, arrays behave as grow-only sets, and numbers merge by addition, so counters converge without coordination. @@ -47,5 +57,5 @@ features: linkText: Browser - icon: ๐Ÿ“ฆ title: Zero runtime dependencies - details: Nothing is installed alongside it. The browser bundle is under 5 KB min+gzip, and CI fails the build if that budget is exceeded. + details: Nothing is installed alongside it. The browser bundle is under 14 KB min+gzip, and CI fails the build if that budget is exceeded. --- diff --git a/docs/public/llms.txt b/docs/public/llms.txt index f295d37..9cf18a0 100644 --- a/docs/public/llms.txt +++ b/docs/public/llms.txt @@ -1,32 +1,40 @@ # @alexify/syncom > Fast, zero-dependency distributed data sync engine for JavaScript. Local-first -> storage with a hash-linked integrity log, CRDT deltas, and JavaScript smart +> storage with a hash-linked integrity log, an operation log replicated between +> peers over an injected transport (wrpc), CRDT deltas, and JavaScript smart > contracts โ€” one API across Node.js and the browser. ## Key facts - Package: `@alexify/syncom` (npm), MIT, source at https://github.com/Alexis-Technologies/syncom - Requires Node.js >= 22, or a browser with OPFS and Web Crypto (secure context) -- **Zero runtime dependencies.** Browser bundle is ~4.7 KB min+gzip +- **Zero runtime dependencies.** Browser bundle is ~13 KB min+gzip; the wrpc adapter under 1 KB - CommonJS, no build step; `src/` ships as published -- Entry points: `.` (Node `index.js` / browser `browser.js` via the `browser` export condition) -- Exports: `open`, `Storage`, `Blockchain`, `SmartContract`, `DataReader`, `SyncManager`, - `diff`, `merge`, `apply`, `calculateHash`, `generateKeys`, `encrypt`, `decrypt`, - `loadKeys`, `read`, `write`, `exists`, `remove` -- `Storage`, `Blockchain` and `SyncManager` initialise asynchronously: `await new Blockchain(path)` +- Entry points: `.` (Node `index.js` / browser `browser.js` via the `browser` export condition) and `./wrpc` +- Exports: `open`, `Storage`, `SyncManager`, `isSyncLink`, `OpLog`, `applyOp`, `mergeEntry`, `isAlive`, + `Blockchain`, `SmartContract`, `DataReader`, `diff`, `merge`, `apply`, `calculateHash`, `generateKeys`, + `encrypt`, `decrypt`, `loadKeys`, `read`, `write`, `exists`, `remove`, `list` +- `@alexify/syncom/wrpc` exports `createWrpcLink`, `isSyncLink`, `SYNC_INTROSPECTION` +- `Storage`, `Blockchain`, `SyncManager` and `OpLog` initialise asynchronously: `await new Blockchain(path)` - In the browser `calculateHash`, `encrypt` and `decrypt` return promises; in Node.js they are synchronous -- `merge` ADDS numbers rather than replacing them โ€” this is the counter CRDT and is deliberate -- **Not implemented:** `storage.sync` registers and persists peer nodes, but no - transport exists; `sync.start()` is a no-op +- Every write is an op: `write` (insert/set), `set` (update/swap/record.save), `inc` (counters), `delete` (tombstone). + A field is a last-writer-wins register OR a per-origin counter, never both โ€” mixing throws +- Replication: every node dials the nodes in its registry through `sync.connect(node)` and hosts + `storage.sync.unit()`; ops relay hop by hop, de-duplicated by version vector; a peer that cannot be + resumed bootstraps from a snapshot. wrpc is injected, never a dependency +- `merge` ADDS numbers rather than replacing them โ€” this is the counter CRDT and is deliberate; storage + does not replicate through `merge` +- Encrypted records (`encrypted: true`) are sealed for this node's key and never leave it ## Documentation -- [Getting Started](https://syncomjs.vercel.app/guide/getting-started): install, open a storage, records, validation. -- [Storage](https://syncomjs.vercel.app/guide/storage): explicit API, compare-and-swap, records, references, schema collections, on-disk layout. +- [Getting Started](https://syncomjs.vercel.app/guide/getting-started): install, open a storage, records, validation, a first sync. +- [Storage](https://syncomjs.vercel.app/guide/storage): explicit API, registers vs counters, compare-and-swap, records, references, schema collections, encryption at rest, on-disk layout. +- [Sync](https://syncomjs.vercel.app/guide/sync): the operation model, the log and its two cursors, the protocol, bootstrap, scopes, the registry, the link contract, guarantees and limits. +- [wrpc transport](https://syncomjs.vercel.app/guide/wrpc): hosting `storage.sync.unit()` on a wrpc server, dialing with `createWrpcLink`, the browser, rooms and presence. - [CRDT deltas](https://syncomjs.vercel.app/guide/crdt): diff/merge/apply semantics, grow-only sets, the numeric-addition rule. - [Blockchain log](https://syncomjs.vercel.app/guide/chain): block structure, hash linking, validate cost, per-record verification. - [Smart contracts](https://syncomjs.vercel.app/guide/contracts): storing procedures in the chain, DataReader, the `new Function` risk, serialisation limits. -- [Sync](https://syncomjs.vercel.app/guide/sync): the node registry and what a transport would still need. - [Browser](https://syncomjs.vercel.app/guide/browser): OPFS and Web Crypto, the async signature differences, bundle size. - [API reference](https://syncomjs.vercel.app/reference/api): every export with signatures and return types. diff --git a/docs/reference/api.md b/docs/reference/api.md index e79e586..cd6a89b 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -2,7 +2,7 @@ Everything below is exported from the package root. The browser build exports the same names โ€” see [Browser](../guide/browser) for the three signatures that -differ. +differ. The wrpc adapter is a separate subpath, [`@alexify/syncom/wrpc`](#wrpc-adapter). ```js const syncom = require('@alexify/syncom'); @@ -13,39 +13,52 @@ const syncom = require('@alexify/syncom'); ### `open(options?)` Returns `Promise`. Options: `path` (default `'syncom'`), `name` -(alias for `path`), `schema`, `getId`. +(alias for `path`), `schema`, `getId`, `logger`, `encrypted`, `sync` (see +[Sync options](../guide/sync#options)). ### `Storage` -`await new Storage(options)` โ€” `open()` is the shorthand. +`await new Storage(options)` โ€” `open()` is the shorthand. Extends `Emitter`. | Method | Returns | Notes | | --- | --- | --- | -| `insert(data)` | `Promise` | Generates an id via `getId` (a UUID by default) and saves | -| `get(id)` | `Promise` | `null` when absent; verifies against the chain | -| `has(id)` | `Promise` | Filesystem check, no read | -| `set(id, data)` | `Promise` | Replaces; emits `update` with the computed delta | -| `update(id, delta)` | `Promise` | Shallow merge; throws if the record is absent | +| `insert(data, options?)` | `Promise` | Generates an id via `getId` (a UUID by default) and writes | +| `get(id)` | `Promise` | `null` when absent or deleted; verifies against the chain | +| `has(id)` | `Promise` | `false` for a tombstone | +| `list()` | `Promise` | Every live id | +| `set(id, data, options?)` | `Promise` | Replaces the record (a `write` op) | +| `update(id, delta)` | `Promise` | Sets the listed fields (a `set` op); throws if absent, or on a counter field | +| `inc(id, key, value = 1)` | `Promise` | Adds to a counter field (an `inc` op); throws on a register field | | `swap(id, changes, prev)` | `Promise` | CAS โ€” `false`, and no write, on mismatch | -| `delete(id)` | `Promise` | | +| `delete(id)` | `Promise` | Leaves a tombstone | | `record(id)` | `Promise` | Cached per id | -| `saveData(id, data, options?)` | `Promise` | `{ encrypted: true }` for RSA at rest | -| `loadData(id)` | `Promise` | Throws if the record no longer matches its block | +| `saveData(id, data, options?)` | `Promise` | `{ encrypted: true }` seals the record for this node only | +| `loadData(id)` | `Promise` | Alias of `get()`; throws if the record no longer matches its block | | `validate(options?)` | `Promise` | `{ last, from }`; no argument walks the whole chain | | `validate(id, data, blockHash)` | `Promise` | One record against its block | | `getCachedData(id)` | `unknown \| null` | Synchronous cache peek | | `hasRecord(id)` | `boolean` | Whether the id is known to the cache | -| `addUpdate(update)` | `void` | Appends to the pending update log | +| `applyRemote(ops, { via, scopes }?)` | `Promise<{ applied, duplicates, gaps }>` | Applies a peer's ops, in order per origin | +| `snapshot({ cursor, limit, scopes }?)` | `Promise<{ records, cursor, eventId, vector }>` | One page of state, tombstones included | +| `mergeSnapshot(records, { via, from }?)` | `Promise` | Joins a page (as `merge` ops of this node) | +| `adoptVector(vector, { via, from }?)` | `Promise` | Records what a joined snapshot covers (an `advance` op) | +| `close()` | `Promise` | Stops sync and seals the log; further writes are refused | -Properties: `schema`, `sync`. Statics: `Storage.Collection`, `Storage.Record`, -`Storage.generateId()` โ€” the default `getId`, synchronous in both halves. +Properties: `schema`, `sync`, `nodeId`, `path`, `log`, `vector`, `logger`. +Statics: `Storage.Collection`, `Storage.Record`, `Storage.generateId()` โ€” the +default `getId`, synchronous in both halves. + +Events: `change` `(op, via)` for every op applied here (`via` is `null` for a +local write, otherwise the link it came from); `error` for a peer's op refused +by the register/counter rule. ### `Record` -Extends `Emitter`. Emits `update` with `(data, delta)`. +Extends `Emitter`. Emits `update` with `(data, delta)` โ€” on a staged +assignment, and when the record changes in storage (`null` data once deleted). -`id`, `data()`, `delta()`, `save()`, `delete()`, plus a live property per -field of the loaded object. +`id`, `data()`, `delta()`, `save()`, `inc(key, value?)`, `delete()`, plus a +live property per field of the loaded object. ### `Collection` @@ -54,6 +67,54 @@ Created per schema entity and exposed as `storage.`. `name`, `insert(data)`, `get(id)`, `update(id, delta)`, `delete(id)`, `record(id)`. +## Sync + +### `SyncManager` + +`storage.sync`; `await new SyncManager(storage, options)` builds one by hand. +Extends `Emitter`. See [Sync](../guide/sync). + +| Method | Returns | Notes | +| --- | --- | --- | +| `addNode({ url, token?, scopes? })` | `Promise` | Registers a peer to dial; dials it at once when started | +| `removeNode(url)` | `Promise` | Hangs up too | +| `enableNode(url, enabled = true)` | `Promise` | Keep a node without dialing it | +| `listNodes()` | `Node[]` | `{ url, token, scopes, enabled, lastSync, cursor, remote }` | +| `start()` | `Promise` | Dials every enabled node; needs `sync.connect` | +| `stop()` | `Promise` | Closes every link | +| `sync(url?)` | `Promise>` | Pushes what the peers lack now | +| `peers()` | `Array<{ url, node, connected, subscribed, cursor, lastSync, pending }>` | Live view | +| `unit({ access, authorize, tracked, pageSize, pageBytes }?)` | router unit | The procedures a host mounts: `hello`, `push`, `snapshot`, `changes` | + +Properties: `storage`, `started`. Events: `peer`, `sync`, `bootstrap`, `error`. + +### `isSyncLink(value)` + +Structural check for the [link contract](../guide/sync#the-link-contract). + +## Operation log and model + +### `OpLog` + +`await new OpLog(basePath, { rollAfter, retention, logger })` โ€” `storage.log` +is the one you want. + +| Member | Notes | +| --- | --- | +| `epoch`, `floor`, `next`, `vector`, `length`, `lastEventId` | See [The log](../guide/sync#the-log) | +| `append(ops)` | `Promise` โ€” entries with their seqs, once on disk | +| `since(eventId, { scopes, limit }?)` | `LogEntry[]` after the id, or `null` when it cannot be resumed | +| `advance(vector)` | Adopts a vector (after a snapshot) | +| `tail()` | The unsealed entries a reopen replays | +| `flush()` | Seals the open files now | +| `close()` | Writes the index | + +### `applyOp(entry, op)` / `mergeEntry(a, b)` / `isAlive(entry)` + +The record-level model: apply one op to an entry (`{ entry, changed, delta, +rejected }`), join two entries, and whether an entry is a live record or a +tombstone. Pure functions; `entry` may be `null`. + ## CRDT | Function | Returns | @@ -77,7 +138,7 @@ replace. | `readBlock(hash)` | `Promise` โ€” throws if unknown | | `writeBlock(block)` | `Promise` | | `writeChain()` | `Promise` โ€” persists the tail pointer | -| `validate(options?)` | `Promise` โ€” `{ last, from }` | +| `validate(options?)` | `Promise` โ€” `{ last, from }`; an unknown `from` is `false` | Properties: `path`, `tailHash`. A block is `{ prev, timestamp, data }`. @@ -112,27 +173,28 @@ storage access a contract has. | `decrypt(data, privateKey)` | `unknown` | `Promise` | | `calculateHash(data)` | `string` | `Promise` | -`KeyPair` is `{ publicKey, privateKey }`, PEM-encoded RSA-2048. -`loadKeys` reads `public.pem` / `private.pem` from `basePath`, generating and -writing them if absent. +`KeyPair` is `{ publicKey, privateKey }`, PEM-encoded RSA-2048. `encrypt` +returns a JSON envelope โ€” AES-256-GCM under a fresh key, that key wrapped with +RSA-OAEP (SHA-256) โ€” identical on both platforms. `loadKeys` reads +`public.pem` / `private.pem` from `basePath`, generating and writing them if +either is absent. ## Filesystem interface -Backed by `node:fs/promises` on the server and OPFS in the browser. All four -take a directory and a filename. +Backed by `node:fs/promises` on the server and OPFS in the browser. `read(dir, file)` โ†’ `Promise`, `write(dir, file, content)`, -`exists(dir, file)` โ†’ `Promise`, `remove(dir, file)`. +`exists(dir, file)` โ†’ `Promise`, `remove(dir, file)`, +`list(dir)` โ†’ `Promise` (file names, sorted; an absent directory is +empty). `write` creates the directory; `remove` is a no-op when the file is absent. -## Sync - -### `SyncManager` +## wrpc adapter -`await new SyncManager(basePath, storage, chain)` โ€” `storage.sync` is the one -you want. - -`addNode(nodeData)`, `removeNode(url)`, `listNodes()`, `start()`. +```js +const { createWrpcLink, isSyncLink, SYNC_INTROSPECTION } = require('@alexify/syncom/wrpc'); +``` -See [Sync](../guide/sync) โ€” `start()` is a no-op today. +`createWrpcLink(client, { unit = 'syncom' }?)` wraps a `WrpcClient` into a +[link](../guide/sync#the-link-contract); see [wrpc transport](../guide/wrpc). diff --git a/index.d.ts b/index.d.ts index 21504d7..3378927 100644 --- a/index.d.ts +++ b/index.d.ts @@ -20,13 +20,112 @@ export function read(dir: string, filename: string): Promise; export function write(dir: string, filename: string, content: string): Promise; export function exists(dir: string, filename: string): Promise; export function remove(dir: string, filename: string): Promise; +export function list(dir: string): Promise; + +// --------------------------------------------------------------------------- +// Replication model (src/ops.js, src/vector.js, src/log.js) + +/** `[ts, origin]` โ€” a millisecond stamp with the node id as the tie-break. */ +export type ClockStamp = readonly [number, string]; + +/** The highest per-origin sequence number a node has applied. */ +export type VersionVector = { readonly [origin: string]: number }; + +export type SyncOpType = 'write' | 'set' | 'inc' | 'delete' | 'merge' | 'advance'; + +export interface SyncOp { + origin: string; + seq: number; + ts: number; + type: SyncOpType; + /** The record; absent only on `advance`. */ + id?: string; + scope?: string; + /** `advance`: the vector a joined snapshot covered. */ + vector?: VersionVector; + /** `merge`/`advance`: the node whose snapshot this came from. */ + from?: string; + /** `write`/`set`: the fields; `merge`: the joined entry's data. */ + data?: Record; + /** `merge`: the joined entry's metadata. */ + meta?: EntryMeta; + /** `inc`: the counter field. */ + key?: string; + /** `inc`: the amount added. */ + value?: number; +} + +export interface EntryMeta { + clock: { [field: string]: ClockStamp }; + base: ClockStamp | null; + deleted: ClockStamp | null; + counters: { [field: string]: { [origin: string]: readonly [number, number, number] } }; +} + +export interface Entry { + data: Record; + meta: EntryMeta; +} + +export interface ApplyResult { + entry: Entry; + changed: boolean; + delta: Record; + rejected: Array<{ field: string; reason: 'register' | 'counter' }>; +} + +export function applyOp(entry: Entry | null, op: SyncOp): ApplyResult; +export function mergeEntry(a: Entry | null, b: Entry | null): Entry | null; +export function isAlive(entry: Entry | null): boolean; + +export interface LogEntry { + seq: number; + op: SyncOp; +} + +export interface OpLogOptions { + rollAfter?: number; + retention?: { ops?: number; days?: number }; + logger?: Logger | false; +} + +export interface OpLog { + readonly epoch: string; + readonly floor: number; + readonly next: number; + readonly vector: VersionVector; + readonly length: number; + readonly lastEventId: string; + formatId(seq: number): string; + append(ops: readonly SyncOp[]): Promise; + since(eventId: string | null | undefined, options?: { scopes?: readonly string[] | null; limit?: number }): LogEntry[] | null; + advance(vector: VersionVector): Promise; + tail(): LogEntry[]; + flush(): Promise; + close(): Promise; +} + +export interface OpLogConstructor { + readonly prototype: OpLog; + new (basePath: string, options?: OpLogOptions): Promise; +} + +export const OpLog: OpLogConstructor; + +/** A Console, a pino-shaped logger, or `false` for silence. */ +export interface Logger { + debug?(...args: unknown[]): void; + info?(...args: unknown[]): void; + warn?(...args: unknown[]): void; + error?(...args: unknown[]): void; +} export type CrdtDelta = Record; export type CrdtHistoryRecord = | { type: 'write'; id?: string; timestamp?: number; data?: unknown } | { type: 'delta'; data: unknown } - | { type: 'delete'; key: string } + | { type: 'delete'; key?: string; id?: string; timestamp?: number } | { type: 'inc'; key: string; value: number } | { type: 'dec'; key: string; value: number }; @@ -122,13 +221,33 @@ export interface Schema { readonly [key: string]: unknown; } -export interface StorageOptions { +export interface SyncOptions { + /** Assigns a scope to a record; unscoped (`'*'`) records reach every peer. */ + scope?: (id: string, data: Record) => string; + retention?: { ops?: number; days?: number }; + log?: { rollAfter?: number }; +} + +export interface StorageOptions { path?: string; name?: string; - schema?: Schema; + schema?: S; getId?: () => string | Promise; + logger?: Logger | false; + sync?: SyncOptions; + /** Encrypt every record at rest by default (records then stay local). */ + encrypted?: boolean; } +// The collections a schema turns into properties on the storage object: +// `open({ schema: { entities: { Post: {} } } })` yields `storage.Post`. A +// `Map` of entities is opaque to the type system and adds nothing. +export type Collections = S['entities'] extends Map + ? {} + : S['entities'] extends { readonly [entityName: string]: SchemaEntity } + ? { readonly [K in keyof S['entities'] & string]: SyncomCollection } + : {}; + export interface StorageDataOptions { encrypted?: boolean; } @@ -141,47 +260,87 @@ export interface StorageEntry { } export interface Node { - type: 'uplink' | 'downlink'; url: string; token?: string; + scopes: string[]; enabled: boolean; - lastSync: number; + lastSync: number | null; } export interface NodeData { url: string; - type: 'uplink' | 'downlink'; token?: string; + scopes?: string[]; } export interface SyncManager { + readonly storage: Storage; addNode(nodeData: NodeData): Promise; removeNode(url: string): Promise; listNodes(): Node[]; start(): Promise; + stop(): Promise; } export interface SyncManagerConstructor { readonly prototype: SyncManager; - new ( - basePath?: string, - storage?: Storage, - chain?: Blockchain, - ): Promise; + new (storage: Storage, options?: SyncOptions): Promise; } export const SyncManager: SyncManagerConstructor; -export interface StorageUpdate { - type: string; - id: string; - timestamp: number; - data: unknown; +/** The structural check on an injected link: call/subscribe/on/off/close. */ +export function isSyncLink(value: unknown): value is SyncLink; + +/** + * What the sync engine needs from a transport. Reconnecting, backing off and + * resuming the subscription from its last event id are the link's job โ€” a + * WrpcClient does all three; `@alexify/syncom/wrpc` wraps one into this. + */ +export interface SyncLink { + call(method: string, args: unknown): Promise; + subscribe( + method: string, + args: unknown, + handlers: { + lastEventId?: string; + onData?: (value: unknown, eventId?: string) => void | Promise; + onError?: (error: Error & { code?: number }) => void; + onEnd?: () => void; + }, + ): { unsubscribe(): void; readonly lastEventId?: string }; + /** Binary-capable transports: the chunks of a streamed snapshot page. */ + stream?(id: string): AsyncIterable; + on(event: 'open' | 'close' | 'reconnect', listener: (...args: unknown[]) => void): void; + off(event: 'open' | 'close' | 'reconnect', listener: (...args: unknown[]) => void): void; + close(): void; +} + +export interface SnapshotPage { + records: Array; + cursor: string | null; + eventId: string; + vector: VersionVector; +} + +export interface ApplyRemoteResult { + applied: number; + duplicates: number; + gaps: number; } export interface Storage { readonly schema: Schema | null; readonly sync: SyncManager; + readonly nodeId: string; + readonly path: string; + readonly log: OpLog; + readonly vector: VersionVector; + + /** Every op applied here โ€” local (`via` null) or from a peer (`via` = the link). */ + on(event: 'change', listener: (op: SyncOp, via: unknown) => void): void; + on(event: 'error', listener: (error: Error & { op?: SyncOp }) => void): void; + off(event: 'change' | 'error', listener?: (...args: unknown[]) => void): void; saveData( id: string, @@ -191,26 +350,45 @@ export interface Storage { loadData(id: string): Promise; validate(options?: { last?: number; from?: string }): Promise; validate(id: string, data: unknown, blockHash: string): Promise; - insert(data: unknown): Promise; + insert(data: unknown, options?: StorageDataOptions): Promise; has(id: string): Promise; get(id: string): Promise; - set(id: string, data: unknown): Promise; + set(id: string, data: unknown, options?: StorageDataOptions): Promise; delete(id: string): Promise; update(id: string, delta: unknown): Promise; + inc(id: string, key: string, value?: number): Promise; swap(id: string, changes: unknown, prev: unknown): Promise; + list(): Promise; record(id: string): Promise; getCachedData(id: string): unknown | null; hasRecord(id: string): boolean; - addUpdate(update: StorageUpdate): void; + applyRemote( + ops: readonly SyncOp[], + options?: { via?: unknown; scopes?: readonly string[] | null }, + ): Promise; + adoptVector(vector: VersionVector, options?: { via?: unknown; from?: string }): Promise; + snapshot(options?: { cursor?: string | null; limit?: number; scopes?: readonly string[] | null }): Promise; + mergeSnapshot(records: readonly (Entry & { id: string })[], options?: { via?: unknown; from?: string }): Promise; + close(): Promise; +} + +// A string field whose value is a known record id reads back as a Reference. +export interface Reference { + record(): Promise; } export interface SyncomRecord { readonly id: string; on(event: 'update', listener: (data: unknown, delta: unknown) => void): void; + once(event: 'update', listener: (data: unknown, delta: unknown) => void): void; + off(event: 'update', listener?: (data: unknown, delta: unknown) => void): void; data(): Promise; delta(): Record; save(): Promise; + inc(key: string, value?: number): Promise; delete(): Promise; + // Live properties: one getter/setter per field of the loaded object. + [field: string]: unknown; } declare class SyncomCollection { @@ -230,7 +408,7 @@ declare const SyncomRecordCtor: { export interface StorageConstructor { readonly prototype: Storage; - new (options?: StorageOptions): Promise; + new (options?: StorageOptions): Promise>; Collection: typeof SyncomCollection; Record: typeof SyncomRecordCtor; // The default `getId` โ€” synchronous in both halves, unlike the crypto three. @@ -244,4 +422,4 @@ export namespace Storage { export type StorageCollection = SyncomCollection; } -export function open(options?: StorageOptions): Promise; +export function open(options?: StorageOptions): Promise>; diff --git a/package.json b/package.json index f618a04..13afb32 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,6 @@ "./src/index.js": "./src/index.browser.js", "./src/crypto.js": "./src/crypto.browser.js", "./src/fsi.js": "./src/fsi.browser.js", - "./src/sync.js": "./src/sync.browser.js", "./src/utils.js": "./src/utils.browser.js" }, "types": "index.d.ts", @@ -21,6 +20,10 @@ "types": "./index.d.ts", "default": "./index.js" }, + "./wrpc": { + "types": "./wrpc.d.ts", + "default": "./wrpc.js" + }, "./package.json": "./package.json" }, "directories": { @@ -57,6 +60,8 @@ "p2p", "peer-to-peer", "replication", + "wrpc", + "websocket", "conflict-resolution", "CRDT", "event-sourcing", @@ -78,6 +83,7 @@ }, "homepage": "https://syncomjs.vercel.app/", "devDependencies": { + "@alexify/wrpc": "1.0.0", "@types/node": "^22.10.0", "@vercel/analytics": "^2.0.1", "@vercel/speed-insights": "^2.0.0", @@ -104,6 +110,8 @@ "index.d.ts", "browser.js", "browser.d.ts", + "wrpc.js", + "wrpc.d.ts", "src/", "LICENSE", "README.md", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cbcd381..813f7ea 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -8,6 +8,9 @@ importers: .: devDependencies: + '@alexify/wrpc': + specifier: 1.0.0 + version: 1.0.0 '@types/node': specifier: ^22.10.0 version: 22.20.1 @@ -50,6 +53,11 @@ importers: packages: + '@alexify/wrpc@1.0.0': + resolution: {integrity: sha512-aFfPWCVgui1i8RPjxZLNFj3v3bUvymcvge2TSIFn+bL4CiSsC6aGFtXcitU+OBTKZ+4+tv+7Ve8rgSkI6+KQxA==} + engines: {node: '>=22'} + hasBin: true + '@algolia/abtesting@1.23.0': resolution: {integrity: sha512-j45MBISstltys9QyQ4xf6quRiN1g7vMuwQL9VM4dx8YuRZvCQ173b9royZAx6iAbRX3IB1VnG1z//NuwyQ8jpQ==} engines: {node: '>= 14.0.0'} @@ -2579,6 +2587,8 @@ packages: snapshots: + '@alexify/wrpc@1.0.0': {} + '@algolia/abtesting@1.23.0': dependencies: '@algolia/client-common': 5.57.0 diff --git a/scripts/size.js b/scripts/size.js index 0b60bef..aeed592 100644 --- a/scripts/size.js +++ b/scripts/size.js @@ -27,10 +27,16 @@ const ROOT = path.join(__dirname, '..'); // One row per package.json#exports entry โ€” tests/package/consistency.test.js // fails if the two ever drift apart. const ENTRIES = [ - // Budget set from the first measured run and treated as a ratchet, not a - // target: raise it deliberately, in the same change that earns the bytes. - { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 5 }, + // A ratchet, not a target: raise it deliberately, in the same change that + // earns the bytes. 5 KB was the migrated storage/chain/crdt core; the + // replication engine (the op model, the persisted log, the sync protocol + // that makes `storage.sync` move data) earned the raise to 14 KB. + { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 14 }, { label: 'main entry โ€” node (@alexify/syncom)', entry: 'index.js', platform: 'node' }, + // Platform-neutral (no node builtins, no browser-only globals): one file + // serves both, bundled as a browser entry so the self-contained check + // guards it too. + { label: 'wrpc adapter โ€” browser (@alexify/syncom/wrpc)', entry: 'wrpc.js', platform: 'browser', budget: 2 }, ]; // A browser entry has to be self-contained: no node builtins, and no packages diff --git a/src/chain.js b/src/chain.js index bc6405c..8152e48 100644 --- a/src/chain.js +++ b/src/chain.js @@ -59,6 +59,10 @@ class Blockchain { if (!currentHash) return false; let count = 0; while (currentHash) { + // An unknown `from` (or a dangling `prev`) is an invalid chain, not an + // exception: validate() answers a boolean for every input. + const known = await fsi.exists(this.path, `${currentHash}.json`); + if (!known) return false; const block = await this.readBlock(currentHash); const expectedHash = await calculateHash(block); const valid = currentHash === expectedHash; diff --git a/src/crdt.js b/src/crdt.js index e7ab411..fdb1317 100644 --- a/src/crdt.js +++ b/src/crdt.js @@ -74,10 +74,14 @@ const apply = (source, history) => { if (!Array.isArray(history)) return source; let result = { ...source }; + // `write` is a full record write: the state becomes `record.data`. A write + // without data (a bare marker) copies the state forward. `delete` names a + // key to drop it; without one it is the record itself that was deleted, + // which is the shape Storage produces (`{ type: 'delete', id }`). const handlers = { - write: (record, state) => ({ ...state }), + write: (record, state) => (record.data && typeof record.data === 'object' ? { ...record.data } : { ...state }), delta: (record, state) => merge(state, record.data), - delete: (record, state) => ({ ...state, [record.key]: undefined }), + delete: (record, state) => (record.key === undefined ? {} : { ...state, [record.key]: undefined }), inc: (record, state) => merge(state, { [record.key]: record.value }), dec: (record, state) => merge(state, { [record.key]: -record.value }), }; diff --git a/src/crypto.browser.js b/src/crypto.browser.js index d625e94..706694d 100644 --- a/src/crypto.browser.js +++ b/src/crypto.browser.js @@ -7,7 +7,12 @@ const PUBLIC_KEY_FOOTER = '-----END PUBLIC KEY-----'; const PRIVATE_KEY_HEADER = '-----BEGIN PRIVATE KEY-----'; const PRIVATE_KEY_FOOTER = '-----END PRIVATE KEY-----'; -const ALG = { name: 'RSA-OAEP', hash: 'SHA-256' }; +const RSA = { name: 'RSA-OAEP', hash: 'SHA-256' }; +const AES = 'AES-GCM'; +// The same envelope as src/crypto.js, so a record encrypted on one platform +// decrypts on the other given the same key pair. +const ALG = 'RSA-OAEP-256+A256GCM'; +const TAG_BYTES = 16; const calculateHash = async (data) => { const block = JSON.stringify(data); @@ -17,14 +22,19 @@ const calculateHash = async (data) => { return hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); }; -const toBase64UrlSafe = (buffer) => { - const base64 = btoa(String.fromCharCode(...new Uint8Array(buffer))); - return base64.replaceAll('+', '-').replaceAll('/', '_').replaceAll('=', ''); +const toBase64 = (buffer) => { + const bytes = new Uint8Array(buffer); + let binary = ''; + for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]); + return btoa(binary); }; -const base64ToBytes = (base64) => { +// Accepts standard AND url-safe base64: keys written by an earlier version +// of this file were url-safe, and node's PEMs are standard. +const fromBase64 = (base64) => { const normalized = base64.replaceAll('-', '+').replaceAll('_', '/'); - const binary = atob(normalized); + const padded = normalized + '='.repeat((4 - (normalized.length % 4)) % 4); + const binary = atob(padded); const bytes = new Uint8Array(binary.length); for (let i = 0; i < binary.length; i++) { bytes[i] = binary.charCodeAt(i); @@ -32,6 +42,12 @@ const base64ToBytes = (base64) => { return bytes; }; +// Standard PEM: 64-character base64 lines, which is what node:crypto parses. +const toPem = (buffer, header, footer) => { + const lines = toBase64(buffer).match(/.{1,64}/g) ?? []; + return [header, ...lines, footer].join('\n'); +}; + const cleanPemKey = (pemKey, header, footer) => { const s = pemKey.replace(header, '').replace(footer, ''); return s.replaceAll(/\s/g, ''); @@ -39,49 +55,72 @@ const cleanPemKey = (pemKey, header, footer) => { const generateKeys = async () => { const keyPair = await crypto.subtle.generateKey( - { ...ALG, modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]) }, + { ...RSA, modulusLength: 2048, publicExponent: new Uint8Array([1, 0, 1]) }, true, ['encrypt', 'decrypt'], ); const pubBuf = await crypto.subtle.exportKey('spki', keyPair.publicKey); const privBuf = await crypto.subtle.exportKey('pkcs8', keyPair.privateKey); - const pub = toBase64UrlSafe(pubBuf); - const priv = toBase64UrlSafe(privBuf); return { - publicKey: [PUBLIC_KEY_HEADER, pub, PUBLIC_KEY_FOOTER].join('\n'), - privateKey: [PRIVATE_KEY_HEADER, priv, PRIVATE_KEY_FOOTER].join('\n'), + publicKey: toPem(pubBuf, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER), + privateKey: toPem(privBuf, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER), }; }; const importPublicKey = async (pemKey) => { - const base64 = cleanPemKey(pemKey, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER); - const bytes = base64ToBytes(base64); - return crypto.subtle.importKey('spki', bytes, ALG, false, ['encrypt']); + const bytes = fromBase64(cleanPemKey(pemKey, PUBLIC_KEY_HEADER, PUBLIC_KEY_FOOTER)); + return crypto.subtle.importKey('spki', bytes, RSA, false, ['encrypt']); }; const importPrivateKey = async (pemKey) => { - const base64 = cleanPemKey(pemKey, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER); - const bytes = base64ToBytes(base64); - return crypto.subtle.importKey('pkcs8', bytes, ALG, false, ['decrypt']); + const bytes = fromBase64(cleanPemKey(pemKey, PRIVATE_KEY_HEADER, PRIVATE_KEY_FOOTER)); + return crypto.subtle.importKey('pkcs8', bytes, RSA, false, ['decrypt']); }; const encrypt = async (data, publicKey) => { - const key = await importPublicKey(publicKey); - const dataBuffer = new TextEncoder().encode(JSON.stringify(data)); - const encryptedBuffer = await crypto.subtle.encrypt(ALG, key, dataBuffer); - return toBase64UrlSafe(encryptedBuffer); + const rsa = await importPublicKey(publicKey); + const raw = crypto.getRandomValues(new Uint8Array(32)); + const iv = crypto.getRandomValues(new Uint8Array(12)); + const aes = await crypto.subtle.importKey('raw', raw, AES, false, ['encrypt']); + const plaintext = new TextEncoder().encode(JSON.stringify(data)); + // Web Crypto appends the GCM tag to the ciphertext; the envelope keeps + // them apart, the way node:crypto hands them over. + const sealed = new Uint8Array(await crypto.subtle.encrypt({ name: AES, iv }, aes, plaintext)); + const body = sealed.subarray(0, sealed.length - TAG_BYTES); + const tag = sealed.subarray(sealed.length - TAG_BYTES); + const wrapped = await crypto.subtle.encrypt(RSA, rsa, raw); + return JSON.stringify({ + v: 1, + alg: ALG, + key: toBase64(wrapped), + iv: toBase64(iv), + tag: toBase64(tag), + data: toBase64(body), + }); }; -const decrypt = async (data, privateKey) => { - const key = await importPrivateKey(privateKey); - const bytes = base64ToBytes(data); - const buffer = await crypto.subtle.decrypt(ALG, key, bytes); - return JSON.parse(new TextDecoder().decode(buffer)); +const decrypt = async (text, privateKey) => { + const envelope = JSON.parse(text); + if (envelope.v !== 1 || envelope.alg !== ALG) throw new Error('Unsupported ciphertext envelope'); + const rsa = await importPrivateKey(privateKey); + const raw = await crypto.subtle.decrypt(RSA, rsa, fromBase64(envelope.key)); + const aes = await crypto.subtle.importKey('raw', raw, AES, false, ['decrypt']); + const body = fromBase64(envelope.data); + const tag = fromBase64(envelope.tag); + const sealed = new Uint8Array(body.length + tag.length); + sealed.set(body, 0); + sealed.set(tag, body.length); + const plain = await crypto.subtle.decrypt({ name: AES, iv: fromBase64(envelope.iv) }, aes, sealed); + return JSON.parse(new TextDecoder().decode(plain)); }; const loadKeys = async (basePath = 'keys') => { let publicKey, privateKey; - if (await fsi.exists(basePath, 'public.pem')) { + // Both halves of the pair, or neither: a public.pem left behind without + // its private.pem used to surface as an ENOENT from the second read. + const hasPublic = await fsi.exists(basePath, 'public.pem'); + const hasPrivate = await fsi.exists(basePath, 'private.pem'); + if (hasPublic && hasPrivate) { publicKey = await fsi.read(basePath, 'public.pem'); privateKey = await fsi.read(basePath, 'private.pem'); } else { diff --git a/src/crypto.js b/src/crypto.js index 78df68e..a2527a5 100644 --- a/src/crypto.js +++ b/src/crypto.js @@ -22,20 +22,47 @@ const generateKeys = () => }); }); +// Hybrid envelope, the same on both halves so ciphertext is portable between +// them: the record is AES-256-GCM encrypted under a fresh random key, and +// that key is RSA-OAEP(SHA-256) wrapped with the storage's public key. RSA +// alone caps the payload at a couple of hundred bytes; this caps nothing. +const ALG = 'RSA-OAEP-256+A256GCM'; +const OAEP = { padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' }; + const encrypt = (data, publicKey) => { - const buffer = Buffer.from(JSON.stringify(data)); - return crypto.publicEncrypt(publicKey, buffer).toString('base64'); + const key = crypto.randomBytes(32); + const iv = crypto.randomBytes(12); + const cipher = crypto.createCipheriv('aes-256-gcm', key, iv); + const body = Buffer.concat([cipher.update(Buffer.from(JSON.stringify(data))), cipher.final()]); + const tag = cipher.getAuthTag(); + const wrapped = crypto.publicEncrypt({ key: publicKey, ...OAEP }, key); + return JSON.stringify({ + v: 1, + alg: ALG, + key: wrapped.toString('base64'), + iv: iv.toString('base64'), + tag: tag.toString('base64'), + data: body.toString('base64'), + }); }; -const decrypt = (data, privateKey) => { - const buffer = Buffer.from(data, 'base64'); - const str = crypto.privateDecrypt(privateKey, buffer).toString(); - return JSON.parse(str); +const decrypt = (text, privateKey) => { + const envelope = JSON.parse(text); + if (envelope.v !== 1 || envelope.alg !== ALG) throw new Error('Unsupported ciphertext envelope'); + const key = crypto.privateDecrypt({ key: privateKey, ...OAEP }, Buffer.from(envelope.key, 'base64')); + const decipher = crypto.createDecipheriv('aes-256-gcm', key, Buffer.from(envelope.iv, 'base64')); + decipher.setAuthTag(Buffer.from(envelope.tag, 'base64')); + const plain = Buffer.concat([decipher.update(Buffer.from(envelope.data, 'base64')), decipher.final()]); + return JSON.parse(plain.toString()); }; const loadKeys = async (basePath) => { let publicKey, privateKey; - if (await fsi.exists(basePath, 'public.pem')) { + // Both halves of the pair, or neither: a public.pem left behind without + // its private.pem used to surface as an ENOENT from the second read. + const hasPublic = await fsi.exists(basePath, 'public.pem'); + const hasPrivate = await fsi.exists(basePath, 'private.pem'); + if (hasPublic && hasPrivate) { publicKey = await fsi.read(basePath, 'public.pem'); privateKey = await fsi.read(basePath, 'private.pem'); } else { diff --git a/src/fsi.browser.js b/src/fsi.browser.js index ac9858d..9bec193 100644 --- a/src/fsi.browser.js +++ b/src/fsi.browser.js @@ -44,4 +44,15 @@ const remove = async (dir, filename) => { } }; -module.exports = { read, write, exists, remove }; +// OPFS twin of the node `list`: `resolveDir` creates the directory on the way +// down, so a directory that never existed lists as empty, like ENOENT above. +const list = async (dir) => { + const dirHandle = await resolveDir(dir); + const names = []; + for await (const [name, handle] of dirHandle.entries()) { + if (handle.kind === 'file') names.push(name); + } + return names.sort(); +}; + +module.exports = { read, write, exists, remove, list }; diff --git a/src/fsi.js b/src/fsi.js index 190da5a..78b6203 100644 --- a/src/fsi.js +++ b/src/fsi.js @@ -18,4 +18,17 @@ const remove = async (dir, filename) => { if (await fileExists(filePath)) await fs.unlink(filePath); }; -module.exports = { read, write, exists, remove }; +// The file names in a directory (no subdirectories); a missing directory is +// an empty listing, not an error โ€” a storage that has never written a record +// simply has no data/ yet. +const list = async (dir) => { + const entries = await fs.readdir(dir, { withFileTypes: true }).catch((error) => { + if (error.code === 'ENOENT') return []; + throw error; + }); + const names = []; + for (const entry of entries) if (entry.isFile()) names.push(entry.name); + return names.sort(); +}; + +module.exports = { read, write, exists, remove, list }; diff --git a/src/index.browser.js b/src/index.browser.js index 7468af5..a7b312a 100644 --- a/src/index.browser.js +++ b/src/index.browser.js @@ -1,15 +1,19 @@ 'use strict'; // The browser twin of ./index.js: it names the browser variant of every split -// module directly. The requires NESTED inside chain.js / storage.js (./fsi.js, -// ./crypto.js, ./sync.js, ./utils.js) are redirected by package.json#browser, -// so a bundler that ignores that field would still pull the Node halves in. +// module directly. The requires NESTED inside chain.js / storage.js / sync.js +// / log.js (./fsi.js, ./crypto.js, ./utils.js) are redirected by +// package.json#browser, so a bundler that ignores that field would still pull +// the Node halves in. const chain = require('./chain.js'); const contract = require('./contract.js'); const crdt = require('./crdt.js'); const crypto = require('./crypto.browser.js'); const fsi = require('./fsi.browser.js'); const storage = require('./storage.js'); -const sync = require('./sync.browser.js'); +const sync = require('./sync.js'); +const log = require('./log.js'); +const { applyOp, mergeEntry, isAlive } = require('./ops.js'); +const ops = { applyOp, mergeEntry, isAlive }; -module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync }; +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops }; diff --git a/src/index.js b/src/index.js index c60a42f..e07aa69 100644 --- a/src/index.js +++ b/src/index.js @@ -7,5 +7,8 @@ const crypto = require('./crypto.js'); const fsi = require('./fsi.js'); const storage = require('./storage.js'); const sync = require('./sync.js'); +const log = require('./log.js'); +const { applyOp, mergeEntry, isAlive } = require('./ops.js'); +const ops = { applyOp, mergeEntry, isAlive }; -module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync }; +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops }; diff --git a/src/link.js b/src/link.js new file mode 100644 index 0000000..eaf4486 --- /dev/null +++ b/src/link.js @@ -0,0 +1,15 @@ +'use strict'; + +// The structural check on an injected link โ€” the whole contract the sync +// engine dials through. In its own file so the `@alexify/syncom/wrpc` +// adapter can share it without pulling the engine into a browser bundle. +// `stream` is optional (a binary-capable transport). +const isSyncLink = (link) => + Boolean(link) && + typeof link.call === 'function' && + typeof link.subscribe === 'function' && + typeof link.on === 'function' && + typeof link.off === 'function' && + typeof link.close === 'function'; + +module.exports = { isSyncLink }; diff --git a/src/log.js b/src/log.js new file mode 100644 index 0000000..ccb107e --- /dev/null +++ b/src/log.js @@ -0,0 +1,332 @@ +'use strict'; + +// The persisted operation log: every op this node applied โ€” its own and the +// ones it accepted from peers โ€” in the order they landed here. Two things read +// it: a subscriber resuming from an event id (`since`), and the node itself on +// reopen (the tail is replayed onto the records, which is what makes a crash +// between "logged" and "written to data/" recoverable). +// +// Layout under `/log/`, over fsi (whole-file read/write, no rename): +// open/.json one file per append batch, never rewritten โ€” a torn +// write can only damage the newest batch, which has not +// been announced to anyone yet +// seg-.json sealed segments: the open files rolled together once +// enough of them accumulate +// index.json { epoch, floor, next, vector, segments } โ€” a summary; +// the files are the source of truth and anything the +// index lacks is rebuilt from a directory listing +// +// Ids are `.`. The epoch is minted once, when the log is created, +// so a subscriber holding an id from a previous incarnation (a wiped or +// restored data directory) is told to bootstrap instead of being fed a numeric +// coincidence. `floor` is the lowest seq still retained: below it, and for an +// unknown epoch, `since()` answers null โ€” "I cannot resume you, take a +// snapshot" โ€” never a silent gap. + +const fsi = require('./fsi.js'); +const { formatEventId, parseEventId, vectorAdvance } = require('./vector.js'); + +const OPEN = 'open'; +const INDEX = 'index.json'; +const DEFAULT_ROLL_AFTER = 64; +const DEFAULT_RETENTION_OPS = 10_000; + +const pad = (n) => String(n).padStart(10, '0'); +const segmentName = (from) => `seg-${pad(from)}.json`; + +const parseJson = (text) => { + try { + return JSON.parse(text); + } catch { + return null; + } +}; + +class OpLog { + #path; + #openPath; + #entries = []; + #openFiles = []; + #segments = []; + #next = 1; + #floor = 1; + #epoch = null; + #vector = {}; + #rollAfter; + #retention; + #log; + #writes = Promise.resolve(); + + constructor(basePath, options = {}) { + this.#path = `${basePath}/log`; + this.#openPath = `${this.#path}/${OPEN}`; + this.#rollAfter = options.rollAfter ?? DEFAULT_ROLL_AFTER; + this.#retention = { ops: DEFAULT_RETENTION_OPS, ...options.retention }; + this.#log = options.logger ?? null; + return this.#init(); + } + + async #init() { + const hasIndex = await fsi.exists(this.#path, INDEX); + const index = hasIndex ? parseJson(await fsi.read(this.#path, INDEX)) : null; + if (index) { + this.#epoch = index.epoch; + this.#floor = index.floor ?? 1; + this.#vector = index.vector ?? {}; + this.#segments = Array.isArray(index.segments) ? index.segments : []; + } else { + // No (readable) index: a new incarnation. Existing segments are still + // picked up below, but their ids belong to an epoch nobody can resume. + this.#epoch = `${Date.now().toString(36)}${Math.random().toString(36).slice(2, 8)}`; + this.#segments = []; + } + await this.#loadSegments(hasIndex); + await this.#loadOpen(); + if (!index) await this.#writeIndex(); + return this; + } + + async #loadSegments(hasIndex) { + const known = new Set(this.#segments.map((segment) => segment.name)); + const names = await fsi.list(this.#path); + for (const name of names) { + if (!name.startsWith('seg-') || known.has(name)) continue; + // A segment the index does not know: rolled, then crashed before the + // index was rewritten. Adopt it. + const from = Number(name.slice(4, -5)); + if (Number.isInteger(from)) this.#segments.push({ name, from, to: from, count: 0 }); + } + this.#segments.sort((a, b) => a.from - b.from); + for (const segment of this.#segments) { + const text = await fsi.read(this.#path, segment.name).catch(() => null); + const parsed = text === null ? null : parseJson(text); + if (!parsed || !Array.isArray(parsed.entries)) { + this.#log?.warn?.({ event: 'log.segment.unreadable', segment: segment.name }); + continue; + } + this.#adopt(parsed.entries); + if (parsed.entries.length > 0) { + segment.to = parsed.entries[parsed.entries.length - 1].seq; + segment.count = parsed.entries.length; + } + } + // The floor is durable in the index; a fresh index means everything on + // disk is retained from its first entry. + if (!hasIndex && this.#entries.length > 0) this.#floor = this.#entries[0].seq; + } + + async #loadOpen() { + const names = await fsi.list(this.#openPath); + for (const name of names) { + const text = await fsi.read(this.#openPath, name).catch(() => null); + const parsed = text === null ? null : parseJson(text); + if (!parsed || !Array.isArray(parsed.entries)) { + // Torn on the way down: the batch was never announced, drop it. + this.#log?.warn?.({ event: 'log.open.unreadable', file: name }); + await fsi.remove(this.#openPath, name); + continue; + } + this.#adopt(parsed.entries); + this.#openFiles.push(name); + } + } + + // Merges loaded entries in, deduplicating by seq (a crash mid-roll leaves + // the same batch both sealed and open) and keeping the order by seq. + #adopt(entries) { + const seen = new Set(this.#entries.map((entry) => entry.seq)); + let added = false; + for (const entry of entries) { + if (!entry || !Number.isInteger(entry.seq) || seen.has(entry.seq)) continue; + seen.add(entry.seq); + this.#entries.push(entry); + added = true; + if (entry.seq >= this.#next) this.#next = entry.seq + 1; + if (entry.op) vectorAdvance(this.#vector, entry.op); + } + if (added) this.#entries.sort((a, b) => a.seq - b.seq); + } + + get epoch() { + return this.#epoch; + } + + get floor() { + return this.#floor; + } + + get next() { + return this.#next; + } + + /** The highest per-origin seq this log has ever held โ€” survives compaction. */ + get vector() { + return { ...this.#vector }; + } + + get length() { + return this.#entries.length; + } + + /** The id of the newest entry; resuming from it replays nothing. */ + get lastEventId() { + return formatEventId(this.#epoch, this.#next - 1); + } + + formatId(seq) { + return formatEventId(this.#epoch, seq); + } + + // Appends run one at a time: two batches must not race for the same seqs. + #serialize(task) { + const run = this.#writes.then(task, task); + this.#writes = run.catch(() => {}); + return run; + } + + /** + * Appends a batch; resolves once it is on disk. Returns the entries with + * their seqs, so the caller can announce them by id. Rolls the open files + * into a segment when enough have accumulated. + */ + append(ops) { + return this.#serialize(async () => { + if (ops.length === 0) return []; + const entries = []; + for (const op of ops) entries.push({ seq: this.#next++, op }); + const name = `${pad(entries[0].seq)}.json`; + await fsi.write(this.#openPath, name, JSON.stringify({ entries })); + for (const entry of entries) this.#entries.push(entry); + for (const op of ops) vectorAdvance(this.#vector, op); + this.#openFiles.push(name); + if (this.#openFiles.length >= this.#rollAfter) await this.#roll(); + return entries; + }); + } + + // Sealing: write the segment, then the index, then remove the open files. + // Any crash in between leaves duplicates, which #adopt tolerates. + async #roll() { + if (this.#openFiles.length === 0) return; + const sealedFrom = this.#segments.length > 0 ? this.#segments[this.#segments.length - 1].to + 1 : this.#floor; + const entries = this.#entries.filter((entry) => entry.seq >= sealedFrom); + if (entries.length === 0) return; + const from = entries[0].seq; + const name = segmentName(from); + await fsi.write(this.#path, name, JSON.stringify({ entries })); + this.#segments.push({ name, from, to: entries[entries.length - 1].seq, count: entries.length }); + await this.#compact(); + await this.#writeIndex(); + const files = this.#openFiles; + this.#openFiles = []; + for (const file of files) await fsi.remove(this.#openPath, file); + } + + // Drops whole sealed segments from the front while the log exceeds its + // retention; the floor moves up and `vector` keeps what was dropped. + async #compact() { + const { ops, days } = this.#retention; + const cutoff = days ? Date.now() - days * 86_400_000 : null; + while (this.#segments.length > 1) { + const oldest = this.#segments[0]; + const overOps = ops ? this.#entries.length - oldest.count >= ops : false; + const lastTs = this.#entries.find((entry) => entry.seq === oldest.to)?.op?.ts ?? Infinity; + const overAge = cutoff !== null && lastTs < cutoff; + const tooMany = ops ? this.#entries.length > ops : false; + if (!(tooMany && overOps) && !overAge) break; + this.#segments.shift(); + this.#entries = this.#entries.filter((entry) => entry.seq > oldest.to); + this.#floor = oldest.to + 1; + await fsi.remove(this.#path, oldest.name); + } + } + + async #writeIndex() { + const index = { + epoch: this.#epoch, + floor: this.#floor, + next: this.#next, + vector: this.#vector, + segments: this.#segments, + }; + await fsi.write(this.#path, INDEX, JSON.stringify(index)); + } + + /** + * Adopts a peer's vector after a snapshot of its state was joined: every op + * up to it is reflected in what was merged, so it need not be offered again. + * Persisted with the index, like the rest of the vector. + */ + advance(vector) { + return this.#serialize(async () => { + let moved = false; + for (const key in vector) { + const seq = vector[key]; + if (Number.isInteger(seq) && (this.#vector[key] ?? 0) < seq) { + this.#vector[key] = seq; + moved = true; + } + } + if (moved) await this.#writeIndex(); + }); + } + + /** Forces a roll + compaction now; otherwise they happen every rollAfter appends. */ + flush() { + return this.#serialize(() => this.#roll()); + } + + /** + * Everything after `eventId`, oldest first, or `null` when the caller + * cannot be resumed: an id from another epoch, or one that fell below the + * floor. No id at all resumes from the very beginning โ€” possible only while + * nothing has been compacted away. + */ + since(eventId, { scopes = null, limit = Infinity } = {}) { + let after; + if (eventId === undefined || eventId === null) { + if (this.#floor > 1) return null; + after = 0; + } else { + const parsed = parseEventId(eventId); + if (!parsed || parsed.epoch !== this.#epoch) return null; + if (parsed.seq < this.#floor - 1) return null; + after = parsed.seq; + } + const entries = []; + let index = this.#lowerBound(after + 1); + for (; index < this.#entries.length && entries.length < limit; index++) { + const entry = this.#entries[index]; + // An op without a scope (or scoped '*') is global and reaches every + // subscriber; a scoped one only those who asked for that scope. + const scope = entry.op.scope ?? '*'; + if (scopes && scope !== '*' && !scopes.includes('*') && !scopes.includes(scope)) continue; + entries.push(entry); + } + return entries; + } + + // Index of the first entry whose seq >= target (entries are sorted). + #lowerBound(target) { + let low = 0; + let high = this.#entries.length; + while (low < high) { + const mid = (low + high) >>> 1; + if (this.#entries[mid].seq < target) low = mid + 1; + else high = mid; + } + return low; + } + + /** The unsealed tail โ€” what a reopen replays onto the records. */ + tail() { + const sealedTo = this.#segments.length > 0 ? this.#segments[this.#segments.length - 1].to : this.#floor - 1; + return this.#entries.filter((entry) => entry.seq > sealedTo); + } + + close() { + return this.#serialize(() => this.#writeIndex()); + } +} + +module.exports = { OpLog }; diff --git a/src/ops.js b/src/ops.js new file mode 100644 index 0000000..00e7ef5 --- /dev/null +++ b/src/ops.js @@ -0,0 +1,311 @@ +'use strict'; + +// The record-level replication model: how ONE operation changes ONE record's +// entry, in a way that gives the same result whatever order the operations +// arrive in, and a no-op when the same operation arrives twice. The version +// vector in the log is an optimization on top of this โ€” it decides what to +// send, never what is correct. +// +// An entry is `{ data, meta }`. `data` is what the application reads; `meta` +// is the bookkeeping that makes merges commute: +// clock[f] โ€” [ts, origin] of the write that set register field f +// base โ€” the record-reset watermark: a field is visible only if its +// clock is not below base. A `write` (whole-record write) and +// a `delete` both raise it, which is what stops an older +// concurrent `set` from resurrecting a field they removed. +// deleted โ€” the tombstone stamp; the record is alive again as soon as +// something newer than it lands (see isAlive) +// counters[f] โ€” { [origin]: [seq, sum, ts] }: a PN-counter, one running sum +// per origin, so re-applying an `inc` (seq already seen) is a +// no-op and two snapshots join by taking the higher seq +// +// Precondition, enforced by Storage: ops from ONE origin are applied in seq +// order. Counters rely on it (a skipped seq would be lost, not reordered); +// registers do not, but the vector-based dedup does. +// +// A field is a REGISTER or a COUNTER, never both: `set`/`write` on a counter +// field and `inc` on a register field are refused (reported in `rejected`) โ€” +// the two kinds cannot be made to commute, and refusing is honest where +// guessing would silently diverge. + +const { compareClock, maxClock } = require('./vector.js'); + +const createEntry = () => ({ data: {}, meta: { clock: {}, base: null, deleted: null, counters: {} } }); + +const cloneEntry = (entry) => { + const counters = {}; + for (const field in entry.meta.counters) counters[field] = { ...entry.meta.counters[field] }; + return { + data: { ...entry.data }, + meta: { clock: { ...entry.meta.clock }, base: entry.meta.base, deleted: entry.meta.deleted, counters }, + }; +}; + +const counterValue = (perOrigin) => { + let sum = 0; + for (const origin in perOrigin) sum += perOrigin[origin][1]; + return sum; +}; + +const isCounter = (meta, field) => { + const perOrigin = meta.counters[field]; + return Boolean(perOrigin) && Object.keys(perOrigin).length > 0; +}; + +const isPlainObject = (value) => typeof value === 'object' && value !== null && !Array.isArray(value); + +// Structural equality over JSON-shaped values, key order ignored โ€” what a +// `merge` op needs to tell "already have this" from "news". +const deepEqual = (a, b) => { + if (a === b) return true; + if (typeof a !== 'object' || typeof b !== 'object' || a === null || b === null) return false; + if (Array.isArray(a) !== Array.isArray(b)) return false; + const keysA = Object.keys(a); + const keysB = Object.keys(b); + if (keysA.length !== keysB.length) return false; + for (const key of keysA) { + if (!Object.hasOwn(b, key) || !deepEqual(a[key], b[key])) return false; + } + return true; +}; + +// A peer-supplied meta, coerced to the shape the rest of this module trusts. +const normalizeMeta = (meta) => ({ + clock: isPlainObject(meta.clock) ? { ...meta.clock } : {}, + base: Array.isArray(meta.base) ? meta.base : null, + deleted: Array.isArray(meta.deleted) ? meta.deleted : null, + counters: isPlainObject(meta.counters) ? meta.counters : {}, +}); + +// Drops every register and counter entry that fell below the (new) base. +const applyBase = (entry, delta) => { + const { meta, data } = entry; + for (const field in meta.clock) { + if (compareClock(meta.clock[field], meta.base) < 0) { + delete meta.clock[field]; + delete data[field]; + delta[field] = undefined; + } + } + for (const field in meta.counters) { + const perOrigin = meta.counters[field]; + for (const origin in perOrigin) { + if (compareClock([perOrigin[origin][2], origin], meta.base) < 0) delete perOrigin[origin]; + } + if (isCounter(meta, field)) { + const value = counterValue(perOrigin); + if (data[field] !== value) delta[field] = value; + data[field] = value; + } else { + delete meta.counters[field]; + delete data[field]; + delta[field] = undefined; + } + } +}; + +// Sets register fields at `stamp`, honouring last-writer-wins per field. +const applyFields = (entry, fields, stamp, delta, rejected) => { + const { meta, data } = entry; + // Own keys only: `fields` may be a JSON-parsed peer payload, where + // "__proto__" is an ordinary own key, or an in-process object with a + // prototype chain nothing here should walk. + for (const field of Object.keys(fields)) { + if (field === '__proto__') continue; + if (isCounter(meta, field)) { + rejected.push({ field, reason: 'counter' }); + continue; + } + if (compareClock(meta.clock[field], stamp) >= 0) continue; + if (compareClock(stamp, meta.base) < 0) continue; + meta.clock[field] = stamp; + data[field] = fields[field]; + delta[field] = fields[field]; + } +}; + +/** + * Applies one op to an entry (null for a record this node has never seen). + * Returns `{ entry, changed, delta, rejected }` โ€” a NEW entry, whether it + * differs, the fields that changed (undefined = removed) and the fields the + * register/counter rule refused. Idempotent: applying an op twice, or an op + * older than what the entry already reflects, changes nothing. + */ +const applyOp = (previous, op) => { + const entry = previous ? cloneEntry(previous) : createEntry(); + const { meta } = entry; + const stamp = [op.ts, op.origin]; + const delta = {}; + const rejected = []; + + if (op.type === 'set') { + if (isPlainObject(op.data)) applyFields(entry, op.data, stamp, delta, rejected); + } else if (op.type === 'write') { + if (isPlainObject(op.data) && compareClock(stamp, meta.base) > 0) { + meta.base = stamp; + // A write above the tombstone is a revival; dropping the stale stamp + // keeps the metadata identical whichever order the two arrived in. + if (compareClock(meta.deleted, meta.base) < 0) meta.deleted = null; + applyBase(entry, delta); + applyFields(entry, op.data, stamp, delta, rejected); + } + } else if (op.type === 'delete') { + if (compareClock(stamp, meta.base) > 0) { + meta.base = stamp; + meta.deleted = stamp; + applyBase(entry, delta); + } + } else if (op.type === 'merge') { + // A whole entry joined in โ€” how a snapshot page becomes log entries a + // downstream peer can receive. Idempotent and commutative by + // construction (see mergeEntry). + if (isPlainObject(op.data) && isPlainObject(op.meta)) { + const joined = mergeEntry(previous, { data: op.data, meta: normalizeMeta(op.meta) }); + // Only the replicated halves count: a cached entry also carries its + // block hash and the encrypted flag. + const same = previous && deepEqual(joined.data, previous.data) && deepEqual(joined.meta, previous.meta); + if (!same) { + for (const field of Object.keys(joined.data)) { + if (!previous || previous.data[field] !== joined.data[field]) delta[field] = joined.data[field]; + } + if (previous) { + for (const field of Object.keys(previous.data)) if (!(field in joined.data)) delta[field] = undefined; + } + const before = previous ? previous.meta : null; + const moved = + joined.meta.base !== (before ? before.base : null) || + joined.meta.deleted !== (before ? before.deleted : null); + return { entry: joined, changed: true, delta, rejected, moved }; + } + } + return { entry, changed: false, delta, rejected }; + } else if (op.type === 'inc') { + const { key: field, value } = op; + if (typeof field === 'string' && field !== '__proto__' && Number.isFinite(value)) { + if (meta.clock[field] !== undefined) { + rejected.push({ field, reason: 'register' }); + } else if (compareClock(stamp, meta.base) >= 0) { + const perOrigin = (meta.counters[field] ??= {}); + const known = perOrigin[op.origin]; + if (!known || known[0] < op.seq) { + perOrigin[op.origin] = [op.seq, (known ? known[1] : 0) + value, op.ts]; + entry.data[field] = counterValue(perOrigin); + delta[field] = entry.data[field]; + } + } + } + } + + const before = previous ? previous.meta : null; + const changed = + Object.keys(delta).length > 0 || + meta.base !== (before ? before.base : null) || + meta.deleted !== (before ? before.deleted : null); + return { entry, changed, delta, rejected }; +}; + +// Alive unless a tombstone stands and nothing newer than it has landed. +const isAlive = (entry) => { + if (!entry) return false; + const { meta } = entry; + if (!meta.deleted) return true; + for (const field in meta.clock) { + if (compareClock(meta.clock[field], meta.deleted) > 0) return true; + } + for (const field in meta.counters) { + const perOrigin = meta.counters[field]; + for (const origin in perOrigin) { + if (compareClock([perOrigin[origin][2], origin], meta.deleted) > 0) return true; + } + } + return false; +}; + +/** + * Joins two entries for the same record โ€” the snapshot merge. Commutative, + * associative and idempotent, and equal to applying both histories in any + * interleaving (up to the documented counter/reset straddle). + */ +const mergeEntry = (a, b) => { + if (!a) return b ? cloneEntry(b) : null; + if (!b) return cloneEntry(a); + const entry = createEntry(); + const { meta, data } = entry; + meta.base = maxClock(a.meta.base, b.meta.base); + meta.deleted = maxClock(a.meta.deleted, b.meta.deleted); + if (compareClock(meta.deleted, meta.base) < 0) meta.deleted = null; + for (const source of [a, b]) { + for (const field in source.meta.clock) { + if (compareClock(source.meta.clock[field], meta.clock[field]) > 0) { + meta.clock[field] = source.meta.clock[field]; + data[field] = source.data[field]; + } + } + for (const field in source.meta.counters) { + const perOrigin = (meta.counters[field] ??= {}); + const from = source.meta.counters[field]; + for (const origin in from) { + if (!perOrigin[origin] || perOrigin[origin][0] < from[origin][0]) perOrigin[origin] = from[origin]; + } + } + } + applyBase(entry, {}); + // A register that also became a counter on the other side: the register + // wins (the rule is refuse-on-write, and the writes already happened). + for (const field in meta.counters) { + if (meta.clock[field] !== undefined) { + delete meta.counters[field]; + data[field] = a.meta.clock[field] === meta.clock[field] ? a.data[field] : b.data[field]; + } + } + return entry; +}; + +// The wire/log shape of a batch: one positional array per op, type as a +// single letter, absent fields as null โ€” no key names repeated per op. +const TYPE_CODES = { write: 'w', set: 's', inc: 'i', delete: 'd', merge: 'm', advance: 'a' }; +const CODE_TYPES = { w: 'write', s: 'set', i: 'inc', d: 'delete', m: 'merge', a: 'advance' }; + +const packOps = (ops) => { + const packed = new Array(ops.length); + for (let i = 0; i < ops.length; i++) { + const op = ops[i]; + packed[i] = [ + op.origin, + op.seq, + op.ts, + TYPE_CODES[op.type] ?? op.type, + op.id, + op.scope ?? null, + op.data === undefined ? null : op.data, + op.meta === undefined ? null : op.meta, + op.key === undefined ? null : op.key, + op.value === undefined ? null : op.value, + op.vector === undefined ? null : op.vector, + op.from === undefined ? null : op.from, + ]; + } + return packed; +}; + +const unpackOps = (packed) => { + if (!Array.isArray(packed)) return []; + const ops = []; + for (const row of packed) { + if (!Array.isArray(row) || row.length < 5) continue; + const [origin, seq, ts, code, id, scope, data, meta, key, value, vector, from] = row; + const op = { origin, seq, ts, type: CODE_TYPES[code] ?? code }; + if (id !== null && id !== undefined) op.id = id; + if (vector !== null && vector !== undefined) op.vector = vector; + if (from !== null && from !== undefined) op.from = from; + if (scope !== null && scope !== undefined) op.scope = scope; + if (data !== null && data !== undefined) op.data = data; + if (meta !== null && meta !== undefined) op.meta = meta; + if (key !== null && key !== undefined) op.key = key; + if (value !== null && value !== undefined) op.value = value; + ops.push(op); + } + return ops; +}; + +module.exports = { createEntry, cloneEntry, applyOp, mergeEntry, isAlive, isCounter, packOps, unpackOps }; diff --git a/src/storage.js b/src/storage.js index bd27874..d32444e 100644 --- a/src/storage.js +++ b/src/storage.js @@ -1,20 +1,24 @@ 'use strict'; -const { Emitter, generateUUID } = require('./utils.js'); +const { Emitter, generateUUID, createLogger } = require('./utils.js'); const { encrypt, decrypt, loadKeys, calculateHash } = require('./crypto.js'); const { Blockchain } = require('./chain.js'); const { SyncManager } = require('./sync.js'); +const { OpLog } = require('./log.js'); +const { applyOp, isAlive } = require('./ops.js'); +const { Clock, opKey, keyScope, vectorHas, vectorNext, vectorAdvance } = require('./vector.js'); const fsi = require('./fsi.js'); -const calculateDelta = (oldData, newData) => { - const delta = {}; - for (const key in newData) { - if (oldData[key] !== newData[key]) { - delta[key] = newData[key]; - } - } - return delta; -}; +const NODE_FILE = 'node.json'; +const GLOBAL_SCOPE = '*'; + +// Module-private entry points between Storage and Record: a Record commits +// through the same path every other write takes and is refreshed when the +// record changes underneath it. Symbols keep both off the public surface. +const COMMIT = Symbol('syncom.commit'); +const REFRESH = Symbol('syncom.refresh'); + +const isPlainObject = (value) => typeof value === 'object' && value !== null && !Array.isArray(value); class Reference { #id; @@ -46,14 +50,16 @@ class Record extends Emitter { async #load() { if (this.#data !== null) return this; const rawData = await this.#storage.get(this.#id); - if (rawData) { - this.#data = JSON.parse(JSON.stringify(rawData)); - this.#convertToReferences(this.#data); - this.#defineProperties(); - } + if (rawData) this.#adopt(rawData); return this; } + #adopt(rawData) { + this.#data = JSON.parse(JSON.stringify(rawData)); + this.#convertToReferences(this.#data); + this.#defineProperties(); + } + #convertToReferences(data) { if (!data || typeof data !== 'object') return; for (const key in data) { @@ -81,11 +87,7 @@ class Record extends Emitter { if (Object.prototype.hasOwnProperty.call(this, key)) continue; Object.defineProperty(this, key, { get() { - const value = this.#data?.[key]; - if (value instanceof Reference) { - return value; - } - return value; + return this.#data?.[key]; }, set(newValue) { const oldValue = this.#data?.[key]; @@ -101,6 +103,22 @@ class Record extends Emitter { } } + // The record changed in storage โ€” a remote op, a `storage.update()`, or + // its own save landing. The view is rebuilt from the authoritative data, + // unsaved local edits are laid back on top (they are still pending), and + // listeners hear about it unless the record itself caused the change. + [REFRESH](data, delta, source) { + if (data === null) { + this.#data = null; + this.#changes = {}; + if (source !== this) this.emit('update', null, delta); + return; + } + this.#adopt(data); + for (const key in this.#changes) this.#data[key] = this.#changes[key]; + if (source !== this) this.emit('update', this.#data, delta); + } + async data() { return this.#data; } @@ -112,15 +130,13 @@ class Record extends Emitter { async save() { if (Object.keys(this.#changes).length === 0) return; const delta = this.delta(); - const data = this.#data; - await this.#storage.saveData(this.#id, data); + await this.#storage[COMMIT]('set', this.#id, { data: delta }, { source: this }); this.#changes = {}; - this.#storage.addUpdate({ - type: 'delta', - id: this.#id, - timestamp: Date.now(), - data: delta, - }); + } + + /** Adds to a counter field; the storage-level `inc` with this record's id. */ + async inc(key, value = 1) { + await this.#storage.inc(this.#id, key, value); } async delete() { @@ -166,16 +182,24 @@ class Collection { } } -class Storage { +class Storage extends Emitter { #path; #keys = null; #chain = null; + #log = null; #records = null; #sync = null; - #updates = []; #cache = new Map(); #schema = null; #getId = null; + #logger; + #options; + #nodeId = null; + #clock = new Clock(); + #scopeOf; + #encrypted = false; + #queue = Promise.resolve(); + #closed = false; // The default id generator, exposed so it stays reachable and stubbable. // `generateUUID` never touches `this`, so the reference is safe to store @@ -185,24 +209,74 @@ class Storage { } constructor(options = {}) { + super(); + this.#options = options; this.#getId = Storage.generateId; this.#path = options.path || options.name || 'syncom'; this.#records = new Map(); + this.#logger = createLogger(options.logger); if (options.schema) this.#schema = options.schema; // A non-function is ignored rather than rejected: the id generator is a // convenience seam, not a security boundary, and the default still holds. if (typeof options.getId === 'function') this.#getId = options.getId; + // Encrypt every record at rest by default; a per-call `{ encrypted }` + // still overrides. Encrypted records are local โ€” see saveData(). + this.#encrypted = options.encrypted === true; + const scope = options.sync?.scope; + this.#scopeOf = typeof scope === 'function' ? scope : () => GLOBAL_SCOPE; return this.#init(); } async #init() { this.#keys = await loadKeys(`${this.#path}/keys`); this.#chain = await new Blockchain(`${this.#path}/blockchain`); - this.#sync = await new SyncManager(this.#path, this, this.#chain); + this.#log = await new OpLog(this.#path, { ...this.#options.sync?.log, retention: this.#options.sync?.retention }); + await this.#loadIdentity(); + await this.#replayTail(); + this.#sync = await new SyncManager(this, this.#options.sync); this.#initCollections(); return this; } + // The node id is minted once and never changes; the own sequence numbers + // (one per scope) live in the log's vector, which survives compaction โ€” so + // a seq is durable the moment its op is appended, before anyone hears of it. + async #loadIdentity() { + if (await fsi.exists(this.#path, NODE_FILE)) { + const raw = await fsi.read(this.#path, NODE_FILE); + this.#nodeId = JSON.parse(raw).id; + } else { + this.#nodeId = generateUUID(); + await fsi.write(this.#path, NODE_FILE, JSON.stringify({ id: this.#nodeId })); + } + } + + // The log is written before the record files are: a crash in between + // leaves ops the records do not reflect yet. Re-applying the unsealed tail + // is idempotent, so replaying more than necessary costs only time. + async #replayTail() { + const tail = this.#log.tail(); + if (tail.length === 0) return; + const touched = new Map(); + for (const { seq, op } of tail) { + this.#clock.observe(op.ts); + let current; + try { + current = touched.get(op.id) ?? (await this.#load(op.id)); + } catch (error) { + // A record that fails its integrity check is left as it is: the next + // read reports it, which is where the caller expects to hear about it. + this.#logger.warn({ event: 'replay.skipped', id: op.id, err: error }); + continue; + } + if (current?.encrypted) continue; + const { entry, changed } = applyOp(current, op); + if (changed) touched.set(op.id, { ...entry, seq }); + } + if (touched.size === 0) return; + await this.#persistBatch(touched, tail[0].seq, tail[tail.length - 1].seq); + } + #initCollections() { const entities = this.#schema?.entities; if (!entities) return; @@ -215,33 +289,59 @@ class Storage { return this.#schema; } - #emitUpdate(id, data, delta) { - const emitter = this.#records.get(id); - if (emitter) { - emitter.emit('update', data, delta); - } + /** This node's stable identity โ€” the `origin` of every op it creates. */ + get nodeId() { + return this.#nodeId; } - addUpdate(update) { - this.#updates.push(update); + get path() { + return this.#path; } - async saveData(id, data, options = {}) { - const { encrypted = false } = options; - const record = encrypted ? encrypt(data, this.#keys.publicKey) : data; - const timestamp = Date.now(); + get log() { + return this.#log; + } - const hash = await calculateHash(record); - const blockHash = await this.#chain.addBlock({ id, hash }); + get logger() { + return this.#logger; + } - const entry = { data: record, encrypted, timestamp, block: blockHash }; - await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); - this.#cache.set(id, data); + get sync() { + return this.#sync; + } - this.#updates.push({ type: 'write', id, timestamp, data }); + /** The highest per-origin seq applied here โ€” what a peer compares against. */ + get vector() { + return this.#log.vector; } - async loadData(id) { + // Every mutation passes through one queue: the chain's tail pointer, the + // log's seq counter and a record's entry are all read-then-written, and two + // interleaved writers would race all three. + #run(task) { + if (this.#closed) return Promise.reject(new Error('Storage is closed')); + const result = this.#queue.then(task, task); + this.#queue = result.catch(() => {}); + return result; + } + + #emitUpdate(id, data, delta, source = null) { + const record = this.#records.get(id); + if (record) record[REFRESH](data, delta, source); + } + + #scope(id, data) { + const scope = this.#scopeOf(id, data); + return typeof scope === 'string' && scope.length > 0 ? scope : GLOBAL_SCOPE; + } + + // --------------------------------------------------------------------- + // Entries on disk: `{ data, meta, encrypted, timestamp, block }`. `meta` is + // the replication bookkeeping (see ops.js); an encrypted entry has none โ€” + // it is stored as ciphertext for this node's key alone and is not + // replicated. + + async #load(id) { if (this.#cache.has(id)) return this.#cache.get(id); const dataDir = `${this.#path}/data`; const exists = await fsi.exists(dataDir, `${id}.json`); @@ -250,20 +350,122 @@ class Storage { return null; } const raw = await fsi.read(dataDir, `${id}.json`); - const { data, encrypted, block } = JSON.parse(raw); + const { data, meta, encrypted, block } = JSON.parse(raw); const isValid = await this.validate(id, data, block); if (!isValid) throw new Error(`Storage record ${id} is invalid`); - const decrypted = encrypted ? decrypt(data, this.#keys.privateKey) : data; - this.#cache.set(id, decrypted); - return decrypted; + const entry = encrypted + ? { data: await decrypt(data, this.#keys.privateKey), meta: null, encrypted: true, block } + : { data, meta, encrypted: false, block }; + this.#cache.set(id, entry); + return entry; + } + + async #writeEntry(id, entry, blockHash) { + const stored = { data: entry.data, meta: entry.meta, encrypted: false, timestamp: Date.now(), block: blockHash }; + await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(stored)); + this.#cache.set(id, { data: entry.data, meta: entry.meta, encrypted: false, block: blockHash }); + } + + // One chain block per applied batch: `records` maps every touched id to + // the hash of its data after the batch, `from`/`to` are the log positions. + async #persistBatch(touched, fromSeq, toSeq) { + const records = {}; + for (const [id, entry] of touched) records[id] = await calculateHash(entry.data); + const from = this.#log.formatId(fromSeq); + const to = this.#log.formatId(toSeq); + const blockHash = await this.#chain.addBlock({ records, from, to }); + for (const [id, entry] of touched) await this.#writeEntry(id, entry, blockHash); + return blockHash; + } + + // --------------------------------------------------------------------- + // The one local write path. Builds the op, applies it, logs it (durable + // seq), persists the entry, then announces it. + + [COMMIT](type, id, payload, { source = null, via = null, from = null } = {}) { + return this.#run(async () => { + const current = await this.#load(id); + if (current?.encrypted) { + // An encrypted record has no op log: writes to it stay local. Set + // and write map onto the encrypted save; a counter needs ops. + if (type === 'write') return void (await this.#writeEncrypted(id, payload.data, source)); + if (type === 'set') return void (await this.#writeEncrypted(id, { ...current.data, ...payload.data }, source)); + if (type === 'delete') return void (await this.#removeEncrypted(id, source)); + throw new Error(`Record ${id} is encrypted; ${type} needs a replicated record`); + } + const data = payload.data ?? current?.data ?? {}; + const scope = this.#scope(id, data); + const op = { origin: this.#nodeId, seq: 0, ts: this.#clock.now(), type, id }; + if (scope !== GLOBAL_SCOPE) op.scope = scope; + if (from) op.from = from; + // Sequence numbers run per stream (see vector.js opKey): the scope's, + // or the bootstrap stream for a merge. + op.seq = (this.#log.vector[opKey(op)] ?? 0) + 1; + if (payload.data !== undefined) op.data = payload.data; + if (payload.meta !== undefined) op.meta = payload.meta; + if (payload.key !== undefined) op.key = payload.key; + if (payload.value !== undefined) op.value = payload.value; + + const { entry, changed, delta, rejected } = applyOp(current, op); + if (rejected.length > 0) { + const [{ field, reason }] = rejected; + const hint = reason === 'counter' ? 'a counter; use inc()' : 'a register; use update()'; + throw new Error(`Field ${field} of record ${id} is ${hint}`); + } + if (!changed) return null; + + const [{ seq: position }] = await this.#log.append([op]); + const hash = await calculateHash(entry.data); + const blockHash = await this.#chain.addBlock({ id, hash, from: this.#log.formatId(position) }); + await this.#writeEntry(id, entry, blockHash); + + const alive = isAlive(entry); + this.#emitUpdate(id, alive ? entry.data : null, delta, source); + // Announced after everything is durable. `via` is null for a local + // write; a merge made from a peer's snapshot names that peer, so the + // relay does not hand its own state straight back to it. + await this.emit('change', op, via); + return op; + }); + } + + // --------------------------------------------------------------------- + // Public API + + // Encrypted at rest for THIS node's key: written directly, outside the op + // log, because no peer could read it. Runs inside the mutation queue. + async #writeEncrypted(id, data, source = null) { + const record = await encrypt(data, this.#keys.publicKey); + const hash = await calculateHash(record); + const blockHash = await this.#chain.addBlock({ id, hash }); + const entry = { data: record, meta: null, encrypted: true, timestamp: Date.now(), block: blockHash }; + await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); + this.#cache.set(id, { data, meta: null, encrypted: true, block: blockHash }); + this.#emitUpdate(id, data, { ...data }, source); + } + + async #removeEncrypted(id, source = null) { + await fsi.remove(`${this.#path}/data`, `${id}.json`); + this.#cache.set(id, null); + this.#emitUpdate(id, null, {}, source); + } + + async saveData(id, data, options = {}) { + const { encrypted = this.#encrypted } = options; + if (!encrypted) return void (await this[COMMIT]('write', id, { data })); + const current = await this.#load(id); + if (current && !current.encrypted) throw new Error(`Record ${id} is replicated and cannot become encrypted`); + await this.#run(() => this.#writeEncrypted(id, data)); + } + + async loadData(id) { + return this.get(id); } async validate(idOrOptions = {}, data, blockHash) { // Two shapes share one method: `validate(id, data, blockHash)` checks a // single record against its block, anything else walks the chain. A record - // id is always a string, so that is the discriminator โ€” keying off `'last' - // in idOrOptions` instead made the no-argument and `{}` calls fall through - // to the record branch, where they threw or silently returned false. + // id is always a string, so that is the discriminator. const isChainValidation = typeof idOrOptions !== 'string' && !data && !blockHash; if (isChainValidation) { return await this.#chain.validate(idOrOptions); @@ -273,65 +475,72 @@ class Storage { const block = await this.#chain.readBlock(blockHash); const expectedHash = await calculateHash(data); if (!block) return false; + // A local write commits `{ id, hash }`; a replicated batch commits + // `{ records: { [id]: hash } }` for every record it touched. + if (block.data.records) return block.data.records[id] === expectedHash; return block.data.hash === expectedHash && block.data.id === id; } catch { return false; } } - async insert(data) { + async insert(data, options = {}) { // `getId` may be sync or async โ€” both call sites already await this. const id = await this.#getId(); - await this.saveData(id, data); + await this.saveData(id, data, options); return id; } async has(id) { - return fsi.exists(`${this.#path}/data`, `${id}.json`); + const entry = await this.#load(id); + return entry !== null && (entry.encrypted || isAlive(entry)); } hasRecord(id) { - return this.#cache.has(id) || this.#records.has(id); + return (this.#cache.has(id) && this.#cache.get(id) !== null) || this.#records.has(id); } async get(id) { - return await this.loadData(id); - } - - async set(id, data) { - const oldData = await this.get(id); - await this.saveData(id, data); - if (oldData) { - const delta = calculateDelta(oldData, data); - this.#emitUpdate(id, data, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); + const entry = await this.#load(id); + if (entry === null) return null; + if (entry.encrypted) return entry.data; + return isAlive(entry) ? entry.data : null; + } + + /** Every live record id โ€” tombstones excluded, encrypted records included. */ + async list() { + const names = await fsi.list(`${this.#path}/data`); + const ids = []; + for (const name of names) { + if (!name.endsWith('.json')) continue; + const id = name.slice(0, -5); + const entry = await this.#load(id); + if (entry && (entry.encrypted || isAlive(entry))) ids.push(id); } + return ids; + } + + async set(id, data, options = {}) { + if (!isPlainObject(data)) throw new TypeError('set() takes a plain object'); + await this.saveData(id, data, options); } async delete(id) { - await fsi.remove(`${this.#path}/data`, `${id}.json`); + await this[COMMIT]('delete', id, {}); this.#records.delete(id); - this.#cache.delete(id); - this.#updates.push({ type: 'delete', id, timestamp: Date.now() }); } async update(id, delta) { - const data = await this.get(id); - if (!data) throw new Error(`Record ${id} not found`); - const updated = { ...data, ...delta }; - await this.saveData(id, updated); - this.#emitUpdate(id, updated, delta); - this.#updates.push({ - type: 'delta', - id, - timestamp: Date.now(), - data: delta, - }); + if (!(await this.has(id))) throw new Error(`Record ${id} not found`); + if (!isPlainObject(delta)) throw new TypeError('update() takes a plain object'); + await this[COMMIT]('set', id, { data: delta }); + } + + /** Adds `value` to counter field `key` โ€” the commutative alternative to update(). */ + async inc(id, key, value = 1) { + if (typeof key !== 'string') throw new TypeError('inc() takes a field name'); + if (!Number.isFinite(value)) throw new TypeError('inc() takes a finite number'); + await this[COMMIT]('inc', id, { key, value }); } async swap(id, changes, prev) { @@ -339,9 +548,7 @@ class Storage { if (!data) throw new Error(`Record ${id} not found`); const matches = Object.keys(prev).every((key) => data[key] === prev[key]); if (!matches) return false; - const updated = { ...data, ...changes }; - await this.saveData(id, updated); - this.#emitUpdate(id, updated, changes); + await this[COMMIT]('set', id, { data: changes }); return true; } @@ -354,11 +561,172 @@ class Storage { } getCachedData(id) { - return this.#cache.has(id) ? this.#cache.get(id) : null; + const entry = this.#cache.get(id); + if (!entry) return null; + if (entry.encrypted) return entry.data; + return isAlive(entry) ? entry.data : null; + } + + // --------------------------------------------------------------------- + // Replication + + /** + * Applies ops received from a peer. Each origin's ops must arrive in seq + * order: an op already covered by this node's vector is a duplicate and is + * skipped, one that skips ahead is a gap and is skipped too (the peer will + * be asked again from the last position). Accepted ops are logged, whether + * or not they changed anything, so the vector advances and they are not + * offered again; `via` names the link they arrived on, so the relay does + * not echo them straight back. + */ + applyRemote(ops, { via = null, scopes = null } = {}) { + return this.#run(async () => { + const accepted = []; + const touched = new Map(); + const deltas = new Map(); + const vector = this.#log.vector; + let adopted = null; + let duplicates = 0; + let gaps = 0; + for (const op of ops) { + const isAdvance = isPlainObject(op) && op.type === 'advance' && isPlainObject(op.vector); + if ( + !isPlainObject(op) || + typeof op.origin !== 'string' || + !Number.isInteger(op.seq) || + (!isAdvance && typeof op.id !== 'string') + ) { + this.#logger.warn({ event: 'sync.op.malformed', op }); + continue; + } + if (op.origin === this.#nodeId || vectorHas(vector, op)) { + duplicates++; + continue; + } + if (!vectorNext(vector, op)) { + gaps++; + continue; + } + if (isAdvance) { + // The peer joined a snapshot covering these seqs; so, through its + // merges, have we โ€” for the scopes we take from it. + for (const key in op.vector) { + const scope = keyScope(key); + const inScope = scope === null || !scopes || scopes.includes('*') || scopes.includes(scope); + if (inScope && Number.isInteger(op.vector[key]) && (vector[key] ?? 0) < op.vector[key]) { + vector[key] = op.vector[key]; + (adopted ??= {})[key] = op.vector[key]; + } + } + vectorAdvance(vector, op); + accepted.push(op); + continue; + } + const current = touched.get(op.id) ?? (await this.#load(op.id)); + if (current?.encrypted) { + this.#logger.warn({ event: 'sync.op.encrypted', id: op.id }); + vectorAdvance(vector, op); + accepted.push(op); + continue; + } + this.#clock.observe(op.ts); + const { entry, changed, delta, rejected } = applyOp(current, op); + if (rejected.length > 0) { + this.#logger.warn({ event: 'sync.op.rejected', id: op.id, rejected }); + await this.emit('error', Object.assign(new Error(`Refused op on ${op.id}`), { op, rejected })); + } + vectorAdvance(vector, op); + accepted.push(op); + if (!changed) continue; + touched.set(op.id, entry); + deltas.set(op.id, { ...(deltas.get(op.id) ?? {}), ...delta }); + } + if (accepted.length === 0) return { applied: 0, duplicates, gaps }; + + const entries = await this.#log.append(accepted); + if (adopted) await this.#log.advance(adopted); + if (touched.size > 0) { + await this.#persistBatch(touched, entries[0].seq, entries[entries.length - 1].seq); + for (const [id, entry] of touched) { + this.#emitUpdate(id, isAlive(entry) ? entry.data : null, deltas.get(id)); + } + } + for (const op of accepted) await this.emit('change', op, via); + return { applied: accepted.length, duplicates, gaps }; + }); } - get sync() { - return this.#sync; + /** + * One page of this node's state for a peer that cannot be resumed from + * the log: live records AND tombstones, with their metadata, in id order. + * `cursor` is the last id of the previous page; `null` back means done. + */ + async snapshot({ cursor = null, limit = 500, scopes = null } = {}) { + const names = await fsi.list(`${this.#path}/data`); + const records = []; + let last = cursor; + for (const name of names) { + if (!name.endsWith('.json')) continue; + const id = name.slice(0, -5); + if (cursor !== null && id <= cursor) continue; + const entry = await this.#load(id); + last = id; + if (!entry || entry.encrypted) continue; + const scope = this.#scope(id, entry.data); + if (scopes && scope !== GLOBAL_SCOPE && !scopes.includes(GLOBAL_SCOPE) && !scopes.includes(scope)) continue; + records.push({ id, data: entry.data, meta: entry.meta }); + if (records.length >= limit) { + return { records, cursor: last, eventId: this.#log.lastEventId, vector: this.#log.vector }; + } + } + return { records, cursor: null, eventId: this.#log.lastEventId, vector: this.#log.vector }; + } + + /** + * Joins snapshot records into this node's state. Each becomes a `merge` op + * of this node's own, so the joined state also reaches the peers that + * resume from THIS node's log โ€” bootstrap is transitive. + */ + async mergeSnapshot(records, { via = null, from = null } = {}) { + for (const record of records) { + if (!isPlainObject(record) || typeof record.id !== 'string') continue; + await this[COMMIT]('merge', record.id, { data: record.data, meta: record.meta }, { via, from }); + } + } + + /** + * Records that everything up to `vector` is reflected in this node's state + * (a snapshot of it was just joined). It travels as an `advance` op in + * this node's own log, so a peer replaying the merges that carried the + * state also learns which ops they cover โ€” and does not report the next + * live op of those origins as a gap. + */ + adoptVector(vector, { via = null, from = null } = {}) { + return this.#run(async () => { + const known = this.#log.vector; + const news = {}; + for (const key in vector) { + if (Number.isInteger(vector[key]) && (known[key] ?? 0) < vector[key]) news[key] = vector[key]; + } + if (Object.keys(news).length === 0) return null; + const op = { origin: this.#nodeId, seq: 0, ts: this.#clock.now(), type: 'advance', vector: news }; + if (from) op.from = from; + op.seq = (known[opKey(op)] ?? 0) + 1; + await this.#log.append([op]); + await this.#log.advance(news); + await this.emit('change', op, via); + return op; + }); + } + + /** Stops replication and seals the log. Further writes are refused. */ + async close() { + if (this.#closed) return; + await this.#sync.stop(); + await this.#run(async () => { + await this.#log.close(); + }); + this.#closed = true; } } diff --git a/src/sync.browser.js b/src/sync.browser.js deleted file mode 100644 index c345956..0000000 --- a/src/sync.browser.js +++ /dev/null @@ -1,54 +0,0 @@ -'use strict'; - -class SyncManager { - #nodes = new Set(); - - constructor() { - return this.#init(); - } - - async #init() { - await this.start(); - return this; - } - - async start() { - // TODO: no transport yet โ€” see the note in ./sync.js. A browser - // WebSocket client goes here; WebRTC would follow for peer-to-peer. - const uplinks = Array.from(this.#nodes).filter((node) => node.type === 'uplink' && node.enabled); - if (uplinks.length === 0) return; - for (const uplink of uplinks) { - try { - // Sync here - } catch (error) { - console.error(`Sync failed with ${uplink.url}:`, error.message); - } - } - } - - async addNode(nodeData) { - const { url, type, token } = nodeData; - if (!url) throw new Error('Node URL is required'); - if (type !== 'uplink' && type !== 'downlink') { - throw new Error('Node type must be uplink or downlink'); - } - const existing = Array.from(this.#nodes).find((n) => n.url === url); - if (existing) throw new Error(`Node with URL ${url} already exists`); - const now = Date.now(); - const node = { type, url, token, enabled: true, lastSync: now }; - this.#nodes.add(node); - await this.start(); - } - - async removeNode(url) { - const node = Array.from(this.#nodes).find((n) => n.url === url); - if (!node) throw new Error(`Node with URL ${url} not found`); - this.#nodes.delete(node); - } - - listNodes() { - return Array.from(this.#nodes); - } -} - -module.exports = { SyncManager }; diff --git a/src/sync.js b/src/sync.js index dfc7f97..2bfd58a 100644 --- a/src/sync.js +++ b/src/sync.js @@ -1,77 +1,635 @@ 'use strict'; -const fs = require('node:fs/promises'); -const path = require('node:path'); -const { between } = require('./utils.js'); +// Replication between nodes. Two halves live here: +// +// SyncManager โ€” the DIALING side. For every registered node it asks the +// injected `connect(node)` factory for a link (a duck-typed SyncLink โ€” the +// `@alexify/syncom/wrpc` subpath wraps a WrpcClient into one), then runs +// the handshake: hello, push what the host lacks, subscribe to what the +// host learns from now on. Reconnecting, backing off and resuming the +// subscription from the last event id are the LINK's job (wrpc already +// does all three); on `reconnect` the manager only repeats hello + push, +// the outbound half a transport cannot know about. +// +// unit() โ€” the HOSTING side: a plain object of `{ access, handler }` +// procedures shaped for wrpc's defineRouter, with no import of wrpc. +// `changes` is an async generator, which wrpc treats as a subscription; +// its values are labelled with this node's log position through the +// `tracked` shape (Symbol.for('wrpc.tracked'), part of wrpc's frozen wire +// protocol), so a resuming subscriber replays exactly what it missed. +// +// Every node is both: a mesh is nodes dialing whichever nodes they can +// reach, with ops relayed transitively and de-duplicated by version vector. -class SyncManager { +const fsi = require('./fsi.js'); +const { Emitter } = require('./utils.js'); +const { packOps, unpackOps } = require('./ops.js'); +const { vectorHas, keyScope } = require('./vector.js'); +const { isSyncLink } = require('./link.js'); + +const REMOTE = 'remote'; +const PROTOCOL = 1; +const GLOBAL_SCOPE = '*'; +const DEFAULT_PAGE_SIZE = 500; +const DEFAULT_PAGE_BYTES = 4 << 20; +const TRACKED = Symbol.for('wrpc.tracked'); + +// Node registry entries are keyed by url; the file name is the url itself, +// percent-encoded โ€” unlike the old `.json`, two peers on one host or a +// url without a port cannot collide. +const fileName = (url) => `${encodeURIComponent(url)}.json`; + +const coded = (message, code) => { + const error = new Error(message); + error.code = code; + error.expose = true; + return error; +}; + +// The part of a vector that concerns `scopes`: global keys (`origin`) and +// the `origin@scope` keys of the scopes asked for. +const scopedVector = (vector, scopes) => { + if (!scopes || scopes.includes(GLOBAL_SCOPE)) return vector; + const out = {}; + for (const key in vector) { + const scope = keyScope(key); + if (scope === null || scopes.includes(scope)) out[key] = vector[key]; + } + return out; +}; + +const inScope = (op, scopes) => { + const scope = op.scope ?? GLOBAL_SCOPE; + return !scopes || scope === GLOBAL_SCOPE || scopes.includes(GLOBAL_SCOPE) || scopes.includes(scope); +}; + +// Resolves on the next change (or abort). The manager keeps ONE listener on +// the storage and wakes every waiting subscription from it, because Emitter +// refuses more than ten listeners and a host has as many `changes` feeds as +// it has peers. +const waitFor = (wakers, signal) => + new Promise((resolve) => { + if (signal?.aborted) return void resolve(); + const done = () => { + wakers.delete(done); + signal?.removeEventListener?.('abort', done); + resolve(); + }; + wakers.add(done); + signal?.addEventListener?.('abort', done, { once: true }); + }); + +class SyncManager extends Emitter { + #storage; #path; - #nodes = new Set(); + #options; + #nodes = new Map(); + #peers = new Map(); + #wakers = new Set(); + #onChange = null; + #started = false; + #saving = new Map(); - constructor(basePath) { - this.#path = path.join(basePath, 'remote'); + constructor(storage, options = {}) { + super(); + this.#storage = storage; + this.#options = options ?? {}; + this.#path = `${storage.path}/${REMOTE}`; return this.#init(); } async #init() { - await fs.mkdir(this.#path, { recursive: true }); - const files = await fs.readdir(this.#path); + const files = await fsi.list(this.#path); for (const file of files) { if (!file.endsWith('.json')) continue; - const filePath = path.join(this.#path, file); - const raw = await fs.readFile(filePath, 'utf8'); + const raw = await fsi.read(this.#path, file); const node = JSON.parse(raw); - this.#nodes.add(node); + if (node && typeof node.url === 'string') this.#nodes.set(node.url, node); } - await this.start(); + // One listener for both halves: the dialing side's fan-out to its + // links, and the hosting side's wake-up of every live `changes` feed. + this.#onChange = (op, via) => this.#fanOut(op, via); + this.#storage.on('change', this.#onChange); + const { connect, autoStart = typeof connect === 'function' } = this.#options; + if (autoStart && typeof connect === 'function') await this.start(); return this; } - async start() { - // TODO: no transport yet โ€” start() walks the enabled uplinks and does - // nothing with them. A WebSocket client that exchanges CRDT delta logs - // goes here. - const uplinks = Array.from(this.#nodes).filter((node) => node.type === 'uplink' && node.enabled); - if (uplinks.length === 0) return; - for (const uplink of uplinks) { - try { - // Sync here - } catch (error) { - console.error(`Sync failed with ${uplink.url}:`, error.message); - } - } + get storage() { + return this.#storage; } + get started() { + return this.#started; + } + + // --------------------------------------------------------------------- + // Registry + async addNode(nodeData) { - const { url, type, token } = nodeData; - if (!url) throw new Error('Node URL is required'); - if (type !== 'uplink' && type !== 'downlink') { - throw new Error('Node type must be uplink or downlink'); + const { url, token, scopes } = nodeData ?? {}; + if (typeof url !== 'string' || url.length === 0) throw new Error('Node URL is required'); + if (this.#nodes.has(url)) throw new Error(`Node with URL ${url} already exists`); + if (scopes !== undefined && !(Array.isArray(scopes) && scopes.every((s) => typeof s === 'string'))) { + throw new Error('Node scopes must be an array of strings'); } - const existing = Array.from(this.#nodes).find((n) => n.url === url); - if (existing) throw new Error(`Node with URL ${url} already exists`); - const now = Date.now(); - const node = { type, url, token, enabled: true, lastSync: now }; - this.#nodes.add(node); - await fs.mkdir(this.#path, { recursive: true }); - const domain = between(url, '://', ':'); - const filePath = path.join(this.#path, `${domain}.json`); - await fs.writeFile(filePath, JSON.stringify(node)); - await this.start(); + const node = { + url, + token, + scopes: scopes ?? [GLOBAL_SCOPE], + enabled: true, + lastSync: null, + cursor: null, + remote: null, + }; + this.#nodes.set(url, node); + await fsi.write(this.#path, fileName(url), JSON.stringify(node)); + if (this.#started) this.#connect(node); } async removeNode(url) { - const node = Array.from(this.#nodes).find((n) => n.url === url); + const node = this.#nodes.get(url); if (!node) throw new Error(`Node with URL ${url} not found`); - this.#nodes.delete(node); - const domain = between(url, '://', ':'); - const filePath = path.join(this.#path, `${domain}.json`); - await fs.unlink(filePath); + this.#nodes.delete(url); + this.#disconnect(url); + await fsi.remove(this.#path, fileName(url)); } listNodes() { - return Array.from(this.#nodes); + return Array.from(this.#nodes.values(), (node) => ({ ...node })); + } + + async enableNode(url, enabled = true) { + const node = this.#nodes.get(url); + if (!node) throw new Error(`Node with URL ${url} not found`); + node.enabled = enabled; + await this.#saveNode(node); + if (!enabled) this.#disconnect(url); + else if (this.#started) this.#connect(node); + } + + // Registry writes are coalesced per node: cursors move on every batch. + #saveNode(node) { + if (!this.#nodes.has(node.url)) return Promise.resolve(); + let pending = this.#saving.get(node.url); + if (pending) return pending; + pending = Promise.resolve().then(async () => { + this.#saving.delete(node.url); + if (!this.#nodes.has(node.url)) return; + await fsi.write(this.#path, fileName(node.url), JSON.stringify(node)); + }); + this.#saving.set(node.url, pending); + return pending; + } + + // --------------------------------------------------------------------- + // The dialing side + + async start() { + if (this.#started) return; + if (typeof this.#options.connect !== 'function') { + throw new Error('sync.start() needs a connect(node) factory in the storage sync options'); + } + this.#started = true; + for (const node of this.#nodes.values()) { + if (node.enabled) this.#connect(node); + } + } + + async stop() { + this.#started = false; + for (const url of Array.from(this.#peers.keys())) this.#disconnect(url); + await Promise.all(Array.from(this.#saving.values())); + } + + /** Pushes whatever the connected peers lack right now; resolves when done. */ + async sync(url) { + const peers = url ? [this.#peers.get(url)].filter(Boolean) : Array.from(this.#peers.values()); + const results = []; + for (const peer of peers) { + if (!peer.remote) continue; + results.push({ url: peer.node.url, pushed: await this.#push(peer) }); + } + return results; + } + + /** Live view of every dialed peer. */ + peers() { + return Array.from(this.#peers.values(), (peer) => ({ + url: peer.node.url, + node: peer.remote?.id ?? null, + connected: peer.connected, + subscribed: Boolean(peer.subscription), + cursor: peer.node.cursor, + lastSync: peer.node.lastSync, + pending: peer.outbox.length + (peer.pushing ? 1 : 0), + })); + } + + #connect(node) { + if (this.#peers.has(node.url)) return; + const peer = { + node, + link: null, + remote: null, + connected: false, + subscription: null, + outbox: [], + pushing: null, + bootstrapping: false, + handshaking: null, + closed: false, + handlers: null, + }; + this.#peers.set(node.url, peer); + Promise.resolve() + .then(() => this.#options.connect(node)) + .then((link) => { + if (peer.closed) return void link?.close?.(); + if (!isSyncLink(link)) { + throw new TypeError('connect(node) must return a SyncLink (call/subscribe/on/off/close)'); + } + peer.link = link; + peer.handlers = { + open: () => { + peer.connected = true; + }, + close: () => { + peer.connected = false; + peer.subscription = null; + void this.emit('peer', { url: node.url, connected: false }); + }, + reconnect: () => { + peer.connected = true; + void this.emit('peer', { url: node.url, connected: true, reconnected: true }); + // The link restored the subscription itself (when the host still + // knows us); what it cannot restore is the outbound half. + this.#handshake(peer, { resubscribe: !peer.subscription }).catch((error) => this.#fail(peer, error)); + }, + }; + for (const event in peer.handlers) link.on(event, peer.handlers[event]); + peer.connected = true; + return this.#handshake(peer, { resubscribe: true }); + }) + .catch((error) => this.#fail(peer, error)); + } + + #disconnect(url) { + const peer = this.#peers.get(url); + if (!peer) return; + peer.closed = true; + this.#peers.delete(url); + if (peer.link) { + if (peer.handlers) for (const event in peer.handlers) peer.link.off(event, peer.handlers[event]); + peer.subscription?.unsubscribe?.(); + try { + peer.link.close(); + } catch { + // A link already gone is what close() was for. + } + } + } + + #fail(peer, error) { + if (peer.closed) return; + this.#storage.logger.warn({ event: 'sync.error', url: peer.node.url, err: error }); + if (this.listenerCount('error') > 0) void this.emit('error', Object.assign(error, { url: peer.node.url })); + } + + // Single-flight: a reconnect and a refused restored subscription can both + // ask for one in the same tick. + #handshake(peer, options) { + if (peer.handshaking) return peer.handshaking; + peer.handshaking = this.#hello(peer, options).finally(() => { + peer.handshaking = null; + }); + return peer.handshaking; + } + + async #hello(peer, { resubscribe }) { + const { node, link } = peer; + const answer = await link.call('syncom/hello', { + node: this.#storage.nodeId, + vector: this.#storage.vector, + version: PROTOCOL, + scopes: node.scopes, + // For `authorize(context, hello)`; a transport-level credential (a + // wrpc `headers`/`authenticate` option) is the other way to carry it. + token: node.token, + }); + if (peer.closed) return; + if (!answer || typeof answer.node !== 'string') throw new Error(`Bad hello from ${node.url}`); + if (answer.duplicate) { + // The host already dials us; one link is enough. + this.#disconnect(node.url); + return; + } + // A host reinstalled since we last spoke: its log speaks another epoch, + // so our cursor is void and the subscription will say `bootstrap`. + if (node.remote && node.remote.epoch !== answer.epoch) node.cursor = null; + peer.remote = { id: answer.node, vector: answer.vector ?? {}, epoch: answer.epoch }; + node.remote = { id: answer.node, epoch: answer.epoch }; + await this.#saveNode(node); + await this.emit('peer', { url: node.url, connected: true, node: answer.node }); + await this.#push(peer); + if (resubscribe && !peer.subscription) this.#subscribe(peer); + } + + // Everything in our log the host has not seen, by its vector โ€” ops of its + // own origin excluded (it has those by definition). A host that is behind + // what we still retain cannot be caught up from here; it will bootstrap + // from a node that dials it, or we from it. + async #push(peer) { + const { node } = peer; + const entries = + this.#storage.log.since(undefined) ?? + this.#storage.log.since(this.#storage.log.formatId(this.#storage.log.floor - 1)) ?? + []; + const ops = []; + for (const { op } of entries) { + if (op.origin === peer.remote.id || op.from === peer.remote.id) continue; + if (vectorHas(peer.remote.vector, op)) continue; + if (!inScope(op, node.scopes)) continue; + ops.push(op); + } + return this.#send(peer, ops); + } + + async #send(peer, ops) { + const pageSize = this.#options.pageSize ?? DEFAULT_PAGE_SIZE; + let sent = 0; + for (let index = 0; index < ops.length; index += pageSize) { + const page = ops.slice(index, index + pageSize); + const answer = await peer.link.call('syncom/push', { ops: packOps(page) }); + if (answer?.vector) peer.remote.vector = answer.vector; + sent += page.length; + } + if (sent > 0) { + peer.node.lastSync = Date.now(); + await this.#saveNode(peer.node); + await this.emit('sync', { url: peer.node.url, pushed: sent }); + } + return sent; + } + + #subscribe(peer) { + const { node, link } = peer; + peer.subscription = link.subscribe( + 'syncom/changes', + { scopes: node.scopes }, + { + lastEventId: node.cursor ?? undefined, + onData: (value, eventId) => void this.#receive(peer, value, eventId).catch((error) => this.#fail(peer, error)), + onError: (error) => { + peer.subscription = null; + if (peer.closed) return; + // 428 "say hello first": the host forgot us โ€” on wrpc a reconnect + // is a new server-side client, and its restored subscription + // arrives before our hello does. Greet again, then subscribe. + if (error?.code === 428 && peer.connected) { + this.#handshake(peer, { resubscribe: true }).catch((e) => this.#fail(peer, e)); + return; + } + this.#fail(peer, error); + }, + onEnd: () => { + peer.subscription = null; + }, + }, + ); + } + + async #receive(peer, value, eventId) { + if (peer.closed || !value) return; + if (value.bootstrap) return void (await this.#bootstrap(peer)); + const ops = unpackOps(value.ops); + const result = await this.#storage.applyRemote(ops, { via: peer, scopes: peer.node.scopes }); + if (eventId) peer.node.cursor = eventId; + peer.node.lastSync = Date.now(); + await this.#saveNode(peer.node); + await this.emit('sync', { url: peer.node.url, received: ops.length, ...result }); + } + + // Bootstrap: page through the host's whole state, joining each page, then + // subscribe from the position the FIRST page was taken at โ€” anything that + // landed while paging is replayed and folds in as a no-op. + async #bootstrap(peer) { + if (peer.bootstrapping) return; + peer.bootstrapping = true; + const { node, link } = peer; + try { + peer.subscription?.unsubscribe?.(); + peer.subscription = null; + let cursor = null; + let startEventId = null; + let startVector = null; + let total = 0; + do { + const page = await link.call('syncom/snapshot', { cursor, scopes: node.scopes }); + if (peer.closed) return; + startEventId ??= page.eventId; + startVector ??= page.vector ?? null; + const records = page.id !== undefined ? await this.#readSnapshot(link, page) : page.records; + await this.#storage.mergeSnapshot(records, { via: peer, from: peer.remote.id }); + total += records.length; + cursor = page.cursor ?? null; + } while (cursor !== null); + // Everything up to the host's vector at the FIRST page is inside what + // was joined; adopting it (for the scopes asked for) means nobody + // offers those ops again. Later ops replay from the cursor. + if (startVector) { + await this.#storage.adoptVector(scopedVector(startVector, node.scopes), { via: peer, from: peer.remote.id }); + } + node.cursor = startEventId; + node.lastSync = Date.now(); + await this.#saveNode(node); + await this.emit('bootstrap', { url: node.url, records: total }); + if (!peer.closed) this.#subscribe(peer); + } finally { + peer.bootstrapping = false; + } + } + + // A streamed page: NDJSON, one `{ id, data, meta }` per line, parsed as the + // chunks arrive so a page costs one record of memory, not one page. + async #readSnapshot(link, page) { + if (typeof link.stream !== 'function') { + throw new Error('The host streamed a snapshot but the link cannot read streams'); + } + const decoder = new TextDecoder(); + const records = []; + let rest = ''; + for await (const chunk of link.stream(page.id)) { + rest += typeof chunk === 'string' ? chunk : decoder.decode(chunk, { stream: true }); + let newline = rest.indexOf('\n'); + while (newline !== -1) { + const line = rest.slice(0, newline); + rest = rest.slice(newline + 1); + if (line.length > 0) records.push(JSON.parse(line)); + newline = rest.indexOf('\n'); + } + } + rest += decoder.decode(); + if (rest.trim().length > 0) records.push(JSON.parse(rest)); + return records; + } + + // One op landed here (locally or from a peer): wake the hosting feeds, and + // queue it for every dialed peer except the one it came from. + #fanOut(op, via) { + for (const wake of Array.from(this.#wakers)) wake(); + for (const peer of this.#peers.values()) { + if (peer === via || !peer.remote || !peer.connected) continue; + if (op.origin === peer.remote.id || op.from === peer.remote.id) continue; + if (vectorHas(peer.remote.vector, op)) continue; + if (!inScope(op, peer.node.scopes)) continue; + peer.outbox.push(op); + this.#scheduleFlush(peer); + } + } + + // Ops that land in one tick travel as one push; pushes to a peer never + // overlap, so its vector answer is always the latest. + #scheduleFlush(peer) { + if (peer.pushing) return; + peer.pushing = Promise.resolve() + .then(async () => { + while (peer.outbox.length > 0 && !peer.closed && peer.connected) { + const ops = peer.outbox; + peer.outbox = []; + await this.#send(peer, ops); + } + }) + .catch((error) => { + // Dropped on purpose: the next hello/push recomputes from the vector. + peer.outbox = []; + this.#fail(peer, error); + }) + .finally(() => { + peer.pushing = null; + if (peer.outbox.length > 0 && !peer.closed && peer.connected) this.#scheduleFlush(peer); + }); + } + + // --------------------------------------------------------------------- + // The hosting side + + /** + * The router unit a wrpc server (or anything shaped like one) hosts: + * `defineRouter({ syncom: storage.sync.unit({ access: 'public' }) })`. + * `access` defaults to wrpc's own default, 'session'; `authorize(context, + * hello)` gates a peer by whatever the transport presented. + */ + unit(options = {}) { + const { + access = 'session', + authorize = null, + tracked = null, + pageSize = DEFAULT_PAGE_SIZE, + pageBytes = DEFAULT_PAGE_BYTES, + } = options; + const storage = this.#storage; + const wakers = this.#wakers; + const track = tracked ?? ((id, data) => ({ [TRACKED]: true, id: String(id), data })); + + const peerOf = (context) => { + const state = context?.client?.data?.syncom; + if (!state) throw coded('Say hello first', 428); + return state; + }; + + return { + hello: { + access, + handler: async (context, args) => { + if (!args || typeof args.node !== 'string') throw coded('hello needs a node id', 400); + if (args.version !== undefined && args.version !== PROTOCOL) { + throw coded(`Unsupported protocol ${args.version}`, 400); + } + if (args.node === storage.nodeId) throw coded('A node cannot sync with itself (copied data directory?)', 409); + if (authorize && !(await authorize(context, args))) throw coded('Not authorized to sync', 403); + const scopes = Array.isArray(args.scopes) ? args.scopes.filter((s) => typeof s === 'string') : [GLOBAL_SCOPE]; + if (context?.client) { + context.client.data ??= {}; + context.client.data.syncom = { node: args.node, scopes, vector: args.vector ?? {} }; + if (typeof context.client.join === 'function') { + for (const scope of scopes) context.client.join(`syncom:${scope}`); + } + } + return { node: storage.nodeId, vector: storage.vector, epoch: storage.log.epoch, version: PROTOCOL }; + }, + }, + + push: { + access, + handler: async (context, args) => { + const { scopes } = peerOf(context); + const ops = unpackOps(args?.ops); + const result = await storage.applyRemote(ops, { via: context.client, scopes }); + return { vector: storage.vector, ...result }; + }, + }, + + snapshot: { + access, + handler: async (context, args) => { + peerOf(context); + const page = await storage.snapshot({ + cursor: args?.cursor ?? null, + limit: pageSize, + scopes: args?.scopes ?? null, + }); + const client = context?.client; + const canStream = client && client.binary !== false && typeof client.createStream === 'function'; + if (!canStream || page.records.length === 0) return page; + // NDJSON, serialized first because a wrpc stream declares its size + // up front; a page is bounded, so that buffer is too. + let text = ''; + for (const record of page.records) text += `${JSON.stringify(record)}\n`; + const bytes = new TextEncoder().encode(text); + if (bytes.length > pageBytes) return page; + const stream = client.createStream('syncom/snapshot', bytes.length); + // The opening packet is already out; writing in this tick would + // race the caller's chance to attach a consumer. + queueMicrotask(() => { + stream.write(bytes); + stream.end(); + }); + return { id: stream.id, size: bytes.length, cursor: page.cursor, eventId: page.eventId, vector: page.vector }; + }, + }, + + changes: { + access, + handler: async function* (context, args, subscription = {}) { + const { node } = peerOf(context); + const { lastEventId, signal } = subscription; + const scopes = Array.isArray(args?.scopes) ? args.scopes : null; + let cursor = lastEventId ?? undefined; + for (;;) { + if (signal?.aborted) return; + const entries = storage.log.since(cursor, { scopes, limit: pageSize }); + if (entries === null) { + // Cannot resume: another epoch, or compacted past the cursor. + yield { bootstrap: true }; + return; + } + if (entries.length === 0) { + await waitFor(wakers, signal); + continue; + } + const last = entries[entries.length - 1].seq; + const ops = []; + for (const { op } of entries) if (op.origin !== node && op.from !== node) ops.push(op); + cursor = storage.log.formatId(last); + // Labelled even when every op was the peer's own echo: the + // position still moves, and a resume must not replay it. + yield track(cursor, { ops: packOps(ops), eventId: cursor }); + } + }, + }, + }; } } -module.exports = { SyncManager }; +module.exports = { SyncManager, isSyncLink }; diff --git a/src/utils.browser.js b/src/utils.browser.js index 4640daa..c4def1f 100644 --- a/src/utils.browser.js +++ b/src/utils.browser.js @@ -108,6 +108,16 @@ class Emitter { const generateUUID = () => crypto.randomUUID(); +// A duck-typed logger writer: a Console, a pino-shaped logger, or `false` to +// go silent. Every level is a function, so callers never guard. +const NOOP = () => {}; +const createLogger = (logger) => { + if (logger === false || logger === null) return { debug: NOOP, info: NOOP, warn: NOOP, error: NOOP }; + const target = logger ?? globalThis.console; + const bind = (level) => (typeof target[level] === 'function' ? target[level].bind(target) : NOOP); + return { debug: bind('debug'), info: bind('info'), warn: bind('warn'), error: bind('error') }; +}; + const between = (s, prefix, suffix) => { let i = s.indexOf(prefix); if (i === -1) return ''; @@ -120,4 +130,4 @@ const between = (s, prefix, suffix) => { return s; }; -module.exports = { Emitter, generateUUID, between }; +module.exports = { Emitter, createLogger, generateUUID, between }; diff --git a/src/utils.js b/src/utils.js index fc3930f..40e6b5e 100644 --- a/src/utils.js +++ b/src/utils.js @@ -108,6 +108,16 @@ class Emitter { const generateUUID = crypto.randomUUID; +// A duck-typed logger writer: a Console, a pino-shaped logger, or `false` to +// go silent. Every level is a function, so callers never guard. +const NOOP = () => {}; +const createLogger = (logger) => { + if (logger === false || logger === null) return { debug: NOOP, info: NOOP, warn: NOOP, error: NOOP }; + const target = logger ?? globalThis.console; + const bind = (level) => (typeof target[level] === 'function' ? target[level].bind(target) : NOOP); + return { debug: bind('debug'), info: bind('info'), warn: bind('warn'), error: bind('error') }; +}; + const between = (s, prefix, suffix) => { let i = s.indexOf(prefix); if (i === -1) return ''; @@ -134,6 +144,7 @@ const directoryExists = async (path) => { module.exports = { Emitter, + createLogger, generateUUID, between, fileExists, diff --git a/src/vector.js b/src/vector.js new file mode 100644 index 0000000..8f719b7 --- /dev/null +++ b/src/vector.js @@ -0,0 +1,119 @@ +'use strict'; + +// Clocks, version vectors and event ids โ€” the three ordering primitives the +// sync engine is built on. Pure functions over plain values, shared by both +// halves. +// +// A CLOCK is `[ts, origin]`: a millisecond timestamp with the node id as the +// tie-break, so two stamps are never equal unless they are the same stamp. +// A VECTOR is `{ [origin]: seq }` โ€” the highest per-origin sequence number a +// node has applied. An EVENT ID is `'.'` โ€” one node's position in +// its own log, the cursor a subscriber hands back to resume (the shape wrpc's +// createEventLog uses, on purpose: it is what rides in `lastEventId`). + +const compareClock = (a, b) => { + if (!a) return b ? -1 : 0; + if (!b) return 1; + if (a[0] !== b[0]) return a[0] < b[0] ? -1 : 1; + if (a[1] === b[1]) return 0; + return a[1] < b[1] ? -1 : 1; +}; + +const maxClock = (a, b) => (compareClock(a, b) >= 0 ? a : b); + +const vectorMax = (a = {}, b = {}) => { + const result = { ...a }; + for (const origin in b) { + if (!(result[origin] >= b[origin])) result[origin] = b[origin]; + } + return result; +}; + +// True when `a` has seen everything `b` has. +const vectorDominates = (a = {}, b = {}) => { + for (const origin in b) { + if (!((a[origin] ?? 0) >= b[origin])) return false; + } + return true; +}; + +// Sequence numbers run per ORIGIN AND SCOPE, so a peer that syncs one scope +// still sees every seq of that scope in order โ€” a global counter would show +// it holes wherever an op of another scope sat. The vector key spells that +// out: `origin` for the global scope, `origin@scope` otherwise. +const vectorKey = (origin, scope) => (scope && scope !== '*' ? `${origin}@${scope}` : origin); + +// Ops born from joining a snapshot (`merge`, `advance`) run in a stream of +// their own: they never travel back to the node the snapshot came from, and +// a stream that node never sees must not be one whose holes it would count. +const BOOT_SCOPE = '~'; +const isBootstrapOp = (op) => op.type === 'merge' || op.type === 'advance'; +const opKey = (op) => vectorKey(op.origin, isBootstrapOp(op) ? BOOT_SCOPE : op.scope); + +// The scope a vector key belongs to: null for the global scope and for the +// bootstrap stream, which every subscriber takes. +const keyScope = (key) => { + const at = key.indexOf('@'); + if (at === -1) return null; + const scope = key.slice(at + 1); + return scope === BOOT_SCOPE ? null : scope; +}; + +// Whether an op is news to a node holding `vector`. +const vectorHas = (vector, op) => (vector[opKey(op)] ?? 0) >= op.seq; + +// The op is the next one expected from its origin in its scope. +const vectorNext = (vector, op) => (vector[opKey(op)] ?? 0) + 1 === op.seq; + +// Mutates: records that `op` has been applied. +const vectorAdvance = (vector, op) => { + const key = opKey(op); + if ((vector[key] ?? 0) < op.seq) vector[key] = op.seq; + return vector; +}; + +const formatEventId = (epoch, seq) => `${epoch}.${seq}`; + +const parseEventId = (value) => { + if (typeof value !== 'string') return null; + const dot = value.lastIndexOf('.'); + if (dot <= 0) return null; + const seq = Number(value.slice(dot + 1)); + if (!Number.isInteger(seq) || seq < 0) return null; + return { epoch: value.slice(0, dot), seq }; +}; + +// A Lamport-style wall clock: never repeats, never runs backwards, and never +// falls behind a stamp it has seen from a peer โ€” so a local write always wins +// last-writer-wins against everything this node already knows about, even +// when a peer's wall clock is ahead of ours. +class Clock { + #last = 0; + + now() { + const ts = Math.max(Date.now(), this.#last + 1); + this.#last = ts; + return ts; + } + + observe(ts) { + if (typeof ts === 'number' && ts > this.#last) this.#last = ts; + } +} + +module.exports = { + compareClock, + maxClock, + vectorMax, + vectorDominates, + vectorKey, + opKey, + keyScope, + isBootstrapOp, + vectorHas, + vectorNext, + vectorAdvance, + formatEventId, + parseEventId, + Clock, +}; diff --git a/src/wrpc.js b/src/wrpc.js new file mode 100644 index 0000000..dc42a47 --- /dev/null +++ b/src/wrpc.js @@ -0,0 +1,92 @@ +'use strict'; + +// The `@alexify/syncom/wrpc` subpath: adapts a wrpc client to the SyncLink +// contract the sync engine dials through. wrpc is INJECTED โ€” the caller +// constructs the WrpcClient and hands it over; nothing here requires +// '@alexify/wrpc', and the shape is checked structurally at the boundary, +// the way wrpc itself takes uws, fastify or a Redis client. +// +// const { WrpcClient } = require('@alexify/wrpc'); +// const { createWrpcLink } = require('@alexify/syncom/wrpc'); +// const storage = await open({ +// path: 'data', +// sync: { connect: async (node) => createWrpcLink(await WrpcClient.connect(node.url)) }, +// }); +// +// Everything the engine leaves to the transport โ€” reconnecting with backoff, +// the heartbeat, re-opening the `changes` subscription from its last event id +// โ€” is what a WrpcClient already does; the adapter only translates names. + +const { isSyncLink } = require('./link.js'); + +const UNIT = 'syncom'; + +// The introspection `client.use()` scaffolds `client.api.syncom` from, with +// no round trip: the engine calls procedures by name through `client.call`, +// and the one subscription needs a scaffolded handle to be resumable. +const SYNC_INTROSPECTION = Object.freeze({ + [UNIT]: Object.freeze({ + hello: Object.freeze({}), + push: Object.freeze({}), + snapshot: Object.freeze({}), + changes: Object.freeze({ kind: 'subscription' }), + }), +}); + +const isWrpcClient = (client) => + Boolean(client) && + typeof client.call === 'function' && + typeof client.use === 'function' && + typeof client.on === 'function' && + typeof client.off === 'function' && + typeof client.close === 'function'; + +/** + * Wraps a WrpcClient (or anything shaped like one) into a SyncLink. `unit` + * renames the router unit when the host mounted it under something other + * than `syncom`. + */ +const createWrpcLink = (client, { unit = UNIT } = {}) => { + if (!isWrpcClient(client)) { + throw new TypeError('createWrpcLink: expected a WrpcClient (call/use/on/off/close)'); + } + client.use({ [unit]: SYNC_INTROSPECTION[UNIT] }); + const target = (method) => { + const slash = method.indexOf('/'); + const name = slash === -1 ? method : method.slice(slash + 1); + return `${unit}/${name}`; + }; + + const link = { + call: (method, args) => client.call(target(method), args), + + subscribe(method, args, { lastEventId, onData, onError, onEnd } = {}) { + const name = target(method).slice(unit.length + 1); + const procedure = client.api?.[unit]?.[name]; + if (!procedure || typeof procedure.subscribe !== 'function') { + throw new Error(`createWrpcLink: ${unit}/${name} is not a subscription on this client`); + } + const handle = procedure.subscribe(args, { + lastEventId, + // wrpc updates the handle's lastEventId from the packet BEFORE it + // delivers the value, so the id read here labels this very value. + onData: (value) => onData?.(value, handle.lastEventId), + onError, + onEnd, + }); + return handle; + }, + + // A WrpcReadable is async-iterable; the engine reads the NDJSON page from it. + stream: (id) => client.getStream(id), + + on: (event, listener) => client.on(event, listener), + off: (event, listener) => client.off(event, listener), + close: () => client.close(), + // The wrapped client, for callers that need to reach past the contract. + client, + }; + return link; +}; + +module.exports = { createWrpcLink, isSyncLink, SYNC_INTROSPECTION }; diff --git a/tests/browser.test-d.ts b/tests/browser.test-d.ts index e17e6de..6a8aa8f 100644 --- a/tests/browser.test-d.ts +++ b/tests/browser.test-d.ts @@ -21,3 +21,6 @@ expectType>(browser.read('dir', 'file.json')); declare const storage: Storage; expectType(storage.sync); +expectType>(browser.list('dir')); +expectType>(storage.inc('id', 'hits')); +expectType(browser.isAlive(null)); diff --git a/tests/chain.test.js b/tests/chain.test.js index e5218d1..f0f7bbc 100644 --- a/tests/chain.test.js +++ b/tests/chain.test.js @@ -65,6 +65,8 @@ test('Chain module', async (t) => { assert.strictEqual(isValid, true); const isValidLast = await blockchain.validate({ last: 2 }); + // An unknown starting hash is an invalid chain, not an exception. + assert.strictEqual(await blockchain.validate({ from: 'deadbeef' }), false); assert.strictEqual(isValidLast, true); } finally { await cleanupTemp(tempDir); diff --git a/tests/crdt.test.js b/tests/crdt.test.js index 2439541..8d64676 100644 --- a/tests/crdt.test.js +++ b/tests/crdt.test.js @@ -112,7 +112,20 @@ test('CRDT module', async (t) => { const history = [{ type: 'write', data: { c: 3, d: 4 } }]; const result = syncom.apply(source, history); - assert.deepStrictEqual(result, { a: 1, b: 2 }); + // A write is a full record write: the state IS the written data. + assert.deepStrictEqual(result, { c: 3, d: 4 }); + assert.deepStrictEqual(source, { a: 1, b: 2 }); + + // A write with no data (a bare marker) copies the state forward. + assert.deepStrictEqual(syncom.apply(source, [{ type: 'write' }]), { a: 1, b: 2 }); + }); + + await t.test('apply function - delete record vs delete key', () => { + const source = { a: 1, b: 2 }; + // The deletion marker merge() understands: the key is present, set to undefined. + assert.deepStrictEqual(syncom.apply(source, [{ type: 'delete', key: 'a' }]), { a: undefined, b: 2 }); + // Storage's own shape: no key means the record itself was deleted. + assert.deepStrictEqual(syncom.apply(source, [{ type: 'delete', id: 'r1' }]), {}); }); await t.test('apply function - delta record', () => { @@ -162,15 +175,16 @@ test('CRDT module', async (t) => { assert.deepStrictEqual(result, { a: 6, b: 22, c: 3 }); }); - await t.test('apply function - write preserves state', () => { + await t.test('apply function - write replaces the accumulated state', () => { const source = { a: 1, b: 2, c: 3 }; const history = [ { type: 'delta', data: { d: 4 } }, { type: 'write', data: { x: 10 } }, + { type: 'delta', data: { x: 1 } }, ]; const result = syncom.apply(source, history); - assert.deepStrictEqual(result, { a: 1, b: 2, c: 3, d: 4 }); + assert.deepStrictEqual(result, { x: 11 }); }); await t.test('apply function - invalid history', () => { diff --git a/tests/crypto.browser.test.js b/tests/crypto.browser.test.js index ee80a35..20f81bf 100644 --- a/tests/crypto.browser.test.js +++ b/tests/crypto.browser.test.js @@ -30,8 +30,10 @@ test('crypto.browser module', async (t) => { } const cryptoBrowser = require('../src/crypto.browser.js'); + const fsi = require('../src/fsi.browser.js'); await runCryptoKeysTests(t, assert, cryptoBrowser, { + fsi, getKeysBase: async () => `keys-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`, }); }); diff --git a/tests/crypto.cross.test.js b/tests/crypto.cross.test.js new file mode 100644 index 0000000..19e98ee --- /dev/null +++ b/tests/crypto.cross.test.js @@ -0,0 +1,37 @@ +'use strict'; + +// Ciphertext and keys are portable between the two halves: a record sealed +// by the browser half (Web Crypto) opens with node:crypto and vice versa, +// with either half's PEM key pair. + +const test = require('node:test'); +const assert = require('node:assert'); +const { installBrowserGlobals } = require('./utils/browser-env.js'); +const node = require('../src/crypto.js'); + +test('crypto halves interoperate', async (t) => { + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = require('../src/crypto.browser.js'); + const payload = { secret: 'both ways', n: 42, nested: { list: [1, 2, 3] }, long: 'y'.repeat(5000) }; + + for (const [label, keys] of [ + ['node keys', await node.generateKeys()], + ['browser keys', await browser.generateKeys()], + ]) { + await t.test(`${label}: browser encrypts, node decrypts, and back`, async () => { + const fromBrowser = await browser.encrypt(payload, keys.publicKey); + assert.deepStrictEqual(node.decrypt(fromBrowser, keys.privateKey), payload); + const fromNode = node.encrypt(payload, keys.publicKey); + assert.deepStrictEqual(await browser.decrypt(fromNode, keys.privateKey), payload); + assert.strictEqual(await browser.calculateHash(payload), node.calculateHash(payload)); + }); + } + + await t.test('url-safe PEMs from the earlier browser half still import', async () => { + const keys = await browser.generateKeys(); + const urlSafe = (pem) => pem.replaceAll('+', '-').replaceAll('/', '_').replaceAll('=', ''); + const sealed = await browser.encrypt(payload, urlSafe(keys.publicKey)); + assert.deepStrictEqual(await browser.decrypt(sealed, urlSafe(keys.privateKey)), payload); + }); +}); diff --git a/tests/index.browser.test.js b/tests/index.browser.test.js index 8a78c21..0eed2b6 100644 --- a/tests/index.browser.test.js +++ b/tests/index.browser.test.js @@ -55,10 +55,8 @@ test('browser barrel', async (t) => { assert.strictEqual(browser.SmartContract, node.SmartContract); }); - await t.test('SyncManager is the in-memory browser one', async () => { - const sync = await new browser.SyncManager(); - assert.notStrictEqual(browser.SyncManager, node.SyncManager); - await sync.addNode({ url: 'ws://peer.example.com:9000', type: 'uplink' }); - assert.strictEqual(sync.listNodes().length, 1); + await t.test('SyncManager is shared: one registry over fsi in both halves', () => { + assert.strictEqual(browser.SyncManager, node.SyncManager); + assert.strictEqual(browser.OpLog, node.OpLog); }); }); diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 3e01373..10333fd 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -6,6 +6,7 @@ import type { CrdtDelta, KeyPair, Node, + Reference, Schema, Storage, StorageOptions, @@ -27,6 +28,7 @@ expectType>(syncom.read('dir', 'file.json')); expectType>(syncom.write('dir', 'file.json', '{}')); expectType>(syncom.exists('dir', 'file.json')); expectType>(syncom.remove('dir', 'file.json')); +expectType>(syncom.list('dir')); // crdt expectType(syncom.diff({ a: 1 }, { a: 2 })); @@ -52,6 +54,13 @@ expectType>(syncom.SmartContract.load('hash', { st // storage expectType>(syncom.open()); expectType>(syncom.open({ path: 'data' })); +// A schema turns entity names into typed collections on the storage object. +syncom.open({ path: 'data', schema: { entities: { Post: {}, Author: {} } } }).then((typed) => { + expectType(typed.Post); + expectType(typed.Author); + expectError(typed.Nope); + expectType>(typed.insert({ a: 1 })); +}); expectAssignable({ path: 'data', schema: {} as Schema }); expectAssignable({ getId: () => 'id' }); expectAssignable({ getId: async () => 'id' }); @@ -59,6 +68,9 @@ expectType(syncom.Storage.generateId()); expectError(syncom.open({ path: 1 })); expectError(syncom.open({ getId: 'nope' })); expectType>(storage.insert({ a: 1 })); +expectType>(storage.insert({ a: 1 }, { encrypted: true })); +expectType>(storage.set('id', { a: 1 }, { encrypted: false })); +expectAssignable({ encrypted: true }); expectType>(storage.get('id')); expectType>(storage.has('id')); expectType>(storage.swap('id', { a: 2 }, { a: 1 })); @@ -75,10 +87,48 @@ record.on('update', (data, delta) => { expectType(data); expectType(delta); }); +record.off('update'); +expectType(record.title); +declare const reference: Reference; +expectType>(reference.record()); +expectType(syncom.apply({}, [{ type: 'delete', id: 'r1' }])); +expectType(syncom.apply({}, [{ type: 'delete', key: 'a' }])); // sync expectType(storage.sync); declare const sync: SyncManager; expectType(sync.listNodes()); -expectType>(sync.addNode({ url: 'ws://a', type: 'uplink' })); -expectError(sync.addNode({ url: 'ws://a', type: 'sideways' })); +expectType>(sync.addNode({ url: 'ws://a', token: 't', scopes: ['*'] })); +expectType>(sync.addNode({ url: 'ws://a' })); +expectError(sync.addNode({ url: 'ws://a', type: 'uplink' })); +expectError(sync.addNode({ token: 't' })); +expectType>(sync.start()); +expectType>(sync.stop()); +expectType>(new syncom.SyncManager(storage)); + +// replication model +expectType(storage.nodeId); +expectType(storage.vector); +expectType>(storage.inc('id', 'hits')); +expectType>(storage.inc('id', 'hits', 2)); +expectType>(storage.list()); +expectType>(storage.close()); +expectType>(storage.applyRemote([], { via: 'link' })); +expectType>(storage.snapshot({ cursor: null, limit: 10 })); +storage.on('change', (op, via) => { + expectType(op); + expectType(via); +}); +declare const entry: syncom.Entry; +const applied = syncom.applyOp(null, { origin: 'a', seq: 1, ts: 1, type: 'set', id: 'r', data: { x: 1 } }); +expectType(applied.entry); +expectType(applied.changed); +expectType(syncom.mergeEntry(entry, null)); +expectType(syncom.isAlive(entry)); +expectError(syncom.applyOp(null, { origin: 'a', seq: 1, ts: 1, type: 'nope', id: 'r' })); +expectType>(new syncom.OpLog('base', { rollAfter: 8, retention: { ops: 100 } })); +declare const log: syncom.OpLog; +expectType(log.since(undefined, { scopes: ['*'], limit: 10 })); +expectType>(log.append([])); +expectAssignable({ logger: false, sync: { scope: () => 'docs', retention: { ops: 10 } } }); +expectError(syncom.open({ sync: { scope: 'docs' } })); diff --git a/tests/keys.test.js b/tests/keys.test.js index 71e1cf8..e234a10 100644 --- a/tests/keys.test.js +++ b/tests/keys.test.js @@ -3,11 +3,13 @@ const test = require('node:test'); const assert = require('node:assert'); const cryptoMod = require('../src/crypto.js'); +const fsi = require('../src/fsi.js'); const { createTemp, cleanupTemp } = require('./utils/temp.js'); const { runCryptoKeysTests } = require('./suites/crypto-keys.js'); test('Keys module', async (t) => { await runCryptoKeysTests(t, assert, cryptoMod, { + fsi, getKeysBase: async (t2) => { const tempDir = await createTemp(); t2.after(() => cleanupTemp(tempDir)); diff --git a/tests/log.test.js b/tests/log.test.js new file mode 100644 index 0000000..bf29ae3 --- /dev/null +++ b/tests/log.test.js @@ -0,0 +1,198 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const path = require('node:path'); +const { OpLog } = require('../src/log.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +const op = (origin, seq, extra = {}) => ({ + type: 'set', + origin, + seq, + ts: seq, + id: `r${seq}`, + data: { v: seq }, + ...extra, +}); + +test('OpLog', async (t) => { + await t.test('append assigns seqs, tracks the vector and answers since()', async () => { + const dir = await createTemp(); + try { + const log = await new OpLog(dir); + assert.strictEqual(log.floor, 1); + assert.strictEqual(log.next, 1); + assert.deepStrictEqual(log.since(undefined), []); + assert.deepStrictEqual(await log.append([]), []); + + const first = await log.append([op('A', 1), op('A', 2)]); + assert.deepStrictEqual( + first.map((entry) => entry.seq), + [1, 2], + ); + await log.append([op('B', 1, { scope: 's' })]); + // Seqs run per origin AND scope: a scoped op has its own vector key. + assert.deepStrictEqual(log.vector, { A: 2, 'B@s': 1 }); + assert.strictEqual(log.lastEventId, `${log.epoch}.3`); + + assert.strictEqual(log.since(undefined).length, 3); + assert.strictEqual(log.since(log.formatId(0)).length, 3); + assert.deepStrictEqual( + log.since(log.formatId(2)).map((entry) => entry.op.origin), + ['B'], + ); + assert.deepStrictEqual(log.since(log.formatId(3)), []); + assert.strictEqual(log.since(log.formatId(1), { limit: 1 }).length, 1); + // Unscoped ops are global: every subscriber gets them; scoped ones + // reach only the subscribers that asked for the scope. + assert.strictEqual(log.since(undefined, { scopes: ['s'] }).length, 3); + assert.strictEqual(log.since(undefined, { scopes: ['*'] }).length, 3); + assert.strictEqual(log.since(undefined, { scopes: ['other'] }).length, 2); + assert.strictEqual(log.since(undefined, { scopes: ['other'] })[0].op.scope, undefined); + + assert.strictEqual(log.since('other-epoch.1'), null); + assert.strictEqual(log.since('garbage'), null); + await log.close(); + } finally { + await cleanupTemp(dir); + } + }); + + await t.test('reopen restores epoch, next, vector and entries', async () => { + const dir = await createTemp(); + try { + const log = await new OpLog(dir); + await log.append([op('A', 1), op('A', 2)]); + const { epoch } = log; + await log.close(); + + const reopened = await new OpLog(dir); + assert.strictEqual(reopened.epoch, epoch); + assert.strictEqual(reopened.next, 3); + assert.deepStrictEqual(reopened.vector, { A: 2 }); + assert.strictEqual(reopened.since(reopened.formatId(0)).length, 2); + // Unsealed entries are the tail a reopen replays. + assert.strictEqual(reopened.tail().length, 2); + const more = await reopened.append([op('A', 3)]); + assert.strictEqual(more[0].seq, 3); + } finally { + await cleanupTemp(dir); + } + }); + + await t.test('rolls open files into a segment and compacts past retention', async () => { + const dir = await createTemp(); + try { + const log = await new OpLog(dir, { rollAfter: 2, retention: { ops: 4 } }); + for (let seq = 1; seq <= 8; seq++) await log.append([op('A', seq)]); + const names = (await fs.readdir(path.join(dir, 'log'))).sort(); + assert.ok(names.some((name) => name.startsWith('seg-'))); + // Retained ops are bounded; the floor moved up; the vector did not forget. + assert.ok(log.length <= 6, `retained ${log.length}`); + assert.ok(log.floor > 1); + assert.deepStrictEqual(log.vector, { A: 8 }); + assert.strictEqual(log.since(undefined), null, 'a fresh subscriber must bootstrap once history was compacted'); + assert.strictEqual(log.since(log.formatId(log.floor - 2)), null); + assert.ok(Array.isArray(log.since(log.formatId(log.floor - 1)))); + assert.deepStrictEqual( + log.since(log.formatId(7)).map((entry) => entry.seq), + [8], + ); + assert.strictEqual(log.tail().length, 0); + + const reopened = await new OpLog(dir, { rollAfter: 2, retention: { ops: 4 } }); + assert.strictEqual(reopened.floor, log.floor); + assert.strictEqual(reopened.length, log.length); + assert.deepStrictEqual(reopened.vector, { A: 8 }); + } finally { + await cleanupTemp(dir); + } + }); + + await t.test('age-based retention drops old segments', async () => { + const dir = await createTemp(); + try { + const log = await new OpLog(dir, { rollAfter: 2, retention: { ops: 0, days: 1 } }); + const old = Date.now() - 3 * 86_400_000; + await log.append([op('A', 1, { ts: old })]); + await log.append([op('A', 2, { ts: old })]); + await log.append([op('A', 3)]); + await log.append([op('A', 4)]); + assert.deepStrictEqual( + log.since(log.formatId(log.floor - 1)).map((entry) => entry.seq), + [3, 4], + ); + } finally { + await cleanupTemp(dir); + } + }); + + await t.test('survives a torn open file, a lost index and a duplicated roll', async () => { + const dir = await createTemp(); + try { + const warnings = []; + const logger = { warn: (entry) => warnings.push(entry) }; + const log = await new OpLog(dir, { rollAfter: 3, logger }); + for (let seq = 1; seq <= 4; seq++) await log.append([op('A', seq)]); + const { epoch } = log; + await log.close(); + + const logDir = path.join(dir, 'log'); + // Crash mid-roll: the sealed batch is still in open/ too. + const segment = (await fs.readdir(logDir)).find((name) => name.startsWith('seg-')); + const sealed = JSON.parse(await fs.readFile(path.join(logDir, segment), 'utf8')); + await fs.writeFile(path.join(logDir, 'open', '0000000001.json'), JSON.stringify(sealed)); + // Crash mid-append: a torn file for a batch nobody was told about. + await fs.writeFile(path.join(logDir, 'open', '0000000009.json'), '{"entries":[{"seq":9,'); + + const reopened = await new OpLog(dir, { rollAfter: 3, logger }); + assert.strictEqual(reopened.epoch, epoch); + assert.deepStrictEqual( + reopened.since(reopened.formatId(0)).map((entry) => entry.seq), + [1, 2, 3, 4], + ); + assert.strictEqual(reopened.next, 5, 'the torn batch must not advance the seq'); + assert.ok(warnings.some((entry) => entry.event === 'log.open.unreadable')); + await reopened.close(); + + // A lost index is a new epoch: old ids cannot resume, data is kept. + await fs.rm(path.join(logDir, 'index.json')); + const fresh = await new OpLog(dir, { rollAfter: 3, logger }); + assert.notStrictEqual(fresh.epoch, epoch); + assert.strictEqual(fresh.since(`${epoch}.2`), null); + assert.strictEqual(fresh.since(fresh.formatId(0)).length, 4); + assert.deepStrictEqual(fresh.vector, { A: 4 }); + assert.strictEqual(fresh.next, 5); + + // An unreadable segment is reported, not fatal: what it held is gone, + // what the open files hold survives. + await fs.rm(path.join(logDir, 'open', '0000000001.json')); + await fs.writeFile(path.join(logDir, segment), 'not json'); + const damaged = await new OpLog(dir, { rollAfter: 3, logger }); + assert.ok(warnings.some((entry) => entry.event === 'log.segment.unreadable')); + assert.deepStrictEqual( + damaged.since(damaged.formatId(0)).map((entry) => entry.seq), + [4], + ); + } finally { + await cleanupTemp(dir); + } + }); + + await t.test('flush seals the open files on demand', async () => { + const dir = await createTemp(); + try { + const log = await new OpLog(dir); + await log.flush(); + await log.append([op('A', 1)]); + await log.flush(); + assert.strictEqual(log.tail().length, 0); + assert.strictEqual((await fs.readdir(path.join(dir, 'log', 'open'))).length, 0); + assert.strictEqual(log.since(log.formatId(0)).length, 1); + } finally { + await cleanupTemp(dir); + } + }); +}); diff --git a/tests/ops.test.js b/tests/ops.test.js new file mode 100644 index 0000000..e284844 --- /dev/null +++ b/tests/ops.test.js @@ -0,0 +1,187 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { applyOp, mergeEntry, isAlive, createEntry, isCounter } = require('../src/ops.js'); + +const op = (type, ts, origin, extra = {}) => ({ type, ts, origin, seq: extra.seq ?? ts, id: 'r', ...extra }); + +// Every permutation of a list โ€” the commutativity check runs them all. +const permutations = (items) => { + if (items.length <= 1) return [items]; + const out = []; + items.forEach((item, index) => { + const rest = [...items.slice(0, index), ...items.slice(index + 1)]; + for (const tail of permutations(rest)) out.push([item, ...tail]); + }); + return out; +}; + +const replay = (ops) => { + let entry = null; + for (const item of ops) entry = applyOp(entry, item).entry; + return entry; +}; + +// Storage applies one origin's ops in seq order, so only interleavings that +// keep that order are reachable โ€” those are the ones that must agree. +const orderedPerOrigin = (sequence) => { + const last = {}; + for (const item of sequence) { + if ((last[item.origin] ?? 0) > item.seq) return false; + last[item.origin] = item.seq; + } + return true; +}; + +const assertCommutes = (ops, expectedData) => { + const results = permutations(ops).filter(orderedPerOrigin).map(replay); + for (const result of results) { + assert.deepStrictEqual(result.data, expectedData); + assert.deepStrictEqual(result.meta, results[0].meta); + } +}; + +test('ops module', async (t) => { + await t.test('set is last-writer-wins per field', () => { + const a = applyOp(null, op('set', 1, 'A', { data: { x: 1, y: 1 } })); + assert.strictEqual(a.changed, true); + assert.deepStrictEqual(a.delta, { x: 1, y: 1 }); + const b = applyOp(a.entry, op('set', 2, 'B', { data: { x: 2 } })); + assert.deepStrictEqual(b.entry.data, { x: 2, y: 1 }); + // An older concurrent write to x loses; y is untouched. + const c = applyOp(b.entry, op('set', 1, 'C', { data: { x: 9 } })); + assert.strictEqual(c.changed, false); + assert.deepStrictEqual(c.entry.data, { x: 2, y: 1 }); + // Equal ts: the origin breaks the tie deterministically. + assertCommutes([op('set', 5, 'A', { data: { x: 'a' } }), op('set', 5, 'B', { data: { x: 'b' } })], { x: 'b' }); + }); + + await t.test('applying the same op twice is a no-op', () => { + const first = applyOp(null, op('set', 1, 'A', { data: { x: 1 } })); + const again = applyOp(first.entry, op('set', 1, 'A', { data: { x: 1 } })); + assert.strictEqual(again.changed, false); + assert.deepStrictEqual(again.entry, first.entry); + const inc = applyOp(first.entry, op('inc', 2, 'A', { seq: 2, key: 'n', value: 3 })); + const incAgain = applyOp(inc.entry, op('inc', 2, 'A', { seq: 2, key: 'n', value: 3 })); + assert.strictEqual(incAgain.changed, false); + assert.strictEqual(incAgain.entry.data.n, 3); + }); + + await t.test('write resets the record and commutes with a concurrent set', () => { + // set@7 to a field the write@6 does not carry survives; set@5 does not. + assertCommutes( + [ + op('set', 5, 'S', { data: { old: 1 } }), + op('write', 6, 'W', { data: { fresh: true } }), + op('set', 7, 'T', { data: { later: 2 } }), + ], + { fresh: true, later: 2 }, + ); + // A write and a set on the SAME field: the newer stamp wins either way. + assertCommutes([op('write', 6, 'W', { data: { x: 'w' } }), op('set', 7, 'T', { data: { x: 't' } })], { x: 't' }); + assertCommutes([op('write', 8, 'W', { data: { x: 'w' } }), op('set', 7, 'T', { data: { x: 't' } })], { x: 'w' }); + }); + + await t.test('delete tombstones; a newer write revives; an older set does not', () => { + const live = replay([op('set', 1, 'A', { data: { x: 1 } })]); + const dead = applyOp(live, op('delete', 2, 'B')).entry; + assert.strictEqual(isAlive(dead), false); + assert.deepStrictEqual(dead.data, {}); + const stale = applyOp(dead, op('set', 1, 'C', { data: { x: 9 } })); + assert.strictEqual(isAlive(stale.entry), false); + assert.strictEqual(stale.changed, false); + const revived = applyOp(dead, op('set', 3, 'C', { data: { y: 2 } })); + assert.strictEqual(isAlive(revived.entry), true); + assert.deepStrictEqual(revived.entry.data, { y: 2 }); + assertCommutes( + [op('set', 1, 'A', { data: { x: 1 } }), op('delete', 2, 'B'), op('write', 3, 'C', { data: { z: 1 } })], + { + z: 1, + }, + ); + assertCommutes([op('delete', 2, 'B'), op('set', 1, 'A', { data: { x: 1 } })], {}); + assert.strictEqual(isAlive(null), false); + assert.strictEqual(isAlive(createEntry()), true); + }); + + await t.test('inc is a per-origin counter: commutative, idempotent, summed', () => { + const ops = [ + op('inc', 1, 'A', { seq: 1, key: 'n', value: 2 }), + op('inc', 2, 'B', { seq: 1, key: 'n', value: 3 }), + op('inc', 3, 'A', { seq: 2, key: 'n', value: -1 }), + ]; + assertCommutes(ops, { n: 4 }); + const entry = replay(ops); + assert.strictEqual(isCounter(entry.meta, 'n'), true); + assert.deepStrictEqual(entry.meta.counters.n, { A: [2, 1, 3], B: [1, 3, 2] }); + // A counter revives a tombstone like any newer write. + const dead = applyOp(entry, op('delete', 5, 'D')).entry; + assert.strictEqual(isAlive(dead), false); + const back = applyOp(dead, op('inc', 6, 'A', { seq: 3, key: 'n', value: 1 })).entry; + assert.strictEqual(isAlive(back), true); + assert.deepStrictEqual(back.data, { n: 1 }); + }); + + await t.test('a field is a register or a counter, never both', () => { + const register = replay([op('set', 1, 'A', { data: { n: 1 } })]); + const incOnRegister = applyOp(register, op('inc', 2, 'B', { seq: 1, key: 'n', value: 1 })); + assert.deepStrictEqual(incOnRegister.rejected, [{ field: 'n', reason: 'register' }]); + assert.strictEqual(incOnRegister.changed, false); + const counter = replay([op('inc', 1, 'A', { seq: 1, key: 'n', value: 1 })]); + const setOnCounter = applyOp(counter, op('set', 2, 'B', { data: { n: 5, other: 1 } })); + assert.deepStrictEqual(setOnCounter.rejected, [{ field: 'n', reason: 'counter' }]); + assert.deepStrictEqual(setOnCounter.entry.data, { n: 1, other: 1 }); + // Malformed ops are ignored rather than applied half-way. + assert.strictEqual(applyOp(counter, op('set', 3, 'B', { data: 'nope' })).changed, false); + assert.strictEqual(applyOp(counter, op('inc', 3, 'B', { key: 'n', value: 'nope' })).changed, false); + assert.strictEqual(applyOp(counter, op('bogus', 3, 'B')).changed, false); + // Inherited keys are not fields; a JSON "__proto__" key is skipped. + assert.strictEqual(applyOp(counter, op('set', 3, 'B', { data: { __proto__: { evil: 1 } } })).changed, false); + const poisoned = applyOp(counter, op('set', 3, 'B', { data: JSON.parse('{"__proto__":{"evil":1},"ok":1}') })); + assert.deepStrictEqual(poisoned.entry.data, { n: 1, ok: 1 }); + assert.strictEqual(Object.getPrototypeOf(poisoned.entry.data), Object.prototype); + }); + + await t.test('a reset drops counter entries older than it', () => { + assertCommutes([op('inc', 1, 'A', { seq: 1, key: 'n', value: 5 }), op('write', 2, 'W', { data: { x: 1 } })], { + x: 1, + }); + assertCommutes([op('write', 2, 'W', { data: { x: 1 } }), op('inc', 3, 'A', { seq: 2, key: 'n', value: 5 })], { + x: 1, + n: 5, + }); + }); + + await t.test('mergeEntry joins two histories like replaying both', () => { + const left = [op('set', 1, 'A', { data: { x: 1, y: 1 } }), op('inc', 2, 'A', { seq: 2, key: 'n', value: 2 })]; + const right = [ + op('set', 3, 'B', { data: { x: 2 } }), + op('inc', 4, 'B', { seq: 1, key: 'n', value: 3 }), + op('delete', 5, 'B'), + op('set', 6, 'B', { data: { z: 1 } }), + ]; + const merged = mergeEntry(replay(left), replay(right)); + const replayed = replay([...left, ...right]); + assert.deepStrictEqual(merged.data, replayed.data); + assert.deepStrictEqual(merged.meta, replayed.meta); + // Idempotent and commutative. + assert.deepStrictEqual(mergeEntry(merged, merged), merged); + assert.deepStrictEqual(mergeEntry(replay(right), replay(left)), merged); + assert.deepStrictEqual(mergeEntry(null, merged), merged); + assert.deepStrictEqual(mergeEntry(merged, null), merged); + assert.strictEqual(mergeEntry(null, null), null); + // Overlapping counter histories join by the higher seq, not by adding. + const a = replay([ + op('inc', 1, 'A', { seq: 1, key: 'n', value: 1 }), + op('inc', 2, 'A', { seq: 2, key: 'n', value: 1 }), + ]); + const b = replay([op('inc', 1, 'A', { seq: 1, key: 'n', value: 1 })]); + assert.deepStrictEqual(mergeEntry(a, b).data, { n: 2 }); + // A register on one side and a counter on the other: the register wins. + const reg = replay([op('set', 9, 'R', { data: { n: 'text' } })]); + const cnt = replay([op('inc', 1, 'C', { seq: 1, key: 'n', value: 4 })]); + assert.deepStrictEqual(mergeEntry(reg, cnt).data, { n: 'text' }); + assert.deepStrictEqual(mergeEntry(cnt, reg).data, { n: 'text' }); + }); +}); diff --git a/tests/storage.browser.test.js b/tests/storage.browser.test.js new file mode 100644 index 0000000..d4cd66d --- /dev/null +++ b/tests/storage.browser.test.js @@ -0,0 +1,41 @@ +'use strict'; + +// Storage over the BROWSER halves: OPFS for files, Web Crypto for hashing and +// encryption. The shared storage.js is loaded through tests/utils/browser-env.js +// so its nested requires resolve the way package.json#browser says they would. + +const test = require('node:test'); +const assert = require('node:assert'); +const { installBrowserGlobals, requireBrowserBarrel } = require('./utils/browser-env.js'); + +test('Storage on the browser halves', async (t) => { + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = requireBrowserBarrel(); + + await t.test('the shared modules see the browser crypto and fsi', async () => { + assert.ok(browser.calculateHash({ a: 1 }) instanceof Promise); + const storage = await browser.open({ path: 'store-basic' }); + const id = await storage.insert({ title: 'Draft', hits: 0 }); + assert.deepStrictEqual(await storage.get(id), { title: 'Draft', hits: 0 }); + assert.strictEqual(await storage.validate(), true); + // Written to the mock OPFS, not to disk. + assert.strictEqual(await browser.exists('store-basic/data', `${id}.json`), true); + }); + + await t.test('encryption at rest awaits the async Web Crypto', async () => { + const storage = await browser.open({ path: 'store-encrypted' }); + const secret = { ssn: '123-45-6789' }; + await storage.saveData('s1', secret, { encrypted: true }); + + // The on-disk entry holds ciphertext, not a serialized pending promise. + const raw = JSON.parse(await browser.read('store-encrypted/data', 's1.json')); + assert.strictEqual(raw.encrypted, true); + assert.strictEqual(typeof raw.data, 'string'); + assert.notDeepStrictEqual(raw.data, {}); + + // A cold read (fresh Storage, empty cache) decrypts it back. + const reopened = await browser.open({ path: 'store-encrypted' }); + assert.deepStrictEqual(await reopened.get('s1'), secret); + }); +}); diff --git a/tests/storage.encrypted.test.js b/tests/storage.encrypted.test.js new file mode 100644 index 0000000..3bf40a8 --- /dev/null +++ b/tests/storage.encrypted.test.js @@ -0,0 +1,87 @@ +'use strict'; + +// Encryption at rest through the whole API: a storage opened with +// `encrypted: true`, per-call overrides, and what stays local. + +const test = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const path = require('node:path'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +test('encrypted storage', async (t) => { + await t.test('every write path encrypts, reads decrypt, nothing becomes an op', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ + path: dir, + logger: false, + encrypted: true, + schema: { entities: { Note: {} } }, + }); + const ops = []; + storage.on('change', (op) => ops.push(op)); + + const id = await storage.insert({ title: 'Secret', body: 'x'.repeat(10_000) }); + const raw = JSON.parse(await fs.readFile(path.join(dir, 'data', `${id}.json`), 'utf8')); + assert.strictEqual(raw.encrypted, true); + assert.ok(!raw.data.includes('Secret')); + assert.strictEqual((await storage.get(id)).title, 'Secret'); + + await storage.update(id, { title: 'Still secret' }); + await storage.set(id, { title: 'Replaced' }); + const record = await storage.record(id); + const updates = []; + record.on('update', (data, delta) => updates.push(delta)); + record.title = 'Via record'; + await record.save(); + assert.strictEqual((await storage.get(id)).title, 'Via record'); + assert.strictEqual(await storage.swap(id, { title: 'Swapped' }, { title: 'Via record' }), true); + assert.strictEqual(record.title, 'Swapped'); + assert.deepStrictEqual(updates.at(-1), { title: 'Swapped' }); + + const note = await storage.Note.insert({ text: 'n' }); + assert.deepStrictEqual((await storage.list()).sort(), [id, note.id].sort()); + assert.strictEqual(await storage.has(id), true); + await assert.rejects(() => storage.inc(id, 'hits', 1), /encrypted/); + assert.strictEqual(ops.length, 0, 'encrypted records never leave the node'); + assert.deepStrictEqual((await storage.snapshot()).records, []); + assert.strictEqual(await storage.validate(), true); + + await storage.delete(id); + assert.strictEqual(await storage.get(id), null); + assert.strictEqual(await storage.has(id), false); + assert.deepStrictEqual(await storage.list(), [note.id]); + assert.strictEqual(await record.data(), null); + + // A cold reopen decrypts with the stored key pair. + await storage.close(); + const reopened = await syncom.open({ path: dir, logger: false, encrypted: true }); + assert.deepStrictEqual(await reopened.get(note.id), { text: 'n', $type: 'Note' }); + await reopened.close(); + }); + + await t.test('per-call overrides go both ways, and a record never changes kind', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, encrypted: true }); + const ops = []; + storage.on('change', (op) => ops.push(op)); + const shared = await storage.insert({ shared: true }, { encrypted: false }); + assert.strictEqual(ops.length, 1, 'an explicitly replicated record is an op'); + await assert.rejects( + () => storage.set(shared, { x: 1 }, { encrypted: true }), + /replicated and cannot become encrypted/, + ); + + const plain = await syncom.open({ path: await createTemp(), logger: false }); + const local = await plain.insert({ local: true }, { encrypted: true }); + assert.deepStrictEqual(await plain.get(local), { local: true }); + assert.strictEqual(plain.log.length, 0); + await plain.set(local, { local: 'still' }); + assert.strictEqual(plain.log.length, 0, 'set on an encrypted record stays encrypted and local'); + await storage.close(); + await plain.close(); + }); +}); diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js new file mode 100644 index 0000000..3580743 --- /dev/null +++ b/tests/storage.ops.test.js @@ -0,0 +1,256 @@ +'use strict'; + +// The replication half of Storage: ops, counters, tombstones, the change +// event, applying a peer's ops, snapshots, and what survives a reopen. + +const test = require('node:test'); +const assert = require('node:assert'); +const fs = require('node:fs/promises'); +const path = require('node:path'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); + +const openTemp = async (t, options = {}) => { + const dir = await createTemp(); + t.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, ...options }); + return { dir, storage }; +}; + +const remoteOp = (origin, seq, type, id, extra = {}) => ({ origin, seq, ts: Date.now() + seq, type, id, ...extra }); + +test('Storage replication model', async (t) => { + await t.test('every local write is an op with this node as origin', async (t2) => { + const { storage } = await openTemp(t2); + const ops = []; + storage.on('change', (op, via) => ops.push({ op, via })); + + const id = await storage.insert({ title: 'Draft', hits: 0 }); + await storage.update(id, { title: 'Final' }); + await storage.swap(id, { hits: 1 }, { hits: 0 }); + await storage.delete(id); + + assert.deepStrictEqual( + ops.map(({ op }) => [op.type, op.seq]), + [ + ['write', 1], + ['set', 2], + ['set', 3], + ['delete', 4], + ], + ); + assert.ok(ops.every(({ op, via }) => op.origin === storage.nodeId && op.id === id && via === null)); + assert.deepStrictEqual(ops[1].op.data, { title: 'Final' }); + assert.deepStrictEqual(storage.vector, { [storage.nodeId]: 4 }); + assert.strictEqual(storage.log.length, 4); + }); + + await t.test('inc is a counter; a field is a register or a counter, never both', async (t2) => { + const { storage } = await openTemp(t2); + const id = await storage.insert({ name: 'Ada' }); + await storage.inc(id, 'hits', 2); + await storage.inc(id, 'hits', 3); + await storage.inc(id, 'hits'); + assert.deepStrictEqual(await storage.get(id), { name: 'Ada', hits: 6 }); + + await assert.rejects(() => storage.update(id, { hits: 1 }), /counter; use inc\(\)/); + await assert.rejects(() => storage.inc(id, 'name', 1), /register; use update\(\)/); + await assert.rejects(() => storage.inc(id, 'hits', 'x'), TypeError); + await assert.rejects(() => storage.inc(id, 7), TypeError); + + const record = await storage.record(id); + await record.inc('hits', -1); + assert.strictEqual(record.hits, 5); + // A whole-record write resets the counter along with everything else. + await storage.set(id, { name: 'Ada' }); + assert.deepStrictEqual(await storage.get(id), { name: 'Ada' }); + }); + + await t.test('delete leaves a tombstone: hidden from get/has/list, kept on disk', async (t2) => { + const { dir, storage } = await openTemp(t2); + const keep = await storage.insert({ n: 1 }); + const gone = await storage.insert({ n: 2 }); + await storage.delete(gone); + + assert.strictEqual(await storage.get(gone), null); + assert.strictEqual(await storage.has(gone), false); + assert.strictEqual(storage.getCachedData(gone), null); + assert.deepStrictEqual(await storage.list(), [keep].sort()); + const entry = JSON.parse(await fs.readFile(path.join(dir, 'data', `${gone}.json`), 'utf8')); + assert.ok(entry.meta.deleted, 'the tombstone stamp is on disk'); + // A later write to the same id revives it. + await storage.set(gone, { n: 3 }); + assert.deepStrictEqual(await storage.get(gone), { n: 3 }); + assert.strictEqual((await storage.list()).length, 2); + // update() on a dead record is "not found". + await storage.delete(gone); + await assert.rejects(() => storage.update(gone, { n: 4 }), /not found/); + }); + + await t.test('applyRemote: in order per origin, duplicates and gaps skipped', async (t2) => { + const { storage } = await openTemp(t2); + const changes = []; + storage.on('change', (op, via) => changes.push({ op, via })); + + const first = await storage.applyRemote([remoteOp('peer', 1, 'write', 'r1', { data: { a: 1 } })], { via: 'L' }); + assert.deepStrictEqual(first, { applied: 1, duplicates: 0, gaps: 0 }); + assert.deepStrictEqual(await storage.get('r1'), { a: 1 }); + assert.deepStrictEqual(changes[0].via, 'L'); + assert.strictEqual(changes[0].op.origin, 'peer'); + + const again = await storage.applyRemote([ + remoteOp('peer', 1, 'write', 'r1', { data: { a: 1 } }), + remoteOp('peer', 3, 'set', 'r1', { data: { b: 1 } }), + { origin: storage.nodeId, seq: 1, ts: 1, type: 'write', id: 'x', data: {} }, + 'garbage', + ]); + assert.deepStrictEqual(again, { applied: 0, duplicates: 2, gaps: 1 }); + assert.deepStrictEqual(await storage.get('r1'), { a: 1 }); + + const next = await storage.applyRemote([ + remoteOp('peer', 2, 'set', 'r1', { data: { b: 2 } }), + remoteOp('peer', 3, 'inc', 'r1', { key: 'n', value: 4 }), + remoteOp('peer', 4, 'delete', 'r1'), + ]); + assert.deepStrictEqual(next, { applied: 3, duplicates: 0, gaps: 0 }); + assert.strictEqual(await storage.get('r1'), null); + assert.deepStrictEqual(storage.vector, { peer: 4 }); + // Accepted-but-unchanged ops still advance the vector and are logged. + const noop = await storage.applyRemote([remoteOp('peer', 5, 'set', 'r1', { data: { b: 0 }, ts: 1 })]); + assert.strictEqual(noop.applied, 1); + assert.strictEqual(storage.vector.peer, 5); + assert.strictEqual(storage.log.length, 5); + assert.strictEqual(await storage.validate(), true); + }); + + await t.test('a refused remote op surfaces as an error event, not a throw', async (t2) => { + const { storage } = await openTemp(t2); + const errors = []; + storage.on('error', (error) => errors.push(error)); + const id = await storage.insert({ hits: 'text' }); + const result = await storage.applyRemote([remoteOp('peer', 1, 'inc', id, { key: 'hits', value: 1 })]); + assert.strictEqual(result.applied, 1); + assert.strictEqual(errors.length, 1); + assert.deepStrictEqual(errors[0].rejected, [{ field: 'hits', reason: 'register' }]); + assert.deepStrictEqual(await storage.get(id), { hits: 'text' }); + }); + + await t.test('a live Record follows remote changes and keeps unsaved edits', async (t2) => { + const { storage } = await openTemp(t2); + const id = await storage.insert({ title: 'Draft', views: 0 }); + const record = await storage.record(id); + const updates = []; + record.on('update', (data, delta) => updates.push({ data: data === null ? null : { ...data }, delta })); + + record.title = 'Local edit'; // staged, not saved + await storage.applyRemote([remoteOp('peer', 1, 'set', id, { data: { views: 7 } })]); + assert.strictEqual(record.views, 7); + assert.strictEqual(record.title, 'Local edit', 'the pending edit is laid back on top'); + assert.deepStrictEqual(updates.at(-1).delta, { views: 7 }); + + await record.save(); + assert.deepStrictEqual(await storage.get(id), { title: 'Local edit', views: 7 }); + // The save itself does not echo an update to its own listeners. + assert.strictEqual(updates.length, 2); + + await storage.applyRemote([remoteOp('peer', 2, 'delete', id)]); + assert.strictEqual(updates.at(-1).data, null); + assert.strictEqual(await record.data(), null); + }); + + await t.test('snapshot pages every entry, tombstones included; mergeSnapshot joins them', async (t2) => { + const { storage: a } = await openTemp(t2); + const { storage: b } = await openTemp(t2); + const ids = []; + for (let i = 0; i < 5; i++) ids.push(await a.insert({ i })); + await a.inc(ids[0], 'hits', 3); + await a.delete(ids[4]); + await a.saveData('secret', { s: 1 }, { encrypted: true }); + + const page1 = await a.snapshot({ limit: 3 }); + assert.strictEqual(page1.records.length, 3); + assert.ok(page1.cursor); + assert.strictEqual(page1.eventId, a.log.lastEventId); + const page2 = await a.snapshot({ cursor: page1.cursor, limit: 3 }); + assert.strictEqual(page2.cursor, null); + const records = [...page1.records, ...page2.records]; + assert.strictEqual(records.length, 5, 'the tombstone travels, the encrypted record does not'); + + await b.mergeSnapshot(records); + const replicated = (await a.list()).filter((id) => id !== 'secret').sort(); + assert.deepStrictEqual((await b.list()).sort(), replicated); + assert.deepStrictEqual(await b.get(ids[0]), { i: 0, hits: 3 }); + assert.strictEqual(await b.get(ids[4]), null); + // Joined state is in B's own log as merge ops โ€” a peer of B can get it. + assert.strictEqual(b.log.since(b.log.formatId(0)).filter((e) => e.op.type === 'merge').length, 5); + // Merging the same snapshot again changes nothing. + const before = b.log.length; + await b.mergeSnapshot(records); + assert.strictEqual(b.log.length, before); + // Ops from A that were already folded in are harmless duplicates of state. + const aOps = a.log.since(a.log.formatId(0)).map((e) => e.op); + await b.applyRemote(aOps); + assert.deepStrictEqual(await b.get(ids[0]), { i: 0, hits: 3 }); + }); + + await t.test('scopes ride on ops and filter snapshots', async (t2) => { + const { storage } = await openTemp(t2, { sync: { scope: (id, data) => data.$type ?? '*' } }); + const ops = []; + storage.on('change', (op) => ops.push(op)); + const global = await storage.insert({ n: 1 }); + const doc = await storage.insert({ $type: 'Doc', n: 2 }); + assert.strictEqual(ops[0].scope, undefined); + assert.strictEqual(ops[1].scope, 'Doc'); + const docs = await storage.snapshot({ scopes: ['Doc'] }); + assert.deepStrictEqual(docs.records.map((r) => r.id).sort(), [global, doc].sort()); + const other = await storage.snapshot({ scopes: ['Other'] }); + assert.deepStrictEqual( + other.records.map((r) => r.id), + [global], + ); + }); + + await t.test('identity, seq and vector survive a reopen; the log tail is replayed', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const first = await syncom.open({ path: dir, logger: false }); + const id = await first.insert({ n: 1 }); + await first.applyRemote([remoteOp('peer', 1, 'write', 'r', { data: { p: 1 } })]); + const { nodeId } = first; + await first.close(); + await assert.rejects(() => first.insert({ n: 2 }), /closed/); + + // Crash between "logged" and "written": the record file is gone. + await fs.rm(path.join(dir, 'data', `${id}.json`)); + + const second = await syncom.open({ path: dir, logger: false }); + assert.strictEqual(second.nodeId, nodeId); + assert.deepStrictEqual(second.vector, { [nodeId]: 1, peer: 1 }); + assert.deepStrictEqual(await second.get(id), { n: 1 }, 'replayed from the log'); + const ops = []; + second.on('change', (op) => ops.push(op)); + await second.insert({ n: 3 }); + assert.strictEqual(ops[0].seq, 2, 'the own seq continues, never repeats'); + assert.strictEqual(await second.validate(), true); + }); + + await t.test('encrypted records stay local', async (t2) => { + const { storage } = await openTemp(t2); + const ops = []; + storage.on('change', (op) => ops.push(op)); + await storage.saveData('s', { secret: 1 }, { encrypted: true }); + assert.strictEqual(ops.length, 0); + assert.deepStrictEqual(await storage.get('s'), { secret: 1 }); + assert.strictEqual(await storage.has('s'), true); + assert.deepStrictEqual(await storage.list(), ['s']); + // Local writes work; they simply never become ops. + await storage.update('s', { secret: 2 }); + assert.deepStrictEqual(await storage.get('s'), { secret: 2 }); + await assert.rejects(() => storage.inc('s', 'n', 1), /encrypted/); + assert.strictEqual(ops.length, 0); + const result = await storage.applyRemote([remoteOp('peer', 1, 'set', 's', { data: { secret: 3 } })]); + assert.strictEqual(result.applied, 1); + assert.deepStrictEqual(await storage.get('s'), { secret: 2 }); + assert.deepStrictEqual((await storage.snapshot()).records, []); + }); +}); diff --git a/tests/storage.test.js b/tests/storage.test.js index 746730d..8c7ac47 100644 --- a/tests/storage.test.js +++ b/tests/storage.test.js @@ -109,7 +109,8 @@ test('Storage module', async (t) => { entry.data = { ok: false }; await fs.writeFile(filePath, JSON.stringify(entry), 'utf8'); - const storage2 = await syncom.open({ path: tempDir }); + // Quiet: the reopen's log replay notices the damage and warns. + const storage2 = await syncom.open({ path: tempDir, logger: false }); await assert.rejects(() => storage2.loadData('tampered'), /invalid/); } finally { await cleanupTemp(tempDir); diff --git a/tests/suites/crypto-keys.js b/tests/suites/crypto-keys.js index d543730..99e42f3 100644 --- a/tests/suites/crypto-keys.js +++ b/tests/suites/crypto-keys.js @@ -1,9 +1,10 @@ 'use strict'; -const runCryptoKeysTests = async (t, assert, cryptoMod, { getKeysBase } = {}) => { +const runCryptoKeysTests = async (t, assert, cryptoMod, { getKeysBase, fsi } = {}) => { if (typeof getKeysBase !== 'function') { throw new TypeError('runCryptoKeysTests requires getKeysBase'); } + if (!fsi) throw new TypeError('runCryptoKeysTests requires the fsi half the crypto module writes through'); await t.test('generateKeys function', async () => { const keys = await Promise.resolve(cryptoMod.generateKeys()); @@ -25,6 +26,22 @@ const runCryptoKeysTests = async (t, assert, cryptoMod, { getKeysBase } = {}) => assert.deepStrictEqual(decrypted, testData); }); + await t.test('encrypt handles payloads far beyond the RSA ceiling, with a fresh key each time', async () => { + const keys = await Promise.resolve(cryptoMod.generateKeys()); + const big = { blob: 'x'.repeat(200_000), items: Array.from({ length: 1000 }, (_, i) => ({ i })) }; + const a = await Promise.resolve(cryptoMod.encrypt(big, keys.publicKey)); + const b = await Promise.resolve(cryptoMod.encrypt(big, keys.publicKey)); + assert.notStrictEqual(a, b, 'a random data key and iv per call'); + assert.deepStrictEqual(await Promise.resolve(cryptoMod.decrypt(a, keys.privateKey)), big); + const envelope = JSON.parse(a); + assert.strictEqual(envelope.alg, 'RSA-OAEP-256+A256GCM'); + // Tampering with the body fails the GCM tag rather than yielding garbage. + envelope.data = envelope.data.slice(0, -4) + 'AAAA'; + // `async` so a synchronous throw (the node half) becomes a rejection too. + await assert.rejects(async () => cryptoMod.decrypt(JSON.stringify(envelope), keys.privateKey)); + await assert.rejects(async () => cryptoMod.decrypt(JSON.stringify({ v: 9 }), keys.privateKey), /envelope/); + }); + await t.test('loadKeys function', async (t2) => { const base = await getKeysBase(t2); const keys1 = await Promise.resolve(cryptoMod.loadKeys(base)); @@ -35,6 +52,18 @@ const runCryptoKeysTests = async (t, assert, cryptoMod, { getKeysBase } = {}) => assert.strictEqual(keys2.publicKey, keys1.publicKey); assert.strictEqual(keys2.privateKey, keys1.privateKey); }); + + await t.test('loadKeys regenerates a pair missing its private half', async (t2) => { + const base = await getKeysBase(t2); + const keys1 = await Promise.resolve(cryptoMod.loadKeys(base)); + await fsi.remove(base, 'private.pem'); + + const keys2 = await Promise.resolve(cryptoMod.loadKeys(base)); + assert.strictEqual(typeof keys2.privateKey, 'string'); + // A fresh pair, not the orphaned public key with no private key behind it. + assert.notStrictEqual(keys2.publicKey, keys1.publicKey); + assert.strictEqual(await fsi.exists(base, 'private.pem'), true); + }); }; module.exports = { runCryptoKeysTests }; diff --git a/tests/suites/fsi.js b/tests/suites/fsi.js index f45c4d8..a2c0bac 100644 --- a/tests/suites/fsi.js +++ b/tests/suites/fsi.js @@ -47,6 +47,20 @@ const runFsiTests = async (t, fsi, options = {}) => { } }); + await t.test('list names files only, sorted, and an absent directory is empty', async () => { + const root = await getRoot(); + try { + assert.deepStrictEqual(await fsi.list(joinPath(root, 'never')), []); + await fsi.write(root, 'b.json', '{}'); + await fsi.write(root, 'a.json', '{}'); + await fsi.write(joinPath(root, 'sub'), 'c.json', '{}'); + assert.deepStrictEqual(await fsi.list(root), ['a.json', 'b.json']); + assert.deepStrictEqual(await fsi.list(joinPath(root, 'sub')), ['c.json']); + } finally { + await finish(root); + } + }); + await t.test('read when missing rejects', async () => { const dir = await getRoot(); try { diff --git a/tests/suites/sync-engine.js b/tests/suites/sync-engine.js new file mode 100644 index 0000000..568e61b --- /dev/null +++ b/tests/suites/sync-engine.js @@ -0,0 +1,333 @@ +'use strict'; + +// The convergence properties of the sync engine, over memory links, against +// whichever storage half the caller builds: `createStorage(t, name, options)` +// must return an open Storage and give the SAME directory for the same name +// within one subtest (that is how the reopen case works). + +const { createMesh, settle, waitUntil, prng } = require('../utils/mesh.js'); + +// Two nodes' clocks are independent: a write on B one tick after a write on +// A is only "later" once the millisecond has moved on. +const tick = () => new Promise((resolve) => setTimeout(resolve, 3)); + +const runSyncEngineTests = async (t, assert, createStorage) => { + const dump = async (storage) => { + const out = {}; + for (const id of (await storage.list()).sort()) out[id] = await storage.get(id); + return out; + }; + + const assertConverged = async (storages) => { + const [first, ...rest] = storages; + const expected = await dump(first); + for (const other of rest) assert.deepStrictEqual(await dump(other), expected); + return expected; + }; + + await t.test('two nodes converge in both directions', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + const a = mesh.get('A'); + const b = mesh.get('B'); + const fromA = await a.insert({ who: 'A' }); + const fromB = await b.insert({ who: 'B' }); + await b.update(fromB, { seen: true }); + await settle([a, b]); + const state = await assertConverged([a, b]); + assert.deepStrictEqual(state[fromA], { who: 'A' }); + assert.deepStrictEqual(state[fromB], { who: 'B', seen: true }); + // Nothing was applied twice: every op sits in each log exactly once. + for (const storage of [a, b]) { + const keys = storage.log.since(storage.log.formatId(0)).map(({ op }) => `${op.origin}:${op.seq}`); + assert.strictEqual(new Set(keys).size, keys.length); + } + const [peer] = a.sync.peers(); + assert.strictEqual(peer.connected, true); + assert.strictEqual(peer.subscribed, true); + assert.strictEqual(peer.node, b.nodeId); + assert.ok(peer.cursor, 'the cursor is remembered'); + assert.strictEqual(b.sync.peers().length, 0, 'the host dials nobody'); + }); + + await t.test('ops relay across a line and around a cycle', async (t2) => { + const line = await createMesh( + t2, + ['A', 'B', 'C'], + [ + ['A', 'B'], + ['C', 'B'], + ], + createStorage, + ); + const [a, b, c] = ['A', 'B', 'C'].map((name) => line.get(name)); + const idA = await a.insert({ from: 'A' }); + const idC = await c.insert({ from: 'C' }); + await settle([a, b, c]); + await assertConverged([a, b, c]); + assert.deepStrictEqual(await c.get(idA), { from: 'A' }); + assert.deepStrictEqual(await a.get(idC), { from: 'C' }); + + const ring = await createMesh( + t2, + ['X', 'Y', 'Z'], + [ + ['X', 'Y'], + ['Y', 'Z'], + ['Z', 'X'], + ], + createStorage, + ); + const [x, y, z] = ['X', 'Y', 'Z'].map((name) => ring.get(name)); + const applied = new Map([x, y, z].map((s) => [s, 0])); + for (const s of [x, y, z]) s.on('change', () => applied.set(s, applied.get(s) + 1)); + await x.insert({ n: 1 }); + await y.insert({ n: 2 }); + await z.insert({ n: 3 }); + await settle([x, y, z]); + await assertConverged([x, y, z]); + // Three ops exist; each node applied each exactly once โ€” no echo storm. + for (const s of [x, y, z]) assert.strictEqual(applied.get(s), 3); + }); + + await t.test('concurrent writes converge: registers by last writer, counters by sum', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + const a = mesh.get('A'); + const b = mesh.get('B'); + // A counter is created by inc(): a field initialised in insert() is a register. + const id = await a.insert({ title: 'v0' }); + await settle([a, b]); + const link = mesh.links.get('mem://B'); + + // Partition, then write on both sides. + link.disconnect(); + await a.update(id, { title: 'from A', tag: 'a' }); + await tick(); + await b.update(id, { title: 'from B' }); + await a.inc(id, 'hits', 2); + await b.inc(id, 'hits', 3); + await link.reconnect(); + await settle([a, b]); + const state = await assertConverged([a, b]); + assert.strictEqual(state[id].tag, 'a'); + assert.strictEqual(state[id].hits, 5); + assert.strictEqual(state[id].title, 'from B', 'the later write wins on both nodes'); + // The register/counter rule holds across the wire too. + await assert.rejects(() => b.update(id, { hits: 9 }), /counter/); + }); + + await t.test('delete versus a concurrent write converges either way', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + const a = mesh.get('A'); + const b = mesh.get('B'); + const first = await a.insert({ n: 1 }); + const second = await a.insert({ n: 2 }); + await settle([a, b]); + const link = mesh.links.get('mem://B'); + link.disconnect(); + await a.delete(first); + await tick(); + await b.update(first, { n: 10 }); // newer than the delete: revives + await b.delete(second); + await tick(); + await a.update(second, { n: 20 }); // newer than B's delete: revives + await link.reconnect(); + await settle([a, b]); + const state = await assertConverged([a, b]); + assert.deepStrictEqual(state[first], { n: 10 }); + assert.deepStrictEqual(state[second], { n: 20 }); + // Now a delete that is genuinely last stays a delete everywhere. + await b.delete(first); + await settle([a, b]); + assert.strictEqual(await a.get(first), null); + assert.strictEqual(await b.get(first), null); + }); + + await t.test('a random workload over a mesh converges (seeded)', async (t2) => { + const names = ['A', 'B', 'C', 'D']; + const edges = [ + ['A', 'B'], + ['B', 'C'], + ['C', 'D'], + ['D', 'A'], + ['A', 'C'], + ]; + const mesh = await createMesh(t2, names, edges, createStorage); + const storages = names.map((name) => mesh.get(name)); + const random = prng(20260905); + const ids = ['r1', 'r2', 'r3']; + for (const id of ids) await storages[0].set(id, { v: 0 }); + await settle(storages); + for (let step = 0; step < 40; step++) { + const storage = storages[Math.floor(random() * storages.length)]; + const id = ids[Math.floor(random() * ids.length)]; + const roll = random(); + try { + if (roll < 0.4) await storage.set(id, { v: step, by: storage.nodeId.slice(0, 4) }); + else if (roll < 0.7) await storage.update(id, { [`f${step % 3}`]: step }); + else if (roll < 0.85) await storage.inc(id, 'count', 1); + else await storage.delete(id); + } catch (error) { + // A record deleted elsewhere is a legitimate outcome of the workload. + if (!/not found/.test(error.message)) throw error; + } + } + await settle(storages, { timeout: 10_000 }); + await assertConverged(storages); + }); + + await t.test('a late joiner bootstraps from a snapshot once history is compacted', async (t2) => { + const options = { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }; + const mesh = await createMesh(t2, ['A', 'B'], [], createStorage, options); + const a = mesh.get('A'); + const id = await a.insert({ n: 0 }); + for (let i = 0; i < 12; i++) await a.inc(id, 'hits', 1); + await a.insert({ other: true }); + assert.ok(a.log.floor > 1, 'history was compacted'); + + const b = mesh.get('B'); + const events = []; + b.sync.on('bootstrap', (event) => events.push(event)); + await b.sync.addNode({ url: 'mem://A' }); + await settle([a, b]); + assert.strictEqual(events.length, 1); + await assertConverged([a, b]); + assert.strictEqual((await b.get(id)).hits, 12, 'counters are exact after a snapshot'); + // Live after bootstrap, both ways. + await a.inc(id, 'hits', 1); + const fromB = await b.insert({ late: true }); + await settle([a, b]); + assert.strictEqual((await b.get(id)).hits, 13); + assert.deepStrictEqual(await a.get(fromB), { late: true }); + // A snapshot merged into B is in B's log, so a peer of B gets it too. + const c = await createStorage(t2, 'C', { + sync: { connect: () => new (require('../utils/memory-link.js').MemoryLink)(b.sync.unit({ access: 'public' })) }, + }); + await c.sync.addNode({ url: 'mem://B' }); + await settle([a, b, c]); + await assertConverged([a, b, c]); + }); + + await t.test('a disconnect and reconnect resumes from the last event id, both ways', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + const a = mesh.get('A'); + const b = mesh.get('B'); + const id = await a.insert({ n: 0 }); + await settle([a, b]); + const link = mesh.links.get('mem://B'); + const peers = []; + a.sync.on('peer', (event) => peers.push(event)); + + link.disconnect(); + await waitUntil(() => a.sync.peers()[0].connected === false); + for (let i = 1; i <= 3; i++) await a.update(id, { [`a${i}`]: i }); + for (let i = 1; i <= 2; i++) await b.update(id, { [`b${i}`]: i }); + const received = []; + a.sync.on('sync', (event) => received.push(event)); + await link.reconnect(); + await settle([a, b]); + await assertConverged([a, b]); + assert.deepStrictEqual(await a.get(id), { n: 0, a1: 1, a2: 2, a3: 3, b1: 1, b2: 2 }); + assert.ok(peers.some((event) => event.reconnected)); + // Exactly one live subscription afterwards, and no bootstrap was needed. + const [peer] = a.sync.peers(); + assert.strictEqual(peer.subscribed, true); + assert.strictEqual(a.sync.peers().length, 1); + assert.ok(!received.some((event) => event.bootstrap)); + // Resume delivered B's two ops, nothing older. + const replayed = received.filter((event) => event.received !== undefined).reduce((sum, e) => sum + e.received, 0); + assert.strictEqual(replayed, 2); + }); + + await t.test('scopes: a node syncs only what it asked for, plus everything global', async (t2) => { + const scope = (id, data) => data.$type ?? '*'; + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage, { + sync: { scope }, + scopes: { A: ['Doc'] }, + }); + const a = mesh.get('A'); + const b = mesh.get('B'); + const doc = await b.insert({ $type: 'Doc', t: 1 }); + const other = await b.insert({ $type: 'Other', t: 2 }); + const global = await b.insert({ t: 3 }); + const fromA = await a.insert({ $type: 'Note', t: 4 }); + await waitUntil(async () => (await a.get(doc)) !== null && (await a.get(global)) !== null); + await waitUntil(async () => (await b.get(fromA)) === null || true); + await new Promise((resolve) => setTimeout(resolve, 50)); + assert.deepStrictEqual(await a.get(doc), { $type: 'Doc', t: 1 }); + assert.deepStrictEqual(await a.get(global), { t: 3 }); + assert.strictEqual(await a.get(other), null, 'out of scope'); + assert.strictEqual(await b.get(fromA), null, 'A pushes only its scopes too'); + assert.deepStrictEqual(a.sync.peers()[0].node, b.nodeId); + // The host put the subscriber in the scope's room. + assert.ok(mesh.links.get('mem://B').client.rooms.has('syncom:Doc')); + }); + + await t.test('a live Record on one node follows writes made on another', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + const a = mesh.get('A'); + const b = mesh.get('B'); + const id = await a.insert({ title: 'Draft', views: 0 }); + await settle([a, b]); + const record = await b.record(id); + const seen = []; + record.on('update', (data, delta) => seen.push(delta)); + record.title = 'B edit'; + await a.update(id, { views: 9 }); + await waitUntil(() => record.views === 9); + assert.strictEqual(record.title, 'B edit', 'unsaved local edit survives'); + assert.deepStrictEqual(seen.at(-1), { views: 9 }); + await record.save(); + await settle([a, b]); + assert.deepStrictEqual(await a.get(id), { title: 'B edit', views: 9 }); + }); + + await t.test('identity, cursor and vector survive a reopen; no bootstrap is needed', async (t2) => { + const mesh = await createMesh(t2, ['A', 'B'], [['A', 'B']], createStorage); + let a = mesh.get('A'); + const b = mesh.get('B'); + const id = await a.insert({ n: 1 }); + await settle([a, b]); + const nodeId = a.nodeId; + const { cursor } = a.sync.peers()[0]; + await a.close(); + assert.strictEqual(a.sync.started, false); + + await b.update(id, { n: 2 }); // while A is away + // A fresh link, as a real connect(node) would make a fresh client. + const connect = (node) => new (require('../utils/memory-link.js').MemoryLink)(mesh.units.get(node.url.slice(6))); + a = await createStorage(t2, 'A', { sync: { connect, autoStart: false } }); + assert.strictEqual(a.nodeId, nodeId); + assert.strictEqual(a.sync.listNodes()[0].cursor, cursor); + const bootstraps = []; + a.sync.on('bootstrap', (event) => bootstraps.push(event)); + await a.sync.start(); + await settle([a, b]); + await assertConverged([a, b]); + assert.deepStrictEqual(await a.get(id), { n: 2 }); + assert.strictEqual(bootstraps.length, 0, 'resumed from the cursor, not re-bootstrapped'); + await a.sync.stop(); + assert.strictEqual(a.sync.peers().length, 0); + }); + + await t.test('the hosting unit refuses what it must', async (t2) => { + const mesh = await createMesh(t2, ['A'], [], createStorage); + const a = mesh.get('A'); + const unit = a.sync.unit({ authorize: (context, hello) => hello.token === 'ok' }); + const { MemoryLink } = require('../utils/memory-link.js'); + const link = new MemoryLink(unit); + await assert.rejects(() => link.call('syncom/push', { ops: [] }), /hello first/); + await assert.rejects(() => link.call('syncom/hello', { node: a.nodeId, token: 'ok' }), /itself/); + await assert.rejects(() => link.call('syncom/hello', { node: 'peer', token: 'no' }), /Not authorized/); + await assert.rejects(() => link.call('syncom/hello', { node: 'peer', token: 'ok', version: 99 }), /protocol/); + await assert.rejects(() => link.call('syncom/hello', {}), /node id/); + const hello = await link.call('syncom/hello', { node: 'peer', token: 'ok', version: 1 }); + assert.strictEqual(hello.node, a.nodeId); + assert.strictEqual(hello.epoch, a.log.epoch); + assert.strictEqual(unit.hello.access, 'session', 'the default access is wrpcโ€™s default'); + assert.strictEqual(a.sync.unit({ access: 'public' }).push.access, 'public'); + const solo = await createStorage(t2, 'solo'); + await assert.rejects(() => solo.sync.start(), /connect/); + }); +}; + +module.exports = { runSyncEngineTests }; diff --git a/tests/suites/sync-manager.js b/tests/suites/sync-manager.js index 40705ad..d9632f0 100644 --- a/tests/suites/sync-manager.js +++ b/tests/suites/sync-manager.js @@ -4,54 +4,52 @@ const runSyncManagerCoreTests = async (t, assert, getSync) => { await t.test('addNode listNodes and removeNode', async (t2) => { const sync = await getSync(t2); - await sync.addNode({ - url: 'http://peer.example.com:9000', - type: 'uplink', - token: 't1', - }); + await sync.addNode({ url: 'wss://peer.example.com/api', token: 't1' }); const nodes = sync.listNodes(); assert.strictEqual(nodes.length, 1); - assert.strictEqual(nodes[0].url, 'http://peer.example.com:9000'); - assert.strictEqual(nodes[0].type, 'uplink'); + assert.strictEqual(nodes[0].url, 'wss://peer.example.com/api'); + assert.strictEqual(nodes[0].token, 't1'); + assert.deepStrictEqual(nodes[0].scopes, ['*']); + assert.strictEqual(nodes[0].enabled, true); + assert.strictEqual(nodes[0].lastSync, null); - await sync.removeNode('http://peer.example.com:9000'); + await sync.removeNode('wss://peer.example.com/api'); assert.strictEqual(sync.listNodes().length, 0); }); await t.test('addNode rejects bad input', async (t2) => { const sync = await getSync(t2); - await assert.rejects(() => sync.addNode({ type: 'uplink' }), /URL is required/); - - await assert.rejects( - () => - sync.addNode({ - url: 'http://a.com:1', - type: 'relay', - }), - /uplink or downlink/, - ); - - await sync.addNode({ - url: 'http://dup.example.com:80', - type: 'downlink', - }); - - await assert.rejects( - () => - sync.addNode({ - url: 'http://dup.example.com:80', - type: 'uplink', - }), - /already exists/, - ); + await assert.rejects(() => sync.addNode({ token: 'x' }), /URL is required/); + await assert.rejects(() => sync.addNode({ url: 'ws://a.com:1', scopes: 'nope' }), /scopes/); + + await sync.addNode({ url: 'ws://dup.example.com:80' }); + await assert.rejects(() => sync.addNode({ url: 'ws://dup.example.com:80' }), /already exists/); + }); + + await t.test('two peers on one host, or a url without a port, do not collide', async (t2) => { + const sync = await getSync(t2); + await sync.addNode({ url: 'ws://host.example.com:9000' }); + await sync.addNode({ url: 'ws://host.example.com:9001' }); + await sync.addNode({ url: 'wss://host.example.com/api' }); + assert.strictEqual(sync.listNodes().length, 3); }); await t.test('removeNode throws when unknown', async (t2) => { const sync = await getSync(t2); await assert.rejects(() => sync.removeNode('http://nope.example.com:1'), /not found/); }); + + await t.test('the registry survives a reopen', async (t2) => { + const sync = await getSync(t2); + await sync.addNode({ url: 'wss://persist.example.com/api', token: 'k', scopes: ['docs'] }); + const reopened = await getSync(t2, sync); + const [node] = reopened.listNodes(); + assert.strictEqual(node.url, 'wss://persist.example.com/api'); + assert.strictEqual(node.token, 'k'); + assert.deepStrictEqual(node.scopes, ['docs']); + }); }; module.exports = { runSyncManagerCoreTests }; diff --git a/tests/sync.browser.test.js b/tests/sync.browser.test.js index a2a1ede..fd335f0 100644 --- a/tests/sync.browser.test.js +++ b/tests/sync.browser.test.js @@ -2,9 +2,21 @@ const test = require('node:test'); const assert = require('node:assert'); -const { SyncManager } = require('../src/sync.browser.js'); +const { installBrowserGlobals, requireBrowserBarrel } = require('./utils/browser-env.js'); const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); test('SyncManager (browser)', async (t) => { - await runSyncManagerCoreTests(t, assert, async () => new SyncManager()); + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = requireBrowserBarrel(); + + // The registry lives in OPFS now, so the reopen case is real in the browser too. + const dirs = new WeakMap(); + let n = 0; + await runSyncManagerCoreTests(t, assert, async (t2, previous) => { + const path = (previous && dirs.get(previous)) || `sync-${n++}`; + const storage = await browser.open({ path }); + dirs.set(storage.sync, path); + return storage.sync; + }); }); diff --git a/tests/sync.engine.browser.test.js b/tests/sync.engine.browser.test.js new file mode 100644 index 0000000..b507fa3 --- /dev/null +++ b/tests/sync.engine.browser.test.js @@ -0,0 +1,22 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { installBrowserGlobals, requireBrowserBarrel } = require('./utils/browser-env.js'); +const { runSyncEngineTests } = require('./suites/sync-engine.js'); + +test('Sync engine (browser halves)', async (t) => { + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = requireBrowserBarrel(); + + let n = 0; + const paths = new Map(); + await runSyncEngineTests(t, assert, async (t2, name, options = {}) => { + const key = `${t2.name}:${name}`; + if (!paths.has(key)) paths.set(key, `mesh-${n++}-${name}`); + const storage = await browser.open({ path: paths.get(key), logger: false, ...options }); + t2.after(() => storage.close().catch(() => {})); + return storage; + }); +}); diff --git a/tests/sync.engine.test.js b/tests/sync.engine.test.js new file mode 100644 index 0000000..2c46929 --- /dev/null +++ b/tests/sync.engine.test.js @@ -0,0 +1,24 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runSyncEngineTests } = require('./suites/sync-engine.js'); + +test('Sync engine (node)', async (t) => { + // The same name within one subtest reopens the same directory. + const dirs = new Map(); + await runSyncEngineTests(t, assert, async (t2, name, options = {}) => { + const key = `${t2.name}:${name}`; + let dir = dirs.get(key); + if (!dir) { + dir = await createTemp(); + dirs.set(key, dir); + t2.after(() => cleanupTemp(dir)); + } + const storage = await syncom.open({ path: dir, logger: false, ...options }); + t2.after(() => storage.close().catch(() => {})); + return storage; + }); +}); diff --git a/tests/sync.manager.test.js b/tests/sync.manager.test.js new file mode 100644 index 0000000..ba1022a --- /dev/null +++ b/tests/sync.manager.test.js @@ -0,0 +1,213 @@ +'use strict'; + +// The dialing side's edges: registry toggles, manual sync, refusals from the +// link factory, a host that already dials us, a link that cannot stream, and +// failures on the way out. + +const test = require('node:test'); +const assert = require('node:assert'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { MemoryLink } = require('./utils/memory-link.js'); +const { waitUntil, settle } = require('./utils/mesh.js'); + +const openTemp = async (t, options = {}) => { + const dir = await createTemp(); + t.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, ...options }); + t.after(() => storage.close().catch(() => {})); + return storage; +}; + +test('SyncManager edges', async (t) => { + await t.test('enableNode disconnects and reconnects; sync(url) pushes on demand', async (t2) => { + const host = await openTemp(t2); + const unit = host.sync.unit({ access: 'public' }); + const links = []; + const dialer = await openTemp(t2, { + sync: { + autoStart: false, + connect: () => { + const link = new MemoryLink(unit); + links.push(link); + return link; + }, + }, + }); + assert.strictEqual(dialer.sync.storage, dialer); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + + await dialer.sync.enableNode('mem://host', false); + assert.strictEqual(dialer.sync.peers().length, 0); + assert.strictEqual(dialer.sync.listNodes()[0].enabled, false); + const id = await dialer.insert({ offline: true }); + assert.strictEqual(await host.get(id), null); + + await dialer.sync.enableNode('mem://host'); + await settle([host, dialer]); + assert.deepStrictEqual(await host.get(id), { offline: true }); + assert.strictEqual(links.length, 2, 'a fresh link per connect'); + await assert.rejects(() => dialer.sync.enableNode('mem://nope'), /not found/); + + const results = await dialer.sync.sync('mem://host'); + assert.deepStrictEqual(results, [{ url: 'mem://host', pushed: 0 }]); + assert.deepStrictEqual(await dialer.sync.sync('mem://unknown'), []); + assert.strictEqual((await dialer.sync.sync()).length, 1); + // start() twice is a no-op; removing a connected node drops its link. + await dialer.sync.start(); + await dialer.sync.removeNode('mem://host'); + assert.strictEqual(dialer.sync.peers().length, 0); + }); + + await t.test('a connect() that does not yield a link is reported, not thrown', async (t2) => { + const errors = []; + const dialer = await openTemp(t2, { sync: { autoStart: false, connect: () => ({ call() {} }) } }); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://bad' }); + await dialer.sync.start(); + await waitUntil(() => errors.length > 0); + assert.match(errors[0].message, /must return a SyncLink/); + assert.strictEqual(errors[0].url, 'mem://bad'); + // Without an error listener the failure goes to the logger only. + const quiet = await openTemp(t2, { + sync: { + autoStart: false, + connect: () => { + throw new Error('boom'); + }, + }, + }); + await quiet.sync.addNode({ url: 'mem://bad' }); + await quiet.sync.start(); + await new Promise((resolve) => setTimeout(resolve, 20)); + assert.strictEqual(quiet.sync.peers()[0].connected, false); + }); + + await t.test('a host that already dials us answers duplicate and we hang up', async (t2) => { + const host = await openTemp(t2); + const unit = host.sync.unit({ access: 'public' }); + class Duplicate extends MemoryLink { + async call(method, args) { + if (method === 'syncom/hello') return { node: 'other', duplicate: true }; + return super.call(method, args); + } + + close() { + super.close(); + throw new Error('already closed'); + } + } + const dialer = await openTemp(t2, { sync: { autoStart: false, connect: () => new Duplicate(unit) } }); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => dialer.sync.peers().length === 0); + // The throwing close() was swallowed; a bad hello is an error event. + const errors = []; + class Bad extends MemoryLink { + async call() { + return { nope: true }; + } + } + const other = await openTemp(t2, { sync: { autoStart: false, connect: () => new Bad(unit) } }); + other.sync.on('error', (error) => errors.push(error)); + await other.sync.addNode({ url: 'mem://host' }); + await other.sync.start(); + await waitUntil(() => errors.length > 0); + assert.match(errors[0].message, /Bad hello/); + }); + + await t.test('a streamed snapshot needs a link that can read streams', async (t2) => { + const host = await openTemp(t2, { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }); + for (let i = 0; i < 8; i++) await host.insert({ i }); + const unit = host.sync.unit({ access: 'public' }); + // A link whose "client" claims binary support but that cannot stream. + class Binary extends MemoryLink { + constructor(u) { + super(u); + this.client.binary = true; + this.client.createStream = (name, size) => ({ id: 's1', size, write() {}, end() {} }); + } + } + const errors = []; + const dialer = await openTemp(t2, { sync: { autoStart: false, connect: () => new Binary(unit) } }); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => errors.length > 0); + assert.match(errors[0].message, /cannot read streams/); + }); + + await t.test('a push that fails mid-flight is reported and recomputed from the vector', async (t2) => { + const host = await openTemp(t2); + const unit = host.sync.unit({ access: 'public' }); + let failing = false; + class Flaky extends MemoryLink { + async call(method, args) { + if (failing && method === 'syncom/push') throw Object.assign(new Error('flaky'), { code: 503 }); + return super.call(method, args); + } + } + const link = new Flaky(unit); + const errors = []; + const dialer = await openTemp(t2, { sync: { autoStart: false, connect: () => link } }); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + + failing = true; + const id = await dialer.insert({ n: 1 }); + await waitUntil(() => errors.length > 0); + assert.match(errors[0].message, /flaky/); + assert.strictEqual(await host.get(id), null); + failing = false; + // The next handshake (a reconnect) pushes what the host still lacks. + link.disconnect(); + await link.reconnect(); + await settle([host, dialer]); + assert.deepStrictEqual(await host.get(id), { n: 1 }); + }); + + await t.test('a scoped late joiner adopts only the vector keys of its scopes', async (t2) => { + const scope = (id, data) => data.$type ?? '*'; + const host = await openTemp(t2, { sync: { scope, log: { rollAfter: 2 }, retention: { ops: 4 } } }); + const doc = await host.insert({ $type: 'Doc', n: 1 }); + for (let i = 0; i < 8; i++) await host.insert({ $type: 'Other', i }); + const unit = host.sync.unit({ access: 'public' }); + const dialer = await openTemp(t2, { sync: { scope, autoStart: false, connect: () => new MemoryLink(unit) } }); + await dialer.sync.addNode({ url: 'mem://host', scopes: ['Doc'] }); + const bootstraps = []; + dialer.sync.on('bootstrap', (event) => bootstraps.push(event)); + await dialer.sync.start(); + await waitUntil(() => bootstraps.length > 0); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + assert.deepStrictEqual(await dialer.get(doc), { $type: 'Doc', n: 1 }); + assert.strictEqual((await dialer.list()).length, 1, 'Other records were not in the snapshot'); + const keys = Object.keys(dialer.vector); + assert.ok(keys.includes(`${host.nodeId}@Doc`)); + assert.ok(!keys.includes(`${host.nodeId}@Other`), 'a scope we did not ask for is not claimed'); + }); + + await t.test('a subscription error that is not "hello first" is reported', async (t2) => { + const host = await openTemp(t2); + const unit = host.sync.unit({ access: 'public' }); + const link = new MemoryLink(unit); + const errors = []; + const dialer = await openTemp(t2, { sync: { autoStart: false, connect: () => link } }); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + // Break the feed from the host side: the generator throws on its next pull. + const original = unit.changes.handler; + unit.changes.handler = async function* () { + throw new Error('feed broke'); + }; + link.disconnect(); + await link.reconnect({ freshClient: false }); + await waitUntil(() => errors.some((error) => /feed broke/.test(error.message))); + unit.changes.handler = original; + }); +}); diff --git a/tests/sync.test.js b/tests/sync.test.js index 9432b31..f5d4fa6 100644 --- a/tests/sync.test.js +++ b/tests/sync.test.js @@ -7,10 +7,16 @@ const { createTemp, cleanupTemp } = require('./utils/temp.js'); const { runSyncManagerCoreTests } = require('./suites/sync-manager.js'); test('SyncManager', async (t) => { - await runSyncManagerCoreTests(t, assert, async (t2) => { - const tempDir = await createTemp(); - t2.after(() => cleanupTemp(tempDir)); + // `previous` asks for the SAME directory again โ€” the reopen case. + const dirs = new WeakMap(); + await runSyncManagerCoreTests(t, assert, async (t2, previous) => { + let tempDir = previous ? dirs.get(previous) : null; + if (!tempDir) { + tempDir = await createTemp(); + t2.after(() => cleanupTemp(tempDir)); + } const storage = await syncom.open({ path: tempDir }); + dirs.set(storage.sync, tempDir); return storage.sync; }); }); diff --git a/tests/utils/browser-env.js b/tests/utils/browser-env.js new file mode 100644 index 0000000..9a90a02 --- /dev/null +++ b/tests/utils/browser-env.js @@ -0,0 +1,65 @@ +'use strict'; + +// Runs the SHARED modules (storage.js, chain.js, ...) the way a bundler that +// honours package.json#browser would: every `./src/x.js` the map redirects is +// primed in require.cache with its `x.browser.js` twin BEFORE the shared +// module is loaded, so its nested requires resolve to the browser halves. +// Requiring `src/index.browser.js` alone does not do that under Node โ€” it +// names the browser variants directly, but storage.js still says +// `require('./crypto.js')`, which Node resolves literally. +// +// Use this from a test file that never loads the node barrel: the cache +// priming is process-wide, and `node --test` gives every file its own process. + +const path = require('node:path'); +const Module = require('node:module'); +const { webcrypto } = require('node:crypto'); +const { installMockOpfs } = require('./mock-opfs.js'); + +const ROOT = path.join(__dirname, '..', '..'); +const pkg = require('../../package.json'); + +const installBrowserGlobals = () => { + const opfs = installMockOpfs(); + const prevCrypto = Object.getOwnPropertyDescriptor(globalThis, 'crypto'); + Object.defineProperty(globalThis, 'crypto', { + value: webcrypto, + configurable: true, + writable: true, + enumerable: true, + }); + if (typeof globalThis.btoa !== 'function') { + globalThis.btoa = (s) => Buffer.from(s, 'latin1').toString('base64'); + } + if (typeof globalThis.atob !== 'function') { + globalThis.atob = (s) => Buffer.from(s, 'base64').toString('latin1'); + } + const uninstall = () => { + opfs.uninstall(); + if (prevCrypto) Object.defineProperty(globalThis, 'crypto', prevCrypto); + else delete globalThis.crypto; + }; + return { opfs, uninstall }; +}; + +// Primes the cache from the package.json#browser map, then returns the +// browser barrel โ€” the same module graph a browser bundle contains. +const requireBrowserBarrel = () => { + for (const [from, to] of Object.entries(pkg.browser)) { + // Only the split leaves are primed; the barrel itself is required LAST, + // once every twin is in place โ€” requiring it first would load the whole + // graph with the node halves before the loop reached them. + if (!from.startsWith('./src/') || from === './src/index.js') continue; + const nodePath = path.join(ROOT, from); + if (require.cache[nodePath]) continue; + const twin = new Module(nodePath, module); + twin.filename = nodePath; + twin.paths = Module._nodeModulePaths(path.dirname(nodePath)); + twin.exports = require(path.join(ROOT, to)); + twin.loaded = true; + require.cache[nodePath] = twin; + } + return require('../../src/index.browser.js'); +}; + +module.exports = { installBrowserGlobals, requireBrowserBarrel }; diff --git a/tests/utils/memory-link.js b/tests/utils/memory-link.js new file mode 100644 index 0000000..0a1d5cf --- /dev/null +++ b/tests/utils/memory-link.js @@ -0,0 +1,123 @@ +'use strict'; + +// An in-process SyncLink over a host's `unit()` โ€” what the engine tests run +// on instead of a socket. It mimics what wrpc does at the boundary: arguments +// and results cross as JSON (no shared references, the tracked symbol +// stripped), `changes` is driven as an async generator with `{ lastEventId, +// signal }`, a dropped connection aborts the generator, and a reconnect +// re-opens every subscription from the last event id it delivered โ€” exactly +// wrpc's restore โ€” before announcing `reconnect`. + +const { Emitter } = require('../../src/utils.js'); + +const TRACKED = Symbol.for('wrpc.tracked'); +const clone = (value) => (value === undefined ? undefined : JSON.parse(JSON.stringify(value))); + +class MemoryLink extends Emitter { + #unit; + #subscriptions = new Set(); + #down = false; + #options; + calls = []; + + constructor(unit, options = {}) { + super(); + this.#unit = unit; + this.#options = options; + // What wrpc's `context.client` looks like from a handler's point of view. + this.client = { + data: {}, + binary: false, + rooms: new Set(), + join: (room) => void this.client.rooms.add(room), + }; + this.context = { client: this.client }; + } + + async call(method, args) { + if (this.#down) throw Object.assign(new Error('Connection closed'), { code: 503 }); + const name = method.split('/')[1]; + const procedure = this.#unit[name]; + if (!procedure) throw Object.assign(new Error(`Unknown method ${method}`), { code: 404 }); + this.calls.push(method); + if (this.#options.latency) await new Promise((resolve) => setTimeout(resolve, this.#options.latency)); + return clone(await procedure.handler(this.context, clone(args))); + } + + subscribe(method, args, handlers) { + const record = { method, args: clone(args), handlers, lastEventId: handlers.lastEventId, controller: null }; + this.#subscriptions.add(record); + this.#open(record); + return { + get lastEventId() { + return record.lastEventId; + }, + unsubscribe: () => { + this.#subscriptions.delete(record); + record.controller?.abort(); + }, + }; + } + + #open(record) { + const controller = new AbortController(); + record.controller = controller; + const name = record.method.split('/')[1]; + const procedure = this.#unit[name]; + const { onData, onError, onEnd } = record.handlers; + const run = async () => { + const iterator = procedure.handler(this.context, clone(record.args), { + lastEventId: record.lastEventId, + signal: controller.signal, + }); + for await (const value of iterator) { + if (controller.signal.aborted) break; + const isTracked = value && typeof value === 'object' && value[TRACKED] === true; + const eventId = isTracked ? value.id : undefined; + const data = clone(isTracked ? value.data : value); + if (eventId !== undefined) record.lastEventId = eventId; + if (this.#options.drop?.(data)) continue; + await onData?.(data, eventId); + } + if (!controller.signal.aborted) { + this.#subscriptions.delete(record); + onEnd?.(); + } + }; + run().catch((error) => { + if (controller.signal.aborted) return; + this.#subscriptions.delete(record); + onError?.(error); + }); + } + + // --- test hooks ------------------------------------------------------- + + /** The socket died: live feeds stop, calls fail, `close` is announced. */ + disconnect() { + this.#down = true; + for (const record of this.#subscriptions) record.controller?.abort(); + void this.emit('close'); + } + + /** + * wrpc's reconnect: a NEW server-side client (its `data` bag is empty, so + * the host has to be greeted again), every subscription re-opened from the + * last event id it delivered, then `reconnect` announced. + */ + async reconnect({ freshClient = true } = {}) { + this.#down = false; + if (freshClient) this.client.data = {}; + for (const record of Array.from(this.#subscriptions)) this.#open(record); + await this.emit('open'); + await this.emit('reconnect'); + } + + close() { + this.#down = true; + for (const record of this.#subscriptions) record.controller?.abort(); + this.#subscriptions.clear(); + } +} + +module.exports = { MemoryLink }; diff --git a/tests/utils/mesh.js b/tests/utils/mesh.js new file mode 100644 index 0000000..6ab956f --- /dev/null +++ b/tests/utils/mesh.js @@ -0,0 +1,79 @@ +'use strict'; + +// Builds N storages wired by memory links from an adjacency list โ€” `['A', +// 'B']` means A dials B โ€” and knows when the whole thing has gone quiet. + +const { MemoryLink } = require('./memory-link.js'); + +const createMesh = async (t, names, edges, createStorage, options = {}) => { + const units = new Map(); + const links = new Map(); + const storages = new Map(); + const connect = (node) => { + const [, host] = node.url.split('mem://'); + const unit = units.get(host); + if (!unit) throw new Error(`no host ${host}`); + const link = new MemoryLink(unit, options.link); + links.set(node.url, link); + return link; + }; + for (const name of names) { + const storage = await createStorage(t, name, { sync: { connect, autoStart: false, ...options.sync } }); + storages.set(name, storage); + units.set(name, storage.sync.unit(options.unit ?? { access: 'public' })); + } + for (const [from, to] of edges) { + await storages.get(from).sync.addNode({ url: `mem://${to}`, scopes: options.scopes?.[from] }); + } + for (const storage of storages.values()) await storage.sync.start(); + return { storages, links, units, get: (name) => storages.get(name) }; +}; + +const sameVectors = (storages) => { + const vectors = storages.map((storage) => storage.vector); + const keys = new Set(vectors.flatMap((vector) => Object.keys(vector))); + for (const key of keys) { + // A node's bootstrap stream never reaches the node it bootstrapped from. + if (key.endsWith('@~')) continue; + const first = vectors[0][key] ?? 0; + if (!vectors.every((vector) => (vector[key] ?? 0) === first)) return false; + } + return true; +}; + +// Quiet = every vector equal and nothing in flight, twice in a row. +const settle = async (storages, { timeout = 5000, step = 10 } = {}) => { + const deadline = Date.now() + timeout; + let stable = 0; + while (Date.now() < deadline) { + const idle = storages.every((storage) => storage.sync.peers().every((peer) => peer.pending === 0)); + if (idle && sameVectors(storages)) { + if (++stable >= 3) return; + } else { + stable = 0; + } + await new Promise((resolve) => setTimeout(resolve, step)); + } + const vectors = storages.map((storage) => JSON.stringify(storage.vector)); + throw new Error(`mesh did not settle within ${timeout} ms:\n${vectors.join('\n')}`); +}; + +const waitUntil = async (predicate, { timeout = 5000, step = 10 } = {}) => { + const deadline = Date.now() + timeout; + while (Date.now() < deadline) { + if (await predicate()) return; + await new Promise((resolve) => setTimeout(resolve, step)); + } + throw new Error(`condition not met within ${timeout} ms`); +}; + +// Deterministic pseudo-random numbers for the convergence property. +const prng = (seed) => { + let state = seed >>> 0; + return () => { + state = (state * 1664525 + 1013904223) >>> 0; + return state / 0x100000000; + }; +}; + +module.exports = { createMesh, settle, waitUntil, sameVectors, prng }; diff --git a/tests/utils/mock-opfs.js b/tests/utils/mock-opfs.js index 7e7e91e..110d048 100644 --- a/tests/utils/mock-opfs.js +++ b/tests/utils/mock-opfs.js @@ -2,6 +2,7 @@ class MockFileHandle { #entry = null; + kind = 'file'; constructor(entry) { this.#entry = entry; @@ -30,11 +31,23 @@ class MockFileHandle { class MockDirectoryHandle { #entries = null; + kind = 'directory'; constructor(entries) { this.#entries = entries; } + // The FileSystemDirectoryHandle iterator: [name, handle] pairs. + async *entries() { + for (const [name, entry] of Array.from(this.#entries.entries())) { + yield [name, entry.type === 'dir' ? new MockDirectoryHandle(entry.map) : new MockFileHandle(entry)]; + } + } + + async *keys() { + for (const name of Array.from(this.#entries.keys())) yield name; + } + async getDirectoryHandle(name, { create } = {}) { if (!this.#entries.has(name)) { if (!create) { diff --git a/tests/utils/temp.js b/tests/utils/temp.js index 7450b02..31a661f 100644 --- a/tests/utils/temp.js +++ b/tests/utils/temp.js @@ -7,7 +7,7 @@ const os = require('node:os'); const createTemp = async () => { const id = Math.random().toString(36).slice(2, 11); const tmpRoot = os.tmpdir(); - const dirName = `gs-test-${Date.now()}-${id}`; + const dirName = `syncom-test-${Date.now()}-${id}`; const tempDir = path.join(tmpRoot, dirName); const mkdirOptions = { recursive: true }; await fs.mkdir(tempDir, mkdirOptions); diff --git a/tests/vector.test.js b/tests/vector.test.js new file mode 100644 index 0000000..1f9aec8 --- /dev/null +++ b/tests/vector.test.js @@ -0,0 +1,53 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const vector = require('../src/vector.js'); + +test('vector module', async (t) => { + await t.test('compareClock orders by ts, then origin, and treats null as oldest', () => { + assert.strictEqual(vector.compareClock([1, 'a'], [2, 'a']), -1); + assert.strictEqual(vector.compareClock([2, 'a'], [1, 'z']), 1); + assert.strictEqual(vector.compareClock([1, 'a'], [1, 'b']), -1); + assert.strictEqual(vector.compareClock([1, 'b'], [1, 'a']), 1); + assert.strictEqual(vector.compareClock([1, 'a'], [1, 'a']), 0); + assert.strictEqual(vector.compareClock(null, [1, 'a']), -1); + assert.strictEqual(vector.compareClock([1, 'a'], null), 1); + assert.strictEqual(vector.compareClock(null, null), 0); + assert.deepStrictEqual(vector.maxClock([1, 'a'], [1, 'b']), [1, 'b']); + assert.deepStrictEqual(vector.maxClock(null, [1, 'b']), [1, 'b']); + }); + + await t.test('vectorMax, vectorDominates and vectorHas', () => { + assert.deepStrictEqual(vector.vectorMax({ a: 2, b: 1 }, { a: 1, c: 3 }), { a: 2, b: 1, c: 3 }); + assert.strictEqual(vector.vectorDominates({ a: 2, b: 1 }, { a: 2 }), true); + assert.strictEqual(vector.vectorDominates({ a: 1 }, { a: 2 }), false); + assert.strictEqual(vector.vectorDominates({}, {}), true); + assert.strictEqual(vector.vectorHas({ a: 3 }, { origin: 'a', seq: 3 }), true); + assert.strictEqual(vector.vectorHas({ a: 3 }, { origin: 'a', seq: 4 }), false); + assert.strictEqual(vector.vectorHas({}, { origin: 'a', seq: 1 }), false); + }); + + await t.test('event ids round-trip and reject garbage', () => { + assert.strictEqual(vector.formatEventId('ep', 7), 'ep.7'); + assert.deepStrictEqual(vector.parseEventId('ep.7'), { epoch: 'ep', seq: 7 }); + assert.deepStrictEqual(vector.parseEventId('a.b.7'), { epoch: 'a.b', seq: 7 }); + assert.strictEqual(vector.parseEventId('nodot'), null); + assert.strictEqual(vector.parseEventId('.7'), null); + assert.strictEqual(vector.parseEventId('ep.x'), null); + assert.strictEqual(vector.parseEventId('ep.-1'), null); + assert.strictEqual(vector.parseEventId(7), null); + }); + + await t.test('Clock is monotonic and never falls behind an observed stamp', () => { + const clock = new vector.Clock(); + const a = clock.now(); + const b = clock.now(); + assert.ok(b > a); + clock.observe(b + 1_000_000); + assert.ok(clock.now() > b + 1_000_000); + clock.observe('nope'); + clock.observe(1); + assert.ok(clock.now() > b + 1_000_000); + }); +}); diff --git a/tests/wrpc.integration.test.js b/tests/wrpc.integration.test.js new file mode 100644 index 0000000..e6f389e --- /dev/null +++ b/tests/wrpc.integration.test.js @@ -0,0 +1,152 @@ +'use strict'; + +// The real thing: syncom nodes hosted by wrpc Servers, dialed through +// WrpcClients over WebSocket. Skips (never fails) when @alexify/wrpc is not +// installed, the way wrpc's own adapter suites treat their frameworks. + +const test = require('node:test'); +const assert = require('node:assert'); +const syncom = require('..'); +const { createWrpcLink } = require('../wrpc.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { settle, waitUntil } = require('./utils/mesh.js'); + +const optional = (name) => { + try { + return require(name); + } catch { + return null; + } +}; + +const wrpc = optional('@alexify/wrpc'); + +const openTemp = async (t, options = {}) => { + const dir = await createTemp(); + t.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, ...options }); + t.after(() => storage.close().catch(() => {})); + return storage; +}; + +// wrpc's own boot pattern: port 0, quiet logger, teardown registered at once. +const host = async (t, storage, unitOptions = { access: 'public' }, port = 0) => { + const router = wrpc.defineRouter({ syncom: storage.sync.unit(unitOptions) }); + const server = new wrpc.Server({ router, host: '127.0.0.1', port, protocol: 'http', logger: false }); + await server.listen(); + t.after(() => server.close().catch(() => {})); + const address = server.address(); + return { server, url: `ws://127.0.0.1:${address.port}${server.rpc.basePath}`, port: address.port }; +}; + +const connect = + (clientOptions = {}) => + async (node) => + createWrpcLink( + await wrpc.WrpcClient.connect(node.url, { + heartbeat: false, + reconnect: { minDelay: 50, maxDelay: 200, jitter: false }, + logger: false, + ...clientOptions, + }), + ); + +test('syncom over wrpc', { skip: wrpc ? false : '@alexify/wrpc unavailable' }, async (t) => { + await t.test('two servers and a dialer: transitive replication over WebSocket', async (t2) => { + const a = await openTemp(t2); + const b = await openTemp(t2); + const c = await openTemp(t2, { sync: { connect: connect(), autoStart: false } }); + const hostA = await host(t2, a); + const hostB = await host(t2, b); + await c.sync.addNode({ url: hostA.url }); + await c.sync.addNode({ url: hostB.url }); + await c.sync.start(); + + const fromA = await a.insert({ from: 'A' }); + const fromB = await b.insert({ from: 'B' }); + const fromC = await c.insert({ from: 'C' }); + await settle([a, b, c], { timeout: 10_000 }); + for (const storage of [a, b, c]) { + assert.deepStrictEqual(await storage.get(fromA), { from: 'A' }); + assert.deepStrictEqual(await storage.get(fromB), { from: 'B' }); + assert.deepStrictEqual(await storage.get(fromC), { from: 'C' }); + } + // The dialer sits in the scope room on both hosts. + assert.strictEqual(hostA.server.rooms.count('syncom:*'), 1); + assert.strictEqual(hostB.server.rooms.count('syncom:*'), 1); + const peers = c.sync.peers(); + assert.deepStrictEqual(peers.map((peer) => peer.node).sort(), [a.nodeId, b.nodeId].sort()); + assert.ok(peers.every((peer) => peer.subscribed)); + }); + + await t.test('a server restart on the same port: wrpc reconnects, the feed resumes, offline ops flow', async (t2) => { + const a = await openTemp(t2); + const c = await openTemp(t2, { sync: { connect: connect(), autoStart: false } }); + let hostA = await host(t2, a); + await c.sync.addNode({ url: hostA.url }); + await c.sync.start(); + const id = await a.insert({ n: 0 }); + await settle([a, c], { timeout: 10_000 }); + + const events = []; + c.sync.on('peer', (event) => events.push(event)); + c.sync.on('bootstrap', (event) => events.push({ bootstrap: true, ...event })); + await hostA.server.close(); + await waitUntil(() => c.sync.peers()[0].connected === false, { timeout: 5000 }); + await a.update(id, { whileDown: 'a' }); + await c.update(id, { whileDown: 'c' }); + + hostA = await host(t2, a, { access: 'public' }, hostA.port); + await settle([a, c], { timeout: 15_000 }); + assert.deepStrictEqual(await a.get(id), await c.get(id)); + assert.ok( + events.some((event) => event.reconnected), + 'the link announced its reconnect', + ); + assert.ok(!events.some((event) => event.bootstrap), 'resumed from the cursor, no snapshot needed'); + assert.strictEqual(c.sync.peers().length, 1); + assert.strictEqual(c.sync.peers()[0].subscribed, true); + }); + + await t.test('a late joiner bootstraps through a binary stream', async (t2) => { + const a = await openTemp(t2, { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }); + const id = await a.insert({ n: 0 }); + for (let i = 0; i < 10; i++) await a.inc(id, 'hits', 1); + for (let i = 0; i < 30; i++) await a.insert({ i, padding: 'x'.repeat(200) }); + assert.ok(a.log.floor > 1); + const hostA = await host(t2, a); + + const c = await openTemp(t2, { sync: { connect: connect(), autoStart: false } }); + const bootstraps = []; + c.sync.on('bootstrap', (event) => bootstraps.push(event)); + await c.sync.addNode({ url: hostA.url }); + await c.sync.start(); + await settle([a, c], { timeout: 15_000 }); + assert.strictEqual(bootstraps.length, 1); + assert.strictEqual(bootstraps[0].records, 31); + assert.deepStrictEqual((await c.list()).sort(), (await a.list()).sort()); + assert.strictEqual((await c.get(id)).hits, 10); + await a.inc(id, 'hits', 1); + await settle([a, c], { timeout: 10_000 }); + assert.strictEqual((await c.get(id)).hits, 11); + }); + + await t.test('authorize gates the handshake by the token the node carries', async (t2) => { + const a = await openTemp(t2); + const hostA = await host(t2, a, { access: 'public', authorize: (context, hello) => hello.token === 'secret' }); + const good = await openTemp(t2, { sync: { connect: connect(), autoStart: false } }); + const bad = await openTemp(t2, { sync: { connect: connect(), autoStart: false } }); + const refused = []; + bad.sync.on('error', (error) => refused.push(error)); + await good.sync.addNode({ url: hostA.url, token: 'secret' }); + await bad.sync.addNode({ url: hostA.url, token: 'nope' }); + await good.sync.start(); + await bad.sync.start(); + const id = await a.insert({ n: 1 }); + await settle([a, good], { timeout: 10_000 }); + assert.deepStrictEqual(await good.get(id), { n: 1 }); + await waitUntil(() => refused.length > 0, { timeout: 5000 }); + assert.match(refused[0].message, /Not authorized/); + assert.strictEqual(await bad.get(id), null); + }); +}); diff --git a/tests/wrpc.test-d.ts b/tests/wrpc.test-d.ts new file mode 100644 index 0000000..726c1a2 --- /dev/null +++ b/tests/wrpc.test-d.ts @@ -0,0 +1,18 @@ +import { expectAssignable, expectError, expectType } from 'tsd'; +import { createWrpcLink, isSyncLink, SYNC_INTROSPECTION } from '../wrpc.js'; +import type { SyncLink, WrpcClientLike, WrpcLink } from '../wrpc.js'; + +declare const client: WrpcClientLike; +const link = createWrpcLink(client); +expectType(link); +expectAssignable(link); +expectType>(link.call('syncom/hello', { node: 'a' })); +expectType>(link.stream('id')); +expectType(link.client); +expectType<'subscription'>(SYNC_INTROSPECTION.syncom.changes.kind); +createWrpcLink(client, { unit: 'replication' }); +expectError(createWrpcLink(client, { unit: 1 })); +expectError(createWrpcLink({})); + +declare const value: unknown; +if (isSyncLink(value)) expectType(value); diff --git a/tests/wrpc.test.js b/tests/wrpc.test.js new file mode 100644 index 0000000..5c58379 --- /dev/null +++ b/tests/wrpc.test.js @@ -0,0 +1,145 @@ +'use strict'; + +// The adapter against a wrpc-SHAPED client, so `src/wrpc.js` is covered with +// or without @alexify/wrpc installed; the real thing runs in +// tests/wrpc.integration.test.js. + +const test = require('node:test'); +const assert = require('node:assert'); +const { createWrpcLink, isSyncLink, SYNC_INTROSPECTION } = require('../wrpc.js'); +const { Emitter } = require('../src/utils.js'); + +class FakeWrpcClient extends Emitter { + calls = []; + used = null; + api = {}; + closed = false; + streams = new Map(); + + async call(method, args) { + this.calls.push([method, args]); + return { echoed: method }; + } + + use(introspection) { + this.used = introspection; + for (const unit in introspection) { + this.api[unit] = {}; + for (const name in introspection[unit]) { + const info = introspection[unit][name]; + if (info.kind !== 'subscription') continue; + this.api[unit][name] = { + kind: 'subscription', + subscribe: (args, options) => { + const record = { args, lastEventId: options.lastEventId, options, closed: false }; + this.last = record; + return { + get lastEventId() { + return record.lastEventId; + }, + unsubscribe: () => { + record.closed = true; + }, + }; + }, + }; + } + } + return this; + } + + // Emulates a `data` packet: the id moves BEFORE the value is delivered. + deliver(eventId, value) { + if (eventId !== undefined) this.last.lastEventId = eventId; + this.last.options.onData(value); + } + + getStream(id) { + return this.streams.get(id); + } + + close() { + this.closed = true; + } +} + +test('createWrpcLink', async (t) => { + await t.test('refuses anything not shaped like a WrpcClient', () => { + assert.throws(() => createWrpcLink({}), /expected a WrpcClient/); + assert.throws(() => createWrpcLink(null), TypeError); + assert.strictEqual(isSyncLink(createWrpcLink(new FakeWrpcClient())), true); + assert.strictEqual(isSyncLink({ call() {} }), false); + }); + + await t.test('scaffolds the unit with use() and routes calls by method name', async () => { + const client = new FakeWrpcClient(); + const link = createWrpcLink(client); + assert.deepStrictEqual(client.used, { syncom: SYNC_INTROSPECTION.syncom }); + assert.deepStrictEqual(await link.call('syncom/hello', { node: 'a' }), { echoed: 'syncom/hello' }); + assert.deepStrictEqual(client.calls, [['syncom/hello', { node: 'a' }]]); + assert.strictEqual(link.client, client); + }); + + await t.test('a renamed unit is honoured on calls and subscriptions', async () => { + const client = new FakeWrpcClient(); + const link = createWrpcLink(client, { unit: 'replication' }); + await link.call('syncom/push', { ops: [] }); + assert.deepStrictEqual(client.calls[0][0], 'replication/push'); + assert.ok(client.api.replication.changes); + link.subscribe('syncom/changes', { scopes: ['*'] }, {}); + assert.deepStrictEqual(client.last.args, { scopes: ['*'] }); + }); + + await t.test('subscribe passes lastEventId in and labels every value with the current one', () => { + const client = new FakeWrpcClient(); + const link = createWrpcLink(client); + const received = []; + const ended = []; + const handle = link.subscribe( + 'syncom/changes', + { scopes: ['*'] }, + { + lastEventId: 'ep.5', + onData: (value, eventId) => received.push([value, eventId]), + onEnd: () => ended.push('end'), + }, + ); + assert.strictEqual(client.last.lastEventId, 'ep.5'); + client.deliver('ep.6', { ops: [] }); + client.deliver(undefined, { bootstrap: true }); + assert.deepStrictEqual(received, [ + [{ ops: [] }, 'ep.6'], + [{ bootstrap: true }, 'ep.6'], + ]); + assert.strictEqual(handle.lastEventId, 'ep.6'); + client.last.options.onEnd(); + assert.deepStrictEqual(ended, ['end']); + handle.unsubscribe(); + assert.strictEqual(client.last.closed, true); + assert.throws(() => link.subscribe('syncom/push', {}, {}), /not a subscription/); + }); + + await t.test('streams, events and close reach the client', async () => { + const client = new FakeWrpcClient(); + client.streams.set( + 's1', + (async function* () { + yield new Uint8Array([1]); + })(), + ); + const link = createWrpcLink(client); + const chunks = []; + for await (const chunk of link.stream('s1')) chunks.push(chunk); + assert.strictEqual(chunks.length, 1); + + const seen = []; + const listener = () => seen.push('reconnect'); + link.on('reconnect', listener); + await client.emit('reconnect'); + link.off('reconnect', listener); + await client.emit('reconnect'); + assert.deepStrictEqual(seen, ['reconnect']); + link.close(); + assert.strictEqual(client.closed, true); + }); +}); diff --git a/wrpc.d.ts b/wrpc.d.ts new file mode 100644 index 0000000..f6a5c71 --- /dev/null +++ b/wrpc.d.ts @@ -0,0 +1,40 @@ +// The `@alexify/syncom/wrpc` subpath: a WrpcClient adapted to the SyncLink +// contract. wrpc itself is injected by the caller and never imported here. + +import type { SyncLink } from './index.js'; + +export type { SyncLink } from './index.js'; +export { isSyncLink } from './index.js'; + +/** The part of a WrpcClient the adapter relies on โ€” checked structurally. */ +export interface WrpcClientLike { + call(method: string, args?: unknown, options?: unknown): Promise; + use(introspection: Record>): unknown; + on(event: string, listener: (...args: unknown[]) => void): void; + off(event: string, listener: (...args: unknown[]) => void): void; + close(): void; + getStream?(id: string): AsyncIterable; + readonly api?: Record; +} + +export interface WrpcLinkOptions { + /** The router unit the host mounted `storage.sync.unit()` under. Default `'syncom'`. */ + unit?: string; +} + +export interface WrpcLink extends SyncLink { + stream(id: string): AsyncIterable; + readonly client: WrpcClientLike; +} + +export function createWrpcLink(client: WrpcClientLike, options?: WrpcLinkOptions): WrpcLink; + +/** What `client.use()` is fed so `client.api.syncom.changes` exists without a `load()`. */ +export const SYNC_INTROSPECTION: { + readonly syncom: { + readonly hello: {}; + readonly push: {}; + readonly snapshot: {}; + readonly changes: { readonly kind: 'subscription' }; + }; +}; diff --git a/wrpc.js b/wrpc.js new file mode 100644 index 0000000..d97942b --- /dev/null +++ b/wrpc.js @@ -0,0 +1,3 @@ +'use strict'; + +module.exports = require('./src/wrpc.js'); From a51df712f6e7f9584a893eeba42ff240787b59de Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 17:32:34 +0300 Subject: [PATCH 06/67] fix: never stamp a local write below state the node already holds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A local write was silently dropped (no op, no event, no error) whenever a stamp in the record's metadata was above the node's clock. Two paths led there: a snapshot joined from a peer whose clock ran ahead โ€” its field clocks, base, tombstone and counter stamps were never observed โ€” and a reopen, where the clock was rebuilt only from the unsealed log tail, so a rolled log or a wall clock corrected backwards left the node stamping below its own earlier writes. - ops.js: latestClock(meta), the newest stamp an entry holds, tolerant of raw peer-shaped meta. - storage.js: [COMMIT] observes the record's meta (and the joined meta for a merge) before stamping; #init seeds the clock from log.latestTs. - log.js: latestTs, maintained on load and append. - Tests for both paths and an end-to-end case in the sync-engine suite; a "Clocks" section in docs/guide/sync.md and a rule in CLAUDE.md. Co-Authored-By: Claude Fable 5.1 --- CLAUDE.md | 8 ++++++ docs/guide/sync.md | 28 ++++++++++++++++++++- src/log.js | 25 +++++++++++++++++-- src/ops.js | 24 +++++++++++++++++- src/storage.js | 15 +++++++++++- tests/ops.test.js | 21 +++++++++++++++- tests/storage.ops.test.js | 49 +++++++++++++++++++++++++++++++++++++ tests/suites/sync-engine.js | 8 ++++++ 8 files changed, 172 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index c7fcb9d..2872015 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,6 +67,14 @@ docs/ VitePress site - **Log first, then data.** `OpLog.append` resolves before the record file is written and before the op is announced; a reopen replays the unsealed tail. A node's own seq must never repeat โ€” it is recovered from the log's vector. +- **Stamps come from `#clock.now()` after the clock has observed everything + the node holds for the record.** `[COMMIT]` in `src/storage.js` does that + (`latestClock` over the entry's meta, and the joined meta for a `merge`) and + `Storage.#init` seeds the clock from `log.latestTs`; `applyRemote` observes + every peer op. A stamp taken any other way can sort below state the node + already has, and `applyOp` then drops the write silently (`changed` false, + `COMMIT` returns null). Do not add a second write path; go through + `[COMMIT]`. - **Reconnect/backoff/resume belong to the link.** `SyncManager` re-runs only `hello` + `push` on `reconnect`; a refused restored subscription (428 "say hello first" โ€” wrpc gives a reconnect a NEW server-side client) triggers a diff --git a/docs/guide/sync.md b/docs/guide/sync.md index b8cae77..4e71c90 100644 --- a/docs/guide/sync.md +++ b/docs/guide/sync.md @@ -45,7 +45,33 @@ in the log: `origin` is the node that made the op, `seq` its sequence number on that node, `ts` a millisecond stamp that never runs backwards and never falls behind a -stamp the node has seen from a peer. +stamp the node has seen from a peer (see [Clocks](#clocks)). + +### Clocks + +`ts` is `Date.now()`: milliseconds since the epoch, in UTC. Time zones never +enter into it โ€” two nodes in different zones stamp the same instant the same +way. Whether their clocks *agree* is a different matter, and syncom is built so +that they do not have to: + +- **Convergence does not depend on clocks.** Ops apply in `seq` order per + origin, and a register conflict is settled by the pair `(ts, origin)`, which + is a total order. Two nodes holding the same ops reach the same state + whatever their wall clocks say. +- **A node's clock never runs backwards and never falls behind what the node + has seen** โ€” a peer's op, a joined snapshot, or its own entries on disk, + including across a reopen (the log seeds the clock when it opens). So a write + that follows a read always outranks what was read, even when the peer that + wrote it has a clock an hour ahead: a local write can only lose to a write + it has not seen yet. +- **Concurrent writes go to the faster clock.** When two nodes write the same + field without seeing each other, the higher `ts` wins on both sides. Skew + changes *who* wins, never *whether* they agree. Keep clocks NTP-synced if + "last writer" should mean what a human means by it; for correctness it is + not required. +- `retention.days` compares stamps with this node's wall clock, so ops stamped + by a peer far ahead age out late and ops from a peer far behind age out + early. Retention by `ops` is unaffected. ### Registers and counters diff --git a/src/log.js b/src/log.js index ccb107e..5104c3f 100644 --- a/src/log.js +++ b/src/log.js @@ -49,6 +49,7 @@ class OpLog { #openFiles = []; #segments = []; #next = 1; + #latestTs = 0; #floor = 1; #epoch = null; #vector = {}; @@ -142,7 +143,10 @@ class OpLog { this.#entries.push(entry); added = true; if (entry.seq >= this.#next) this.#next = entry.seq + 1; - if (entry.op) vectorAdvance(this.#vector, entry.op); + if (entry.op) { + vectorAdvance(this.#vector, entry.op); + this.#observe(entry.op); + } } if (added) this.#entries.sort((a, b) => a.seq - b.seq); } @@ -159,6 +163,20 @@ class OpLog { return this.#next; } + #observe(op) { + if (typeof op.ts === 'number' && op.ts > this.#latestTs) this.#latestTs = op.ts; + } + + /** + * The newest `ts` this log has held since it was opened, over everything + * still on disk at open time and everything appended since. A reopened + * node seeds its clock from it, so its stamps never run backwards even + * when the wall clock did โ€” or when a peer's ran ahead. + */ + get latestTs() { + return this.#latestTs; + } + /** The highest per-origin seq this log has ever held โ€” survives compaction. */ get vector() { return { ...this.#vector }; @@ -197,7 +215,10 @@ class OpLog { const name = `${pad(entries[0].seq)}.json`; await fsi.write(this.#openPath, name, JSON.stringify({ entries })); for (const entry of entries) this.#entries.push(entry); - for (const op of ops) vectorAdvance(this.#vector, op); + for (const op of ops) { + vectorAdvance(this.#vector, op); + this.#observe(op); + } this.#openFiles.push(name); if (this.#openFiles.length >= this.#rollAfter) await this.#roll(); return entries; diff --git a/src/ops.js b/src/ops.js index 00e7ef5..32b92eb 100644 --- a/src/ops.js +++ b/src/ops.js @@ -77,6 +77,28 @@ const normalizeMeta = (meta) => ({ counters: isPlainObject(meta.counters) ? meta.counters : {}, }); +// The newest stamp an entry's metadata holds, over registers, the reset +// watermark, the tombstone and the counters โ€” what a node must have observed +// before it stamps a write to that record, or the write sorts below state the +// node already has and is dropped as stale. Takes raw peer-shaped meta too. +const latestClock = (meta) => { + if (!isPlainObject(meta)) return null; + const { clock, base, deleted, counters } = normalizeMeta(meta); + let latest = maxClock(base, deleted); + for (const field in clock) { + if (Array.isArray(clock[field])) latest = maxClock(latest, clock[field]); + } + for (const field in counters) { + const perOrigin = counters[field]; + if (!isPlainObject(perOrigin)) continue; + for (const origin in perOrigin) { + const ts = Array.isArray(perOrigin[origin]) ? perOrigin[origin][2] : undefined; + if (typeof ts === 'number') latest = maxClock(latest, [ts, origin]); + } + } + return latest; +}; + // Drops every register and counter entry that fell below the (new) base. const applyBase = (entry, delta) => { const { meta, data } = entry; @@ -308,4 +330,4 @@ const unpackOps = (packed) => { return ops; }; -module.exports = { createEntry, cloneEntry, applyOp, mergeEntry, isAlive, isCounter, packOps, unpackOps }; +module.exports = { createEntry, cloneEntry, applyOp, mergeEntry, isAlive, isCounter, latestClock, packOps, unpackOps }; diff --git a/src/storage.js b/src/storage.js index d32444e..f4bc2c0 100644 --- a/src/storage.js +++ b/src/storage.js @@ -5,7 +5,7 @@ const { encrypt, decrypt, loadKeys, calculateHash } = require('./crypto.js'); const { Blockchain } = require('./chain.js'); const { SyncManager } = require('./sync.js'); const { OpLog } = require('./log.js'); -const { applyOp, isAlive } = require('./ops.js'); +const { applyOp, isAlive, latestClock } = require('./ops.js'); const { Clock, opKey, keyScope, vectorHas, vectorNext, vectorAdvance } = require('./vector.js'); const fsi = require('./fsi.js'); @@ -231,6 +231,9 @@ class Storage extends Emitter { this.#keys = await loadKeys(`${this.#path}/keys`); this.#chain = await new Blockchain(`${this.#path}/blockchain`); this.#log = await new OpLog(this.#path, { ...this.#options.sync?.log, retention: this.#options.sync?.retention }); + // The clock picks up where the log left off: every stamp this node ever + // logged is below what it will stamp next, whatever the wall clock says. + this.#clock.observe(this.#log.latestTs); await this.#loadIdentity(); await this.#replayTail(); this.#sync = await new SyncManager(this, this.#options.sync); @@ -378,6 +381,10 @@ class Storage extends Emitter { return blockHash; } + #observe(stamp) { + if (stamp) this.#clock.observe(stamp[0]); + } + // --------------------------------------------------------------------- // The one local write path. Builds the op, applies it, logs it (durable // seq), persists the entry, then announces it. @@ -395,6 +402,12 @@ class Storage extends Emitter { } const data = payload.data ?? current?.data ?? {}; const scope = this.#scope(id, data); + // Stamp above everything this node already holds for the record โ€” its + // entry and, for a merge, the one being joined โ€” so a local write never + // sorts below state the node knows about and is silently dropped. The + // log seeds the clock at open; this covers what the log no longer has. + this.#observe(latestClock(current?.meta)); + if (type === 'merge') this.#observe(latestClock(payload.meta)); const op = { origin: this.#nodeId, seq: 0, ts: this.#clock.now(), type, id }; if (scope !== GLOBAL_SCOPE) op.scope = scope; if (from) op.from = from; diff --git a/tests/ops.test.js b/tests/ops.test.js index e284844..c429c92 100644 --- a/tests/ops.test.js +++ b/tests/ops.test.js @@ -2,7 +2,7 @@ const test = require('node:test'); const assert = require('node:assert'); -const { applyOp, mergeEntry, isAlive, createEntry, isCounter } = require('../src/ops.js'); +const { applyOp, mergeEntry, isAlive, createEntry, isCounter, latestClock } = require('../src/ops.js'); const op = (type, ts, origin, extra = {}) => ({ type, ts, origin, seq: extra.seq ?? ts, id: 'r', ...extra }); @@ -153,6 +153,25 @@ test('ops module', async (t) => { }); }); + await t.test('latestClock is the newest stamp an entry holds, or null', () => { + assert.strictEqual(latestClock(null), null); + assert.strictEqual(latestClock(createEntry().meta), null); + const meta = { + clock: { a: [5, 'x'], b: [9, 'y'] }, + base: [3, 'x'], + deleted: null, + counters: { hits: { x: [1, 2, 7], z: [4, 1, 9] } }, + }; + assert.deepStrictEqual(latestClock(meta), [9, 'z'], 'ties break by origin, like compareClock'); + meta.deleted = [12, 'x']; + assert.deepStrictEqual(latestClock(meta), [12, 'x']); + // Peer-shaped garbage is tolerated, not trusted. + assert.deepStrictEqual(latestClock({ clock: { a: 'nope', b: [2, 'b'] }, counters: { c: 3, d: { o: 'x' } } }), [ + 2, + 'b', + ]); + }); + await t.test('mergeEntry joins two histories like replaying both', () => { const left = [op('set', 1, 'A', { data: { x: 1, y: 1 } }), op('inc', 2, 'A', { seq: 2, key: 'n', value: 2 })]; const right = [ diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js index 3580743..9a60a0a 100644 --- a/tests/storage.ops.test.js +++ b/tests/storage.ops.test.js @@ -234,6 +234,55 @@ test('Storage replication model', async (t) => { assert.strictEqual(await second.validate(), true); }); + await t.test('a local write outranks a snapshot joined from a peer whose clock is ahead', async (t2) => { + const { storage } = await openTemp(t2); + const ahead = Date.now() + 3_600_000; + const meta = (clock, extra = {}) => ({ clock, base: null, deleted: null, counters: {}, ...extra }); + await storage.mergeSnapshot([ + { id: 'reg', data: { title: 'remote' }, meta: meta({ title: [ahead, 'peer'] }) }, + { id: 'reset', data: { n: 1 }, meta: meta({ n: [ahead, 'peer'] }, { base: [ahead, 'peer'] }) }, + { id: 'gone', data: {}, meta: meta({}, { base: [ahead, 'peer'], deleted: [ahead, 'peer'] }) }, + ]); + const ops = []; + storage.on('change', (op) => ops.push(op)); + await storage.update('reg', { title: 'local' }); + assert.deepStrictEqual(await storage.get('reg'), { title: 'local' }); + assert.ok(ops.at(-1).ts > ahead, 'stamped above the joined clock'); + await storage.delete('reset'); + assert.strictEqual(await storage.get('reset'), null, 'a delete lands above a joined reset'); + await storage.inc('gone', 'hits', 1); + assert.deepStrictEqual(await storage.get('gone'), { hits: 1 }, 'a counter lands above a joined tombstone'); + // Joining the same, now older, snapshot again is still a no-op. + const before = ops.length; + await storage.mergeSnapshot([{ id: 'reg', data: { title: 'remote' }, meta: meta({ title: [ahead, 'peer'] }) }]); + assert.strictEqual(ops.length, before); + assert.deepStrictEqual(await storage.get('reg'), { title: 'local' }); + }); + + await t.test('the clock survives a reopen: a node never stamps below what it already holds', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const ahead = Date.now() + 3_600_000; + // rollAfter 1 seals every append: the reopen has no tail to replay from. + const options = { path: dir, logger: false, sync: { log: { rollAfter: 1 } } }; + const first = await syncom.open(options); + await first.applyRemote([{ origin: 'peer', seq: 1, ts: ahead, type: 'write', id: 'r', data: { n: 1 } }]); + await first.close(); + + const second = await syncom.open(options); + assert.strictEqual(second.log.tail().length, 0, 'everything is sealed'); + assert.ok(second.log.latestTs >= ahead, 'the log remembers the newest stamp it holds'); + const ops = []; + second.on('change', (op) => ops.push(op)); + await second.update('r', { n: 2 }); + assert.deepStrictEqual(await second.get('r'), { n: 2 }, 'an update to the peer-stamped record lands'); + assert.ok(ops[0].ts > ahead); + // A write to an unrelated record is stamped above too: the node's own + // stream never runs backwards across a reopen. + await second.insert({ fresh: true }); + assert.ok(ops[1].ts > ops[0].ts); + }); + await t.test('encrypted records stay local', async (t2) => { const { storage } = await openTemp(t2); const ops = []; diff --git a/tests/suites/sync-engine.js b/tests/suites/sync-engine.js index 568e61b..4a6cbad 100644 --- a/tests/suites/sync-engine.js +++ b/tests/suites/sync-engine.js @@ -183,6 +183,10 @@ const runSyncEngineTests = async (t, assert, createStorage) => { for (let i = 0; i < 12; i++) await a.inc(id, 'hits', 1); await a.insert({ other: true }); assert.ok(a.log.floor > 1, 'history was compacted'); + // A holds state stamped by a peer whose clock runs an hour ahead; it + // reaches B only through the snapshot, never as an op B could observe. + const ahead = Date.now() + 3_600_000; + await a.applyRemote([{ origin: 'fast', seq: 1, ts: ahead, type: 'write', id: 'skewed', data: { v: 'fast' } }]); const b = mesh.get('B'); const events = []; @@ -198,6 +202,10 @@ const runSyncEngineTests = async (t, assert, createStorage) => { await settle([a, b]); assert.strictEqual((await b.get(id)).hits, 13); assert.deepStrictEqual(await a.get(fromB), { late: true }); + // B's write to the skew-stamped record wins over what it joined, everywhere. + await b.update('skewed', { v: 'B' }); + await settle([a, b]); + assert.deepStrictEqual(await a.get('skewed'), { v: 'B' }, 'a local write outranks the joined state'); // A snapshot merged into B is in B's log, so a peer of B gets it too. const c = await createStorage(t2, 'C', { sync: { connect: () => new (require('../utils/memory-link.js').MemoryLink)(b.sync.unit({ access: 'public' })) }, From 2ee511d3eec97c2f2fe59edba0fac0bbfbbe319e Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 18:47:44 +0300 Subject: [PATCH 07/67] feat: inject OpenTelemetry the way wrpc does, without carrying trace context MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `open({ telemetry })` takes the same object wrpc's option does โ€” `{ api }` or `{ tracer, meter }` โ€” and everything is duck-typed; @opentelemetry/* stays a devDependency. Six INTERNAL spans (syncom.commit, apply, replay, push, bootstrap, log.roll) and nine metrics cover what only the engine can see: the local write path, what a push came to, replication lag, log rolls, bootstraps as one unit, pushes as one batch. Nothing wrpc already emits is repeated, and syncom puts no trace context into ops: its spans open with startActiveSpan and nest under wrpc's, so a write on one node and its application on the next are one trace through the socket. Merge commits are counted, not spanned. Identity (record id, peer url/node) rides on spans behind includeIdentity and is never a metric label. The browser budget is ratcheted 14 โ†’ 17 KB for it (16.1 measured). Two bugs on the same seam: applyRemote emitted `error` unguarded, so a refused op with no listener failed the push before the vector moved and the peer offered the same page forever; OpLog never received the storage logger. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 22 + CLAUDE.md | 11 + README.md | 5 +- browser.d.ts | 7 + docs/.vitepress/config.mts | 6 + docs/guide/sync.md | 3 + docs/guide/telemetry.md | 191 +++++++++ docs/guide/wrpc.md | 9 + docs/reference/api.md | 10 +- index.d.ts | 73 ++++ package.json | 9 +- pnpm-lock.yaml | 99 +++++ scripts/size.js | 6 +- src/log.js | 26 +- src/storage.js | 129 +++++- src/sync.js | 127 ++++-- src/telemetry.js | 338 ++++++++++++++++ tests/browser.test-d.ts | 1 + tests/index.test-d.ts | 19 + tests/storage.browser.test.js | 19 + tests/storage.ops.test.js | 31 ++ tests/telemetry.test.js | 720 +++++++++++++++++++++++++++++++++ tests/wrpc.integration.test.js | 84 ++++ 23 files changed, 1892 insertions(+), 53 deletions(-) create mode 100644 docs/guide/telemetry.md create mode 100644 src/telemetry.js create mode 100644 tests/telemetry.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index f787b96..a8e5e44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,20 @@ section until the first publish; everything below lands there on release day. ### Added +- **OpenTelemetry, injected.** `open({ telemetry: { api } })` (the + `@opentelemetry/api` module) or `{ tracer, meter }` โ€” the same object wrpc's + `telemetry` option takes, and no dependency on either. Spans: + `syncom.commit`, `syncom.apply`, `syncom.replay`, `syncom.push`, + `syncom.bootstrap`, `syncom.log.roll`. Metrics: `syncom.commits`, + `syncom.write.duration`, `syncom.ops.received`, `syncom.ops.rejected`, + `syncom.replication.lag`, `syncom.push.ops`, `syncom.bootstraps`, + `syncom.log.segments`, `syncom.log.append.duration`. syncom carries no + trace context of its own: its spans nest under whatever is active, so with + wrpc telemetry on both ends a write on one node and its application on the + next are one trace. `includeIdentity: false` keeps record ids and peer + urls off spans; they are never metric labels. `storage.telemetry` exposes + the writer; `OpLog` takes the option directly. Browser budget raised + 14 โ†’ 17 KB for it. - **Replication.** Every write is now an operation โ€” `write` (`insert`/`set`), `set` (`update`/`swap`/`record.save`), `inc`, `delete` โ€” appended to a persisted log (`src/log.js`) and applied through a record-level model @@ -58,6 +72,14 @@ section until the first publish; everything below lands there on release day. ### Fixed +- A peer's op refused by the register/counter rule no longer fails the whole + push when nobody listens for `error`: the emit is guarded, the vector + advances, and the peer stops offering the same page on every hello. +- `OpLog` now receives the storage `logger`, so an unreadable segment or open + file is reported rather than silently skipped. + +### Fixed + - `Storage.saveData()` did not await `encrypt`/`decrypt`, so encryption at rest in the browser stored a serialized promise (`{}`) instead of ciphertext. - `apply()` ignored the data of a `write` record and only understood `delete` diff --git a/CLAUDE.md b/CLAUDE.md index 2872015..cf13a7e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,6 +27,7 @@ src/log.js OpLog โ€” the persisted operation log over fsi src/sync.js SyncManager (dialing side) + unit() (hosting side) src/link.js isSyncLink โ€” the link contract check src/wrpc.js createWrpcLink โ€” a WrpcClient as a SyncLink (no wrpc import) +src/telemetry.js the injected-OpenTelemetry writer (duck-typed; @opentelemetry/* only devDeps) src/ ships as-is โ€” no build step, CommonJS, 'use strict' tests/ *.test.js are tests; anything else is a helper bench/ one file per subject + support/harness.js @@ -96,6 +97,16 @@ docs/ VitePress site fields is not a plain patch. Do not "correct" it, and do not write docs or examples that assume replacement. Storage does NOT replicate through `merge`; the op model in `src/ops.js` is what converges. +- **Telemetry is injected, never required.** `createTelemetry` in + `src/telemetry.js` duck-types whatever `open({ telemetry })` was given and + returns a writer; `DISABLED` is a frozen null-object whose `withSpan` still + runs its callback, so call sites bracket work the same way on or off (but + guard with `otel.enabled` before allocating attributes or reading the + clock). syncom never puts trace context into ops โ€” `tp`/`ts` are wrpc's, + ops are durable โ€” and its spans nest under the active context instead. + Never mint a metric label from a record id, an origin, a peer url or a + scope. Every `catch {}` in the writer has a hostile-implementation test in + `tests/telemetry.test.js`; keep it that way or the branch threshold drops. - **Encrypted records are local.** `{ encrypted: true }` seals a record for this node's key; it produces no op and is excluded from snapshots. Do not route an encrypted entry through `applyOp`. diff --git a/README.md b/README.md index c957270..64431e7 100644 --- a/README.md +++ b/README.md @@ -76,8 +76,8 @@ that fails the build when exceeded. | Entry | raw | min | min+gzip | budget | | ----- | ---:| ---:| --------:| ------:| -| main entry โ€” browser (@alexify/syncom) | 96.1 KB | 39.4 KB | 13.7 KB | 14.0 KB | -| main entry โ€” node (@alexify/syncom) | 89.0 KB | 38.5 KB | 13.4 KB | โ€” | +| main entry โ€” browser (@alexify/syncom) | 114.6 KB | 46.6 KB | 16.1 KB | 17.0 KB | +| main entry โ€” node (@alexify/syncom) | 106.6 KB | 45.6 KB | 15.8 KB | โ€” | | wrpc adapter โ€” browser (@alexify/syncom/wrpc) | 3.3 KB | 1.4 KB | 0.7 KB | 2.0 KB | ## Quick Start @@ -144,6 +144,7 @@ nodes converges. | **CRDT** | `diff` / `merge` / `apply` over plain objects โ€” nested maps merge recursively, arrays are grow-only sets, numbers add so counters converge | | **Contracts** | Store a procedure in the chain by hash, load it anywhere, run it against a read-only view of storage; failures are recorded as blocks | | **Portability** | One API over `node:fs` + `node:crypto` on the server and OPFS + Web Crypto in the browser, chosen by the `browser` export condition | +| **Observability** | Injected OpenTelemetry (`open({ telemetry })`): spans for local writes, applied batches, pushes, bootstraps and log rolls, plus outcome and lag metrics โ€” nested under wrpc's spans, which carry the trace across the wire. See [OpenTelemetry](https://syncomjs.vercel.app/guide/telemetry) | | **Dependencies** | None. wrpc is a devDependency for one integration test and is injected at runtime | ## Browser usage diff --git a/browser.d.ts b/browser.d.ts index 51a08b8..c48b8ab 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -36,6 +36,13 @@ export type { NodeData, SyncManagerConstructor, SyncOptions, + SyncomTelemetryOptions, + SyncomTelemetryApi, + SyncomTracer, + SyncomSpan, + SyncomMeter, + SyncomCounter, + SyncomHistogram, SyncLink, StorageConstructor, ClockStamp, diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index d82e543..d81edb2 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -38,6 +38,8 @@ const keywords = [ 'javascript', 'nodejs', 'browser', + 'opentelemetry', + 'observability', ].join(', '); // schema.org structured data โ€” helps search and AI engines understand the @@ -156,6 +158,10 @@ export default withMermaid({ text: 'Platforms', items: [{ text: 'Browser', link: '/guide/browser' }], }, + { + text: 'Operations', + items: [{ text: 'OpenTelemetry', link: '/guide/telemetry' }], + }, ], '/reference/': [ { diff --git a/docs/guide/sync.md b/docs/guide/sync.md index 4e71c90..ad239d7 100644 --- a/docs/guide/sync.md +++ b/docs/guide/sync.md @@ -270,6 +270,9 @@ storage.on('error', (error) => {}); // a peer's op refused by the register/c | `log.rollAfter` | `64` | Appended batches per sealed segment | | `pageSize` | `500` | Ops per push, records per snapshot page | +The `telemetry` option sits beside `sync` on `open()`, not inside it โ€” see +[OpenTelemetry](./telemetry) for the spans and metrics replication emits. + ## Hosting The unit a wrpc server (or anything shaped like one) mounts: diff --git a/docs/guide/telemetry.md b/docs/guide/telemetry.md new file mode 100644 index 0000000..4495e57 --- /dev/null +++ b/docs/guide/telemetry.md @@ -0,0 +1,191 @@ +# OpenTelemetry + +syncom emits OpenTelemetry spans and metrics **without depending on +OpenTelemetry**: you inject the API (or a tracer and a meter), and everything +is duck-typed. With nothing injected, every recording path is a no-op and the +write path does not even read the clock. + +The option has the same shape as wrpc's, so one object serves both packages โ€” +and that is the point of the design, see +[What wrpc already gives you](#what-wrpc-already-gives-you). + +## Two injection modes + +```js +const otelApi = require('@opentelemetry/api'); +const { open } = require('@alexify/syncom'); + +// The api module: syncom derives its own tracer and meter, so spans carry +// the `@alexify/syncom` instrumentation scope. +const storage = await open({ path: 'data', telemetry: { api: otelApi } }); + +// Or pre-built instances โ€” either alone is fine. +const traced = await open({ path: 'data', telemetry: { tracer, meter } }); +``` + +`includeIdentity: false` keeps record ids and peer urls/node ids off spans. +They are never metric labels at any setting. + +The writer is `storage.telemetry` (opaque, `@experimental`), and `OpLog` +takes the same option directly. The signals will keep improving in minor +releases; the option and the injection modes are stable. + +## What wrpc already gives you + +When syncom replicates over wrpc and the host passes `telemetry` to its +`Server` and `WrpcClient`, wrpc emits โ€” with no help from syncom: + +| Signal | Where it comes from | +| --- | --- | +| A span and a duration sample for every `syncom/hello`, `push`, `snapshot`, `changes` call, on both ends | wrpc's `rpc.server.duration`, `rpc.client.duration`, `wrpc.server.calls` | +| W3C trace context across the socket (`tp`/`ts` on each packet) | wrpc's propagation | +| Connections, reconnects, subscriptions, stream bytes, backpressure | wrpc's server and client instruments | + +syncom does not repeat any of it. What it adds is what only the engine can +see: + +| Signal | Why wrpc cannot see it | +| --- | --- | +| The local write path โ€” commit, log append, chain block, record file | never crosses a socket | +| What a push came to: applied, duplicate, gap, malformed, refused | wrpc sees a call that returned 200 | +| Replication lag, origin stamp to applied here | needs the op | +| The log rolling and compacting, the tail replayed at open | local I/O | +| A bootstrap as one unit, with its pages and records | many calls, one operation | +| A push as one batch, however many pages, and why it happened | the coalescing is syncom's | +| Any link that is not wrpc | there is no wrpc | + +## Spans + +All syncom spans are `INTERNAL`; the `CLIENT` and `SERVER` kinds belong to +wrpc's spans around the actual calls. + +| Span | When | Attributes | +| --- | --- | --- | +| `syncom.commit` | one local write, from the moment it runs to after the `change` listeners | `syncom.op.type`, `syncom.record.id`โ€ , `syncom.result` (`applied` / `noop` / `encrypted`), `syncom.scope` | +| `syncom.apply` | one batch of a peer's ops | `syncom.ops.count`, `syncom.ops.applied`, `syncom.ops.duplicates`, `syncom.ops.gaps` | +| `syncom.replay` | the unsealed log tail replayed at open | `syncom.ops.count`, `syncom.records` | +| `syncom.push` | one push to a peer, all its pages | `syncom.push.reason` (`handshake` / `manual` / `relay`), `syncom.ops.count`, `syncom.pages`, `syncom.peer.url`โ€ , `syncom.peer.node`โ€  | +| `syncom.bootstrap` | the whole snapshot walk from a peer | `syncom.peer.url`โ€ , `syncom.peer.node`โ€ , `syncom.pages`, `syncom.records` | +| `syncom.log.roll` | sealing open files into a segment and compacting | `syncom.log.sealed`, `syncom.log.dropped`, `syncom.log.floor` | + +โ€  off the span when `includeIdentity` is false. + +A span is marked failed (status `ERROR`, the exception recorded, `error.type` +set) when the work it brackets throws: a write refused by the +register/counter rule, a push whose call rejected, a bootstrap that could not +read a snapshot. A peer's op refused inside `applyRemote` is a protocol +outcome, not a failure of the batch: it is counted (below), the batch's span +stays green. + +A `merge` โ€” one commit per snapshot record during a bootstrap โ€” is counted +but not spanned, or a bootstrap of ten thousand records would open ten +thousand spans under one `syncom.bootstrap`. + +## Metrics + +Labels are closed sets. A record id, an origin, a peer url or a scope never +appears on one. + +| Metric | Instrument | Unit | Labels | +| --- | --- | --- | --- | +| `syncom.commits` | Counter | `{op}` | `syncom.op.type`, `syncom.result` (`applied` / `noop` / `encrypted` / `error`) | +| `syncom.write.duration` | Histogram | `ms` | `syncom.path` (`commit` / `apply` / `replay`) | +| `syncom.ops.received` | Counter | `{op}` | `syncom.outcome` (`applied` / `duplicate` / `gap` / `malformed`) | +| `syncom.ops.rejected` | Counter | `{op}` | `syncom.reason` (`register` / `counter`) | +| `syncom.replication.lag` | Histogram | `ms` | โ€” | +| `syncom.push.ops` | Histogram | `{op}` | `syncom.push.reason`, `syncom.result` (`ok` / `error`) | +| `syncom.bootstraps` | Counter | `{bootstrap}` | `syncom.result` | +| `syncom.log.segments` | Counter | `{segment}` | `syncom.log.event` (`sealed` / `dropped`) | +| `syncom.log.append.duration` | Histogram | `ms` | โ€” | + +`syncom.push.ops` answers three questions with one instrument: its count is +the number of pushes, its sum the ops pushed, and its distribution how many +ops coalesce into one push. + +`syncom.replication.lag` is the receiving node's clock minus the op's stamp, +which is the origin's wall clock at commit time โ€” so it includes clock skew +between the two nodes along with transit and queueing. Read it as a trend, +not as an absolute. + +Two storages sharing one meter create the same instruments twice; the SDK +returns the same instrument for an identical descriptor, so this is fine. + +## One trace across nodes + +syncom carries **no trace context** in its ops โ€” ops are durable, traces are +not, and one push carries ops of many origins. Instead, every syncom span is +opened with `startActiveSpan`, so it nests under whatever the host's context +manager says is active: + +- On the host, wrpc runs the `syncom/push` handler inside its `SERVER` span, + so `syncom.apply` becomes its child. +- On the dialing node, `syncom.push` wraps the `syncom/push` call, so wrpc's + `CLIENT` span becomes *its* child โ€” and wrpc injects that context into the + packet. +- The relay of an op that just landed is scheduled from inside the commit (or + apply) that produced it, and inherits that context. + +With telemetry on wrpc and syncom at both ends, a write and everything it +caused is one trace: + +``` +syncom.commit (node A) +โ””โ”€ syncom.push {relay} (node A) + โ””โ”€ syncom/push CLIENT, wrpc (node A) + โ””โ”€ syncom/push SERVER, wrpc (node B) + โ””โ”€ syncom.apply (node B) + โ””โ”€ syncom.push {relay} (node B, on to C) + โ””โ”€ โ€ฆ +``` + +```js +const otelApi = require('@opentelemetry/api'); +const { Server, WrpcClient, defineRouter } = require('@alexify/wrpc'); +const { open } = require('@alexify/syncom'); +const { createWrpcLink } = require('@alexify/syncom/wrpc'); + +const telemetry = { api: otelApi }; + +// the host +const hub = await open({ path: 'hub', telemetry }); +const router = defineRouter({ syncom: hub.sync.unit({ access: 'public' }) }); +await new Server({ router, host: '0.0.0.0', port: 8000, protocol: 'http', telemetry }).listen(); + +// a node +const node = await open({ + path: 'node', + telemetry, + sync: { connect: async (peer) => createWrpcLink(await WrpcClient.connect(peer.url, { telemetry })) }, +}); +``` + +Two things to know when reading such a trace: + +- Ops that land in one tick travel as one push, and that push is parented to + the **first** commit or apply that scheduled it; the others in the batch + ride along. +- A push outlives the commit that scheduled it โ€” the commit ends when its + caller's `await` resolves, the push runs on a later tick. A child ending + after its parent is legal in OpenTelemetry, and viewers draw it correctly. + +In a browser without a context manager (`ZoneContextManager` or +`StackContextManager` from the OTel web SDK), `startActiveSpan` still opens a +span but nothing sees it as active: every syncom span is a root, as wrpc's +client spans are there. + +## Timing + +`syncom.commit` and `syncom.apply` start when the write **runs**, not when it +was queued: `Storage` serialises writes, and time spent waiting on an earlier +write is the caller's contention, not this write's cost. They end after the +`change` listeners have run, because that is what the caller awaits โ€” a slow +listener shows up in `syncom.write.duration`, and it should. + +## Failures + +Telemetry never breaks a write. A tracer that throws, a tracer that does not +call back, a span whose `end` throws, a meter whose factories throw, an +instrument that throws on use โ€” each is contained, and the write goes through +exactly once. The one thing that is *not* swallowed is an error thrown by the +work a span brackets: that is the write's own failure and it propagates as it +always did. The hostile-implementation cases are in `tests/telemetry.test.js`. diff --git a/docs/guide/wrpc.md b/docs/guide/wrpc.md index 6ee7932..672c2d4 100644 --- a/docs/guide/wrpc.md +++ b/docs/guide/wrpc.md @@ -103,6 +103,15 @@ wrpc's `WrpcReadable`. `syncom`. `isSyncLink()` is exported too, and `SYNC_INTROSPECTION` if you want to scaffold the unit yourself. +## Tracing across the wire + +The adapter adds nothing for tracing, and needs nothing: give wrpc's `Server` +and `WrpcClient` their `telemetry` option and syncom its own, and a write on +the dialing node, its `syncom/push` call, the handler on the host and the +`syncom.apply` inside it are one trace โ€” wrpc carries the context per packet, +syncom's spans nest under whatever is active. See +[OpenTelemetry](./telemetry). + ## Rooms and presence On `hello` the host joins the peer to `syncom:` for every scope it asked diff --git a/docs/reference/api.md b/docs/reference/api.md index cd6a89b..e2d412d 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -14,7 +14,8 @@ const syncom = require('@alexify/syncom'); Returns `Promise`. Options: `path` (default `'syncom'`), `name` (alias for `path`), `schema`, `getId`, `logger`, `encrypted`, `sync` (see -[Sync options](../guide/sync#options)). +[Sync options](../guide/sync#options)), `telemetry` (see +[OpenTelemetry](../guide/telemetry)). ### `Storage` @@ -44,7 +45,8 @@ Returns `Promise`. Options: `path` (default `'syncom'`), `name` | `adoptVector(vector, { via, from }?)` | `Promise` | Records what a joined snapshot covers (an `advance` op) | | `close()` | `Promise` | Stops sync and seals the log; further writes are refused | -Properties: `schema`, `sync`, `nodeId`, `path`, `log`, `vector`, `logger`. +Properties: `schema`, `sync`, `nodeId`, `path`, `log`, `vector`, `logger`, +`telemetry` (the writer built from the option โ€” opaque). Statics: `Storage.Collection`, `Storage.Record`, `Storage.generateId()` โ€” the default `getId`, synchronous in both halves. @@ -96,8 +98,8 @@ Structural check for the [link contract](../guide/sync#the-link-contract). ### `OpLog` -`await new OpLog(basePath, { rollAfter, retention, logger })` โ€” `storage.log` -is the one you want. +`await new OpLog(basePath, { rollAfter, retention, logger, telemetry })` โ€” +`storage.log` is the one you want. | Member | Notes | | --- | --- | diff --git a/index.d.ts b/index.d.ts index 3378927..c0de18b 100644 --- a/index.d.ts +++ b/index.d.ts @@ -87,6 +87,7 @@ export interface OpLogOptions { rollAfter?: number; retention?: { ops?: number; days?: number }; logger?: Logger | false; + telemetry?: SyncomTelemetryOptions; } export interface OpLog { @@ -120,6 +121,70 @@ export interface Logger { error?(...args: unknown[]): void; } +// --------------------------------------------------------------------------- +// OpenTelemetry +// +// Structural views of the OTel objects, so a real SDK is assignable without +// syncom importing (or depending on) @opentelemetry/api. Only `Span.end` is +// required โ€” everything a span might not implement is optional, and syncom +// calls it optionally. The shapes match wrpc's, so one object serves both. + +export interface SyncomSpan { + setAttribute?(key: string, value: unknown): unknown; + addEvent?(name: string, attributes?: Record): unknown; + recordException?(error: unknown): void; + setStatus?(status: { code: number; message?: string }): unknown; + end(): void; +} + +export interface SyncomTracer { + startSpan?(name: string, options?: unknown): SyncomSpan; + startActiveSpan?(name: string, options: unknown, fn: (span: SyncomSpan) => T): T; +} + +export interface SyncomCounter { + add(value: number, attributes?: Record): void; +} + +export interface SyncomHistogram { + record(value: number, attributes?: Record): void; +} + +export interface SyncomMeter { + createCounter(name: string, options?: unknown): SyncomCounter; + createHistogram(name: string, options?: unknown): SyncomHistogram; +} + +export interface SyncomTelemetryApi { + trace?: { getTracer(name: string, version?: string): SyncomTracer }; + metrics?: { getMeter(name: string, version?: string): SyncomMeter }; +} + +/** + * Two injection modes. `{ api }` โ€” the `@opentelemetry/api` module, from + * which syncom derives its own tracer and meter so spans carry the + * `@alexify/syncom` instrumentation scope. `{ tracer, meter }` โ€” pre-built + * instances; either alone is a supported configuration. + * + * With neither, telemetry is off and every recording path is a no-op. + * syncom carries no trace context of its own: its spans nest under whatever + * is active โ€” wrpc's server span inside a `syncom/push` handler โ€” through + * the context manager the host installed. + * + * @experimental The writer shape, span names and metric set may change in a + * minor release. + */ +export interface SyncomTelemetryOptions { + api?: SyncomTelemetryApi; + tracer?: SyncomTracer; + meter?: SyncomMeter; + /** + * Default true. When false, record ids and peer urls/node ids are left off + * spans. They are never metric labels at any setting. + */ + includeIdentity?: boolean; +} + export type CrdtDelta = Record; export type CrdtHistoryRecord = @@ -237,6 +302,8 @@ export interface StorageOptions { sync?: SyncOptions; /** Encrypt every record at rest by default (records then stay local). */ encrypted?: boolean; + /** Injected OpenTelemetry โ€” see SyncomTelemetryOptions. */ + telemetry?: SyncomTelemetryOptions; } // The collections a schema turns into properties on the storage object: @@ -336,6 +403,12 @@ export interface Storage { readonly path: string; readonly log: OpLog; readonly vector: VersionVector; + /** + * The telemetry writer built from `options.telemetry`. Opaque on purpose: + * its shape is an implementation detail. + * @experimental + */ + readonly telemetry: unknown; /** Every op applied here โ€” local (`via` null) or from a peer (`via` = the link). */ on(event: 'change', listener: (op: SyncOp, via: unknown) => void): void; diff --git a/package.json b/package.json index 13afb32..5a7734a 100644 --- a/package.json +++ b/package.json @@ -74,7 +74,9 @@ "zero-dependency", "javascript", "nodejs", - "browser" + "browser", + "opentelemetry", + "observability" ], "author": "Alex Dolid ", "license": "MIT", @@ -84,6 +86,11 @@ "homepage": "https://syncomjs.vercel.app/", "devDependencies": { "@alexify/wrpc": "1.0.0", + "@opentelemetry/api": "^1.9.1", + "@opentelemetry/context-async-hooks": "^2.11.0", + "@opentelemetry/core": "^2.11.0", + "@opentelemetry/sdk-metrics": "^2.11.0", + "@opentelemetry/sdk-trace-base": "^2.11.0", "@types/node": "^22.10.0", "@vercel/analytics": "^2.0.1", "@vercel/speed-insights": "^2.0.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 813f7ea..36ed0f3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -11,6 +11,21 @@ importers: '@alexify/wrpc': specifier: 1.0.0 version: 1.0.0 + '@opentelemetry/api': + specifier: ^1.9.1 + version: 1.9.1 + '@opentelemetry/context-async-hooks': + specifier: ^2.11.0 + version: 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/core': + specifier: ^2.11.0 + version: 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-metrics': + specifier: ^2.11.0 + version: 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace-base': + specifier: ^2.11.0 + version: 2.11.0(@opentelemetry/api@1.9.1) '@types/node': specifier: ^22.10.0 version: 22.20.1 @@ -544,6 +559,50 @@ packages: resolution: {integrity: sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==} engines: {node: '>= 8'} + '@opentelemetry/api@1.9.1': + resolution: {integrity: sha512-gLyJlPHPZYdAk1JENA9LeHejZe1Ti77/pTeFm/nMXmQH/HFZlcS/O2XJB+L8fkbrNSqhdtlvjBVjxwUYanNH5Q==} + engines: {node: '>=8.0.0'} + + '@opentelemetry/context-async-hooks@2.11.0': + resolution: {integrity: sha512-Tr79DyWI8itsBdg+jH+opjfrwLzX+erk1/ExkIwhWoAVjVrJIn2y5+cGjTC0Vy8fyNIA/y8wuJPZwr1T3xCZeQ==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.0.0 <1.10.0' + + '@opentelemetry/core@2.11.0': + resolution: {integrity: sha512-7YP44XH0tV6+Mb54x2YGf84i7yi+31MBZlE8JwvozkxyTvXbSp10X7cI7YE49ChJ3shMJoBmCJF3+1QFBJctGA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.0.0 <1.10.0' + + '@opentelemetry/resources@2.11.0': + resolution: {integrity: sha512-Ie7+8q8MDF4FAEQCKVMTx3ReUvxiIAgIiiW3c9JdmP8+HMcDy20puT+AHjexnExgnbvBxjQ9fjkFDWrikJ2jQA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/sdk-metrics@2.11.0': + resolution: {integrity: sha512-7GXXcObyHyDUUSG+L+kJoquty01bzm7ivE7+SSgXXJcHuPzGviptxwARmI2c+bnnxjexGQbJnyNlN8HxBP/Y7A==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.9.0 <1.10.0' + + '@opentelemetry/sdk-trace-base@2.11.0': + resolution: {integrity: sha512-H19x/TX/LZdqiYOjM7fqtSxwlplC5pgelavqbQdHbhdq0q/AI/TGkM2dfGuuynTXmJPeF2HoZVoPDu+TGoW78A==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/sdk-trace@2.11.0': + resolution: {integrity: sha512-fFnTqGm8/G73GQVnxYi7LXa1ZVYEUvgL6XI1LpvV0bPC7WQ/ZGgKxCSl8FnlZBKto9JHHEFTO6s6CUpvvtwFrA==} + engines: {node: ^18.19.0 || >=20.6.0} + peerDependencies: + '@opentelemetry/api': '>=1.3.0 <1.10.0' + + '@opentelemetry/semantic-conventions@1.43.0': + resolution: {integrity: sha512-eSYWTm620tTk45EKSedaUL8MFYI8hW164hIXsgIHyxu3VobUB3fFCu5t0hQby6OoWRPsG1KkKUG2M5UadiLiVg==} + engines: {node: '>=14'} + '@oxfmt/binding-android-arm-eabi@0.62.0': resolution: {integrity: sha512-pdsv0C4gPjJ8H1+sd8u0BDx+yLACTL+rgeMIOL1ln4ihSnhw8CWXtYWgvcSkyTfgGBIzFKab+d8rx9Xl4en/Kw==} engines: {node: ^20.19.0 || >=22.12.0} @@ -2972,6 +3031,46 @@ snapshots: '@nodelib/fs.scandir': 2.1.5 fastq: 1.20.3 + '@opentelemetry/api@1.9.1': {} + + '@opentelemetry/context-async-hooks@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + + '@opentelemetry/core@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/resources@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-metrics@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.11.0(@opentelemetry/api@1.9.1) + + '@opentelemetry/sdk-trace-base@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/sdk-trace': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/sdk-trace@2.11.0(@opentelemetry/api@1.9.1)': + dependencies: + '@opentelemetry/api': 1.9.1 + '@opentelemetry/core': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/resources': 2.11.0(@opentelemetry/api@1.9.1) + '@opentelemetry/semantic-conventions': 1.43.0 + + '@opentelemetry/semantic-conventions@1.43.0': {} + '@oxfmt/binding-android-arm-eabi@0.62.0': optional: true diff --git a/scripts/size.js b/scripts/size.js index aeed592..8c590e7 100644 --- a/scripts/size.js +++ b/scripts/size.js @@ -30,8 +30,10 @@ const ENTRIES = [ // A ratchet, not a target: raise it deliberately, in the same change that // earns the bytes. 5 KB was the migrated storage/chain/crdt core; the // replication engine (the op model, the persisted log, the sync protocol - // that makes `storage.sync` move data) earned the raise to 14 KB. - { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 14 }, + // that makes `storage.sync` move data) earned the raise to 14 KB; the + // injected OpenTelemetry writer (src/telemetry.js and its call sites in + // storage, log and sync) the raise to 17 KB. + { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 17 }, { label: 'main entry โ€” node (@alexify/syncom)', entry: 'index.js', platform: 'node' }, // Platform-neutral (no node builtins, no browser-only globals): one file // serves both, bundled as a browser entry so the self-contained check diff --git a/src/log.js b/src/log.js index 5104c3f..bb6d50d 100644 --- a/src/log.js +++ b/src/log.js @@ -25,6 +25,7 @@ const fsi = require('./fsi.js'); const { formatEventId, parseEventId, vectorAdvance } = require('./vector.js'); +const { createTelemetry, traced, now } = require('./telemetry.js'); const OPEN = 'open'; const INDEX = 'index.json'; @@ -56,6 +57,7 @@ class OpLog { #rollAfter; #retention; #log; + #otel; #writes = Promise.resolve(); constructor(basePath, options = {}) { @@ -64,6 +66,7 @@ class OpLog { this.#rollAfter = options.rollAfter ?? DEFAULT_ROLL_AFTER; this.#retention = { ops: DEFAULT_RETENTION_OPS, ...options.retention }; this.#log = options.logger ?? null; + this.#otel = createTelemetry(options.telemetry); return this.#init(); } @@ -213,7 +216,9 @@ class OpLog { const entries = []; for (const op of ops) entries.push({ seq: this.#next++, op }); const name = `${pad(entries[0].seq)}.json`; + const started = this.#otel.enabled ? now() : 0; await fsi.write(this.#openPath, name, JSON.stringify({ entries })); + if (started) this.#otel.recordAppend(now() - started); for (const entry of entries) this.#entries.push(entry); for (const op of ops) { vectorAdvance(this.#vector, op); @@ -232,22 +237,39 @@ class OpLog { const sealedFrom = this.#segments.length > 0 ? this.#segments[this.#segments.length - 1].to + 1 : this.#floor; const entries = this.#entries.filter((entry) => entry.seq >= sealedFrom); if (entries.length === 0) return; + const otel = this.#otel; + if (!otel.enabled) return void (await this.#seal(entries)); + // Spanned on its own: a roll is why one append (and so one commit) is + // slow every `rollAfter` writes, which a duration histogram alone hides. + const after = ({ dropped }) => ({ + 'syncom.log.sealed': entries.length, + 'syncom.log.dropped': dropped, + 'syncom.log.floor': this.#floor, + }); + const { dropped } = await traced(otel, 'syncom.log.roll', {}, () => this.#seal(entries), after); + otel.recordLogRoll(entries.length, dropped); + } + + async #seal(entries) { const from = entries[0].seq; const name = segmentName(from); await fsi.write(this.#path, name, JSON.stringify({ entries })); this.#segments.push({ name, from, to: entries[entries.length - 1].seq, count: entries.length }); - await this.#compact(); + const dropped = await this.#compact(); await this.#writeIndex(); const files = this.#openFiles; this.#openFiles = []; for (const file of files) await fsi.remove(this.#openPath, file); + return { dropped }; } // Drops whole sealed segments from the front while the log exceeds its // retention; the floor moves up and `vector` keeps what was dropped. + // Returns how many segments went. async #compact() { const { ops, days } = this.#retention; const cutoff = days ? Date.now() - days * 86_400_000 : null; + let dropped = 0; while (this.#segments.length > 1) { const oldest = this.#segments[0]; const overOps = ops ? this.#entries.length - oldest.count >= ops : false; @@ -259,7 +281,9 @@ class OpLog { this.#entries = this.#entries.filter((entry) => entry.seq > oldest.to); this.#floor = oldest.to + 1; await fsi.remove(this.#path, oldest.name); + dropped++; } + return dropped; } async #writeIndex() { diff --git a/src/storage.js b/src/storage.js index f4bc2c0..238ae11 100644 --- a/src/storage.js +++ b/src/storage.js @@ -8,6 +8,7 @@ const { OpLog } = require('./log.js'); const { applyOp, isAlive, latestClock } = require('./ops.js'); const { Clock, opKey, keyScope, vectorHas, vectorNext, vectorAdvance } = require('./vector.js'); const fsi = require('./fsi.js'); +const { createTelemetry, traced, now } = require('./telemetry.js'); const NODE_FILE = 'node.json'; const GLOBAL_SCOPE = '*'; @@ -20,6 +21,10 @@ const REFRESH = Symbol('syncom.refresh'); const isPlainObject = (value) => typeof value === 'object' && value !== null && !Array.isArray(value); +// What a commit came to: an op, `null` for a write that changed nothing, and +// `undefined` for an encrypted record (saved locally, no op). +const commitResult = (op) => (op === undefined ? 'encrypted' : op === null ? 'noop' : 'applied'); + class Reference { #id; #storage = null; @@ -193,6 +198,7 @@ class Storage extends Emitter { #schema = null; #getId = null; #logger; + #otel; #options; #nodeId = null; #clock = new Clock(); @@ -215,6 +221,7 @@ class Storage extends Emitter { this.#path = options.path || options.name || 'syncom'; this.#records = new Map(); this.#logger = createLogger(options.logger); + this.#otel = createTelemetry(options.telemetry); if (options.schema) this.#schema = options.schema; // A non-function is ignored rather than rejected: the id generator is a // convenience seam, not a security boundary, and the default still holds. @@ -230,7 +237,12 @@ class Storage extends Emitter { async #init() { this.#keys = await loadKeys(`${this.#path}/keys`); this.#chain = await new Blockchain(`${this.#path}/blockchain`); - this.#log = await new OpLog(this.#path, { ...this.#options.sync?.log, retention: this.#options.sync?.retention }); + this.#log = await new OpLog(this.#path, { + ...this.#options.sync?.log, + retention: this.#options.sync?.retention, + logger: this.#logger, + telemetry: this.#otel, + }); // The clock picks up where the log left off: every stamp this node ever // logged is below what it will stamp next, whatever the wall clock says. this.#clock.observe(this.#log.latestTs); @@ -260,6 +272,25 @@ class Storage extends Emitter { async #replayTail() { const tail = this.#log.tail(); if (tail.length === 0) return; + const otel = this.#otel; + if (!otel.enabled) return void (await this.#replay(tail)); + const started = now(); + try { + const attributes = { 'syncom.ops.count': tail.length }; + await traced( + otel, + 'syncom.replay', + attributes, + () => this.#replay(tail), + (records) => ({ 'syncom.records': records }), + ); + } finally { + otel.recordReplay(now() - started); + } + } + + // Returns how many records the tail touched. + async #replay(tail) { const touched = new Map(); for (const { seq, op } of tail) { this.#clock.observe(op.ts); @@ -276,8 +307,8 @@ class Storage extends Emitter { const { entry, changed } = applyOp(current, op); if (changed) touched.set(op.id, { ...entry, seq }); } - if (touched.size === 0) return; - await this.#persistBatch(touched, tail[0].seq, tail[tail.length - 1].seq); + if (touched.size > 0) await this.#persistBatch(touched, tail[0].seq, tail[tail.length - 1].seq); + return touched.size; } #initCollections() { @@ -309,6 +340,14 @@ class Storage extends Emitter { return this.#logger; } + /** + * The telemetry writer built from `options.telemetry` โ€” what SyncManager + * and OpLog record through. Opaque: its shape is an implementation detail. + */ + get telemetry() { + return this.#otel; + } + get sync() { return this.#sync; } @@ -389,8 +428,40 @@ class Storage extends Emitter { // The one local write path. Builds the op, applies it, logs it (durable // seq), persists the entry, then announces it. - [COMMIT](type, id, payload, { source = null, via = null, from = null } = {}) { - return this.#run(async () => { + [COMMIT](type, id, payload, options = {}) { + return this.#run(() => { + const otel = this.#otel; + if (!otel.enabled) return this.#commit(type, id, payload, options); + return this.#tracedCommit(otel, type, id, payload, options); + }); + } + + // The span starts when the write RUNS, not when it was queued: waiting on + // `#run` is the caller's contention, not the write's cost. It ends after + // the `change` listeners, because that is what the caller awaits. A merge + // โ€” one commit per snapshot record during a bootstrap โ€” is counted but not + // spanned, or a bootstrap would open thousands of spans. + async #tracedCommit(otel, type, id, payload, options) { + const started = now(); + let result = 'error'; + try { + const task = () => this.#commit(type, id, payload, options); + const op = + type === 'merge' + ? await task() + : await traced(otel, 'syncom.commit', { 'syncom.op.type': type, 'syncom.record.id': id }, task, (op) => ({ + 'syncom.result': commitResult(op), + 'syncom.scope': op ? (op.scope ?? GLOBAL_SCOPE) : undefined, + })); + result = commitResult(op); + return op; + } finally { + otel.recordCommit(type, result, now() - started); + } + } + + async #commit(type, id, payload, { source = null, via = null, from = null } = {}) { + { const current = await this.#load(id); if (current?.encrypted) { // An encrypted record has no op log: writes to it stay local. Set @@ -439,7 +510,7 @@ class Storage extends Emitter { // relay does not hand its own state straight back to it. await this.emit('change', op, via); return op; - }); + } } // --------------------------------------------------------------------- @@ -592,8 +663,38 @@ class Storage extends Emitter { * offered again; `via` names the link they arrived on, so the relay does * not echo them straight back. */ - applyRemote(ops, { via = null, scopes = null } = {}) { - return this.#run(async () => { + applyRemote(ops, options = {}) { + return this.#run(() => { + const otel = this.#otel; + if (!otel.enabled) return this.#applyRemote(ops, options); + return this.#tracedApply(otel, ops, options); + }); + } + + async #tracedApply(otel, ops, options) { + const started = now(); + let result = null; + try { + const attributes = { 'syncom.ops.count': ops.length }; + result = await traced( + otel, + 'syncom.apply', + attributes, + () => this.#applyRemote(ops, options), + (r) => ({ + 'syncom.ops.applied': r.applied, + 'syncom.ops.duplicates': r.duplicates, + 'syncom.ops.gaps': r.gaps, + }), + ); + return result; + } finally { + otel.recordApply(result ?? { applied: 0, duplicates: 0, gaps: 0 }, now() - started); + } + } + + async #applyRemote(ops, { via = null, scopes = null } = {}) { + { const accepted = []; const touched = new Map(); const deltas = new Map(); @@ -610,6 +711,7 @@ class Storage extends Emitter { (!isAdvance && typeof op.id !== 'string') ) { this.#logger.warn({ event: 'sync.op.malformed', op }); + this.#otel.recordMalformed(); continue; } if (op.origin === this.#nodeId || vectorHas(vector, op)) { @@ -646,10 +748,17 @@ class Storage extends Emitter { const { entry, changed, delta, rejected } = applyOp(current, op); if (rejected.length > 0) { this.#logger.warn({ event: 'sync.op.rejected', id: op.id, rejected }); - await this.emit('error', Object.assign(new Error(`Refused op on ${op.id}`), { op, rejected })); + this.#otel.recordRejected(rejected[0].reason); + // Guarded: Emitter throws on an unheard `error`, and a throw here + // would fail the push before the vector moved โ€” the peer would + // then offer the same page again on every hello, forever. + if (this.listenerCount('error') > 0) { + await this.emit('error', Object.assign(new Error(`Refused op on ${op.id}`), { op, rejected })); + } } vectorAdvance(vector, op); accepted.push(op); + this.#otel.recordLag(op.ts); if (!changed) continue; touched.set(op.id, entry); deltas.set(op.id, { ...(deltas.get(op.id) ?? {}), ...delta }); @@ -666,7 +775,7 @@ class Storage extends Emitter { } for (const op of accepted) await this.emit('change', op, via); return { applied: accepted.length, duplicates, gaps }; - }); + } } /** diff --git a/src/sync.js b/src/sync.js index 2bfd58a..63b1f35 100644 --- a/src/sync.js +++ b/src/sync.js @@ -26,6 +26,7 @@ const { Emitter } = require('./utils.js'); const { packOps, unpackOps } = require('./ops.js'); const { vectorHas, keyScope } = require('./vector.js'); const { isSyncLink } = require('./link.js'); +const { createTelemetry, traced } = require('./telemetry.js'); const REMOTE = 'remote'; const PROTOCOL = 1; @@ -89,11 +90,13 @@ class SyncManager extends Emitter { #onChange = null; #started = false; #saving = new Map(); + #otel; constructor(storage, options = {}) { super(); this.#storage = storage; this.#options = options ?? {}; + this.#otel = createTelemetry(storage.telemetry); this.#path = `${storage.path}/${REMOTE}`; return this.#init(); } @@ -208,7 +211,7 @@ class SyncManager extends Emitter { const results = []; for (const peer of peers) { if (!peer.remote) continue; - results.push({ url: peer.node.url, pushed: await this.#push(peer) }); + results.push({ url: peer.node.url, pushed: await this.#push(peer, 'manual') }); } return results; } @@ -331,7 +334,7 @@ class SyncManager extends Emitter { node.remote = { id: answer.node, epoch: answer.epoch }; await this.#saveNode(node); await this.emit('peer', { url: node.url, connected: true, node: answer.node }); - await this.#push(peer); + await this.#push(peer, 'handshake'); if (resubscribe && !peer.subscription) this.#subscribe(peer); } @@ -339,7 +342,7 @@ class SyncManager extends Emitter { // own origin excluded (it has those by definition). A host that is behind // what we still retain cannot be caught up from here; it will bootstrap // from a node that dials it, or we from it. - async #push(peer) { + async #push(peer, reason) { const { node } = peer; const entries = this.#storage.log.since(undefined) ?? @@ -352,10 +355,43 @@ class SyncManager extends Emitter { if (!inScope(op, node.scopes)) continue; ops.push(op); } - return this.#send(peer, ops); + return this.#send(peer, ops, reason); } - async #send(peer, ops) { + // One span per push, whatever its page count: with wrpc underneath, the + // `syncom/push` client span of every page nests under it, and the server + // span on the far side under that โ€” the link carries the context. + // `reason` says what caused it: a handshake, a manual sync(), or a relay + // of ops that just landed. + async #send(peer, ops, reason) { + const otel = this.#otel; + if (!otel.enabled || ops.length === 0) return this.#sendPages(peer, ops); + const attributes = { + 'syncom.push.reason': reason, + 'syncom.ops.count': ops.length, + 'syncom.peer.url': peer.node.url, + 'syncom.peer.node': peer.remote.id, + }; + const pages = Math.ceil(ops.length / (this.#options.pageSize ?? DEFAULT_PAGE_SIZE)); + let result = 'error'; + try { + const sent = await traced( + otel, + 'syncom.push', + attributes, + () => this.#sendPages(peer, ops), + () => ({ + 'syncom.pages': pages, + }), + ); + result = 'ok'; + return sent; + } finally { + otel.recordPush(reason, result, ops.length); + } + } + + async #sendPages(peer, ops) { const pageSize = this.#options.pageSize ?? DEFAULT_PAGE_SIZE; let sent = 0; for (let index = 0; index < ops.length; index += pageSize) { @@ -416,40 +452,65 @@ class SyncManager extends Emitter { async #bootstrap(peer) { if (peer.bootstrapping) return; peer.bootstrapping = true; - const { node, link } = peer; try { - peer.subscription?.unsubscribe?.(); - peer.subscription = null; - let cursor = null; - let startEventId = null; - let startVector = null; - let total = 0; - do { - const page = await link.call('syncom/snapshot', { cursor, scopes: node.scopes }); - if (peer.closed) return; - startEventId ??= page.eventId; - startVector ??= page.vector ?? null; - const records = page.id !== undefined ? await this.#readSnapshot(link, page) : page.records; - await this.#storage.mergeSnapshot(records, { via: peer, from: peer.remote.id }); - total += records.length; - cursor = page.cursor ?? null; - } while (cursor !== null); - // Everything up to the host's vector at the FIRST page is inside what - // was joined; adopting it (for the scopes asked for) means nobody - // offers those ops again. Later ops replay from the cursor. - if (startVector) { - await this.#storage.adoptVector(scopedVector(startVector, node.scopes), { via: peer, from: peer.remote.id }); + const otel = this.#otel; + if (!otel.enabled) return void (await this.#join(peer)); + const attributes = { 'syncom.peer.url': peer.node.url, 'syncom.peer.node': peer.remote.id }; + let result = 'error'; + try { + await traced( + otel, + 'syncom.bootstrap', + attributes, + () => this.#join(peer), + (stats) => ({ + 'syncom.pages': stats.pages, + 'syncom.records': stats.records, + }), + ); + result = 'ok'; + } finally { + otel.recordBootstrap(result); } - node.cursor = startEventId; - node.lastSync = Date.now(); - await this.#saveNode(node); - await this.emit('bootstrap', { url: node.url, records: total }); - if (!peer.closed) this.#subscribe(peer); } finally { peer.bootstrapping = false; } } + async #join(peer) { + const { node, link } = peer; + peer.subscription?.unsubscribe?.(); + peer.subscription = null; + let cursor = null; + let startEventId = null; + let startVector = null; + let pages = 0; + let total = 0; + do { + const page = await link.call('syncom/snapshot', { cursor, scopes: node.scopes }); + pages++; + if (peer.closed) return { pages, records: total }; + startEventId ??= page.eventId; + startVector ??= page.vector ?? null; + const records = page.id !== undefined ? await this.#readSnapshot(link, page) : page.records; + await this.#storage.mergeSnapshot(records, { via: peer, from: peer.remote.id }); + total += records.length; + cursor = page.cursor ?? null; + } while (cursor !== null); + // Everything up to the host's vector at the FIRST page is inside what + // was joined; adopting it (for the scopes asked for) means nobody + // offers those ops again. Later ops replay from the cursor. + if (startVector) { + await this.#storage.adoptVector(scopedVector(startVector, node.scopes), { via: peer, from: peer.remote.id }); + } + node.cursor = startEventId; + node.lastSync = Date.now(); + await this.#saveNode(node); + await this.emit('bootstrap', { url: node.url, records: total }); + if (!peer.closed) this.#subscribe(peer); + return { pages, records: total }; + } + // A streamed page: NDJSON, one `{ id, data, meta }` per line, parsed as the // chunks arrive so a page costs one record of memory, not one page. async #readSnapshot(link, page) { @@ -497,7 +558,7 @@ class SyncManager extends Emitter { while (peer.outbox.length > 0 && !peer.closed && peer.connected) { const ops = peer.outbox; peer.outbox = []; - await this.#send(peer, ops); + await this.#send(peer, ops, 'relay'); } }) .catch((error) => { diff --git a/src/telemetry.js b/src/telemetry.js new file mode 100644 index 0000000..0d0e20c --- /dev/null +++ b/src/telemetry.js @@ -0,0 +1,338 @@ +'use strict'; + +// OpenTelemetry with no dependency on OpenTelemetry. +// +// Nothing here imports anything: the caller injects either the +// `@opentelemetry/api` module or pre-built tracer/meter instances, and every +// one of them is duck-typed. The shape mirrors wrpc's `telemetry` option so a +// host can hand the same object to both packages โ€” and the two then meet in +// one trace: wrpc opens the SERVER span for a `syncom/push` call, and the +// `syncom.apply` span opened inside the handler nests under it through the +// active context. syncom itself never carries trace context: ops are durable +// and traces are not, and wrpc already puts W3C context on every packet. +// +// Every recording path is contained โ€” a tracer, meter, span or instrument +// that throws must never break a write. The one thing NOT swallowed is an +// error thrown by the work the span brackets. + +// 0 = UNSET, 1 = OK, 2 = ERROR. Frozen by the OTel specification, so +// hardcoding it is what avoids importing @opentelemetry/api for a constant. +const SPAN_STATUS_ERROR = 2; + +// Instrumentation scope. The version argument of getTracer/getMeter is +// deliberately omitted: reading it would mean requiring package.json, which +// would drag this file's whole dependency graph into browser bundles. +const SCOPE_NAME = '@alexify/syncom'; + +// Span attributes that identify a record or a peer. They stay off the span +// when `includeIdentity` is false; they are never metric labels at any +// setting โ€” a record id or a peer url on a label is unbounded cardinality. +const IDENTITY_KEYS = Object.freeze(['syncom.record.id', 'syncom.peer.url', 'syncom.peer.node']); + +const hasMethod = (value, name) => typeof value?.[name] === 'function'; +const noop = () => {}; + +// Monotonic where the platform has it; wall-clock otherwise. Only ever read +// when telemetry is on, so the disabled path costs no clock read. +const now = () => + typeof performance === 'object' && typeof performance.now === 'function' ? performance.now() : Date.now(); + +/** + * Two injection modes: + * - `{ api }` โ€” the @opentelemetry/api module, from which syncom derives its + * own tracer and meter so spans carry the right instrumentation scope. + * - `{ tracer, meter }` โ€” pre-built instances; either may be absent, and a + * tracer-only or meter-only configuration is fully supported. + */ +const resolveTracerAndMeter = (telemetry) => { + if (telemetry.api) { + const { api } = telemetry; + return { + tracer: hasMethod(api.trace, 'getTracer') ? api.trace.getTracer(SCOPE_NAME) : null, + meter: hasMethod(api.metrics, 'getMeter') ? api.metrics.getMeter(SCOPE_NAME) : null, + }; + } + return { tracer: telemetry.tracer ?? null, meter: telemetry.meter ?? null }; +}; + +// Opens a span (active when the tracer can, detached otherwise) and invokes +// `fn(handle)` exactly once. Unlike wrpc โ€” where a tracer that never calls +// its callback merely means one RPC does not happen โ€” a skipped callback here +// would be a silently lost write, so the callback's own result is what is +// returned, and a tracer that returned without invoking it gets the work run +// detached. +const startSpanWith = (tracer, name, options, handle, fn) => { + if (hasMethod(tracer, 'startActiveSpan')) { + let invoked = false; + let value; + try { + tracer.startActiveSpan(name, options, (span) => { + if (invoked) return value; + handle.span = span ?? null; + invoked = true; + value = fn(handle); + return value; + }); + } catch (error) { + // An error thrown by `fn` itself must propagate untouched; only a + // tracer that broke BEFORE running the callback is swallowed. + if (invoked) throw error; + return fn(handle); + } + return invoked ? value : fn(handle); + } + try { + handle.span = hasMethod(tracer, 'startSpan') ? tracer.startSpan(name, options) : null; + } catch { + handle.span = null; + } + return fn(handle); +}; + +const recordSpanError = (handle, error) => { + try { + handle.error = true; + const span = handle?.span; + if (!span) return; + span.recordException?.(error); + span.setStatus?.({ code: SPAN_STATUS_ERROR, message: error?.message }); + if (error?.name) span.setAttribute?.('error.type', error.name); + } catch { + // Telemetry must never break the write path. + } +}; + +const endSpanHandle = (handle, attributes) => { + try { + const span = handle?.span; + if (!span) return; + if (attributes) { + for (const key in attributes) { + const value = attributes[key]; + if (value !== undefined && value !== null) span.setAttribute?.(key, value); + } + } + span.end(); + } catch {} +}; + +// Label sets for the fixed-attribute samples, built once: they ride on every +// write when metrics are on. +const PATH_COMMIT = Object.freeze({ 'syncom.path': 'commit' }); +const PATH_APPLY = Object.freeze({ 'syncom.path': 'apply' }); +const PATH_REPLAY = Object.freeze({ 'syncom.path': 'replay' }); +const OUTCOME_APPLIED = Object.freeze({ 'syncom.outcome': 'applied' }); +const OUTCOME_DUPLICATE = Object.freeze({ 'syncom.outcome': 'duplicate' }); +const OUTCOME_GAP = Object.freeze({ 'syncom.outcome': 'gap' }); +const OUTCOME_MALFORMED = Object.freeze({ 'syncom.outcome': 'malformed' }); +const SEGMENT_SEALED = Object.freeze({ 'syncom.log.event': 'sealed' }); +const SEGMENT_DROPPED = Object.freeze({ 'syncom.log.event': 'dropped' }); + +const DISABLED = Object.freeze({ + enabled: false, + // The lifecycle wrapper still runs its callback โ€” that is what lets a call + // site bracket a write the same way whether telemetry is on or off. + withSpan(_name, _attributes, fn) { + return fn(null); + }, + recordError: noop, + endSpan: noop, + recordCommit: noop, + recordApply: noop, + recordMalformed: noop, + recordRejected: noop, + recordLag: noop, + recordReplay: noop, + recordPush: noop, + recordBootstrap: noop, + recordLogRoll: noop, + recordAppend: noop, +}); + +const isWriter = (value) => typeof value.enabled === 'boolean' && typeof value.withSpan === 'function'; + +/** + * Builds the writer once from the `telemetry` option. Idempotent on a writer: + * Storage builds its own and hands that same object to OpLog and SyncManager. + */ +const createTelemetry = (telemetry) => { + if (!telemetry || typeof telemetry !== 'object') return DISABLED; + if (isWriter(telemetry)) return telemetry; + + let tracer = null; + let meter = null; + try { + ({ tracer, meter } = resolveTracerAndMeter(telemetry)); + } catch { + return DISABLED; + } + if (!tracer && !meter) return DISABLED; + + const includeIdentity = telemetry.includeIdentity !== false; + + let commits = null; + let writeDuration = null; + let opsReceived = null; + let opsRejected = null; + let lag = null; + let pushOps = null; + let bootstraps = null; + let logSegments = null; + let appendDuration = null; + + if (hasMethod(meter, 'createCounter') && hasMethod(meter, 'createHistogram')) { + try { + commits = meter.createCounter('syncom.commits', { + unit: '{op}', + description: 'Local writes committed, by op type and result', + }); + writeDuration = meter.createHistogram('syncom.write.duration', { + unit: 'ms', + description: 'Duration of a local commit, an applied peer batch or a replayed tail', + }); + opsReceived = meter.createCounter('syncom.ops.received', { + unit: '{op}', + description: 'Peer ops offered to this node, by outcome', + }); + opsRejected = meter.createCounter('syncom.ops.rejected', { + unit: '{op}', + description: 'Peer ops refused because a field is the other kind (register/counter)', + }); + lag = meter.createHistogram('syncom.replication.lag', { + unit: 'ms', + description: 'Origin stamp to applied here, for each accepted peer op (clock skew included)', + }); + pushOps = meter.createHistogram('syncom.push.ops', { + unit: '{op}', + description: 'Ops per push to a peer, by reason and result', + }); + bootstraps = meter.createCounter('syncom.bootstraps', { + unit: '{bootstrap}', + description: 'Snapshot bootstraps from a peer, by result', + }); + logSegments = meter.createCounter('syncom.log.segments', { + unit: '{segment}', + description: 'Log segments sealed and dropped', + }); + appendDuration = meter.createHistogram('syncom.log.append.duration', { + unit: 'ms', + description: 'Duration of writing a batch to the open log', + }); + } catch { + commits = null; + writeDuration = null; + opsReceived = null; + opsRejected = null; + lag = null; + pushOps = null; + bootstraps = null; + logSegments = null; + appendDuration = null; + } + } + + const add = (instrument, value, attributes) => { + try { + instrument?.add(value, attributes); + } catch {} + }; + const record = (instrument, value, attributes) => { + try { + instrument?.record(value, attributes); + } catch {} + }; + + return { + enabled: true, + + /** + * Opens a span around one unit of work and invokes `fn(handle)` exactly + * once. The handle is mutable โ€” `{ span, error }` โ€” rather than a raw + * span, because the span is NOT ended here: `endSpan` does that from the + * caller's `finally`, which is the only place that knows the work is done. + * `attributes` is a literal built by the call site for this span alone. + */ + withSpan(name, attributes, fn) { + const handle = { span: null, error: false }; + if (!tracer) return fn(handle); + if (!includeIdentity) for (const key of IDENTITY_KEYS) delete attributes[key]; + return startSpanWith(tracer, name, { attributes }, handle, fn); + }, + + recordError(handle, error) { + recordSpanError(handle, error); + }, + + endSpan(handle, attributes) { + endSpanHandle(handle, attributes); + }, + + recordCommit(type, result, elapsed) { + add(commits, 1, { 'syncom.op.type': type, 'syncom.result': result }); + record(writeDuration, elapsed, PATH_COMMIT); + }, + + recordApply(result, elapsed) { + if (result.applied > 0) add(opsReceived, result.applied, OUTCOME_APPLIED); + if (result.duplicates > 0) add(opsReceived, result.duplicates, OUTCOME_DUPLICATE); + if (result.gaps > 0) add(opsReceived, result.gaps, OUTCOME_GAP); + record(writeDuration, elapsed, PATH_APPLY); + }, + + recordMalformed() { + add(opsReceived, 1, OUTCOME_MALFORMED); + }, + + recordRejected(reason) { + add(opsRejected, 1, { 'syncom.reason': reason }); + }, + + // The op's stamp is the origin's wall clock at commit time + // (vector.js Clock), so this is skew plus transit plus queueing. + recordLag(ts) { + if (typeof ts === 'number') record(lag, Math.max(0, Date.now() - ts)); + }, + + recordReplay(elapsed) { + record(writeDuration, elapsed, PATH_REPLAY); + }, + + recordPush(reason, result, count) { + record(pushOps, count, { 'syncom.push.reason': reason, 'syncom.result': result }); + }, + + recordBootstrap(result) { + add(bootstraps, 1, { 'syncom.result': result }); + }, + + recordLogRoll(sealed, dropped) { + if (sealed > 0) add(logSegments, 1, SEGMENT_SEALED); + if (dropped > 0) add(logSegments, dropped, SEGMENT_DROPPED); + }, + + recordAppend(elapsed) { + record(appendDuration, elapsed); + }, + }; +}; + +/** + * Runs `task()` inside a span: `attributes` open it, `after(result)` supplies + * the attributes it ends with, and a throw marks it failed before + * propagating. What the task returns is returned; timing and metrics stay + * with the caller, which is the only place that knows what to count. + */ +const traced = (otel, name, attributes, task, after) => + otel.withSpan(name, attributes, async (handle) => { + try { + const result = await task(); + otel.endSpan(handle, after(result)); + return result; + } catch (error) { + otel.recordError(handle, error); + otel.endSpan(handle); + throw error; + } + }); + +module.exports = { createTelemetry, traced, now, DISABLED, SCOPE_NAME, SPAN_STATUS_ERROR }; diff --git a/tests/browser.test-d.ts b/tests/browser.test-d.ts index 6a8aa8f..568dcd4 100644 --- a/tests/browser.test-d.ts +++ b/tests/browser.test-d.ts @@ -15,6 +15,7 @@ expectType(browser.merge({ a: 1 }, { a: 2 })); expectType>(new browser.Blockchain('base')); expectType>(browser.open({ path: 'data' })); expectType>(browser.open({ path: 'data', getId: () => 'id' })); +expectType>(browser.open({ path: 'data', telemetry: { api: {} } })); // The id generator is not part of the async asymmetry: sync in both halves. expectType(browser.Storage.generateId()); expectType>(browser.read('dir', 'file.json')); diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 10333fd..1e3b9ee 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -13,6 +13,7 @@ import type { SyncManager, SyncomCollection, SyncomRecord, + SyncomTelemetryOptions, } from '../index.js'; // crypto โ€” node:crypto is synchronous, unlike the browser half. @@ -132,3 +133,21 @@ expectType(log.since(undefined, { scopes: ['*'], limit expectType>(log.append([])); expectAssignable({ logger: false, sync: { scope: () => 'docs', retention: { ops: 10 } } }); expectError(syncom.open({ sync: { scope: 'docs' } })); + +// telemetry โ€” structural, so a real SDK fits without importing @opentelemetry/api. +expectAssignable({ telemetry: { api: {} } }); +expectAssignable({ telemetry: { tracer: { startSpan: () => ({ end() {} }) } } }); +expectAssignable({ + telemetry: { + meter: { createCounter: () => ({ add() {} }), createHistogram: () => ({ record() {} }) }, + includeIdentity: false, + }, +}); +expectAssignable({ + api: { trace: { getTracer: () => ({ startActiveSpan: (_n, _o, fn) => fn({ end() {} }) }) } }, +}); +expectError(syncom.open({ telemetry: 'yes' })); +expectError(syncom.open({ telemetry: { tracer: 'a tracer' } })); +expectType>(new syncom.OpLog('p', { telemetry: { api: {} } })); +declare const telemetryStorage: Storage; +expectType(telemetryStorage.telemetry); diff --git a/tests/storage.browser.test.js b/tests/storage.browser.test.js index d4cd66d..ef4dff4 100644 --- a/tests/storage.browser.test.js +++ b/tests/storage.browser.test.js @@ -39,3 +39,22 @@ test('Storage on the browser halves', async (t) => { assert.deepStrictEqual(await reopened.get('s1'), secret); }); }); + +test('Storage telemetry on the browser halves', async (t) => { + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = requireBrowserBarrel(); + const { BasicTracerProvider, InMemorySpanExporter, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base'); + + await t.test('a commit is spanned with nothing but the injected tracer', async () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ spanProcessors: [new SimpleSpanProcessor(exporter)] }); + const storage = await browser.open({ path: 'store-telemetry', telemetry: { tracer: provider.getTracer('t') } }); + const id = await storage.insert({ title: 'Traced' }); + const [span] = exporter.getFinishedSpans(); + assert.strictEqual(span.name, 'syncom.commit'); + assert.strictEqual(span.attributes['syncom.op.type'], 'write'); + assert.strictEqual(span.attributes['syncom.record.id'], id); + assert.strictEqual(span.attributes['syncom.result'], 'applied'); + }); +}); diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js index 9a60a0a..30f0b8a 100644 --- a/tests/storage.ops.test.js +++ b/tests/storage.ops.test.js @@ -303,3 +303,34 @@ test('Storage replication model', async (t) => { assert.deepStrictEqual((await storage.snapshot()).records, []); }); }); + +test('Storage: the seams telemetry shares', async (t) => { + await t.test('a refused op with no error listener is still logged and answered', async (t2) => { + // Emitter throws on an unheard `error`; before the guard, that throw + // failed the whole push and the peer offered the same page forever. + const { storage } = await openTemp(t2); + const id = await storage.insert({ hits: 'text' }); + const result = await storage.applyRemote([remoteOp('peer', 1, 'inc', id, { key: 'hits', value: 1 })]); + assert.strictEqual(result.applied, 1); + assert.strictEqual(storage.vector.peer, 1); + assert.deepStrictEqual(await storage.get(id), { hits: 'text' }); + }); + + await t.test('the storage logger reaches the op log', async (t2) => { + const events = []; + const logger = { warn: (entry) => events.push(entry) }; + const { dir, storage } = await openTemp(t2, { logger, sync: { log: { rollAfter: 1 } } }); + await storage.insert({ n: 1 }); + await storage.close(); + // One sealed segment on disk; corrupt it and reopen. + const logDir = path.join(dir, 'log'); + const segment = (await fs.readdir(logDir)).find((name) => name.startsWith('seg-')); + assert.ok(segment, 'a segment was sealed'); + await fs.writeFile(path.join(logDir, segment), '{not json'); + await syncom.open({ path: dir, logger }); + assert.strictEqual( + events.some((entry) => entry.event === 'log.segment.unreadable'), + true, + ); + }); +}); diff --git a/tests/telemetry.test.js b/tests/telemetry.test.js new file mode 100644 index 0000000..96b9670 --- /dev/null +++ b/tests/telemetry.test.js @@ -0,0 +1,720 @@ +'use strict'; + +// The injected OpenTelemetry writer: what enables it, what it emits, and โ€” +// most of the file โ€” that nothing a hostile tracer, meter, span or +// instrument does can break a write. + +const { test } = require('node:test'); +const assert = require('node:assert'); + +const otelApi = require('@opentelemetry/api'); +const { BasicTracerProvider, InMemorySpanExporter, SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base'); +const { + AggregationTemporality, + InMemoryMetricExporter, + MeterProvider, + PeriodicExportingMetricReader, +} = require('@opentelemetry/sdk-metrics'); +const { AsyncLocalStorageContextManager } = require('@opentelemetry/context-async-hooks'); + +const syncom = require('..'); +const { createTelemetry, DISABLED, SCOPE_NAME } = require('../src/telemetry.js'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { createMesh, settle, waitUntil } = require('./utils/mesh.js'); +const { MemoryLink } = require('./utils/memory-link.js'); + +// A real SDK, exporting into memory. Anything less would not prove that what +// syncom emits is actually a valid span or a valid metric. +const createTracing = () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ spanProcessors: [new SimpleSpanProcessor(exporter)] }); + return { + provider, + tracer: provider.getTracer(SCOPE_NAME), + spans: () => exporter.getFinishedSpans(), + named: (name) => exporter.getFinishedSpans().filter((span) => span.name === name), + }; +}; + +const createMetrics = () => { + const exporter = new InMemoryMetricExporter(AggregationTemporality.CUMULATIVE); + // A very long interval: the reader is flushed by hand so the assertions + // are not racing a background timer. + const reader = new PeriodicExportingMetricReader({ exporter, exportIntervalMillis: 3_600_000 }); + const provider = new MeterProvider({ readers: [reader] }); + const collect = async () => { + await reader.forceFlush(); + const metrics = []; + for (const batch of exporter.getMetrics()) { + for (const scope of batch.scopeMetrics) metrics.push(...scope.metrics); + } + return metrics; + }; + const points = async (name) => { + const metric = (await collect()).find((m) => m.descriptor.name === name); + return metric ? metric.dataPoints : []; + }; + return { provider, meter: provider.getMeter(SCOPE_NAME), collect, points }; +}; + +const useActiveContext = (t) => { + const manager = new AsyncLocalStorageContextManager().enable(); + otelApi.context.setGlobalContextManager(manager); + t.after(() => otelApi.context.disable()); +}; + +const openTemp = async (t, options = {}) => { + const dir = await createTemp(); + t.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, ...options }); + t.after(() => storage.close().catch(() => {})); + return { dir, storage }; +}; + +const remoteOp = (origin, seq, type, id, extra = {}) => ({ origin, seq, ts: Date.now() + seq, type, id, ...extra }); + +const sum = (dataPoints, predicate = () => true) => + dataPoints.filter(predicate).reduce((total, point) => total + (point.value.sum ?? point.value), 0); + +const point = (dataPoints, attributes) => + dataPoints.find((candidate) => + Object.entries(attributes).every(([key, value]) => candidate.attributes[key] === value), + ); + +// Never a metric label: anything that identifies a record, a node or a peer. +const IDENTITY_LABELS = ['syncom.record.id', 'syncom.op.origin', 'syncom.peer.url', 'syncom.peer.node', 'syncom.scope']; +const assertNoIdentityLabels = (metrics) => { + for (const metric of metrics) { + for (const dataPoint of metric.dataPoints) { + for (const key of Object.keys(dataPoint.attributes)) { + assert.ok(!IDENTITY_LABELS.includes(key), `${metric.descriptor.name} carries ${key}`); + } + } + } +}; + +test('createTelemetry: what enables it', async (t) => { + await t.test('nothing, a non-object, or an empty bag is the frozen DISABLED writer', () => { + for (const value of [undefined, null, false, 'otel', 42, {}]) { + assert.strictEqual(createTelemetry(value), DISABLED, String(value)); + } + assert.ok(Object.isFrozen(DISABLED)); + }); + + await t.test('an api module with neither trace nor metrics is disabled', () => { + assert.strictEqual(createTelemetry({ api: { trace: {}, metrics: {} } }), DISABLED); + }); + + await t.test('a getter that throws is disabled rather than fatal', () => { + const telemetry = { + get api() { + throw new Error('exploded while resolving'); + }, + }; + assert.strictEqual(createTelemetry(telemetry), DISABLED); + }); + + await t.test('a writer is returned as-is, so Storage can hand its own down', () => { + const { tracer } = createTracing(); + const writer = createTelemetry({ tracer }); + assert.strictEqual(createTelemetry(writer), writer); + assert.strictEqual(createTelemetry(DISABLED), DISABLED); + }); + + await t.test('the disabled writer still runs the callback and answers everything', () => { + let handle = 'untouched'; + const result = DISABLED.withSpan('syncom.commit', {}, (value) => { + handle = value; + return 'returned'; + }); + assert.strictEqual(result, 'returned'); + assert.strictEqual(handle, null); + assert.doesNotThrow(() => { + DISABLED.recordCommit('write', 'applied', 1); + DISABLED.recordApply({ applied: 1, duplicates: 0, gaps: 0 }, 1); + DISABLED.recordError(null, new Error('x')); + DISABLED.endSpan(null); + }); + }); + + await t.test('{ api } derives a tracer and meter with the syncom scope', async () => { + const { provider, spans } = createTracing(); + const metrics = createMetrics(); + const api = { + trace: { getTracer: (name) => provider.getTracer(name) }, + metrics: { getMeter: (name) => metrics.provider.getMeter(name) }, + }; + const otel = createTelemetry({ api }); + assert.strictEqual(otel.enabled, true); + otel.withSpan('syncom.commit', {}, (handle) => otel.endSpan(handle)); + otel.recordCommit('write', 'applied', 2); + assert.strictEqual(spans()[0].instrumentationScope.name, SCOPE_NAME); + const [commits] = await metrics.collect(); + assert.strictEqual(commits.descriptor.name, 'syncom.commits'); + }); + + await t.test('tracer-only and meter-only are both supported', async () => { + const { tracer, spans } = createTracing(); + const traced = createTelemetry({ tracer }); + assert.strictEqual(traced.enabled, true); + traced.withSpan('syncom.commit', {}, (handle) => traced.endSpan(handle)); + traced.recordCommit('write', 'applied', 1); + assert.strictEqual(spans().length, 1); + + const { meter, points } = createMetrics(); + const metered = createTelemetry({ meter }); + assert.strictEqual(metered.enabled, true); + let seen; + metered.withSpan('syncom.commit', {}, (handle) => { + seen = handle; + metered.endSpan(handle); + }); + assert.deepStrictEqual(seen, { span: null, error: false }); + metered.recordCommit('write', 'applied', 1); + assert.strictEqual(sum(await points('syncom.commits')), 1); + }); + + await t.test('a meter missing a factory leaves metrics off, not the writer', () => { + const otel = createTelemetry({ meter: { createCounter: () => ({ add() {} }) } }); + assert.strictEqual(otel.enabled, true); + assert.doesNotThrow(() => otel.recordCommit('write', 'applied', 1)); + }); + + await t.test('a meter whose factories throw leaves the writer enabled and silent', () => { + const meter = { + createCounter: () => { + throw new Error('no instruments today'); + }, + createHistogram: () => ({ record() {} }), + }; + const otel = createTelemetry({ meter }); + assert.strictEqual(otel.enabled, true); + assert.doesNotThrow(() => { + otel.recordCommit('write', 'applied', 1); + otel.recordApply({ applied: 1, duplicates: 1, gaps: 1 }, 1); + otel.recordLogRoll(1, 1); + }); + }); +}); + +test('spans', async (t) => { + await t.test('an error sets status 2 and records the exception', () => { + const { tracer, spans } = createTracing(); + const otel = createTelemetry({ tracer }); + const boom = new TypeError('bad field'); + otel.withSpan('syncom.commit', { 'syncom.op.type': 'inc' }, (handle) => { + otel.recordError(handle, boom); + otel.endSpan(handle, { 'syncom.result': 'error' }); + }); + const [span] = spans(); + assert.strictEqual(span.status.code, 2); + assert.strictEqual(span.status.message, 'bad field'); + assert.strictEqual(span.attributes['error.type'], 'TypeError'); + assert.strictEqual(span.attributes['syncom.op.type'], 'inc'); + assert.strictEqual(span.attributes['syncom.result'], 'error'); + assert.strictEqual(span.events[0].name, 'exception'); + }); + + await t.test('null and undefined end-attributes are skipped', () => { + const { tracer, spans } = createTracing(); + const otel = createTelemetry({ tracer }); + otel.withSpan('syncom.apply', {}, (handle) => otel.endSpan(handle, { a: null, b: undefined, c: 0 })); + const [span] = spans(); + assert.deepStrictEqual(span.attributes, { c: 0 }); + }); + + await t.test('identity attributes are stripped when includeIdentity is false', () => { + const { tracer, spans } = createTracing(); + const otel = createTelemetry({ tracer, includeIdentity: false }); + otel.withSpan( + 'syncom.push', + { 'syncom.peer.url': 'ws://x', 'syncom.peer.node': 'n', 'syncom.record.id': 'r', 'syncom.ops.count': 3 }, + (handle) => otel.endSpan(handle), + ); + const [span] = spans(); + assert.deepStrictEqual(span.attributes, { 'syncom.ops.count': 3 }); + }); + + await t.test('spans nest through startActiveSpan under a context manager', (t2) => { + useActiveContext(t2); + const { provider, tracer, spans } = createTracing(); + const otel = createTelemetry({ tracer }); + otel.withSpan('syncom.commit', {}, (handle) => { + const inner = provider.getTracer('inner').startSpan('host-listener'); + inner.end(); + otel.endSpan(handle); + }); + const [inner, outer] = spans(); + assert.strictEqual(inner.name, 'host-listener'); + assert.strictEqual(inner.parentSpanContext?.spanId, outer.spanContext().spanId); + }); +}); + +test('containment: a hostile tracer never breaks the work', async (t) => { + const work = (otel) => { + let runs = 0; + const result = otel.withSpan('syncom.commit', {}, (handle) => { + runs++; + otel.endSpan(handle); + return 'done'; + }); + return { runs, result }; + }; + + await t.test('a tracer that throws before invoking the callback', () => { + const tracer = { + startActiveSpan() { + throw new Error('tracer down'); + }, + }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test('a tracer that never invokes the callback still gets the work run once', () => { + const tracer = { startActiveSpan: () => 'ignored you' }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test('a tracer that invokes the callback twice runs the work once', () => { + const tracer = { + startActiveSpan(_name, _options, fn) { + fn({ end() {} }); + return fn({ end() {} }); + }, + }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test("the callback's result wins over whatever the tracer returns", () => { + const tracer = { + startActiveSpan(_name, _options, fn) { + fn({ end() {} }); + return 'not the result'; + }, + }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test('an error thrown by the work propagates untouched', () => { + const { tracer } = createTracing(); + const otel = createTelemetry({ tracer }); + const boom = new Error('from the work'); + assert.throws( + () => + otel.withSpan('syncom.commit', {}, () => { + throw boom; + }), + (error) => error === boom, + ); + }); + + await t.test('a tracer with only startSpan opens a detached span', () => { + let ended = 0; + const tracer = { startSpan: () => ({ end: () => ended++ }) }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + assert.strictEqual(ended, 1); + }); + + await t.test('a tracer whose startSpan throws', () => { + const tracer = { + startSpan() { + throw new Error('no span for you'); + }, + }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test('a tracer with neither method', () => { + const tracer = { getSpan: () => null }; + assert.deepStrictEqual(work(createTelemetry({ tracer })), { runs: 1, result: 'done' }); + }); + + await t.test('a span whose methods throw', () => { + const hostile = () => { + throw new Error('span exploded'); + }; + const span = { end: hostile, setAttribute: hostile, setStatus: hostile, recordException: hostile }; + const tracer = { startActiveSpan: (_n, _o, fn) => fn(span) }; + const otel = createTelemetry({ tracer }); + const result = otel.withSpan('syncom.commit', {}, (handle) => { + otel.recordError(handle, new Error('x')); + otel.endSpan(handle, { 'syncom.result': 'error' }); + return 'done'; + }); + assert.strictEqual(result, 'done'); + }); + + await t.test('an instrument that throws on use', () => { + const hostile = () => { + throw new Error('instrument exploded'); + }; + const meter = { createCounter: () => ({ add: hostile }), createHistogram: () => ({ record: hostile }) }; + const otel = createTelemetry({ meter }); + assert.doesNotThrow(() => { + otel.recordCommit('write', 'applied', 1); + otel.recordApply({ applied: 1, duplicates: 1, gaps: 1 }, 1); + otel.recordMalformed(); + otel.recordRejected('register'); + otel.recordLag(Date.now()); + otel.recordLag('not a stamp'); + otel.recordReplay(1); + otel.recordPush('relay', 'ok', 1); + otel.recordBootstrap('ok'); + otel.recordLogRoll(1, 2); + otel.recordAppend(1); + }); + }); + + await t.test('recordError and endSpan tolerate a null handle', () => { + const { tracer } = createTracing(); + const otel = createTelemetry({ tracer }); + assert.doesNotThrow(() => { + otel.recordError(null, new Error('x')); + otel.endSpan(null); + otel.endSpan({ span: null }); + }); + }); +}); + +test('Storage telemetry', async (t) => { + await t.test('a commit span per local write, counted by type and result', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const { storage } = await openTemp(t2, { telemetry: { tracer, meter } }); + const id = await storage.insert({ title: 'Draft' }); + await storage.inc(id, 'views', 2); + await storage.delete(id); + const spans = named('syncom.commit'); + assert.deepStrictEqual( + spans.map((span) => [ + span.attributes['syncom.op.type'], + span.attributes['syncom.result'], + span.attributes['syncom.scope'], + ]), + [ + ['write', 'applied', '*'], + ['inc', 'applied', '*'], + ['delete', 'applied', '*'], + ], + ); + assert.ok(spans.every((span) => span.attributes['syncom.record.id'] === id)); + assert.ok(spans.every((span) => span.status.code === 0)); + assert.strictEqual(spans[0].instrumentationScope.name, SCOPE_NAME); + assert.strictEqual( + sum(await points('syncom.commits'), (p) => p.attributes['syncom.result'] === 'applied'), + 3, + ); + assert.strictEqual(point(await points('syncom.write.duration'), { 'syncom.path': 'commit' }).value.count, 3); + }); + + await t.test('a refused local write is an error span and an error sample', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const { storage } = await openTemp(t2, { telemetry: { tracer, meter } }); + const id = await storage.insert({ hits: 'text' }); + await assert.rejects(() => storage.inc(id, 'hits', 1), /register/); + const span = named('syncom.commit').at(-1); + assert.strictEqual(span.attributes['syncom.op.type'], 'inc'); + assert.strictEqual(span.status.code, 2); + assert.strictEqual(span.attributes['error.type'], 'Error'); + assert.strictEqual(span.events[0].name, 'exception'); + assert.strictEqual( + sum(await points('syncom.commits'), (p) => p.attributes['syncom.result'] === 'error'), + 1, + ); + }); + + await t.test('a write to an encrypted record stays local: result encrypted, no error', async (t2) => { + const { tracer, named } = createTracing(); + const { storage } = await openTemp(t2, { telemetry: { tracer } }); + await storage.saveData('s1', { secret: 1 }, { encrypted: true }); + await storage.set('s1', { secret: 2 }); + assert.deepStrictEqual(await storage.get('s1'), { secret: 2 }); + const [span] = named('syncom.commit'); + assert.strictEqual(span.attributes['syncom.result'], 'encrypted'); + assert.strictEqual(span.attributes['syncom.scope'], undefined); + assert.strictEqual(span.status.code, 0); + }); + + await t.test('a merge is counted but not spanned; joining what is held already is a noop', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const { storage } = await openTemp(t2, { telemetry: { tracer, meter } }); + await storage.insert({ a: 1 }); + const page = await storage.snapshot(); + await storage.mergeSnapshot(page.records); + assert.strictEqual(named('syncom.commit').length, 1, 'only the insert has a span'); + const commits = await points('syncom.commits'); + assert.strictEqual(point(commits, { 'syncom.op.type': 'merge', 'syncom.result': 'noop' }).value, 1); + }); + + await t.test('an apply span counts the batch; ops by outcome, rejections and lag are sampled', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points, collect } = createMetrics(); + const { storage } = await openTemp(t2, { telemetry: { tracer, meter } }); + storage.on('error', () => {}); + const id = await storage.insert({ hits: 'text' }); + const result = await storage.applyRemote([ + remoteOp('peer', 1, 'write', 'r1', { data: { a: 1 } }), + remoteOp('peer', 1, 'write', 'r1', { data: { a: 1 } }), + remoteOp('peer', 3, 'set', 'r1', { data: { b: 1 } }), + 'garbage', + remoteOp('peer', 2, 'inc', id, { key: 'hits', value: 1 }), + ]); + assert.deepStrictEqual(result, { applied: 2, duplicates: 1, gaps: 1 }); + const [span] = named('syncom.apply'); + assert.strictEqual(span.attributes['syncom.ops.count'], 5); + assert.strictEqual(span.attributes['syncom.ops.applied'], 2); + assert.strictEqual(span.attributes['syncom.ops.duplicates'], 1); + assert.strictEqual(span.attributes['syncom.ops.gaps'], 1); + assert.strictEqual(span.status.code, 0); + const received = await points('syncom.ops.received'); + assert.strictEqual(point(received, { 'syncom.outcome': 'applied' }).value, 2); + assert.strictEqual(point(received, { 'syncom.outcome': 'duplicate' }).value, 1); + assert.strictEqual(point(received, { 'syncom.outcome': 'gap' }).value, 1); + assert.strictEqual(point(received, { 'syncom.outcome': 'malformed' }).value, 1); + assert.strictEqual(point(await points('syncom.ops.rejected'), { 'syncom.reason': 'register' }).value, 1); + const [lag] = await points('syncom.replication.lag'); + assert.strictEqual(lag.value.count, 2); + assert.strictEqual(point(await points('syncom.write.duration'), { 'syncom.path': 'apply' }).value.count, 1); + assertNoIdentityLabels(await collect()); + }); + + await t.test('a replayed tail is spanned on reopen', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const { dir, storage: first } = await openTemp(t2); + const id = await first.insert({ n: 1 }); + await first.close(); + // Crash between "logged" and "written": the record file is gone. + await require('node:fs/promises').rm(require('node:path').join(dir, 'data', `${id}.json`)); + const second = await syncom.open({ path: dir, logger: false, telemetry: { tracer, meter } }); + t2.after(() => second.close().catch(() => {})); + assert.deepStrictEqual(await second.get(id), { n: 1 }); + const [span] = named('syncom.replay'); + assert.strictEqual(span.attributes['syncom.ops.count'], 1); + assert.strictEqual(span.attributes['syncom.records'], 1); + assert.strictEqual(point(await points('syncom.write.duration'), { 'syncom.path': 'replay' }).value.count, 1); + }); + + await t.test('includeIdentity: false keeps the record id off the span', async (t2) => { + const { tracer, named } = createTracing(); + const { storage } = await openTemp(t2, { telemetry: { tracer, includeIdentity: false } }); + await storage.insert({ n: 1 }); + const [span] = named('syncom.commit'); + assert.strictEqual(span.attributes['syncom.record.id'], undefined); + assert.strictEqual(span.attributes['syncom.op.type'], 'write'); + }); + + await t.test('a commit nests under whatever span is active โ€” a wrpc handler, say', async (t2) => { + useActiveContext(t2); + const { provider, tracer, named } = createTracing(); + const { storage } = await openTemp(t2, { telemetry: { tracer } }); + const outer = await provider.getTracer('host').startActiveSpan('request', async (span) => { + await storage.insert({ n: 1 }); + span.end(); + return span; + }); + const [commit] = named('syncom.commit'); + assert.strictEqual(commit.parentSpanContext?.spanId, outer.spanContext().spanId); + }); + + await t.test('nothing is recorded, and nothing is measured, when telemetry is off', async (t2) => { + const { storage } = await openTemp(t2); + assert.strictEqual(storage.telemetry, DISABLED); + assert.strictEqual(storage.sync.storage.telemetry.enabled, false); + await storage.insert({ n: 1 }); + }); +}); + +test('Sync telemetry', async (t) => { + const createStorage = + (telemetryOf) => + async (t2, name, options = {}) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const storage = await syncom.open({ path: dir, logger: false, telemetry: telemetryOf(name), ...options }); + t2.after(() => storage.close().catch(() => {})); + return storage; + }; + + await t.test('a write, its relay push and the peer apply form one trace', async (t2) => { + useActiveContext(t2); + const tracing = { A: createTracing(), B: createTracing() }; + const mesh = await createMesh( + t2, + ['A', 'B'], + [['A', 'B']], + createStorage((name) => ({ tracer: tracing[name].tracer })), + ); + const a = mesh.get('A'); + const b = mesh.get('B'); + await a.insert({ n: 1 }); + await settle([a, b]); + const [commit] = tracing.A.named('syncom.commit'); + const [push] = tracing.A.named('syncom.push'); + const [apply] = tracing.B.named('syncom.apply'); + assert.strictEqual(push.attributes['syncom.push.reason'], 'relay'); + assert.strictEqual(push.attributes['syncom.ops.count'], 1); + assert.strictEqual(push.attributes['syncom.pages'], 1); + assert.strictEqual(push.attributes['syncom.peer.url'], 'mem://B'); + assert.strictEqual(push.attributes['syncom.peer.node'], b.nodeId); + assert.strictEqual(push.status.code, 0); + // The relay was scheduled from inside the commit; the peer applied it + // inside the push. One trace, three spans, two nodes. + assert.strictEqual(push.parentSpanContext?.spanId, commit.spanContext().spanId); + assert.strictEqual(apply.parentSpanContext?.spanId, push.spanContext().spanId); + assert.strictEqual(apply.spanContext().traceId, commit.spanContext().traceId); + assert.strictEqual(apply.attributes['syncom.ops.applied'], 1); + }); + + await t.test('a handshake push, a failed relay and a manual retry each say why', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points, collect } = createMetrics(); + const { storage: b } = await openTemp(t2); + const unit = b.sync.unit({ access: 'public' }); + let link = null; + const connect = () => { + link = new MemoryLink(unit); + return link; + }; + const { storage: a } = await openTemp(t2, { telemetry: { tracer, meter }, sync: { connect, autoStart: false } }); + await a.insert({ first: true }); + await a.sync.addNode({ url: 'mem://B' }); + await a.sync.start(); + await settle([a, b]); + const [handshake] = named('syncom.push'); + assert.strictEqual(handshake.attributes['syncom.push.reason'], 'handshake'); + assert.strictEqual(handshake.attributes['syncom.ops.count'], 1); + + // The relay of the next write is refused by the link. + const call = link.call.bind(link); + let refuse = true; + link.call = (method, args) => + refuse && method === 'syncom/push' ? Promise.reject(new Error('push refused')) : call(method, args); + await a.insert({ second: true }); + await waitUntil(() => named('syncom.push').length === 2); + const relay = named('syncom.push')[1]; + assert.strictEqual(relay.attributes['syncom.push.reason'], 'relay'); + assert.strictEqual(relay.status.code, 2); + assert.strictEqual(relay.status.message, 'push refused'); + + refuse = false; + const [{ pushed }] = await a.sync.sync(); + assert.strictEqual(pushed, 1); + const manual = named('syncom.push')[2]; + assert.strictEqual(manual.attributes['syncom.push.reason'], 'manual'); + assert.strictEqual(manual.status.code, 0); + + const pushes = await points('syncom.push.ops'); + assert.strictEqual(point(pushes, { 'syncom.push.reason': 'handshake', 'syncom.result': 'ok' }).value.count, 1); + assert.strictEqual(point(pushes, { 'syncom.push.reason': 'relay', 'syncom.result': 'error' }).value.count, 1); + assert.strictEqual(point(pushes, { 'syncom.push.reason': 'manual', 'syncom.result': 'ok' }).value.sum, 1); + assertNoIdentityLabels(await collect()); + }); + + await t.test('a bootstrap is one span; the host rolls its log under spans of its own', async (t2) => { + const tracing = { A: createTracing(), B: createTracing() }; + const metrics = { A: createMetrics(), B: createMetrics() }; + const options = { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }; + const telemetryOf = (name) => ({ tracer: tracing[name].tracer, meter: metrics[name].meter }); + const mesh = await createMesh(t2, ['A', 'B'], [], createStorage(telemetryOf), options); + const a = mesh.get('A'); + const id = await a.insert({ n: 0 }); + for (let i = 0; i < 12; i++) await a.inc(id, 'hits', 1); + await a.insert({ other: true }); + assert.ok(a.log.floor > 1, 'history was compacted'); + + const rolls = tracing.A.named('syncom.log.roll'); + assert.ok(rolls.length >= 3); + assert.ok(rolls.every((span) => span.attributes['syncom.log.sealed'] === 2)); + assert.ok(rolls.at(-1).attributes['syncom.log.floor'] > 1); + assert.ok(rolls.some((span) => span.attributes['syncom.log.dropped'] >= 1)); + const segments = await metrics.A.points('syncom.log.segments'); + assert.strictEqual(point(segments, { 'syncom.log.event': 'sealed' }).value, rolls.length); + assert.ok(point(segments, { 'syncom.log.event': 'dropped' }).value >= 1); + assert.strictEqual((await metrics.A.points('syncom.log.append.duration'))[0].value.count, 14); + + const b = mesh.get('B'); + await b.sync.addNode({ url: 'mem://A' }); + await settle([a, b]); + assert.strictEqual((await b.get(id)).hits, 12); + const [bootstrap] = tracing.B.named('syncom.bootstrap'); + assert.strictEqual(bootstrap.attributes['syncom.peer.url'], 'mem://A'); + assert.strictEqual(bootstrap.attributes['syncom.peer.node'], a.nodeId); + assert.ok(bootstrap.attributes['syncom.pages'] >= 1); + assert.strictEqual(bootstrap.attributes['syncom.records'], 2); + assert.strictEqual(bootstrap.status.code, 0); + assert.strictEqual(point(await metrics.B.points('syncom.bootstraps'), { 'syncom.result': 'ok' }).value, 1); + // The joined records were merges: counted, never spanned. + const merges = point(await metrics.B.points('syncom.commits'), { + 'syncom.op.type': 'merge', + 'syncom.result': 'applied', + }); + assert.strictEqual(merges.value, 2); + assert.ok(tracing.B.named('syncom.commit').every((span) => span.attributes['syncom.op.type'] !== 'merge')); + }); + + await t.test('a failed bootstrap is an error span and an error sample', async (t2) => { + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const options = { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }; + const { storage: a } = await openTemp(t2, options); + const id = await a.insert({ n: 0 }); + for (let i = 0; i < 12; i++) await a.inc(id, 'hits', 1); + assert.ok(a.log.floor > 1); + const unit = a.sync.unit({ access: 'public' }); + const connect = () => { + const link = new MemoryLink(unit); + const call = link.call.bind(link); + link.call = (method, args) => + method === 'syncom/snapshot' ? Promise.reject(new Error('no snapshots')) : call(method, args); + return link; + }; + const { storage: b } = await openTemp(t2, { telemetry: { tracer, meter }, sync: { connect, autoStart: false } }); + b.sync.on('error', () => {}); + await b.sync.addNode({ url: 'mem://A' }); + await b.sync.start(); + await waitUntil(() => named('syncom.bootstrap').length === 1); + const [span] = named('syncom.bootstrap'); + assert.strictEqual(span.status.code, 2); + assert.strictEqual(span.status.message, 'no snapshots'); + assert.strictEqual(point(await points('syncom.bootstraps'), { 'syncom.result': 'error' }).value, 1); + }); +}); + +test('OpLog telemetry', async (t) => { + await t.test('a bare log takes the option too', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const { tracer, named } = createTracing(); + const { meter, points } = createMetrics(); + const log = await new syncom.OpLog(dir, { telemetry: { tracer, meter }, rollAfter: 2, retention: { ops: 4 } }); + for (let seq = 1; seq <= 8; seq++) { + await log.append([{ origin: 'n', seq, ts: seq, type: 'write', id: 'r', data: {} }]); + } + await log.close(); + const rolls = named('syncom.log.roll'); + assert.strictEqual(rolls.length, 4); + assert.ok(rolls.every((span) => span.attributes['syncom.log.sealed'] === 2)); + assert.strictEqual(rolls[0].attributes['syncom.log.dropped'], 0); + assert.strictEqual(rolls.at(-1).attributes['syncom.log.dropped'], 1); + assert.strictEqual(rolls.at(-1).attributes['syncom.log.floor'], log.floor); + const segments = await points('syncom.log.segments'); + assert.strictEqual(point(segments, { 'syncom.log.event': 'sealed' }).value, 4); + assert.strictEqual(point(segments, { 'syncom.log.event': 'dropped' }).value, 2); + assert.strictEqual((await points('syncom.log.append.duration'))[0].value.count, 8); + }); + + await t.test('an empty append is neither measured nor spanned', async (t2) => { + const dir = await createTemp(); + t2.after(() => cleanupTemp(dir)); + const { tracer, spans } = createTracing(); + const { meter, collect } = createMetrics(); + const log = await new syncom.OpLog(dir, { telemetry: { tracer, meter } }); + assert.deepStrictEqual(await log.append([]), []); + await log.close(); + assert.strictEqual(spans().length, 0); + assert.strictEqual((await collect()).length, 0); + }); +}); diff --git a/tests/wrpc.integration.test.js b/tests/wrpc.integration.test.js index e6f389e..f695d83 100644 --- a/tests/wrpc.integration.test.js +++ b/tests/wrpc.integration.test.js @@ -150,3 +150,87 @@ test('syncom over wrpc', { skip: wrpc ? false : '@alexify/wrpc unavailable' }, a assert.strictEqual(await bad.get(id), null); }); }); + +// The two packages meet in one trace: syncom opens `syncom.push` around the +// call, wrpc's client span nests under it and carries the context across +// the wire, wrpc's server span receives it, and `syncom.apply` runs inside +// that. Nothing in syncom touches a traceparent. +const otel = wrpc && + optional('@opentelemetry/api') && { + api: require('@opentelemetry/api'), + ...require('@opentelemetry/sdk-trace-base'), + ...require('@opentelemetry/context-async-hooks'), + ...require('@opentelemetry/core'), + }; + +test( + 'syncom over wrpc: one trace from a write to the peer apply', + { skip: otel ? false : 'wrpc or otel unavailable' }, + async (t) => { + const manager = new otel.AsyncLocalStorageContextManager().enable(); + otel.api.context.setGlobalContextManager(manager); + otel.api.propagation.setGlobalPropagator(new otel.W3CTraceContextPropagator()); + t.after(() => { + otel.api.context.disable(); + otel.api.propagation.disable(); + }); + // Each side exports to its own memory, so a shared trace id proves the + // context crossed the socket rather than leaking within the process. + const tracing = (name) => { + const exporter = new otel.InMemorySpanExporter(); + const provider = new otel.BasicTracerProvider({ spanProcessors: [new otel.SimpleSpanProcessor(exporter)] }); + return { tracer: provider.getTracer(name), spans: () => exporter.getFinishedSpans() }; + }; + const hostSide = tracing('host'); + const dialSide = tracing('dialer'); + const wire = { propagation: otel.api.propagation, context: otel.api.context }; + + const a = await openTemp(t, { telemetry: { tracer: hostSide.tracer } }); + const router = wrpc.defineRouter({ syncom: a.sync.unit({ access: 'public' }) }); + const server = new wrpc.Server({ + router, + host: '127.0.0.1', + port: 0, + protocol: 'http', + logger: false, + telemetry: { tracer: hostSide.tracer, ...wire }, + }); + await server.listen(); + t.after(() => server.close().catch(() => {})); + const url = `ws://127.0.0.1:${server.address().port}${server.rpc.basePath}`; + + const c = await openTemp(t, { + telemetry: { tracer: dialSide.tracer }, + sync: { connect: connect({ telemetry: { tracer: dialSide.tracer, ...wire } }), autoStart: false }, + }); + await c.sync.addNode({ url }); + await c.sync.start(); + // Greeted and subscribed first: a write that lands mid-handshake rides the + // handshake push AND a relay, and the chain below wants exactly one push. + await waitUntil(() => c.sync.peers().some((peer) => peer.subscribed)); + const id = await c.insert({ from: 'C' }); + await settle([a, c], { timeout: 10_000 }); + assert.deepStrictEqual(await a.get(id), { from: 'C' }); + + const one = (side, predicate) => { + const found = side.spans().filter(predicate); + const describe = (span) => + `${span.name}(${span.attributes['syncom.push.reason'] ?? span.attributes['syncom.ops.count'] ?? ''})`; + assert.strictEqual(found.length, 1, `expected one span, got ${found.map(describe).join(', ')}`); + return found[0]; + }; + const commit = one(dialSide, (span) => span.name === 'syncom.commit'); + const push = one(dialSide, (span) => span.name === 'syncom.push'); + const clientCall = one(dialSide, (span) => span.name === 'syncom/push'); + const serverCall = one(hostSide, (span) => span.name === 'syncom/push'); + const apply = one(hostSide, (span) => span.name === 'syncom.apply'); + assert.strictEqual(clientCall.kind, 2, 'CLIENT'); + assert.strictEqual(serverCall.kind, 1, 'SERVER'); + const chain = [commit, push, clientCall, serverCall, apply]; + for (let i = 1; i < chain.length; i++) { + assert.strictEqual(chain[i].parentSpanContext?.spanId, chain[i - 1].spanContext().spanId, chain[i].name); + assert.strictEqual(chain[i].spanContext().traceId, commit.spanContext().traceId, chain[i].name); + } + assert.strictEqual(apply.attributes['syncom.ops.applied'], 1); + }, +); From bb45ceddcfb709aea0f43d969530e5c5bbbeca47 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:15:11 +0300 Subject: [PATCH 08/67] feat: hook primitives and a contained event path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `src/hooks.js` (no requires โ€” it ships to the browser) brings HookError, normalizeHooks, runHook and notify. Every event the engine emits now goes out through notify(): invoked at once, never awaited, contained. A `change` listener that throws no longer fails the write it observed, one that writes back queues behind it instead of deadlocking the mutation queue, and an `error` with nobody listening is silent rather than an Emitter throw. The fire-and-forget emits in Record and SyncManager stop leaking rejections. HookError is exported now so the hook commits that follow can land their types alongside โ€” the consistency test wants every runtime export declared. Also: a peer hung up while its page was applying no longer writes its registry file after stop(), which raced whoever reopened the directory next. Co-Authored-By: Claude Fable 5.1 --- browser.d.ts | 1 + index.d.ts | 14 ++++ src/hooks.js | 66 +++++++++++++++++++ src/index.browser.js | 4 +- src/index.js | 4 +- src/storage.js | 20 +++--- src/sync.js | 19 ++++-- tests/hooks.test.js | 127 +++++++++++++++++++++++++++++++++++++ tests/index.test-d.ts | 7 ++ tests/storage.ops.test.js | 61 ++++++++++++++++++ tests/sync.manager.test.js | 23 +++++++ 11 files changed, 325 insertions(+), 21 deletions(-) create mode 100644 src/hooks.js create mode 100644 tests/hooks.test.js diff --git a/browser.d.ts b/browser.d.ts index c48b8ab..8a80967 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -82,6 +82,7 @@ export { isSyncLink, Storage, open, + HookError, } from './index.js'; // Web Crypto: async where node:crypto is sync. diff --git a/index.d.ts b/index.d.ts index c0de18b..a96af6a 100644 --- a/index.d.ts +++ b/index.d.ts @@ -74,6 +74,20 @@ export interface ApplyResult { rejected: Array<{ field: string; reason: 'register' | 'counter' }>; } +/** + * Thrown (rejected) by the operation a hook belongs to when that hook threw. + * `hook` names it, `cause` is the original error; the operation itself is + * described by the hook's documentation โ€” a `before*` hook aborts it, an + * `after*` hook fails the call while what it observed stays committed. + */ +export class HookError extends Error { + constructor(hook: string, cause: unknown); + readonly name: 'HookError'; + readonly code: 'HOOK_FAILED'; + readonly hook: string; + readonly cause: unknown; +} + export function applyOp(entry: Entry | null, op: SyncOp): ApplyResult; export function mergeEntry(a: Entry | null, b: Entry | null): Entry | null; export function isAlive(entry: Entry | null): boolean; diff --git a/src/hooks.js b/src/hooks.js new file mode 100644 index 0000000..81821c0 --- /dev/null +++ b/src/hooks.js @@ -0,0 +1,66 @@ +'use strict'; + +// Hooks and the contained event path. No requires on purpose: this file is +// reached from the browser bundle, and scripts/size.js fails on any import +// that is not a relative file. +// +// A hook is one function per name, configured up front (`open({ hooks })`) +// and awaited INSIDE the operation it belongs to: it can alter the operation +// or abort it by throwing. An event is what `on()` subscribes to from the +// outside: many listeners, none of which can affect the operation โ€” a +// listener that throws is logged and forgotten. + +class HookError extends Error { + constructor(hook, cause) { + super(`The ${hook} hook failed: ${cause?.message ?? cause}`, { cause }); + this.name = 'HookError'; + this.code = 'HOOK_FAILED'; + this.hook = hook; + Error.captureStackTrace?.(this, HookError); + } +} + +// Unknown names throw: a hook with a typo that silently never runs is a +// validation that silently never runs. One function per name โ€” a return +// value (a replacement, a verdict) could not be chained across a list. +const normalizeHooks = (hooks, allowed, label) => { + const result = {}; + for (const name of allowed) result[name] = null; + if (hooks === undefined || hooks === null) return result; + if (typeof hooks !== 'object' || Array.isArray(hooks)) throw new TypeError(`${label}: hooks must be an object`); + for (const key of Object.keys(hooks)) { + if (!allowed.includes(key)) throw new TypeError(`${label}: unknown hook '${key}'`); + const fn = hooks[key]; + if (fn === undefined || fn === null) continue; + if (typeof fn !== 'function') throw new TypeError(`${label}: hooks.${key} must be a function`); + result[key] = fn; + } + return result; +}; + +// Returns what the hook returns; a throw becomes a HookError naming the hook, +// with the original as `cause`. An absent hook is a no-op. +const runHook = async (hooks, name, ...args) => { + const fn = hooks[name]; + if (!fn) return undefined; + try { + return await fn(...args); + } catch (error) { + throw new HookError(name, error); + } +}; + +// The one way an event leaves the engine. Listeners run at once (the +// Emitter invokes them synchronously) but are never awaited, so a slow +// listener does not hold the mutation queue and one that writes back to the +// storage queues behind the write it heard about instead of deadlocking it. +// A rejection is contained: logged at debug, never surfaced to the caller. +// The count check is what keeps an unhandled `error` from throwing. +const notify = (emitter, logger, event, ...args) => { + if (emitter.listenerCount(event) === 0) return; + emitter.emit(event, ...args).catch((err) => { + logger.debug({ event: 'listener.error', name: event, err }); + }); +}; + +module.exports = { HookError, normalizeHooks, runHook, notify }; diff --git a/src/index.browser.js b/src/index.browser.js index a7b312a..ff6b2ad 100644 --- a/src/index.browser.js +++ b/src/index.browser.js @@ -15,5 +15,7 @@ const sync = require('./sync.js'); const log = require('./log.js'); const { applyOp, mergeEntry, isAlive } = require('./ops.js'); const ops = { applyOp, mergeEntry, isAlive }; +const { HookError } = require('./hooks.js'); +const hooks = { HookError }; -module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops }; +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops, ...hooks }; diff --git a/src/index.js b/src/index.js index e07aa69..b095490 100644 --- a/src/index.js +++ b/src/index.js @@ -10,5 +10,7 @@ const sync = require('./sync.js'); const log = require('./log.js'); const { applyOp, mergeEntry, isAlive } = require('./ops.js'); const ops = { applyOp, mergeEntry, isAlive }; +const { HookError } = require('./hooks.js'); +const hooks = { HookError }; -module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops }; +module.exports = { ...chain, ...contract, ...crdt, ...crypto, ...fsi, ...storage, ...sync, ...log, ...ops, ...hooks }; diff --git a/src/storage.js b/src/storage.js index 238ae11..6ff3bfb 100644 --- a/src/storage.js +++ b/src/storage.js @@ -7,6 +7,7 @@ const { SyncManager } = require('./sync.js'); const { OpLog } = require('./log.js'); const { applyOp, isAlive, latestClock } = require('./ops.js'); const { Clock, opKey, keyScope, vectorHas, vectorNext, vectorAdvance } = require('./vector.js'); +const { notify } = require('./hooks.js'); const fsi = require('./fsi.js'); const { createTelemetry, traced, now } = require('./telemetry.js'); @@ -100,7 +101,7 @@ class Record extends Emitter { this.#data[key] = newValue; this.#changes[key] = newValue; const delta = { [key]: newValue }; - this.emit('update', this.#data, delta); + notify(this, this.#storage.logger, 'update', this.#data, delta); }, enumerable: true, configurable: true, @@ -116,12 +117,12 @@ class Record extends Emitter { if (data === null) { this.#data = null; this.#changes = {}; - if (source !== this) this.emit('update', null, delta); + if (source !== this) notify(this, this.#storage.logger, 'update', null, delta); return; } this.#adopt(data); for (const key in this.#changes) this.#data[key] = this.#changes[key]; - if (source !== this) this.emit('update', this.#data, delta); + if (source !== this) notify(this, this.#storage.logger, 'update', this.#data, delta); } async data() { @@ -508,7 +509,7 @@ class Storage extends Emitter { // Announced after everything is durable. `via` is null for a local // write; a merge made from a peer's snapshot names that peer, so the // relay does not hand its own state straight back to it. - await this.emit('change', op, via); + notify(this, this.#logger, 'change', op, via); return op; } } @@ -749,12 +750,7 @@ class Storage extends Emitter { if (rejected.length > 0) { this.#logger.warn({ event: 'sync.op.rejected', id: op.id, rejected }); this.#otel.recordRejected(rejected[0].reason); - // Guarded: Emitter throws on an unheard `error`, and a throw here - // would fail the push before the vector moved โ€” the peer would - // then offer the same page again on every hello, forever. - if (this.listenerCount('error') > 0) { - await this.emit('error', Object.assign(new Error(`Refused op on ${op.id}`), { op, rejected })); - } + notify(this, this.#logger, 'error', Object.assign(new Error(`Refused op on ${op.id}`), { op, rejected })); } vectorAdvance(vector, op); accepted.push(op); @@ -773,7 +769,7 @@ class Storage extends Emitter { this.#emitUpdate(id, isAlive(entry) ? entry.data : null, deltas.get(id)); } } - for (const op of accepted) await this.emit('change', op, via); + for (const op of accepted) notify(this, this.#logger, 'change', op, via); return { applied: accepted.length, duplicates, gaps }; } } @@ -836,7 +832,7 @@ class Storage extends Emitter { op.seq = (known[opKey(op)] ?? 0) + 1; await this.#log.append([op]); await this.#log.advance(news); - await this.emit('change', op, via); + notify(this, this.#logger, 'change', op, via); return op; }); } diff --git a/src/sync.js b/src/sync.js index 63b1f35..5857ee8 100644 --- a/src/sync.js +++ b/src/sync.js @@ -27,6 +27,7 @@ const { packOps, unpackOps } = require('./ops.js'); const { vectorHas, keyScope } = require('./vector.js'); const { isSyncLink } = require('./link.js'); const { createTelemetry, traced } = require('./telemetry.js'); +const { notify } = require('./hooks.js'); const REMOTE = 'remote'; const PROTOCOL = 1; @@ -260,11 +261,11 @@ class SyncManager extends Emitter { close: () => { peer.connected = false; peer.subscription = null; - void this.emit('peer', { url: node.url, connected: false }); + notify(this, this.#storage.logger, 'peer', { url: node.url, connected: false }); }, reconnect: () => { peer.connected = true; - void this.emit('peer', { url: node.url, connected: true, reconnected: true }); + notify(this, this.#storage.logger, 'peer', { url: node.url, connected: true, reconnected: true }); // The link restored the subscription itself (when the host still // knows us); what it cannot restore is the outbound half. this.#handshake(peer, { resubscribe: !peer.subscription }).catch((error) => this.#fail(peer, error)); @@ -296,7 +297,7 @@ class SyncManager extends Emitter { #fail(peer, error) { if (peer.closed) return; this.#storage.logger.warn({ event: 'sync.error', url: peer.node.url, err: error }); - if (this.listenerCount('error') > 0) void this.emit('error', Object.assign(error, { url: peer.node.url })); + notify(this, this.#storage.logger, 'error', Object.assign(error, { url: peer.node.url })); } // Single-flight: a reconnect and a refused restored subscription can both @@ -333,7 +334,7 @@ class SyncManager extends Emitter { peer.remote = { id: answer.node, vector: answer.vector ?? {}, epoch: answer.epoch }; node.remote = { id: answer.node, epoch: answer.epoch }; await this.#saveNode(node); - await this.emit('peer', { url: node.url, connected: true, node: answer.node }); + notify(this, this.#storage.logger, 'peer', { url: node.url, connected: true, node: answer.node }); await this.#push(peer, 'handshake'); if (resubscribe && !peer.subscription) this.#subscribe(peer); } @@ -403,7 +404,7 @@ class SyncManager extends Emitter { if (sent > 0) { peer.node.lastSync = Date.now(); await this.#saveNode(peer.node); - await this.emit('sync', { url: peer.node.url, pushed: sent }); + notify(this, this.#storage.logger, 'sync', { url: peer.node.url, pushed: sent }); } return sent; } @@ -440,10 +441,14 @@ class SyncManager extends Emitter { if (value.bootstrap) return void (await this.#bootstrap(peer)); const ops = unpackOps(value.ops); const result = await this.#storage.applyRemote(ops, { via: peer, scopes: peer.node.scopes }); + // Hung up while applying: the cursor is not saved, so a restart replays + // this page as duplicates โ€” better than a registry write racing whoever + // reopens the directory next. + if (peer.closed) return; if (eventId) peer.node.cursor = eventId; peer.node.lastSync = Date.now(); await this.#saveNode(peer.node); - await this.emit('sync', { url: peer.node.url, received: ops.length, ...result }); + notify(this, this.#storage.logger, 'sync', { url: peer.node.url, received: ops.length, ...result }); } // Bootstrap: page through the host's whole state, joining each page, then @@ -506,7 +511,7 @@ class SyncManager extends Emitter { node.cursor = startEventId; node.lastSync = Date.now(); await this.#saveNode(node); - await this.emit('bootstrap', { url: node.url, records: total }); + notify(this, this.#storage.logger, 'bootstrap', { url: node.url, records: total }); if (!peer.closed) this.#subscribe(peer); return { pages, records: total }; } diff --git a/tests/hooks.test.js b/tests/hooks.test.js new file mode 100644 index 0000000..7a186db --- /dev/null +++ b/tests/hooks.test.js @@ -0,0 +1,127 @@ +'use strict'; + +// The hook primitives and the contained event path, in isolation. + +const test = require('node:test'); +const assert = require('node:assert'); +const { Emitter } = require('../src/utils.js'); +const { HookError, normalizeHooks, runHook, notify } = require('../src/hooks.js'); + +const ALLOWED = ['beforeWrite', 'afterWrite']; + +test('normalizeHooks', async (t) => { + await t.test('absent hooks normalize to a map of nulls', () => { + assert.deepStrictEqual(normalizeHooks(undefined, ALLOWED, 'Storage'), { beforeWrite: null, afterWrite: null }); + assert.deepStrictEqual(normalizeHooks(null, ALLOWED, 'Storage'), { beforeWrite: null, afterWrite: null }); + assert.deepStrictEqual(normalizeHooks({}, ALLOWED, 'Storage'), { beforeWrite: null, afterWrite: null }); + }); + + await t.test('keeps every allowed name, functions or null', () => { + const beforeWrite = () => {}; + const hooks = normalizeHooks({ beforeWrite, afterWrite: undefined }, ALLOWED, 'Storage'); + assert.deepStrictEqual(hooks, { beforeWrite, afterWrite: null }); + }); + + await t.test('an unknown name, a non-object or a non-function throws a TypeError naming the label', () => { + assert.throws(() => normalizeHooks({ beforeWrtie: () => {} }, ALLOWED, 'Storage'), { + name: 'TypeError', + message: "Storage: unknown hook 'beforeWrtie'", + }); + assert.throws(() => normalizeHooks({ beforeWrite: 'nope' }, ALLOWED, 'SyncManager'), { + name: 'TypeError', + message: 'SyncManager: hooks.beforeWrite must be a function', + }); + assert.throws(() => normalizeHooks('nope', ALLOWED, 'Storage'), /Storage: hooks must be an object/); + assert.throws(() => normalizeHooks([], ALLOWED, 'Storage'), /Storage: hooks must be an object/); + }); +}); + +test('runHook', async (t) => { + await t.test('an absent hook is a no-op that resolves undefined', async () => { + assert.strictEqual(await runHook({ beforeWrite: null }, 'beforeWrite', 1), undefined); + }); + + await t.test('passes the arguments through and returns what the hook returns', async () => { + const seen = []; + const hooks = { + beforeWrite: async (...args) => { + seen.push(args); + return 'replacement'; + }, + }; + assert.strictEqual(await runHook(hooks, 'beforeWrite', 'a', 2), 'replacement'); + assert.deepStrictEqual(seen, [['a', 2]]); + }); + + await t.test('a throw becomes a HookError naming the hook, with the original as cause', async () => { + const original = new Error('nope'); + const hooks = { beforeWrite: () => Promise.reject(original) }; + const error = await runHook(hooks, 'beforeWrite').then( + () => assert.fail('should reject'), + (e) => e, + ); + assert.ok(error instanceof HookError); + assert.ok(error instanceof Error); + assert.strictEqual(error.name, 'HookError'); + assert.strictEqual(error.code, 'HOOK_FAILED'); + assert.strictEqual(error.hook, 'beforeWrite'); + assert.strictEqual(error.cause, original); + assert.strictEqual(error.message, 'The beforeWrite hook failed: nope'); + // A rejection with a non-error is wrapped the same way. + // oxlint-disable-next-line prefer-promise-reject-errors -- a non-Error cause is the point + const sync = await runHook({ afterWrite: () => Promise.reject('text') }, 'afterWrite').catch((e) => e); + assert.strictEqual(sync.cause, 'text'); + assert.strictEqual(sync.message, 'The afterWrite hook failed: text'); + }); +}); + +test('notify', async (t) => { + const stubLogger = () => { + const entries = []; + return { entries, debug: (entry) => entries.push(entry), info() {}, warn() {}, error() {} }; + }; + + await t.test('with no listener nothing is emitted โ€” not even error, which the Emitter would throw on', () => { + const emitter = new Emitter(); + const logger = stubLogger(); + assert.throws(() => emitter.emit('error', new Error('unheard')), /Unhandled error/); + assert.doesNotThrow(() => notify(emitter, logger, 'error', new Error('unheard'))); + assert.strictEqual(notify(emitter, logger, 'change', 1), undefined); + assert.strictEqual(logger.entries.length, 0); + }); + + await t.test('listeners run at once with the arguments, and are not awaited', async () => { + const emitter = new Emitter(); + const seen = []; + let resolveSlow; + emitter.on('change', (a, b) => seen.push([a, b])); + emitter.on('change', () => new Promise((resolve) => (resolveSlow = resolve))); + notify(emitter, stubLogger(), 'change', 'op', null); + assert.deepStrictEqual(seen, [['op', null]], 'the synchronous listener already ran'); + resolveSlow(); + }); + + await t.test('a throwing or rejecting listener is contained and logged at debug', async () => { + const emitter = new Emitter(); + const logger = stubLogger(); + const seen = []; + emitter.on('change', () => { + throw new Error('sync boom'); + }); + emitter.on('change', () => seen.push('other listener still ran')); + emitter.once('error', async () => { + throw new Error('async boom'); + }); + assert.doesNotThrow(() => notify(emitter, logger, 'change', 1)); + assert.doesNotThrow(() => notify(emitter, logger, 'error', new Error('reported'))); + await new Promise((resolve) => setImmediate(resolve)); + assert.deepStrictEqual(seen, ['other listener still ran']); + assert.deepStrictEqual( + logger.entries.map((entry) => [entry.event, entry.name, entry.err.message]), + [ + ['listener.error', 'change', 'sync boom'], + ['listener.error', 'error', 'async boom'], + ], + ); + }); +}); diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 1e3b9ee..02bc341 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -151,3 +151,10 @@ expectError(syncom.open({ telemetry: { tracer: 'a tracer' } })); expectType>(new syncom.OpLog('p', { telemetry: { api: {} } })); declare const telemetryStorage: Storage; expectType(telemetryStorage.telemetry); + +// hooks +const hookError = new syncom.HookError('beforeWrite', new Error('x')); +expectType<'HOOK_FAILED'>(hookError.code); +expectType(hookError.hook); +expectType(hookError.cause); +expectAssignable(hookError); diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js index 30f0b8a..fab1d26 100644 --- a/tests/storage.ops.test.js +++ b/tests/storage.ops.test.js @@ -334,3 +334,64 @@ test('Storage: the seams telemetry shares', async (t) => { ); }); }); + +test('Storage: events are contained', async (t) => { + const collect = () => { + const entries = []; + return { entries, logger: { debug: (entry) => entries.push(entry), info() {}, warn() {}, error() {} } }; + }; + + await t.test('a change listener that throws does not fail the write it observed', async (t2) => { + const { entries, logger } = collect(); + const { storage } = await openTemp(t2, { logger }); + const seen = []; + storage.on('change', () => { + throw new Error('listener boom'); + }); + storage.on('change', (op) => seen.push(op.type)); + const id = await storage.insert({ a: 1 }); + assert.deepStrictEqual(await storage.get(id), { a: 1 }); + assert.deepStrictEqual(seen, ['write']); + const remote = await storage.applyRemote([remoteOp('peer', 1, 'write', 'r1', { data: { b: 1 } })]); + assert.strictEqual(remote.applied, 1); + await new Promise((resolve) => setImmediate(resolve)); + const logged = entries.filter((entry) => entry.event === 'listener.error'); + assert.strictEqual(logged.length, 2); + assert.strictEqual(logged[0].name, 'change'); + assert.strictEqual(logged[0].err.message, 'listener boom'); + }); + + await t.test('a change listener that writes back queues behind the write instead of deadlocking', async (t2) => { + const { storage } = await openTemp(t2); + const id = await storage.insert({ n: 0 }); + let echoed = null; + storage.once('change', (op) => { + echoed = storage.update(op.id, { echoed: true }); + }); + await storage.update(id, { n: 1 }); + await echoed; + assert.deepStrictEqual(await storage.get(id), { n: 1, echoed: true }); + }); + + await t.test('a record update listener that throws does not break the setter or the refresh', async (t2) => { + const { entries, logger } = collect(); + const { storage } = await openTemp(t2, { logger }); + const id = await storage.insert({ title: 'a' }); + const record = await storage.record(id); + const rejections = []; + const onRejection = (reason) => rejections.push(reason); + process.on('unhandledRejection', onRejection); + t2.after(() => process.off('unhandledRejection', onRejection)); + record.on('update', () => { + throw new Error('update boom'); + }); + assert.doesNotThrow(() => { + record.title = 'b'; + }); + await storage.applyRemote([remoteOp('peer', 1, 'set', id, { data: { views: 1 } })]); + await new Promise((resolve) => setImmediate(resolve)); + assert.strictEqual(record.views, 1); + assert.deepStrictEqual(rejections, []); + assert.strictEqual(entries.filter((entry) => entry.event === 'listener.error').length, 2); + }); +}); diff --git a/tests/sync.manager.test.js b/tests/sync.manager.test.js index ba1022a..53b914c 100644 --- a/tests/sync.manager.test.js +++ b/tests/sync.manager.test.js @@ -211,3 +211,26 @@ test('SyncManager edges', async (t) => { unit.changes.handler = original; }); }); + +test('SyncManager: events are contained', async (t) => { + await t.test('listeners that throw do not break the handshake or the feed', async (t2) => { + const entries = []; + const logger = { debug: (entry) => entries.push(entry), info() {}, warn() {}, error() {} }; + const host = await openTemp(t2, { logger }); + const unit = host.sync.unit({ access: 'public' }); + const dialer = await openTemp(t2, { logger, sync: { autoStart: false, connect: () => new MemoryLink(unit) } }); + for (const event of ['peer', 'sync', 'bootstrap', 'error']) { + dialer.sync.on(event, () => { + throw new Error(`${event} boom`); + }); + } + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + const id = await dialer.insert({ through: true }); + await settle([host, dialer]); + assert.deepStrictEqual(await host.get(id), { through: true }); + await new Promise((resolve) => setImmediate(resolve)); + const names = new Set(entries.filter((e) => e.event === 'listener.error').map((e) => e.name)); + assert.ok(names.has('peer') && names.has('sync'), `logged: ${[...names].join(', ')}`); + }); +}); From 5a8151292862e72e052f2011b502cf48b0bfc80b Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:20:23 +0300 Subject: [PATCH 09/67] fix: serialize a node's registry writes and stop saving after hang-up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #saveNode dropped itself from the pending map before its write began, so stop() only waited for writes that had not started, and a second save during one in flight wrote the same file concurrently โ€” a reopen could read half a JSON document. Writes now chain per node, stop() awaits the chain, and a peer that was disconnected mid-push or mid-bootstrap does not write its file afterwards. Co-Authored-By: Claude Fable 5.1 --- src/sync.js | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/src/sync.js b/src/sync.js index 5857ee8..b2dd6d8 100644 --- a/src/sync.js +++ b/src/sync.js @@ -172,18 +172,31 @@ class SyncManager extends Emitter { else if (this.#started) this.#connect(node); } - // Registry writes are coalesced per node: cursors move on every batch. + // Registry writes are coalesced AND serialized per node: cursors move on + // every batch, so calls that land while a write is pending share it, and + // one that lands while a write is in flight runs after it โ€” two writes to + // one file interleaving is how a reopen reads half a JSON document. #saveNode(node) { - if (!this.#nodes.has(node.url)) return Promise.resolve(); - let pending = this.#saving.get(node.url); - if (pending) return pending; - pending = Promise.resolve().then(async () => { - this.#saving.delete(node.url); - if (!this.#nodes.has(node.url)) return; - await fsi.write(this.#path, fileName(node.url), JSON.stringify(node)); + const { url } = node; + if (!this.#nodes.has(url)) return Promise.resolve(); + let state = this.#saving.get(url); + if (!state) { + state = { last: Promise.resolve(), next: null }; + this.#saving.set(url, state); + } + if (state.next) return state.next; + const write = state.last.then(async () => { + state.next = null; + if (!this.#nodes.has(url)) return; + await fsi.write(this.#path, fileName(url), JSON.stringify(node)); }); - this.#saving.set(node.url, pending); - return pending; + state.next = write; + state.last = write + .catch(() => {}) + .then(() => { + if (state.next === null) this.#saving.delete(url); + }); + return write; } // --------------------------------------------------------------------- @@ -203,7 +216,7 @@ class SyncManager extends Emitter { async stop() { this.#started = false; for (const url of Array.from(this.#peers.keys())) this.#disconnect(url); - await Promise.all(Array.from(this.#saving.values())); + await Promise.all(Array.from(this.#saving.values(), (state) => state.last)); } /** Pushes whatever the connected peers lack right now; resolves when done. */ @@ -401,7 +414,7 @@ class SyncManager extends Emitter { if (answer?.vector) peer.remote.vector = answer.vector; sent += page.length; } - if (sent > 0) { + if (sent > 0 && !peer.closed) { peer.node.lastSync = Date.now(); await this.#saveNode(peer.node); notify(this, this.#storage.logger, 'sync', { url: peer.node.url, pushed: sent }); @@ -508,6 +521,7 @@ class SyncManager extends Emitter { if (startVector) { await this.#storage.adoptVector(scopedVector(startVector, node.scopes), { via: peer, from: peer.remote.id }); } + if (peer.closed) return { pages, records: total }; node.cursor = startEventId; node.lastSync = Date.now(); await this.#saveNode(node); From 1b72a84ef06620e78b0bc0302444ab2a4a5052f6 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:20:23 +0300 Subject: [PATCH 10/67] feat(storage): beforeWrite/afterWrite and beforeRead/afterRead hooks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `open({ hooks })` โ€” one function per name, awaited inside the operation. beforeWrite sees the op being made and the record it lands on, may replace the data of a write or a set, and refuses the write by throwing (HookError; nothing stamped, logged or written). afterWrite sees the durable op; a throw fails the call but the op stays and is announced. beforeRead/afterRead wrap get(): the second may redact or hide what the caller sees, while swap, snapshot and the cache keep reading raw data. Hooks run inside the mutation queue, so a write issued from a hook is refused rather than left to deadlock. The first encrypted write now goes through the same commit path (an `encrypted` option), so the hooks see every write and "one write path" is literally true. Budget ratcheted 17 โ†’ 18 KB. Co-Authored-By: Claude Fable 5.1 --- browser.d.ts | 6 + index.d.ts | 93 ++++++++++ scripts/size.js | 5 +- src/storage.js | 168 +++++++++++------ tests/index.test-d.ts | 30 +++ tests/storage.hooks.browser.test.js | 22 +++ tests/storage.hooks.test.js | 24 +++ tests/suites/hooks.js | 274 ++++++++++++++++++++++++++++ 8 files changed, 567 insertions(+), 55 deletions(-) create mode 100644 tests/storage.hooks.browser.test.js create mode 100644 tests/storage.hooks.test.js create mode 100644 tests/suites/hooks.js diff --git a/browser.d.ts b/browser.d.ts index 8a80967..f696771 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -26,6 +26,12 @@ export type { SchemaEntities, Schema, StorageOptions, + StorageHooks, + WriteContext, + ReadContext, + ApplyContext, + ApplyBatchContext, + PeerRef, Collections, Reference, SyncomRecord, diff --git a/index.d.ts b/index.d.ts index a96af6a..6d56fb3 100644 --- a/index.d.ts +++ b/index.d.ts @@ -307,6 +307,97 @@ export interface SyncOptions { log?: { rollAfter?: number }; } +/** What a write hook sees: the op being made, and the record it lands on. */ +export interface WriteContext { + type: Exclude; + id: string; + /** The whole record for `write`, the delta for `set`, the joined data for `merge`. */ + data?: Record; + /** `inc` only. */ + key?: string; + /** `inc` only. */ + value?: number; + /** `merge` only: the metadata being joined. */ + meta?: EntryMeta; + /** The record before the write โ€” `null` when absent or deleted. Read-only. */ + current: Record | null; + /** Sealed for this node: the write is encrypted, or the record already is. */ + encrypted: boolean; + /** The Record whose `save()` this is, or `null`. */ + source: SyncomRecord | null; + /** The link a `merge` came in on; `null` for a local write. */ + via: unknown; + /** The peer a `merge` came from; `null` for a local write. */ + from: string | null; +} + +export interface ReadContext { + id: string; +} + +/** What a remote op faces before it is applied here. */ +export interface ApplyContext { + op: SyncOp; + /** The link the op arrived on. */ + via: unknown; + /** Who sent it โ€” `url` is `null` when this node is the host. */ + peer: PeerRef | null; + /** The record before the op โ€” `null` when absent or deleted. Read-only. */ + current: Record | null; +} + +export interface PeerRef { + url: string | null; + node: string; +} + +/** One `applyRemote` call, after its batch is durable. */ +export interface ApplyBatchContext extends ApplyRemoteResult { + via: unknown; + peer: PeerRef | null; + /** Every accepted op โ€” the skipped ones included. */ + ops: SyncOp[]; +} + +/** + * Hooks run INSIDE the storage's operations: awaited, one function per + * name, able to alter the operation or abort it by throwing (the operation + * then rejects with a HookError). They also run inside the mutation queue, + * so a hook that awaits holds every other write, and a write issued from a + * hook is refused. Events (`on`) are the contained, observe-only alternative. + */ +export interface StorageHooks { + /** + * Before a local write is stamped and applied. Return a plain object to + * replace `data` (`write` and `set` only); throw to refuse the write โ€” + * nothing is stamped, logged or written. + */ + beforeWrite?: (context: WriteContext) => void | Record | Promise>; + /** + * After the write is durable โ€” `op` is the op it became, `null` for an + * encrypted (local) write. Not called when the write changed nothing. A + * throw fails the call, but the op stays committed and announced. + */ + afterWrite?: (context: WriteContext, op: SyncOp | null) => void | Promise; + /** Before `get(id)` reads; throw to refuse the read. */ + beforeRead?: (context: ReadContext) => void | Promise; + /** + * After `get(id)` read. Return anything but `undefined` (including `null`) + * to replace what the caller sees; the record itself is untouched, and + * `swap`, `snapshot` and `getCachedData` keep reading the raw data. + */ + afterRead?: (context: ReadContext, data: unknown) => unknown | Promise; + /** + * Before a peer's op is applied. Return `false` to skip it: the op is + * still logged and relayed (the vector must move, or the peer would offer + * it forever), only this node's record is left alone. A throw aborts the + * whole batch before anything is persisted. + */ + beforeApply?: (context: ApplyContext) => void | boolean | Promise; + /** After a batch is durable; a throw fails the call, the batch stays. */ + afterApply?: (context: ApplyBatchContext) => void | Promise; +} + export interface StorageOptions { path?: string; name?: string; @@ -318,6 +409,8 @@ export interface StorageOptions { encrypted?: boolean; /** Injected OpenTelemetry โ€” see SyncomTelemetryOptions. */ telemetry?: SyncomTelemetryOptions; + /** Lifecycle hooks โ€” see StorageHooks. */ + hooks?: StorageHooks; } // The collections a schema turns into properties on the storage object: diff --git a/scripts/size.js b/scripts/size.js index 8c590e7..fb693d8 100644 --- a/scripts/size.js +++ b/scripts/size.js @@ -32,8 +32,9 @@ const ENTRIES = [ // replication engine (the op model, the persisted log, the sync protocol // that makes `storage.sync` move data) earned the raise to 14 KB; the // injected OpenTelemetry writer (src/telemetry.js and its call sites in - // storage, log and sync) the raise to 17 KB. - { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 17 }, + // storage, log and sync) the raise to 17 KB; hooks (src/hooks.js and the + // write, read, apply and sync seams they run in) the raise to 18 KB. + { label: 'main entry โ€” browser (@alexify/syncom)', entry: 'browser.js', platform: 'browser', budget: 18 }, { label: 'main entry โ€” node (@alexify/syncom)', entry: 'index.js', platform: 'node' }, // Platform-neutral (no node builtins, no browser-only globals): one file // serves both, bundled as a browser entry so the self-contained check diff --git a/src/storage.js b/src/storage.js index 6ff3bfb..23fb391 100644 --- a/src/storage.js +++ b/src/storage.js @@ -7,13 +7,17 @@ const { SyncManager } = require('./sync.js'); const { OpLog } = require('./log.js'); const { applyOp, isAlive, latestClock } = require('./ops.js'); const { Clock, opKey, keyScope, vectorHas, vectorNext, vectorAdvance } = require('./vector.js'); -const { notify } = require('./hooks.js'); +const { notify, runHook, normalizeHooks } = require('./hooks.js'); const fsi = require('./fsi.js'); const { createTelemetry, traced, now } = require('./telemetry.js'); const NODE_FILE = 'node.json'; const GLOBAL_SCOPE = '*'; +// The hooks a storage accepts (`open({ hooks })`). Writes and reads run +// their hooks inside this storage; the apply pair runs per peer batch. +const STORAGE_HOOKS = ['beforeWrite', 'afterWrite', 'beforeRead', 'afterRead', 'beforeApply', 'afterApply']; + // Module-private entry points between Storage and Record: a Record commits // through the same path every other write takes and is refreshed when the // record changes underneath it. Symbols keep both off the public surface. @@ -205,6 +209,8 @@ class Storage extends Emitter { #clock = new Clock(); #scopeOf; #encrypted = false; + #hooks; + #inHook = 0; #queue = Promise.resolve(); #closed = false; @@ -230,6 +236,7 @@ class Storage extends Emitter { // Encrypt every record at rest by default; a per-call `{ encrypted }` // still overrides. Encrypted records are local โ€” see saveData(). this.#encrypted = options.encrypted === true; + this.#hooks = normalizeHooks(options.hooks, STORAGE_HOOKS, 'Storage'); const scope = options.sync?.scope; this.#scopeOf = typeof scope === 'function' ? scope : () => GLOBAL_SCOPE; return this.#init(); @@ -363,11 +370,33 @@ class Storage extends Emitter { // interleaved writers would race all three. #run(task) { if (this.#closed) return Promise.reject(new Error('Storage is closed')); + // A hook runs inside the queue (or, for a read, while a queued hook may + // be running); a write it issues would wait for the hook that waits for + // it. Refused here, where the deadlock would otherwise begin. + if (this.#inHook > 0) { + return Promise.reject( + new Error( + 'Cannot write to the storage from inside a hook: the write would wait for the hook that waits for it', + ), + ); + } const result = this.#queue.then(task, task); this.#queue = result.catch(() => {}); return result; } + // Every hook of this storage goes through here, so the guard above knows + // one is running. Returns what the hook returns; a throw is a HookError. + async #hook(name, ...args) { + if (!this.#hooks[name]) return undefined; + this.#inHook++; + try { + return await runHook(this.#hooks, name, ...args); + } finally { + this.#inHook--; + } + } + #emitUpdate(id, data, delta, source = null) { const record = this.#records.get(id); if (record) record[REFRESH](data, delta, source); @@ -461,86 +490,110 @@ class Storage extends Emitter { } } - async #commit(type, id, payload, { source = null, via = null, from = null } = {}) { - { - const current = await this.#load(id); - if (current?.encrypted) { - // An encrypted record has no op log: writes to it stay local. Set - // and write map onto the encrypted save; a counter needs ops. - if (type === 'write') return void (await this.#writeEncrypted(id, payload.data, source)); - if (type === 'set') return void (await this.#writeEncrypted(id, { ...current.data, ...payload.data }, source)); - if (type === 'delete') return void (await this.#removeEncrypted(id, source)); - throw new Error(`Record ${id} is encrypted; ${type} needs a replicated record`); + async #commit(type, id, payload, { source = null, via = null, from = null, encrypted = false } = {}) { + const current = await this.#load(id); + if (encrypted && current && !current.encrypted) { + throw new Error(`Record ${id} is replicated and cannot become encrypted`); + } + // Sealed: this write is encrypted, or the record already is. An + // encrypted record has no op log โ€” writes to it stay local. + const sealed = encrypted || current?.encrypted === true; + const visible = current && (current.encrypted || isAlive(current)) ? current.data : null; + const ctx = { type, id, ...payload, current: visible, encrypted: sealed, source, via, from }; + const replacement = await this.#hook('beforeWrite', ctx); + if (replacement !== undefined) { + if (type !== 'write' && type !== 'set') { + throw new TypeError('beforeWrite may only replace data for write and set'); } - const data = payload.data ?? current?.data ?? {}; - const scope = this.#scope(id, data); - // Stamp above everything this node already holds for the record โ€” its - // entry and, for a merge, the one being joined โ€” so a local write never - // sorts below state the node knows about and is silently dropped. The - // log seeds the clock at open; this covers what the log no longer has. - this.#observe(latestClock(current?.meta)); - if (type === 'merge') this.#observe(latestClock(payload.meta)); - const op = { origin: this.#nodeId, seq: 0, ts: this.#clock.now(), type, id }; - if (scope !== GLOBAL_SCOPE) op.scope = scope; - if (from) op.from = from; - // Sequence numbers run per stream (see vector.js opKey): the scope's, - // or the bootstrap stream for a merge. - op.seq = (this.#log.vector[opKey(op)] ?? 0) + 1; - if (payload.data !== undefined) op.data = payload.data; - if (payload.meta !== undefined) op.meta = payload.meta; - if (payload.key !== undefined) op.key = payload.key; - if (payload.value !== undefined) op.value = payload.value; - - const { entry, changed, delta, rejected } = applyOp(current, op); - if (rejected.length > 0) { - const [{ field, reason }] = rejected; - const hint = reason === 'counter' ? 'a counter; use inc()' : 'a register; use update()'; - throw new Error(`Field ${field} of record ${id} is ${hint}`); + if (!isPlainObject(replacement)) throw new TypeError('beforeWrite must return a plain object or undefined'); + payload = { ...payload, data: replacement }; + ctx.data = replacement; + } + if (sealed) { + // Set and write map onto the encrypted save; a counter needs ops. + let data; + if (type === 'write') data = payload.data; + else if (type === 'set') data = { ...(current?.data ?? {}), ...payload.data }; + else if (type === 'delete') data = null; + else throw new Error(`Record ${id} is encrypted; ${type} needs a replicated record`); + if (data === null) await this.#removeEncrypted(id); + else await this.#writeEncrypted(id, data); + try { + await this.#hook('afterWrite', ctx, null); + } finally { + this.#emitUpdate(id, data, data === null ? {} : { ...data }, source); } - if (!changed) return null; + return undefined; + } + const data = payload.data ?? current?.data ?? {}; + const scope = this.#scope(id, data); + // Stamp above everything this node already holds for the record โ€” its + // entry and, for a merge, the one being joined โ€” so a local write never + // sorts below state the node knows about and is silently dropped. The + // log seeds the clock at open; this covers what the log no longer has. + this.#observe(latestClock(current?.meta)); + if (type === 'merge') this.#observe(latestClock(payload.meta)); + const op = { origin: this.#nodeId, seq: 0, ts: this.#clock.now(), type, id }; + if (scope !== GLOBAL_SCOPE) op.scope = scope; + if (from) op.from = from; + // Sequence numbers run per stream (see vector.js opKey): the scope's, + // or the bootstrap stream for a merge. + op.seq = (this.#log.vector[opKey(op)] ?? 0) + 1; + if (payload.data !== undefined) op.data = payload.data; + if (payload.meta !== undefined) op.meta = payload.meta; + if (payload.key !== undefined) op.key = payload.key; + if (payload.value !== undefined) op.value = payload.value; + + const { entry, changed, delta, rejected } = applyOp(current, op); + if (rejected.length > 0) { + const [{ field, reason }] = rejected; + const hint = reason === 'counter' ? 'a counter; use inc()' : 'a register; use update()'; + throw new Error(`Field ${field} of record ${id} is ${hint}`); + } + if (!changed) return null; - const [{ seq: position }] = await this.#log.append([op]); - const hash = await calculateHash(entry.data); - const blockHash = await this.#chain.addBlock({ id, hash, from: this.#log.formatId(position) }); - await this.#writeEntry(id, entry, blockHash); + const [{ seq: position }] = await this.#log.append([op]); + const hash = await calculateHash(entry.data); + const blockHash = await this.#chain.addBlock({ id, hash, from: this.#log.formatId(position) }); + await this.#writeEntry(id, entry, blockHash); - const alive = isAlive(entry); + const alive = isAlive(entry); + // afterWrite sees a durable op; whatever it does, the record views and + // the `change` listeners (the relay among them) hear about the op. + try { + await this.#hook('afterWrite', ctx, op); + } finally { this.#emitUpdate(id, alive ? entry.data : null, delta, source); // Announced after everything is durable. `via` is null for a local // write; a merge made from a peer's snapshot names that peer, so the // relay does not hand its own state straight back to it. notify(this, this.#logger, 'change', op, via); - return op; } + return op; } // --------------------------------------------------------------------- // Public API // Encrypted at rest for THIS node's key: written directly, outside the op - // log, because no peer could read it. Runs inside the mutation queue. - async #writeEncrypted(id, data, source = null) { + // log, because no peer could read it. Reached only through #commit. + async #writeEncrypted(id, data) { const record = await encrypt(data, this.#keys.publicKey); const hash = await calculateHash(record); const blockHash = await this.#chain.addBlock({ id, hash }); const entry = { data: record, meta: null, encrypted: true, timestamp: Date.now(), block: blockHash }; await fsi.write(`${this.#path}/data`, `${id}.json`, JSON.stringify(entry)); this.#cache.set(id, { data, meta: null, encrypted: true, block: blockHash }); - this.#emitUpdate(id, data, { ...data }, source); } - async #removeEncrypted(id, source = null) { + async #removeEncrypted(id) { await fsi.remove(`${this.#path}/data`, `${id}.json`); this.#cache.set(id, null); - this.#emitUpdate(id, null, {}, source); } async saveData(id, data, options = {}) { const { encrypted = this.#encrypted } = options; - if (!encrypted) return void (await this[COMMIT]('write', id, { data })); - const current = await this.#load(id); - if (current && !current.encrypted) throw new Error(`Record ${id} is replicated and cannot become encrypted`); - await this.#run(() => this.#writeEncrypted(id, data)); + await this[COMMIT]('write', id, { data }, { encrypted }); } async loadData(id) { @@ -585,13 +638,22 @@ class Storage extends Emitter { return (this.#cache.has(id) && this.#cache.get(id) !== null) || this.#records.has(id); } - async get(id) { + // The raw read: what `get` returns before its hooks. `swap` compares + // against this, so a redacting afterRead cannot break its CAS. + async #read(id) { const entry = await this.#load(id); if (entry === null) return null; if (entry.encrypted) return entry.data; return isAlive(entry) ? entry.data : null; } + async get(id) { + await this.#hook('beforeRead', { id }); + const data = await this.#read(id); + const replaced = await this.#hook('afterRead', { id }, data); + return replaced === undefined ? data : replaced; + } + /** Every live record id โ€” tombstones excluded, encrypted records included. */ async list() { const names = await fsi.list(`${this.#path}/data`); @@ -629,7 +691,7 @@ class Storage extends Emitter { } async swap(id, changes, prev) { - const data = await this.get(id); + const data = await this.#read(id); if (!data) throw new Error(`Record ${id} not found`); const matches = Object.keys(prev).every((key) => data[key] === prev[key]); if (!matches) return false; diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 02bc341..edb0cb4 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -158,3 +158,33 @@ expectType<'HOOK_FAILED'>(hookError.code); expectType(hookError.hook); expectType(hookError.cause); expectAssignable(hookError); +expectAssignable({ + hooks: { + beforeWrite: async (ctx) => { + expectType(ctx.id); + expectType | null>(ctx.current); + expectType(ctx.encrypted); + return { ...ctx.data }; + }, + afterWrite: (ctx, op) => { + expectType(op); + expectType(ctx.source); + }, + beforeRead: ({ id }) => { + expectType(id); + }, + afterRead: (_ctx, data) => (data === null ? null : { ...(data as object), redacted: true }), + beforeApply: ({ op, peer }) => { + expectType(op); + expectType(peer); + return false; + }, + afterApply: (batch) => { + expectType(batch.applied); + expectType(batch.ops); + }, + }, +}); +expectError(syncom.open({ hooks: { nope: () => {} } })); +expectError(syncom.open({ hooks: { beforeWrite: 'x' } })); +expectError(syncom.open({ hooks: { beforeWrite: () => 'not an object' } })); diff --git a/tests/storage.hooks.browser.test.js b/tests/storage.hooks.browser.test.js new file mode 100644 index 0000000..8b0c629 --- /dev/null +++ b/tests/storage.hooks.browser.test.js @@ -0,0 +1,22 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const { installBrowserGlobals, requireBrowserBarrel } = require('./utils/browser-env.js'); +const { runHooksTests } = require('./suites/hooks.js'); + +test('Hooks (browser halves)', async (t) => { + const env = installBrowserGlobals(); + t.after(() => env.uninstall()); + const browser = requireBrowserBarrel(); + + let n = 0; + const paths = new Map(); + await runHooksTests(t, assert, async (t2, name, options = {}) => { + const key = `${t2.name}:${name}`; + if (!paths.has(key)) paths.set(key, `hooks-${n++}-${name}`); + const storage = await browser.open({ path: paths.get(key), logger: false, ...options }); + t2.after(() => storage.close().catch(() => {})); + return storage; + }); +}); diff --git a/tests/storage.hooks.test.js b/tests/storage.hooks.test.js new file mode 100644 index 0000000..b6494de --- /dev/null +++ b/tests/storage.hooks.test.js @@ -0,0 +1,24 @@ +'use strict'; + +const test = require('node:test'); +const assert = require('node:assert'); +const syncom = require('..'); +const { createTemp, cleanupTemp } = require('./utils/temp.js'); +const { runHooksTests } = require('./suites/hooks.js'); + +test('Hooks (node)', async (t) => { + // The same name within one subtest reopens the same directory. + const dirs = new Map(); + await runHooksTests(t, assert, async (t2, name, options = {}) => { + const key = `${t2.name}:${name}`; + let dir = dirs.get(key); + if (!dir) { + dir = await createTemp(); + dirs.set(key, dir); + t2.after(() => cleanupTemp(dir)); + } + const storage = await syncom.open({ path: dir, logger: false, ...options }); + t2.after(() => storage.close().catch(() => {})); + return storage; + }); +}); diff --git a/tests/suites/hooks.js b/tests/suites/hooks.js new file mode 100644 index 0000000..1f2c2aa --- /dev/null +++ b/tests/suites/hooks.js @@ -0,0 +1,274 @@ +'use strict'; + +// Hooks: run inside the operation, can alter or abort it. Against whichever +// storage half the caller builds โ€” `createStorage(t, name, options)` gives +// the SAME directory for the same name within one subtest (reopen cases). + +const { createMesh, settle, waitUntil } = require('../utils/mesh.js'); +const { MemoryLink } = require('../utils/memory-link.js'); + +const remoteOp = (origin, seq, type, id, extra = {}) => ({ origin, seq, ts: Date.now() + seq, type, id, ...extra }); + +const runHooksTests = async (t, assert, createStorage) => { + const hookError = (hook) => ({ name: 'HookError', code: 'HOOK_FAILED', hook }); + + await t.test('an unknown or malformed hook is refused at open', async (t2) => { + await assert.rejects(() => createStorage(t2, 'bad-name', { hooks: { beforeWrtie: () => {} } }), { + name: 'TypeError', + message: "Storage: unknown hook 'beforeWrtie'", + }); + await assert.rejects(() => createStorage(t2, 'bad-fn', { hooks: { afterWrite: 'nope' } }), TypeError); + await assert.rejects(() => createStorage(t2, 'bad-shape', { hooks: 'nope' }), TypeError); + }); + + await t.test('beforeWrite sees the write and the record it lands on', async (t2) => { + const seen = []; + const storage = await createStorage(t2, 'A', { + hooks: { beforeWrite: (ctx) => void seen.push({ ...ctx, current: ctx.current && { ...ctx.current } }) }, + }); + const id = await storage.insert({ title: 'a' }); + await storage.update(id, { title: 'b' }); + await storage.inc(id, 'hits', 2); + const record = await storage.record(id); + record.title = 'c'; + await record.save(); + await storage.delete(id); + assert.deepStrictEqual( + seen.map(({ type, current, data, key, value, source, encrypted }) => ({ + type, + current, + data, + key, + value, + source: source === record ? 'record' : source, + encrypted, + })), + [ + { + type: 'write', + current: null, + data: { title: 'a' }, + key: undefined, + value: undefined, + source: null, + encrypted: false, + }, + { + type: 'set', + current: { title: 'a' }, + data: { title: 'b' }, + key: undefined, + value: undefined, + source: null, + encrypted: false, + }, + { + type: 'inc', + current: { title: 'b' }, + data: undefined, + key: 'hits', + value: 2, + source: null, + encrypted: false, + }, + { + type: 'set', + current: { title: 'b', hits: 2 }, + data: { title: 'c' }, + key: undefined, + value: undefined, + source: 'record', + encrypted: false, + }, + { + type: 'delete', + current: { title: 'c', hits: 2 }, + data: undefined, + key: undefined, + value: undefined, + source: null, + encrypted: false, + }, + ], + ); + assert.ok(seen.every((ctx) => ctx.id === id && ctx.via === null && ctx.from === null)); + }); + + await t.test('beforeWrite may replace the data of a write or a set, and only those', async (t2) => { + const storage = await createStorage(t2, 'A', { + hooks: { + beforeWrite: ({ type, data }) => { + if (type === 'inc') return { nope: true }; + return { ...data, stamped: type }; + }, + }, + }); + const ops = []; + storage.on('change', (op) => ops.push(op)); + const original = { title: 'a' }; + const id = await storage.insert(original); + assert.deepStrictEqual(await storage.get(id), { title: 'a', stamped: 'write' }); + assert.deepStrictEqual(original, { title: 'a' }, 'the callerโ€™s object is untouched'); + await storage.update(id, { title: 'b' }); + assert.deepStrictEqual(await storage.get(id), { title: 'b', stamped: 'set' }); + assert.deepStrictEqual(ops.at(-1).data, { title: 'b', stamped: 'set' }, 'the op carries the replacement'); + await assert.rejects(() => storage.inc(id, 'hits'), { + name: 'TypeError', + message: 'beforeWrite may only replace data for write and set', + }); + const bad = await createStorage(t2, 'B', { hooks: { beforeWrite: () => 'text' } }); + await assert.rejects(() => bad.insert({ a: 1 }), /plain object or undefined/); + }); + + await t.test('a throwing beforeWrite refuses the write: nothing stamped, logged or written', async (t2) => { + const cause = new Error('not allowed'); + const storage = await createStorage(t2, 'A', { + hooks: { + beforeWrite: ({ data }) => { + if (data?.forbidden) throw cause; + }, + }, + }); + const id = await storage.insert({ ok: true }); + const vector = { ...storage.vector }; + const error = await storage.update(id, { forbidden: true }).then( + () => assert.fail('should reject'), + (e) => e, + ); + assert.strictEqual(error.name, 'HookError'); + assert.strictEqual(error.code, 'HOOK_FAILED'); + assert.strictEqual(error.hook, 'beforeWrite'); + assert.strictEqual(error.cause, cause); + assert.deepStrictEqual(await storage.get(id), { ok: true }); + assert.deepStrictEqual(storage.vector, vector); + assert.strictEqual(storage.log.length, 1); + assert.strictEqual(await storage.validate(), true); + await assert.rejects(() => storage.insert({ forbidden: true }), hookError('beforeWrite')); + assert.strictEqual((await storage.list()).length, 1); + }); + + await t.test('afterWrite sees the durable op; a throw fails the call but the op stays', async (t2) => { + const seen = []; + let fail = false; + const storage = await createStorage(t2, 'A', { + hooks: { + afterWrite: (ctx, op) => { + seen.push([ctx.type, op && op.seq, ctx.encrypted]); + if (fail) throw new Error('after boom'); + }, + }, + }); + const changes = []; + storage.on('change', (op) => changes.push(op.seq)); + const id = await storage.insert({ n: 1 }); + await storage.update(id, {}); // changes nothing: no afterWrite + fail = true; + await assert.rejects(() => storage.update(id, { n: 2 }), hookError('afterWrite')); + assert.deepStrictEqual(await storage.get(id), { n: 2 }, 'the op is committed'); + assert.deepStrictEqual(changes, [1, 2], 'and announced'); + fail = false; + await storage.insert({ secret: true }, { encrypted: true }); + assert.deepStrictEqual(seen, [ + ['write', 1, false], + ['set', 2, false], + ['write', null, true], + ]); + }); + + await t.test('encrypted writes go through the same hooks and stay local', async (t2) => { + const seen = []; + const storage = await createStorage(t2, 'A', { + encrypted: true, + hooks: { beforeWrite: (ctx) => void seen.push([ctx.type, ctx.encrypted, ctx.current]) }, + }); + const id = await storage.insert({ s: 1 }); + await storage.update(id, { t: 2 }); + const record = await storage.record(id); + const updates = []; + record.on('update', (data) => updates.push(data)); + await storage.delete(id); + assert.deepStrictEqual(seen, [ + ['write', true, null], + ['set', true, { s: 1 }], + ['delete', true, { s: 1, t: 2 }], + ]); + assert.deepStrictEqual(updates, [null]); + assert.strictEqual(storage.log.length, 0); + const shared = await storage.insert({ shared: true }, { encrypted: false }); + await assert.rejects( + () => storage.set(shared, { x: 1 }, { encrypted: true }), + /replicated and cannot become encrypted/, + ); + }); + + await t.test('a hook cannot write to the storage, but it can read', async (t2) => { + const reads = []; + const storage = await createStorage(t2, 'A', { + hooks: { + beforeWrite: async ({ id, data }) => { + reads.push(await storage.get(id)); + if (data?.echo) await storage.update(id, { echoed: true }); + }, + }, + }); + const id = await storage.insert({ n: 1 }); + assert.deepStrictEqual(reads, [null]); + const error = await storage.update(id, { echo: true }).catch((e) => e); + assert.strictEqual(error.name, 'HookError'); + assert.match(error.cause.message, /inside a hook/); + assert.deepStrictEqual(await storage.get(id), { n: 1 }); + await storage.update(id, { n: 2 }); + assert.deepStrictEqual(await storage.get(id), { n: 2 }, 'the queue is not stuck'); + }); + + await t.test('beforeRead refuses, afterRead redacts โ€” for get() and what reads through it', async (t2) => { + let deny = false; + const storage = await createStorage(t2, 'A', { + hooks: { + beforeRead: ({ id }) => { + if (deny) throw new Error(`no ${id}`); + }, + afterRead: (ctx, data) => { + if (data === null || data.hidden) return data === null ? undefined : null; + const { ssn, ...rest } = data; + return ssn === undefined ? undefined : { ...rest, ssn: '***' }; + }, + }, + }); + const id = await storage.insert({ name: 'Ada', ssn: '123' }); + const hidden = await storage.insert({ hidden: true }); + assert.deepStrictEqual(await storage.get(id), { name: 'Ada', ssn: '***' }); + assert.deepStrictEqual(await storage.loadData(id), { name: 'Ada', ssn: '***' }); + assert.strictEqual(await storage.get(hidden), null, 'afterRead may hide a record from get'); + assert.strictEqual(await storage.has(hidden), true, 'has() is raw'); + const record = await storage.record(id); + assert.strictEqual(record.ssn, '***', 'a Record view reads through get'); + assert.strictEqual(storage.getCachedData(id).ssn, '123', 'the cache is raw'); + const page = await storage.snapshot(); + assert.strictEqual(page.records.find((r) => r.id === id).data.ssn, '123', 'replication is raw'); + assert.strictEqual(await storage.swap(id, { name: 'Ada L.' }, { ssn: '123' }), true, 'swap compares raw'); + assert.deepStrictEqual(await storage.get(id), { name: 'Ada L.', ssn: '***' }); + deny = true; + await assert.rejects(() => storage.get(id), hookError('beforeRead')); + assert.strictEqual(await storage.has(id), true); + }); + + await t.test('the order of a local write', async (t2) => { + const order = []; + const storage = await createStorage(t2, 'A', { + hooks: { + beforeWrite: () => void order.push('beforeWrite'), + afterWrite: () => void order.push('afterWrite'), + }, + }); + const id = await storage.insert({ n: 1 }); + const record = await storage.record(id); + record.on('update', () => order.push('update')); + storage.on('change', () => order.push('change')); + order.length = 0; + await storage.update(id, { n: 2 }); + assert.deepStrictEqual(order, ['beforeWrite', 'afterWrite', 'update', 'change']); + }); +}; + +module.exports = { runHooksTests, remoteOp, createMesh, settle, waitUntil, MemoryLink }; From e3bfe53326b811ee40be6047a75976aaaebf6a83 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:22:34 +0300 Subject: [PATCH 11/67] feat(storage): beforeApply/afterApply hooks, skipped ops, apply event MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit beforeApply runs per remote op, after the duplicate/gap checks and before the op is applied. `false` skips it HERE only: the op is still logged and relayed and the vector moves โ€” a hole in this node's log would be a gap for every peer resuming from it, and a vector that does not move would have the sender offer the page forever. The log flags such entries so a reopen's replay of the tail leaves them alone too. A throw aborts the batch before anything is durable; the sender sees a failed push and offers it again. afterApply sees the durable batch; whatever it does, the record views, the new `apply` event and the `change` listeners hear about every op. The result grows `skipped`, and applyRemote takes a `peer` descriptor ({ url, node }) so hooks know who sent the page without the link object. Co-Authored-By: Claude Fable 5.1 --- index.d.ts | 5 +- src/log.js | 12 ++- src/storage.js | 47 +++++++--- src/sync.js | 11 ++- tests/index.test-d.ts | 2 + tests/log.test.js | 24 ++++++ tests/storage.ops.test.js | 6 +- tests/suites/hooks.js | 176 ++++++++++++++++++++++++++++++++++++++ tests/telemetry.test.js | 2 +- tests/utils/mesh.js | 5 +- 10 files changed, 267 insertions(+), 23 deletions(-) diff --git a/index.d.ts b/index.d.ts index 6d56fb3..04fa8d3 100644 --- a/index.d.ts +++ b/index.d.ts @@ -498,7 +498,10 @@ export interface SnapshotPage { } export interface ApplyRemoteResult { + /** Accepted ops โ€” the skipped ones included; they moved the vector too. */ applied: number; + /** Accepted but not applied here: a beforeApply hook returned `false`. */ + skipped: number; duplicates: number; gaps: number; } @@ -544,7 +547,7 @@ export interface Storage { hasRecord(id: string): boolean; applyRemote( ops: readonly SyncOp[], - options?: { via?: unknown; scopes?: readonly string[] | null }, + options?: { via?: unknown; scopes?: readonly string[] | null; peer?: PeerRef | null }, ): Promise; adoptVector(vector: VersionVector, options?: { via?: unknown; from?: string }): Promise; snapshot(options?: { cursor?: string | null; limit?: number; scopes?: readonly string[] | null }): Promise; diff --git a/src/log.js b/src/log.js index bb6d50d..5b33b9c 100644 --- a/src/log.js +++ b/src/log.js @@ -208,13 +208,19 @@ class OpLog { /** * Appends a batch; resolves once it is on disk. Returns the entries with * their seqs, so the caller can announce them by id. Rolls the open files - * into a segment when enough have accumulated. + * into a segment when enough have accumulated. Ops in `skipped` were + * accepted but not applied here (a beforeApply hook said no): they are + * logged and served like any other, and flagged so a reopen's replay of + * the tail leaves them alone too. */ - append(ops) { + append(ops, { skipped = null } = {}) { return this.#serialize(async () => { if (ops.length === 0) return []; const entries = []; - for (const op of ops) entries.push({ seq: this.#next++, op }); + for (const op of ops) { + const seq = this.#next++; + entries.push(skipped?.has(op) ? { seq, op, skipped: true } : { seq, op }); + } const name = `${pad(entries[0].seq)}.json`; const started = this.#otel.enabled ? now() : 0; await fsi.write(this.#openPath, name, JSON.stringify({ entries })); diff --git a/src/storage.js b/src/storage.js index 23fb391..a475da6 100644 --- a/src/storage.js +++ b/src/storage.js @@ -300,7 +300,9 @@ class Storage extends Emitter { // Returns how many records the tail touched. async #replay(tail) { const touched = new Map(); - for (const { seq, op } of tail) { + for (const { seq, op, skipped } of tail) { + // Accepted but not applied when it arrived (beforeApply said no). + if (skipped) continue; this.#clock.observe(op.ts); let current; try { @@ -746,23 +748,25 @@ class Storage extends Emitter { () => this.#applyRemote(ops, options), (r) => ({ 'syncom.ops.applied': r.applied, + 'syncom.ops.skipped': r.skipped, 'syncom.ops.duplicates': r.duplicates, 'syncom.ops.gaps': r.gaps, }), ); return result; } finally { - otel.recordApply(result ?? { applied: 0, duplicates: 0, gaps: 0 }, now() - started); + otel.recordApply(result ?? { applied: 0, skipped: 0, duplicates: 0, gaps: 0 }, now() - started); } } - async #applyRemote(ops, { via = null, scopes = null } = {}) { + async #applyRemote(ops, { via = null, scopes = null, peer = null } = {}) { { const accepted = []; const touched = new Map(); const deltas = new Map(); const vector = this.#log.vector; let adopted = null; + let skipped = null; let duplicates = 0; let gaps = 0; for (const op of ops) { @@ -807,6 +811,19 @@ class Storage extends Emitter { accepted.push(op); continue; } + // A `false` from beforeApply skips the op HERE only: it is still + // logged and relayed (the vector must move, or the peer would offer + // it forever, and a hole in this log would be a gap for everyone + // resuming from it). A throw aborts the batch โ€” nothing is durable + // yet, so the peer sees a failed push and offers the page again. + const visible = current && isAlive(current) ? current.data : null; + const verdict = await this.#hook('beforeApply', { op, via, peer, current: visible }); + if (verdict === false) { + vectorAdvance(vector, op); + accepted.push(op); + (skipped ??= new Set()).add(op); + continue; + } this.#clock.observe(op.ts); const { entry, changed, delta, rejected } = applyOp(current, op); if (rejected.length > 0) { @@ -821,18 +838,26 @@ class Storage extends Emitter { touched.set(op.id, entry); deltas.set(op.id, { ...(deltas.get(op.id) ?? {}), ...delta }); } - if (accepted.length === 0) return { applied: 0, duplicates, gaps }; - - const entries = await this.#log.append(accepted); - if (adopted) await this.#log.advance(adopted); - if (touched.size > 0) { - await this.#persistBatch(touched, entries[0].seq, entries[entries.length - 1].seq); + const result = { applied: accepted.length, skipped: skipped?.size ?? 0, duplicates, gaps }; + if (accepted.length > 0) { + const entries = await this.#log.append(accepted, { skipped }); + if (adopted) await this.#log.advance(adopted); + if (touched.size > 0) { + await this.#persistBatch(touched, entries[0].seq, entries[entries.length - 1].seq); + } + } + // afterApply sees a durable batch; whatever it does, the record views + // and the listeners (the relay among them) hear about every op. + try { + await this.#hook('afterApply', { via, peer, ops: accepted, ...result }); + } finally { for (const [id, entry] of touched) { this.#emitUpdate(id, isAlive(entry) ? entry.data : null, deltas.get(id)); } + notify(this, this.#logger, 'apply', { via, peer, ops: accepted, ...result }); + for (const op of accepted) notify(this, this.#logger, 'change', op, via); } - for (const op of accepted) notify(this, this.#logger, 'change', op, via); - return { applied: accepted.length, duplicates, gaps }; + return result; } } diff --git a/src/sync.js b/src/sync.js index b2dd6d8..49e965b 100644 --- a/src/sync.js +++ b/src/sync.js @@ -453,7 +453,11 @@ class SyncManager extends Emitter { if (peer.closed || !value) return; if (value.bootstrap) return void (await this.#bootstrap(peer)); const ops = unpackOps(value.ops); - const result = await this.#storage.applyRemote(ops, { via: peer, scopes: peer.node.scopes }); + const result = await this.#storage.applyRemote(ops, { + via: peer, + scopes: peer.node.scopes, + peer: { url: peer.node.url, node: peer.remote.id }, + }); // Hung up while applying: the cursor is not saved, so a restart replays // this page as duplicates โ€” better than a registry write racing whoever // reopens the directory next. @@ -643,9 +647,10 @@ class SyncManager extends Emitter { push: { access, handler: async (context, args) => { - const { scopes } = peerOf(context); + const { node, scopes } = peerOf(context); const ops = unpackOps(args?.ops); - const result = await storage.applyRemote(ops, { via: context.client, scopes }); + // `url` is null: a host does not dial, so it has no url for the peer. + const result = await storage.applyRemote(ops, { via: context.client, scopes, peer: { url: null, node } }); return { vector: storage.vector, ...result }; }, }, diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index edb0cb4..0b81a26 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -188,3 +188,5 @@ expectAssignable({ expectError(syncom.open({ hooks: { nope: () => {} } })); expectError(syncom.open({ hooks: { beforeWrite: 'x' } })); expectError(syncom.open({ hooks: { beforeWrite: () => 'not an object' } })); +expectType>(storage.applyRemote([], { peer: { url: null, node: 'n' } })); +storage.applyRemote([]).then((result) => expectType(result.skipped)); diff --git a/tests/log.test.js b/tests/log.test.js index bf29ae3..4e9d71c 100644 --- a/tests/log.test.js +++ b/tests/log.test.js @@ -181,6 +181,30 @@ test('OpLog', async (t) => { } }); + await t.test('a skipped op is logged, served and flagged through a reopen', async () => { + const dir = await createTemp(); + try { + let log = await new OpLog(dir); + const skip = op('A', 2); + const entries = await log.append([op('A', 1), skip, op('A', 3)], { skipped: new Set([skip]) }); + assert.deepStrictEqual( + entries.map((entry) => entry.skipped), + [undefined, true, undefined], + ); + assert.deepStrictEqual(log.vector, { A: 3 }, 'a skipped op moves the vector like any other'); + assert.strictEqual(log.since(undefined).length, 3, 'and is served to resuming peers'); + await log.close(); + log = await new OpLog(dir); + assert.deepStrictEqual( + log.tail().map((entry) => entry.skipped), + [undefined, true, undefined], + 'the flag survives a reopen, for the replay to honour', + ); + } finally { + await cleanupTemp(dir); + } + }); + await t.test('flush seals the open files on demand', async () => { const dir = await createTemp(); try { diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js index fab1d26..934347e 100644 --- a/tests/storage.ops.test.js +++ b/tests/storage.ops.test.js @@ -93,7 +93,7 @@ test('Storage replication model', async (t) => { storage.on('change', (op, via) => changes.push({ op, via })); const first = await storage.applyRemote([remoteOp('peer', 1, 'write', 'r1', { data: { a: 1 } })], { via: 'L' }); - assert.deepStrictEqual(first, { applied: 1, duplicates: 0, gaps: 0 }); + assert.deepStrictEqual(first, { applied: 1, skipped: 0, duplicates: 0, gaps: 0 }); assert.deepStrictEqual(await storage.get('r1'), { a: 1 }); assert.deepStrictEqual(changes[0].via, 'L'); assert.strictEqual(changes[0].op.origin, 'peer'); @@ -104,7 +104,7 @@ test('Storage replication model', async (t) => { { origin: storage.nodeId, seq: 1, ts: 1, type: 'write', id: 'x', data: {} }, 'garbage', ]); - assert.deepStrictEqual(again, { applied: 0, duplicates: 2, gaps: 1 }); + assert.deepStrictEqual(again, { applied: 0, skipped: 0, duplicates: 2, gaps: 1 }); assert.deepStrictEqual(await storage.get('r1'), { a: 1 }); const next = await storage.applyRemote([ @@ -112,7 +112,7 @@ test('Storage replication model', async (t) => { remoteOp('peer', 3, 'inc', 'r1', { key: 'n', value: 4 }), remoteOp('peer', 4, 'delete', 'r1'), ]); - assert.deepStrictEqual(next, { applied: 3, duplicates: 0, gaps: 0 }); + assert.deepStrictEqual(next, { applied: 3, skipped: 0, duplicates: 0, gaps: 0 }); assert.strictEqual(await storage.get('r1'), null); assert.deepStrictEqual(storage.vector, { peer: 4 }); // Accepted-but-unchanged ops still advance the vector and are logged. diff --git a/tests/suites/hooks.js b/tests/suites/hooks.js index 1f2c2aa..d1c2716 100644 --- a/tests/suites/hooks.js +++ b/tests/suites/hooks.js @@ -6,6 +6,7 @@ const { createMesh, settle, waitUntil } = require('../utils/mesh.js'); const { MemoryLink } = require('../utils/memory-link.js'); +const { packOps } = require('../../src/ops.js'); const remoteOp = (origin, seq, type, id, extra = {}) => ({ origin, seq, ts: Date.now() + seq, type, id, ...extra }); @@ -253,6 +254,181 @@ const runHooksTests = async (t, assert, createStorage) => { assert.strictEqual(await storage.has(id), true); }); + await t.test('beforeApply(false) skips an op here: logged, relayed, replay-safe', async (t2) => { + const seen = []; + let storage = await createStorage(t2, 'A', { + hooks: { + beforeApply: (ctx) => { + seen.push({ ...ctx, current: ctx.current && { ...ctx.current } }); + if (ctx.op.data?.secret) return false; + }, + }, + }); + const changes = []; + const applies = []; + storage.on('change', (op) => changes.push(op.seq)); + storage.on('apply', (batch) => applies.push(batch)); + const first = await storage.applyRemote( + [ + remoteOp('peer', 1, 'write', 'r1', { data: { secret: true } }), + remoteOp('peer', 2, 'write', 'r2', { data: { n: 2 } }), + ], + { via: 'L', peer: { url: 'mem://peer', node: 'peer' } }, + ); + assert.deepStrictEqual(first, { applied: 2, skipped: 1, duplicates: 0, gaps: 0 }); + assert.strictEqual(await storage.get('r1'), null, 'the skipped op did not touch the record'); + assert.deepStrictEqual(await storage.get('r2'), { n: 2 }); + assert.strictEqual(storage.vector.peer, 2, 'but moved the vector'); + assert.strictEqual(storage.log.length, 2, 'and is in the log'); + assert.deepStrictEqual(changes, [1, 2], 'and was announced'); + assert.deepStrictEqual(applies.length, 1); + assert.deepStrictEqual(applies[0].peer, { url: 'mem://peer', node: 'peer' }); + assert.strictEqual(applies[0].ops.length, 2); + assert.deepStrictEqual( + seen.map(({ op, via, peer, current }) => [op.seq, via, peer.node, current]), + [ + [1, 'L', 'peer', null], + [2, 'L', 'peer', null], + ], + ); + // The next op of that origin is not a gap. + const next = await storage.applyRemote([remoteOp('peer', 3, 'set', 'r2', { data: { n: 3 } })]); + assert.strictEqual(next.applied, 1); + assert.deepStrictEqual(seen.at(-1).current, { n: 2 }); + // A reopen replays the unsealed tail โ€” and leaves the skipped op alone. + await storage.close(); + storage = await createStorage(t2, 'A'); + assert.strictEqual(await storage.get('r1'), null); + assert.deepStrictEqual(await storage.get('r2'), { n: 3 }); + assert.strictEqual(storage.vector.peer, 3); + }); + + await t.test('a throwing beforeApply aborts the batch before anything is durable', async (t2) => { + const storage = await createStorage(t2, 'A', { + hooks: { + beforeApply: ({ op }) => { + if (op.seq === 2) throw new Error('refused'); + }, + }, + }); + await assert.rejects( + () => + storage.applyRemote([ + remoteOp('peer', 1, 'write', 'r1', { data: { n: 1 } }), + remoteOp('peer', 2, 'write', 'r2', { data: { n: 2 } }), + ]), + hookError('beforeApply'), + ); + assert.strictEqual(storage.log.length, 0); + assert.deepStrictEqual(storage.vector, {}); + assert.strictEqual(await storage.get('r1'), null, 'the op before the throw was not persisted either'); + assert.strictEqual(await storage.has('r1'), false); + }); + + await t.test('afterApply sees the durable batch; a throw fails the call but the batch stays', async (t2) => { + const batches = []; + const storage = await createStorage(t2, 'A', { + hooks: { + afterApply: (batch) => { + batches.push(batch); + if (batch.applied > 0) throw new Error('after boom'); + }, + }, + }); + const events = []; + storage.on('apply', (batch) => events.push(batch.applied)); + await assert.rejects( + () => storage.applyRemote([remoteOp('peer', 1, 'write', 'r1', { data: { n: 1 } })]), + hookError('afterApply'), + ); + assert.deepStrictEqual(await storage.get('r1'), { n: 1 }, 'the batch is committed'); + assert.deepStrictEqual(events, [1], 'and announced'); + const empty = await storage.applyRemote([remoteOp('peer', 1, 'write', 'r1', { data: { n: 1 } })]); + assert.deepStrictEqual(empty, { applied: 0, skipped: 0, duplicates: 1, gaps: 0 }); + assert.strictEqual(batches.length, 2, 'an empty batch runs the hook too'); + assert.deepStrictEqual(batches[1].ops, []); + assert.deepStrictEqual(events, [1, 0]); + }); + + await t.test('a skip is local: the op still reaches the peers behind this node', async (t2) => { + const mesh = await createMesh( + t2, + ['A', 'B', 'C'], + [ + ['B', 'A'], + ['C', 'B'], + ], + createStorage, + { nodes: { B: { hooks: { beforeApply: ({ op }) => (op.data?.secret ? false : undefined) } } } }, + ); + const a = mesh.get('A'); + const b = mesh.get('B'); + const c = mesh.get('C'); + const secret = await a.insert({ secret: true }); + const plain = await a.insert({ plain: true }); + await settle([a, b, c]); + assert.strictEqual(await b.get(secret), null, 'B skipped it'); + assert.deepStrictEqual(await b.get(plain), { plain: true }); + assert.deepStrictEqual(await c.get(secret), { secret: true }, 'C, behind B, still got it'); + assert.deepStrictEqual(await c.get(plain), { plain: true }); + }); + + await t.test('the hosting side runs beforeApply for a pushed page', async (t2) => { + const seen = []; + let refuse = false; + const host = await createStorage(t2, 'H', { + hooks: { + beforeApply: (ctx) => { + seen.push(ctx.peer); + if (refuse) throw new Error('not here'); + return ctx.op.data?.secret ? false : undefined; + }, + }, + }); + const unit = host.sync.unit({ access: 'public' }); + const link = new MemoryLink(unit); + await link.call('syncom/hello', { node: 'peer', version: 1 }); + const answer = await link.call('syncom/push', { + ops: packOps([remoteOp('peer', 1, 'write', 'r1', { data: { secret: true } })]), + }); + assert.strictEqual(answer.applied, 1); + assert.strictEqual(answer.skipped, 1); + assert.deepStrictEqual(answer.vector, { peer: 1 }); + assert.deepStrictEqual(seen, [{ url: null, node: 'peer' }]); + refuse = true; + await assert.rejects( + () => link.call('syncom/push', { ops: packOps([remoteOp('peer', 2, 'write', 'r2', { data: {} })]) }), + /beforeApply hook failed: not here/, + ); + assert.deepStrictEqual(host.vector, { peer: 1 }); + }); + + await t.test('a peer whose page is refused hears about it as an error, and recovers', async (t2) => { + let refuse = true; + const host = await createStorage(t2, 'H', { + hooks: { + beforeApply: () => { + if (refuse) throw new Error('not yet'); + }, + }, + }); + const unit = host.sync.unit({ access: 'public' }); + const dialer = await createStorage(t2, 'D', { sync: { autoStart: false, connect: () => new MemoryLink(unit) } }); + const errors = []; + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + const id = await dialer.insert({ n: 1 }); + await dialer.sync.start(); + await waitUntil(() => errors.length > 0); + assert.match(errors[0].message, /beforeApply hook failed: not yet/); + assert.strictEqual(errors[0].url, 'mem://host'); + assert.strictEqual(await host.get(id), null); + refuse = false; + await dialer.sync.sync(); + await settle([host, dialer]); + assert.deepStrictEqual(await host.get(id), { n: 1 }); + }); + await t.test('the order of a local write', async (t2) => { const order = []; const storage = await createStorage(t2, 'A', { diff --git a/tests/telemetry.test.js b/tests/telemetry.test.js index 96b9670..a3c2808 100644 --- a/tests/telemetry.test.js +++ b/tests/telemetry.test.js @@ -461,7 +461,7 @@ test('Storage telemetry', async (t) => { 'garbage', remoteOp('peer', 2, 'inc', id, { key: 'hits', value: 1 }), ]); - assert.deepStrictEqual(result, { applied: 2, duplicates: 1, gaps: 1 }); + assert.deepStrictEqual(result, { applied: 2, skipped: 0, duplicates: 1, gaps: 1 }); const [span] = named('syncom.apply'); assert.strictEqual(span.attributes['syncom.ops.count'], 5); assert.strictEqual(span.attributes['syncom.ops.applied'], 2); diff --git a/tests/utils/mesh.js b/tests/utils/mesh.js index 6ab956f..035ed8f 100644 --- a/tests/utils/mesh.js +++ b/tests/utils/mesh.js @@ -18,7 +18,10 @@ const createMesh = async (t, names, edges, createStorage, options = {}) => { return link; }; for (const name of names) { - const storage = await createStorage(t, name, { sync: { connect, autoStart: false, ...options.sync } }); + const storage = await createStorage(t, name, { + ...options.nodes?.[name], + sync: { connect, autoStart: false, ...options.sync, ...options.nodes?.[name]?.sync }, + }); storages.set(name, storage); units.set(name, storage.sync.unit(options.unit ?? { access: 'public' })); } From 1a06654de5ad37167a16334ee8656a666cbe3321 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:26:42 +0300 Subject: [PATCH 12/67] feat(sync): beforeConnect/beforeBootstrap hooks and lifecycle events MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `open({ sync: { hooks } })`. beforeConnect sees a registry entry about to be dialled; `false` leaves it registered but not dialled. beforeBootstrap runs when a host's log cannot resume us and its whole state is about to be joined; `false` refuses โ€” the peer stays connected for pushes and receives nothing until the next handshake asks again. A throw in either is reported through the manager's `error` event, as any failure on the way out is. There is deliberately no hook on the ops that leave for a peer: seqs run contiguously per origin and scope, so an op withheld from one peer is a gap that stops every later op of that stream at it. Per-peer visibility is what scopes are for. New events: `start`, `stop`, `node` ({ url, action }), `connect` (a link accepted, before hello โ€” `open` used to say nothing) and, on the hosting side, `hello` ({ node, scopes }) once a peer is authorized. SyncOptions gains the `connect`, `autoStart` and `pageSize` it always accepted. Co-Authored-By: Claude Fable 5.1 --- browser.d.ts | 2 + index.d.ts | 36 ++++++++++ src/sync.js | 35 +++++++++- tests/index.test-d.ts | 19 +++++ tests/suites/hooks.js | 159 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 249 insertions(+), 2 deletions(-) diff --git a/browser.d.ts b/browser.d.ts index f696771..b3b602c 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -32,6 +32,8 @@ export type { ApplyContext, ApplyBatchContext, PeerRef, + SyncHooks, + BootstrapContext, Collections, Reference, SyncomRecord, diff --git a/index.d.ts b/index.d.ts index 04fa8d3..cd86224 100644 --- a/index.d.ts +++ b/index.d.ts @@ -300,11 +300,47 @@ export interface Schema { readonly [key: string]: unknown; } +/** The peer whose whole state is about to be joined. */ +export interface BootstrapContext { + url: string; + node: string; +} + +/** + * Hooks of the sync manager (`open({ sync: { hooks } })`). Same contract as + * StorageHooks: awaited inside the operation, a throw becomes a HookError โ€” + * reported through the manager's `error` event on the dialing side. + */ +export interface SyncHooks { + /** + * Before a registered node is dialled; `node` is a copy of its registry + * entry. Return `false` to leave it registered but not dialled. + */ + beforeConnect?: (node: Node) => void | boolean | Promise; + /** + * Before this node joins a peer's whole state (its log could not resume + * us). Return `false` to refuse: the peer stays connected for pushes but + * nothing is received from it until the next handshake asks again. There + * is deliberately no hook on the ops that leave for a peer โ€” seqs run + * contiguously per origin and scope, so a withheld op is a gap that stops + * every later op of that stream; per-peer visibility is what scopes do. + */ + beforeBootstrap?: (peer: BootstrapContext) => void | boolean | Promise; +} + export interface SyncOptions { + /** Returns a link for a registry node; required to dial. */ + connect?: (node: Node) => SyncLink | Promise; + /** Dial on open โ€” `true` when `connect` is given. */ + autoStart?: boolean; + /** Ops per push page and per `changes` page (default 500). */ + pageSize?: number; /** Assigns a scope to a record; unscoped (`'*'`) records reach every peer. */ scope?: (id: string, data: Record) => string; retention?: { ops?: number; days?: number }; log?: { rollAfter?: number }; + /** Lifecycle hooks โ€” see SyncHooks. */ + hooks?: SyncHooks; } /** What a write hook sees: the op being made, and the record it lands on. */ diff --git a/src/sync.js b/src/sync.js index 49e965b..3226133 100644 --- a/src/sync.js +++ b/src/sync.js @@ -27,7 +27,7 @@ const { packOps, unpackOps } = require('./ops.js'); const { vectorHas, keyScope } = require('./vector.js'); const { isSyncLink } = require('./link.js'); const { createTelemetry, traced } = require('./telemetry.js'); -const { notify } = require('./hooks.js'); +const { notify, runHook, normalizeHooks } = require('./hooks.js'); const REMOTE = 'remote'; const PROTOCOL = 1; @@ -36,6 +36,14 @@ const DEFAULT_PAGE_SIZE = 500; const DEFAULT_PAGE_BYTES = 4 << 20; const TRACKED = Symbol.for('wrpc.tracked'); +// The hooks a manager accepts (`open({ sync: { hooks } })`). There is no +// hook on what leaves for a peer: seqs run contiguously per origin and +// scope, so an op withheld from one peer is a gap that stops every later +// op of that stream at it โ€” per-peer visibility is what scopes are for. +const SYNC_HOOKS = ['beforeConnect', 'beforeBootstrap']; +// What #connect's chain carries when beforeConnect said no. +const NOT_DIALLED = Symbol('syncom.notDialled'); + // Node registry entries are keyed by url; the file name is the url itself, // percent-encoded โ€” unlike the old `.json`, two peers on one host or a // url without a port cannot collide. @@ -92,11 +100,13 @@ class SyncManager extends Emitter { #started = false; #saving = new Map(); #otel; + #hooks; constructor(storage, options = {}) { super(); this.#storage = storage; this.#options = options ?? {}; + this.#hooks = normalizeHooks(this.#options.hooks, SYNC_HOOKS, 'SyncManager'); this.#otel = createTelemetry(storage.telemetry); this.#path = `${storage.path}/${REMOTE}`; return this.#init(); @@ -148,6 +158,7 @@ class SyncManager extends Emitter { }; this.#nodes.set(url, node); await fsi.write(this.#path, fileName(url), JSON.stringify(node)); + notify(this, this.#storage.logger, 'node', { url, action: 'add' }); if (this.#started) this.#connect(node); } @@ -157,6 +168,7 @@ class SyncManager extends Emitter { this.#nodes.delete(url); this.#disconnect(url); await fsi.remove(this.#path, fileName(url)); + notify(this, this.#storage.logger, 'node', { url, action: 'remove' }); } listNodes() { @@ -168,6 +180,7 @@ class SyncManager extends Emitter { if (!node) throw new Error(`Node with URL ${url} not found`); node.enabled = enabled; await this.#saveNode(node); + notify(this, this.#storage.logger, 'node', { url, action: enabled ? 'enable' : 'disable' }); if (!enabled) this.#disconnect(url); else if (this.#started) this.#connect(node); } @@ -211,12 +224,14 @@ class SyncManager extends Emitter { for (const node of this.#nodes.values()) { if (node.enabled) this.#connect(node); } + notify(this, this.#storage.logger, 'start'); } async stop() { this.#started = false; for (const url of Array.from(this.#peers.keys())) this.#disconnect(url); await Promise.all(Array.from(this.#saving.values(), (state) => state.last)); + notify(this, this.#storage.logger, 'stop'); } /** Pushes whatever the connected peers lack right now; resolves when done. */ @@ -260,8 +275,16 @@ class SyncManager extends Emitter { }; this.#peers.set(node.url, peer); Promise.resolve() - .then(() => this.#options.connect(node)) + // beforeConnect sees a copy of the registry entry; `false` leaves the + // node registered but not dialled โ€” no peer, no error. + .then(() => runHook(this.#hooks, 'beforeConnect', { ...node })) + .then((verdict) => { + if (verdict !== false) return this.#options.connect(node); + if (this.#peers.get(node.url) === peer) this.#peers.delete(node.url); + return NOT_DIALLED; + }) .then((link) => { + if (link === NOT_DIALLED) return; if (peer.closed) return void link?.close?.(); if (!isSyncLink(link)) { throw new TypeError('connect(node) must return a SyncLink (call/subscribe/on/off/close)'); @@ -286,6 +309,7 @@ class SyncManager extends Emitter { }; for (const event in peer.handlers) link.on(event, peer.handlers[event]); peer.connected = true; + notify(this, this.#storage.logger, 'connect', { url: node.url }); return this.#handshake(peer, { resubscribe: true }); }) .catch((error) => this.#fail(peer, error)); @@ -475,6 +499,12 @@ class SyncManager extends Emitter { if (peer.bootstrapping) return; peer.bootstrapping = true; try { + // A bootstrap replaces this node's view of the peer's scopes wholesale; + // beforeBootstrap may refuse it. The feed has ended by now, so a + // refused peer stays connected for pushes but receives nothing until + // the next handshake asks again. + const verdict = await runHook(this.#hooks, 'beforeBootstrap', { url: peer.node.url, node: peer.remote.id }); + if (verdict === false) return; const otel = this.#otel; if (!otel.enabled) return void (await this.#join(peer)); const attributes = { 'syncom.peer.url': peer.node.url, 'syncom.peer.node': peer.remote.id }; @@ -640,6 +670,7 @@ class SyncManager extends Emitter { for (const scope of scopes) context.client.join(`syncom:${scope}`); } } + notify(this, storage.logger, 'hello', { node: args.node, scopes }); return { node: storage.nodeId, vector: storage.vector, epoch: storage.log.epoch, version: PROTOCOL }; }, }, diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 0b81a26..53b2fff 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -190,3 +190,22 @@ expectError(syncom.open({ hooks: { beforeWrite: 'x' } })); expectError(syncom.open({ hooks: { beforeWrite: () => 'not an object' } })); expectType>(storage.applyRemote([], { peer: { url: null, node: 'n' } })); storage.applyRemote([]).then((result) => expectType(result.skipped)); +expectAssignable({ + sync: { + connect: async () => ({}) as syncom.SyncLink, + autoStart: false, + pageSize: 100, + hooks: { + beforeConnect: (node) => { + expectType(node); + return false; + }, + beforeBootstrap: ({ url, node }) => { + expectType(url); + expectType(node); + }, + }, + }, +}); +expectError(syncom.open({ sync: { hooks: { nope: () => {} } } })); +expectError(syncom.open({ sync: { hooks: { beforeBootstrap: 'nope' } } })); diff --git a/tests/suites/hooks.js b/tests/suites/hooks.js index d1c2716..d42ec82 100644 --- a/tests/suites/hooks.js +++ b/tests/suites/hooks.js @@ -429,6 +429,165 @@ const runHooksTests = async (t, assert, createStorage) => { assert.deepStrictEqual(await host.get(id), { n: 1 }); }); + await t.test('sync hooks are refused at open like storage hooks', async (t2) => { + await assert.rejects(() => createStorage(t2, 'bad', { sync: { hooks: { beforeSned: () => {} } } }), { + name: 'TypeError', + message: "SyncManager: unknown hook 'beforeSned'", + }); + }); + + await t.test('beforeConnect(false) leaves a node registered but not dialled', async (t2) => { + const host = await createStorage(t2, 'H'); + const unit = host.sync.unit({ access: 'public' }); + const seen = []; + let dialled = 0; + let refuse = true; + const dialer = await createStorage(t2, 'D', { + sync: { + autoStart: false, + connect: () => { + dialled++; + return new MemoryLink(unit); + }, + hooks: { + beforeConnect: (node) => { + seen.push(node); + if (refuse) return false; + }, + }, + }, + }); + const errors = []; + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host', scopes: ['*'] }); + await dialer.sync.start(); + await new Promise((resolve) => setTimeout(resolve, 20)); + assert.strictEqual(dialled, 0); + assert.deepStrictEqual(dialer.sync.peers(), []); + assert.deepStrictEqual(errors, []); + assert.strictEqual(dialer.sync.listNodes().length, 1); + assert.strictEqual(seen[0].url, 'mem://host'); + assert.deepStrictEqual(seen[0].scopes, ['*']); + // Enabling again re-asks; this time it is allowed. + refuse = false; + await dialer.sync.enableNode('mem://host', false); + await dialer.sync.enableNode('mem://host'); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + assert.strictEqual(dialled, 1); + }); + + await t.test('a throwing beforeConnect is reported as an error, not thrown', async (t2) => { + const errors = []; + const dialer = await createStorage(t2, 'D', { + sync: { + autoStart: false, + connect: () => assert.fail('never dialled'), + hooks: { + beforeConnect: () => { + throw new Error('no dialling today'); + }, + }, + }, + }); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => errors.length > 0); + assert.strictEqual(errors[0].name, 'HookError'); + assert.strictEqual(errors[0].hook, 'beforeConnect'); + assert.strictEqual(errors[0].url, 'mem://host'); + }); + + await t.test('beforeBootstrap may refuse to join a peerโ€™s state', async (t2) => { + // A host whose log cannot resume a newcomer: the feed says bootstrap. + const host = await createStorage(t2, 'H', { sync: { log: { rollAfter: 2 }, retention: { ops: 4 } } }); + for (let i = 0; i < 8; i++) await host.insert({ i }); + const unit = host.sync.unit({ access: 'public' }); + const seen = []; + let refuse = true; + const dialer = await createStorage(t2, 'D', { + sync: { + autoStart: false, + connect: () => new MemoryLink(unit), + hooks: { + beforeBootstrap: (peer) => { + seen.push(peer); + if (refuse === 'throw') throw new Error('not from you'); + if (refuse) return false; + }, + }, + }, + }); + const bootstraps = []; + const errors = []; + dialer.sync.on('bootstrap', (event) => bootstraps.push(event)); + dialer.sync.on('error', (error) => errors.push(error)); + await dialer.sync.addNode({ url: 'mem://host' }); + await dialer.sync.start(); + await waitUntil(() => seen.length > 0); + await new Promise((resolve) => setTimeout(resolve, 30)); + assert.deepStrictEqual(seen, [{ url: 'mem://host', node: host.nodeId }]); + assert.deepStrictEqual(bootstraps, []); + assert.deepStrictEqual(await dialer.list(), [], 'nothing was joined'); + const [peer] = dialer.sync.peers(); + assert.strictEqual(peer.connected, true, 'still connected'); + assert.strictEqual(peer.subscribed, false, 'but receiving nothing'); + const mine = await dialer.insert({ mine: true }); + await waitUntil(async () => (await host.get(mine)) !== null); + assert.deepStrictEqual(await host.get(mine), { mine: true }, 'pushes still flow'); + // A throw is reported like any other failure on the way in. + refuse = 'throw'; + await dialer.sync.enableNode('mem://host', false); + await dialer.sync.enableNode('mem://host'); + await waitUntil(() => errors.length > 0); + assert.strictEqual(errors[0].name, 'HookError'); + assert.strictEqual(errors[0].hook, 'beforeBootstrap'); + // Allowed: the next handshake bootstraps. + refuse = false; + await dialer.sync.enableNode('mem://host', false); + await dialer.sync.enableNode('mem://host'); + await waitUntil(() => bootstraps.length > 0); + await settle([host, dialer]); + assert.strictEqual((await dialer.list()).length, 9); + }); + + await t.test('the manager announces its lifecycle', async (t2) => { + const host = await createStorage(t2, 'H'); + const hellos = []; + host.sync.on('hello', (event) => hellos.push(event)); + const unit = host.sync.unit({ access: 'public' }); + const dialer = await createStorage(t2, 'D', { sync: { autoStart: false, connect: () => new MemoryLink(unit) } }); + const events = []; + for (const name of ['start', 'stop', 'node', 'connect', 'peer']) { + dialer.sync.on(name, (payload) => events.push([name, payload])); + } + await dialer.sync.addNode({ url: 'mem://host', scopes: ['docs'] }); + await dialer.sync.start(); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + await dialer.sync.enableNode('mem://host', false); + await dialer.sync.enableNode('mem://host'); + await waitUntil(() => dialer.sync.peers()[0]?.subscribed); + await dialer.sync.removeNode('mem://host'); + await dialer.sync.stop(); + assert.deepStrictEqual( + events.filter(([name]) => name !== 'peer'), + [ + ['node', { url: 'mem://host', action: 'add' }], + ['start', undefined], + ['connect', { url: 'mem://host' }], + ['node', { url: 'mem://host', action: 'disable' }], + ['node', { url: 'mem://host', action: 'enable' }], + ['connect', { url: 'mem://host' }], + ['node', { url: 'mem://host', action: 'remove' }], + ['stop', undefined], + ], + ); + assert.deepStrictEqual(hellos, [ + { node: dialer.nodeId, scopes: ['docs'] }, + { node: dialer.nodeId, scopes: ['docs'] }, + ]); + }); + await t.test('the order of a local write', async (t2) => { const order = []; const storage = await createStorage(t2, 'A', { From 587371e84e32c622d84110c1d942ac1d144ec96e Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:28:12 +0300 Subject: [PATCH 13/67] types: event maps, generic on/once/off, the SyncManager surface, a close event MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit StorageEvents, RecordEvents and SyncEvents type every event with its payload, and on/once/off/listenerCount are generic over them on Storage, SyncomRecord and SyncManager. SyncManager was declared with six members while the code and the docs had eleven: started, enableNode, sync, peers, unit and the listeners are declared now, with PeerView, SyncUnitOptions and SyncUnit; Node gains the cursor and remote it always carried. Storage now emits `close` โ€” once, after the log is sealed. Co-Authored-By: Claude Fable 5.1 --- browser.d.ts | 7 +++ index.d.ts | 106 +++++++++++++++++++++++++++++++++++--- src/storage.js | 1 + tests/browser.test-d.ts | 10 +++- tests/index.test-d.ts | 32 ++++++++++++ tests/storage.ops.test.js | 9 ++++ 6 files changed, 156 insertions(+), 9 deletions(-) diff --git a/browser.d.ts b/browser.d.ts index b3b602c..8bb8599 100644 --- a/browser.d.ts +++ b/browser.d.ts @@ -34,6 +34,13 @@ export type { PeerRef, SyncHooks, BootstrapContext, + StorageEvents, + RecordEvents, + SyncEvents, + PeerView, + SyncUnitOptions, + SyncProcedure, + SyncUnit, Collections, Reference, SyncomRecord, diff --git a/index.d.ts b/index.d.ts index cd86224..812b2c9 100644 --- a/index.d.ts +++ b/index.d.ts @@ -475,6 +475,72 @@ export interface Node { scopes: string[]; enabled: boolean; lastSync: number | null; + /** Where the subscription resumes from; `null` until the first page. */ + cursor: string | null; + /** What the last hello learned; `null` before the first. */ + remote: { id: string; epoch: string } | null; +} + +/** One dialed peer, as `sync.peers()` reports it. */ +export interface PeerView { + url: string; + node: string | null; + connected: boolean; + subscribed: boolean; + cursor: string | null; + lastSync: number | null; + /** Ops waiting in the outbox, plus one while a push is in flight. */ + pending: number; +} + +export interface SyncUnitOptions { + /** wrpc's access level for the four procedures (default `'session'`). */ + access?: string; + /** Gates a peer by whatever the transport presented. */ + authorize?: (context: unknown, hello: { node: string; vector?: VersionVector; scopes?: string[]; token?: string }) => boolean | Promise; + /** How a `changes` value is labelled with its position (wrpc's tracked shape by default). */ + tracked?: (id: string, data: unknown) => unknown; + pageSize?: number; + pageBytes?: number; +} + +export interface SyncProcedure { + access: string; + handler: (...args: any[]) => unknown; +} + +/** What a host mounts: `defineRouter({ syncom: storage.sync.unit() })`. */ +export interface SyncUnit { + hello: SyncProcedure; + push: SyncProcedure; + snapshot: SyncProcedure; + changes: SyncProcedure; +} + +/** The events a SyncManager emits; listeners are contained. */ +export interface SyncEvents { + /** A link came up (with the host's node id after hello), went down, or came back. */ + peer: (event: { url: string; connected: boolean; node?: string; reconnected?: boolean }) => void; + /** A push went out (`pushed`) or a page came in (`received` and the apply result). */ + sync: (event: { + url: string; + pushed?: number; + received?: number; + applied?: number; + skipped?: number; + duplicates?: number; + gaps?: number; + }) => void; + bootstrap: (event: { url: string; records: number }) => void; + /** A failure on the way to a peer; `url` names it. */ + error: (error: Error & { url: string }) => void; + start: () => void; + stop: () => void; + node: (event: { url: string; action: 'add' | 'remove' | 'enable' | 'disable' }) => void; + /** A link was accepted, before hello. */ + connect: (event: { url: string }) => void; + /** Hosting side: a peer greeted this node and was authorized. */ + hello: (event: { node: string; scopes: string[] }) => void; } export interface NodeData { @@ -485,11 +551,21 @@ export interface NodeData { export interface SyncManager { readonly storage: Storage; + readonly started: boolean; + on(event: E, listener: SyncEvents[E]): void; + once(event: E, listener: SyncEvents[E]): void; + off(event: E, listener?: SyncEvents[E]): void; + listenerCount(event: keyof SyncEvents): number; addNode(nodeData: NodeData): Promise; removeNode(url: string): Promise; + enableNode(url: string, enabled?: boolean): Promise; listNodes(): Node[]; start(): Promise; stop(): Promise; + /** Pushes whatever the connected peers (or one) lack right now. */ + sync(url?: string): Promise>; + peers(): PeerView[]; + unit(options?: SyncUnitOptions): SyncUnit; } export interface SyncManagerConstructor { @@ -542,6 +618,22 @@ export interface ApplyRemoteResult { gaps: number; } +/** The events a Storage emits; listeners are contained. */ +export interface StorageEvents { + /** Every op applied here โ€” local (`via` null) or from a peer (`via` = the link). */ + change: (op: SyncOp, via: unknown) => void; + /** A peer's op refused by the register/counter rule. */ + error: (error: Error & { op?: SyncOp; rejected?: Array<{ field: string; reason: 'register' | 'counter' }> }) => void; + /** A peer's batch is durable. */ + apply: (batch: ApplyBatchContext) => void; + close: () => void; +} + +export interface RecordEvents { + /** A staged assignment, or the record changing in storage (`null` once deleted). */ + update: (data: unknown, delta: unknown) => void; +} + export interface Storage { readonly schema: Schema | null; readonly sync: SyncManager; @@ -556,10 +648,10 @@ export interface Storage { */ readonly telemetry: unknown; - /** Every op applied here โ€” local (`via` null) or from a peer (`via` = the link). */ - on(event: 'change', listener: (op: SyncOp, via: unknown) => void): void; - on(event: 'error', listener: (error: Error & { op?: SyncOp }) => void): void; - off(event: 'change' | 'error', listener?: (...args: unknown[]) => void): void; + on(event: E, listener: StorageEvents[E]): void; + once(event: E, listener: StorageEvents[E]): void; + off(event: E, listener?: StorageEvents[E]): void; + listenerCount(event: keyof StorageEvents): number; saveData( id: string, @@ -598,9 +690,9 @@ export interface Reference { export interface SyncomRecord { readonly id: string; - on(event: 'update', listener: (data: unknown, delta: unknown) => void): void; - once(event: 'update', listener: (data: unknown, delta: unknown) => void): void; - off(event: 'update', listener?: (data: unknown, delta: unknown) => void): void; + on(event: E, listener: RecordEvents[E]): void; + once(event: E, listener: RecordEvents[E]): void; + off(event: E, listener?: RecordEvents[E]): void; data(): Promise; delta(): Record; save(): Promise; diff --git a/src/storage.js b/src/storage.js index a475da6..1eaeb62 100644 --- a/src/storage.js +++ b/src/storage.js @@ -932,6 +932,7 @@ class Storage extends Emitter { await this.#log.close(); }); this.#closed = true; + notify(this, this.#logger, 'close'); } } diff --git a/tests/browser.test-d.ts b/tests/browser.test-d.ts index 568dcd4..3ec4095 100644 --- a/tests/browser.test-d.ts +++ b/tests/browser.test-d.ts @@ -1,6 +1,6 @@ -import { expectType } from 'tsd'; +import { expectAssignable, expectType } from 'tsd'; import * as browser from '../browser.js'; -import type { Blockchain, CrdtDelta, KeyPair, Storage, SyncManager } from '../browser.js'; +import type { Blockchain, CrdtDelta, KeyPair, Storage, StorageOptions, SyncManager } from '../browser.js'; // The whole point of the split: Web Crypto is async where node:crypto is not. expectType>(browser.calculateHash({ a: 1 })); @@ -25,3 +25,9 @@ expectType(storage.sync); expectType>(browser.list('dir')); expectType>(storage.inc('id', 'hits')); expectType(browser.isAlive(null)); + +// hooks and events reach the browser half unchanged +expectAssignable({ + hooks: { beforeWrite: () => undefined }, + sync: { hooks: { beforeConnect: () => false } }, +}); diff --git a/tests/index.test-d.ts b/tests/index.test-d.ts index 53b2fff..706aed4 100644 --- a/tests/index.test-d.ts +++ b/tests/index.test-d.ts @@ -209,3 +209,35 @@ expectAssignable({ }); expectError(syncom.open({ sync: { hooks: { nope: () => {} } } })); expectError(syncom.open({ sync: { hooks: { beforeBootstrap: 'nope' } } })); + +// events +storage.once('apply', (batch) => { + expectType(batch.skipped); + expectType(batch.ops); +}); +storage.on('error', (error) => { + expectType(error.op); +}); +storage.on('close', () => {}); +expectType(storage.listenerCount('change')); +expectError(storage.on('nope', () => {})); +sync.on('peer', ({ url, connected, node }) => { + expectType(url); + expectType(connected); + expectType(node); +}); +sync.on('sync', (event) => expectType(event.skipped)); +sync.on('node', ({ action }) => expectType<'add' | 'remove' | 'enable' | 'disable'>(action)); +sync.on('hello', ({ node, scopes }) => { + expectType(node); + expectType(scopes); +}); +sync.on('error', (error) => expectType(error.url)); +expectError(sync.on('nope', () => {})); +expectType(sync.started); +expectType>(sync.enableNode('ws://a', false)); +expectType>>(sync.sync()); +expectType(sync.peers()[0].pending); +expectType(sync.listNodes()[0].cursor); +expectType(sync.unit({ access: 'public', authorize: async () => true })); +expectType(sync.unit().hello.access); diff --git a/tests/storage.ops.test.js b/tests/storage.ops.test.js index 934347e..0a6da5a 100644 --- a/tests/storage.ops.test.js +++ b/tests/storage.ops.test.js @@ -335,6 +335,15 @@ test('Storage: the seams telemetry shares', async (t) => { }); }); +test('Storage announces close', async (t) => { + const { storage } = await openTemp(t); + const closed = []; + storage.on('close', () => closed.push(true)); + await storage.close(); + await storage.close(); + assert.deepStrictEqual(closed, [true], 'once, and not again for a second close()'); +}); + test('Storage: events are contained', async (t) => { const collect = () => { const entries = []; From 4d26de9af038c2c9bd5740d5571ef963c285f415 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sat, 5 Sep 2026 19:31:01 +0300 Subject: [PATCH 14/67] docs: hooks and events guide, reference, CLAUDE.md rules A new guide page (docs/guide/hooks.md): what a hook may return and what a throw does, per hook; the order of a local write and of a peer's page; why a skipped op is still logged and relayed; why there is no hook on what leaves for a peer; the events of each class with their payloads. Every example was run before it was written down. The API reference, the sync and storage guides, the README (features, bundle table) and the changelog follow; CLAUDE.md gets the rules that bite. Co-Authored-By: Claude Fable 5.1 --- CHANGELOG.md | 23 +++++ CLAUDE.md | 16 ++- README.md | 5 +- docs/.vitepress/config.mts | 1 + docs/guide/hooks.md | 204 +++++++++++++++++++++++++++++++++++++ docs/guide/storage.md | 1 + docs/guide/sync.md | 13 ++- docs/reference/api.md | 40 ++++++-- 8 files changed, 293 insertions(+), 10 deletions(-) create mode 100644 docs/guide/hooks.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a8e5e44..eb73359 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,29 @@ section until the first publish; everything below lands there on release day. ### Added +- **Hooks.** `open({ hooks })` โ€” `beforeWrite` (sees the op being made and + the record it lands on; may replace the data of a write or a set; a throw + refuses the write), `afterWrite` (the durable op), `beforeRead` / + `afterRead` (refuse or redact what `get()` returns โ€” `swap`, `snapshot` and + the cache stay raw), `beforeApply` (per remote op; `false` skips it here + while it is still logged and relayed, a throw aborts the batch before + anything is durable), `afterApply`. `open({ sync: { hooks } })` โ€” + `beforeConnect` (`false` leaves a node registered but not dialled) and + `beforeBootstrap` (`false` refuses to join a peer's whole state). Hooks run + inside the operation; a throw rejects it with `HookError` (`code: + 'HOOK_FAILED'`, `hook`, `cause`). A write issued from a hook is refused + rather than left to deadlock the queue. `applyRemote` reports `skipped` + and takes a `peer` descriptor. Browser budget raised 17 โ†’ 18 KB. +- **Events are contained.** Every event now goes out through one path: + listeners are invoked at once, never awaited, and a throwing listener is + logged at debug instead of failing the write it observed (or, for the + fire-and-forget emits in `Record` and `SyncManager`, becoming an unhandled + rejection). A `change` listener that writes back queues behind the write. + New events: `apply` and `close` on the storage; `start`, `stop`, `node`, + `connect` and (hosting side) `hello` on the manager. Typed event maps + (`StorageEvents`, `SyncEvents`, `RecordEvents`) and the `SyncManager` + members the types were missing (`started`, `enableNode`, `sync`, `peers`, + `unit`). - **OpenTelemetry, injected.** `open({ telemetry: { api } })` (the `@opentelemetry/api` module) or `{ tracer, meter }` โ€” the same object wrpc's `telemetry` option takes, and no dependency on either. Spans: diff --git a/CLAUDE.md b/CLAUDE.md index cf13a7e..90e49bf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -110,6 +110,19 @@ docs/ VitePress site - **Encrypted records are local.** `{ encrypted: true }` seals a record for this node's key; it produces no op and is excluded from snapshots. Do not route an encrypted entry through `applyOp`. +- **Hooks run inside the operation; events never do.** `src/hooks.js` (no + requires โ€” it ships to the browser) is the whole mechanism. Storage hooks + go through `#hook()` inside the mutation queue, and `#run` refuses a write + while one is running โ€” a write from a hook would wait for the hook that + waits for it. Hook calls live in `#commit`, `get`, `#applyRemote`, + `#connect` and `#bootstrap` only; there is deliberately no hook on the ops + that leave for a peer (a withheld op is a gap that stops that stream โ€” + scopes are the per-peer filter). Every event leaves through `notify()`: + never `await this.emit(...)`, never emit `error` any other way (the Emitter + throws on an unheard `error`, and that throw used to fail a push before the + vector moved). A remote op a `beforeApply` skipped is still appended, with + `skipped: true` on its log entry โ€” that flag is what keeps `#replayTail` + from applying it on reopen; do not strip it. ## Testing @@ -137,7 +150,8 @@ running it before writing it down โ€” several examples inherited from the legacy repo were wrong in exactly that way. A new page needs a sidebar entry in `docs/.vitepress/config.mts`; `docs/guide/sync.md` and `docs/guide/wrpc.md` are the two pages that describe the protocol โ€” keep them in step with -`src/sync.js`. +`src/sync.js`, and `docs/guide/hooks.md` with the hook and event surface of +`src/storage.js` and `src/sync.js`. ## Formatting diff --git a/README.md b/README.md index 64431e7..5fa485c 100644 --- a/README.md +++ b/README.md @@ -76,8 +76,8 @@ that fails the build when exceeded. | Entry | raw | min | min+gzip | budget | | ----- | ---:| ---:| --------:| ------:| -| main entry โ€” browser (@alexify/syncom) | 114.6 KB | 46.6 KB | 16.1 KB | 17.0 KB | -| main entry โ€” node (@alexify/syncom) | 106.6 KB | 45.6 KB | 15.8 KB | โ€” | +| main entry โ€” browser (@alexify/syncom) | 122.3 KB | 49.8 KB | 17.3 KB | 18.0 KB | +| main entry โ€” node (@alexify/syncom) | 113.9 KB | 48.8 KB | 17.0 KB | โ€” | | wrpc adapter โ€” browser (@alexify/syncom/wrpc) | 3.3 KB | 1.4 KB | 0.7 KB | 2.0 KB | ## Quick Start @@ -144,6 +144,7 @@ nodes converges. | **CRDT** | `diff` / `merge` / `apply` over plain objects โ€” nested maps merge recursively, arrays are grow-only sets, numbers add so counters converge | | **Contracts** | Store a procedure in the chain by hash, load it anywhere, run it against a read-only view of storage; failures are recorded as blocks | | **Portability** | One API over `node:fs` + `node:crypto` on the server and OPFS + Web Crypto in the browser, chosen by the `browser` export condition | +| **Hooks and events** | `open({ hooks })` runs your code inside a write, a read or a peer's batch โ€” validate, stamp, redact, skip โ€” and rejects with a typed `HookError` when it throws; `on()` events (`change`, `apply`, `peer`, `sync`, โ€ฆ) observe everything and can never break it. See [Hooks and events](https://syncomjs.vercel.app/guide/hooks) | | **Observability** | Injected OpenTelemetry (`open({ telemetry })`): spans for local writes, applied batches, pushes, bootstraps and log rolls, plus outcome and lag metrics โ€” nested under wrpc's spans, which carry the trace across the wire. See [OpenTelemetry](https://syncomjs.vercel.app/guide/telemetry) | | **Dependencies** | None. wrpc is a devDependency for one integration test and is injected at runtime | diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index d81edb2..3378ea1 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -142,6 +142,7 @@ export default withMermaid({ text: 'Core', items: [ { text: 'Storage', link: '/guide/storage' }, + { text: 'Hooks and events', link: '/guide/hooks' }, { text: 'CRDT deltas', link: '/guide/crdt' }, { text: 'Blockchain log', link: '/guide/chain' }, { text: 'Smart contracts', link: '/guide/contracts' }, diff --git a/docs/guide/hooks.md b/docs/guide/hooks.md new file mode 100644 index 0000000..f768316 --- /dev/null +++ b/docs/guide/hooks.md @@ -0,0 +1,204 @@ +# Hooks and events + +Two ways to attach code to what a storage does, kept deliberately apart: + +- **Hooks** run *inside* an operation. They are configured once, in `open()`, + one function per name, and awaited where the operation happens โ€” so a hook + can alter what is written, refuse it, or skip a peer's op. A hook that + throws fails the operation with a `HookError`; it is never swallowed. +- **Events** are what `on()` subscribes to from the outside, after the fact. + Any number of listeners, none of which can affect the operation: a listener + that throws is logged and forgotten, and one that writes back to the storage + queues behind the write it heard about. + +If the code should be able to change the outcome, it is a hook. If it should +only ever watch โ€” metrics, an audit trail, a UI โ€” it is a listener. + +```js +const { open, HookError } = require('@alexify/syncom'); + +const storage = await open({ + path: 'data', + hooks: { + beforeWrite: ({ type, data, current }) => { + if (type === 'delete') return; + const next = { ...current, ...data }; + if (typeof next.title !== 'string' || next.title.length === 0) { + throw new Error('title is required'); + } + // A plain object replaces the data of a write or a set. + if (type === 'write' || type === 'set') return { ...data, updatedAt: Date.now() }; + }, + afterRead: (ctx, data) => { + if (data === null || data.ssn === undefined) return; + return { ...data, ssn: '***' }; + }, + }, + sync: { + hooks: { + beforeConnect: (node) => node.url.startsWith('wss://'), + }, + }, +}); + +storage.on('change', (op, via) => metrics.count('ops', { local: via === null })); +``` + +An unknown hook name, or a value that is not a function, is a `TypeError` at +`open()` โ€” a hook with a typo that silently never runs is a validation that +silently never runs. + +## Storage hooks + +| Hook | Runs | Return | Throw | +| --- | --- | --- | --- | +| `beforeWrite(ctx)` | Inside every local write, after the current record is loaded and before the op is stamped and applied | A plain object replaces `data` (`write` and `set` only; a `TypeError` for anything else) | The write is refused: nothing stamped, logged or written | +| `afterWrite(ctx, op)` | After the write is durable โ€” `op` is the op it became, `null` for an encrypted (local) write; not called when the write changed nothing | Ignored | The call rejects, but the op stays committed and is announced | +| `beforeRead({ id })` | Inside `get(id)`, before the read | Ignored | The read is refused | +| `afterRead({ id }, data)` | Inside `get(id)`, after the read | Anything but `undefined` (including `null`) replaces what the caller sees | The read rejects | +| `beforeApply(ctx)` | Inside `applyRemote`, once per op, after the duplicate/gap checks and before the op is applied | `false` skips the op *here* (see below) | The whole batch is aborted before anything is durable | +| `afterApply(batch)` | After a peer's batch is durable, once per `applyRemote` call | Ignored | The call rejects; the batch stays and is announced | + +`ctx` for a write is `{ type, id, data?, key?, value?, meta?, current, +encrypted, source, via, from }`: `type` is the op type (`write`, `set`, `inc`, +`delete`, `merge`), `data` is the whole record for a `write` and the delta for +a `set`, `key`/`value` belong to an `inc`, `current` is the record as it is +now (`null` when absent or deleted โ€” read it, do not change it), `encrypted` +says whether the write is sealed for this node, and `source` is the `Record` +whose `save()` this is. For a remote op it is `{ op, via, peer, current }`, +where `peer` is `{ url, node }` โ€” `url` is `null` when this node is the host. + +The order of a local write: + +``` +load the record โ†’ beforeWrite โ†’ apply the op โ†’ append to the log โ†’ chain block +โ†’ record file โ†’ afterWrite โ†’ Record 'update' โ†’ 'change' +``` + +and of a page from a peer: + +``` +per op: malformed / duplicate / gap checks โ†’ beforeApply โ†’ apply +then: append to the log โ†’ persist the records โ†’ afterApply โ†’ 'apply' โ†’ 'change' ร— n +``` + +### Reads stay raw where it matters + +`afterRead` changes what `get()` โ€” and everything that reads through it: a +`Record`'s live properties, `Collection.get`, a contract's `DataReader` โ€” +hands back. It does not touch the record: `swap()` compares against the raw +data, `snapshot()` replicates the raw data, `getCachedData()` and `has()` are +raw. A `Record` that saved a redacted field would write the redaction back; +it does not, because `save()` writes only the fields that were assigned. + +### Skipping a peer's op + +`beforeApply` returning `false` leaves *this node's* record alone โ€” and +nothing else. The op is still appended to this node's log, still counted in +the vector, still relayed to the peers behind this node, and still announced +as a `change`. That is not a shortcut: sequence numbers run contiguously per +origin and scope, so an op missing from a log is a gap that stops every later +op of that stream for everyone who resumes from it, and a vector that does not +move has the sender offer the same page forever. Skipping is a local decision; +peers make their own with their own `beforeApply`. A skipped op is flagged in +the log so a reopen's replay of the unsealed tail leaves it alone too. + +`applyRemote` reports it: `{ applied, skipped, duplicates, gaps }` โ€” `applied` +counts every accepted op, skipped ones included; `applied - skipped` reached a +record. + +A throwing `beforeApply` aborts the batch before anything is durable. The +sender sees a failed push (on the dialing side its `error` event fires, with +`error.url`), keeps its cursor, and offers the page again at the next +handshake. + +### Hooks and the queue + +Every local write and every peer batch runs through one mutation queue, and +hooks run inside it. A hook that awaits something slow holds every other +write for that long. A hook that *writes* would wait for the queue, which is +waiting for the hook โ€” so a write issued from any hook is refused with an +error rather than left to deadlock. Reads (`get`, `has`, `record`) are fine. + +## Sync hooks + +`open({ sync: { hooks } })`: + +| Hook | Runs | Return | Throw | +| --- | --- | --- | --- | +| `beforeConnect(node)` | Before a registered node is dialled โ€” at `start()`, `addNode()` while started, or `enableNode()`. `node` is a copy of the registry entry | `false` leaves it registered but not dialled | Reported through the manager's `error` event, with `error.url` | +| `beforeBootstrap({ url, node })` | When a host's log cannot resume this node and its whole state is about to be joined | `false` refuses: the peer stays connected for pushes but nothing is received from it until the next handshake asks again | Reported through `error` | + +There is deliberately no hook on the ops that leave for a peer. Seqs run +contiguously per origin and scope, so an op withheld from one peer is a gap +that stops every later op of that stream at it. Per-peer visibility is what +[scopes](./sync#scopes) are for: a scope is its own stream, and a node only +receives the scopes it asked for. + +```js +const storage = await open({ + path: 'data', + hooks: { + // Only the admin node may delete anything here. + beforeApply: ({ op, peer }) => { + if (op.type === 'delete' && peer?.node !== ADMIN_NODE) return false; + }, + }, + sync: { + connect, + hooks: { + beforeBootstrap: ({ node }) => trusted.has(node), + }, + }, +}); +``` + +## Events + +Every listener is contained: it is invoked at once, never awaited, and if it +throws the failure is logged at `debug` (`{ event: 'listener.error', name }`) +and the operation carries on. `error` with nobody listening is silent rather +than a throw. + +### Storage + +| Event | Payload | When | +| --- | --- | --- | +| `change` | `(op, via)` | Every op applied here; `via` is `null` for a local write, otherwise the link it came from | +| `apply` | `({ via, peer, ops, applied, skipped, duplicates, gaps })` | A peer's batch is durable โ€” `ops` are the accepted ones, skipped included | +| `error` | `(error)` with `error.op`, `error.rejected` | A peer's op refused by the register/counter rule | +| `close` | โ€” | After `close()` sealed the log | + +A `Record` emits `update` `(data, delta)` on a staged assignment and when the +record changes in storage (`null` data once deleted). + +### SyncManager + +| Event | Payload | When | +| --- | --- | --- | +| `start` / `stop` | โ€” | Dialling began / every link was closed | +| `node` | `({ url, action })` | The registry changed: `'add'`, `'remove'`, `'enable'`, `'disable'` | +| `connect` | `({ url })` | A link was accepted, before hello | +| `peer` | `({ url, connected, node?, reconnected? })` | Hello answered (with the host's `node` id), the link closed, or it came back | +| `sync` | `({ url, pushed })` / `({ url, received, applied, skipped, duplicates, gaps })` | A push went out / a page came in | +| `bootstrap` | `({ url, records })` | A host's whole state was joined | +| `error` | `(error)` with `error.url` | A failure on the way to a peer โ€” a refused link, a failed push, a hook that threw | +| `hello` | `({ node, scopes })` | Hosting side: a peer greeted this node and was authorized | + +## `HookError` + +What an operation rejects with when one of its hooks threw: + +```js +try { + await storage.insert({}); +} catch (error) { + error instanceof HookError; // true + error.code; // 'HOOK_FAILED' + error.hook; // 'beforeWrite' + error.cause.message; // 'title is required' +} +``` + +For a `before*` hook the operation did not happen. For an `after*` hook it +did โ€” the op is committed and announced; only the caller's promise rejects. diff --git a/docs/guide/storage.md b/docs/guide/storage.md index d97fa44..08ccd94 100644 --- a/docs/guide/storage.md +++ b/docs/guide/storage.md @@ -18,6 +18,7 @@ const storage = await open({ path: 'data' }); | `logger` | `console` | A Console or pino-shaped logger, or `false` for silence | | `encrypted` | `false` | Encrypt every record at rest by default ([see below](#encryption-at-rest)) | | `sync` | โ€” | Replication: `connect`, `scope`, `retention`, โ€ฆ โ€” see [Sync](./sync#options) | +| `hooks` | โ€” | `beforeWrite`, `afterRead`, `beforeApply`, โ€ฆ โ€” see [Hooks and events](./hooks) | ## The explicit API diff --git a/docs/guide/sync.md b/docs/guide/sync.md index ad239d7..1ff718c 100644 --- a/docs/guide/sync.md +++ b/docs/guide/sync.md @@ -248,14 +248,24 @@ await storage.sync.sync(); // push what the peers lack right now storage.sync.peers(); // [{ url, node, connected, subscribed, cursor, lastSync, pending }] storage.sync.on('peer', ({ url, connected, node, reconnected }) => {}); -storage.sync.on('sync', ({ url, pushed, received, applied, duplicates, gaps }) => {}); +storage.sync.on('sync', ({ url, pushed, received, applied, skipped, duplicates, gaps }) => {}); storage.sync.on('bootstrap', ({ url, records }) => {}); storage.sync.on('error', (error) => {}); // error.url names the peer +storage.sync.on('start', () => {}); +storage.sync.on('stop', () => {}); +storage.sync.on('node', ({ url, action }) => {}); // 'add' | 'remove' | 'enable' | 'disable' +storage.sync.on('connect', ({ url }) => {}); // a link was accepted, before hello +storage.sync.on('hello', ({ node, scopes }) => {}); // hosting side: a peer was authorized storage.on('change', (op, via) => {}); // every op applied here; via is null for a local write +storage.on('apply', ({ peer, applied, skipped }) => {}); // a peer's batch is durable storage.on('error', (error) => {}); // a peer's op refused by the register/counter rule ``` +Listeners are contained: one that throws is logged and never fails what it +observed. To take part in an operation โ€” refuse a write, skip a peer's op, +decline to dial a node โ€” use [hooks](./hooks) instead. + `await storage.close()` stops sync and seals the log. ## Options @@ -269,6 +279,7 @@ storage.on('error', (error) => {}); // a peer's op refused by the register/c | `retention.days` | โ€” | Age after which a segment is dropped | | `log.rollAfter` | `64` | Appended batches per sealed segment | | `pageSize` | `500` | Ops per push, records per snapshot page | +| `hooks` | โ€” | `beforeConnect`, `beforeBootstrap` โ€” see [Hooks and events](./hooks#sync-hooks) | The `telemetry` option sits beside `sync` on `open()`, not inside it โ€” see [OpenTelemetry](./telemetry) for the spans and metrics replication emits. diff --git a/docs/reference/api.md b/docs/reference/api.md index e2d412d..cf34531 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -15,7 +15,8 @@ const syncom = require('@alexify/syncom'); Returns `Promise`. Options: `path` (default `'syncom'`), `name` (alias for `path`), `schema`, `getId`, `logger`, `encrypted`, `sync` (see [Sync options](../guide/sync#options)), `telemetry` (see -[OpenTelemetry](../guide/telemetry)). +[OpenTelemetry](../guide/telemetry)), `hooks` (see +[Hooks and events](../guide/hooks)). ### `Storage` @@ -39,7 +40,7 @@ Returns `Promise`. Options: `path` (default `'syncom'`), `name` | `validate(id, data, blockHash)` | `Promise` | One record against its block | | `getCachedData(id)` | `unknown \| null` | Synchronous cache peek | | `hasRecord(id)` | `boolean` | Whether the id is known to the cache | -| `applyRemote(ops, { via, scopes }?)` | `Promise<{ applied, duplicates, gaps }>` | Applies a peer's ops, in order per origin | +| `applyRemote(ops, { via, scopes, peer }?)` | `Promise<{ applied, skipped, duplicates, gaps }>` | Applies a peer's ops, in order per origin; `peer` is `{ url, node }` for the hooks | | `snapshot({ cursor, limit, scopes }?)` | `Promise<{ records, cursor, eventId, vector }>` | One page of state, tombstones included | | `mergeSnapshot(records, { via, from }?)` | `Promise` | Joins a page (as `merge` ops of this node) | | `adoptVector(vector, { via, from }?)` | `Promise` | Records what a joined snapshot covers (an `advance` op) | @@ -50,9 +51,19 @@ Properties: `schema`, `sync`, `nodeId`, `path`, `log`, `vector`, `logger`, Statics: `Storage.Collection`, `Storage.Record`, `Storage.generateId()` โ€” the default `getId`, synchronous in both halves. -Events: `change` `(op, via)` for every op applied here (`via` is `null` for a -local write, otherwise the link it came from); `error` for a peer's op refused -by the register/counter rule. +Events โ€” listeners are contained, a throwing one is logged and never fails +the operation (see [Hooks and events](../guide/hooks#events)): + +| Event | Payload | When | +| --- | --- | --- | +| `change` | `(op, via)` | Every op applied here; `via` is `null` for a local write, otherwise the link it came from | +| `apply` | `({ via, peer, ops, applied, skipped, duplicates, gaps })` | A peer's batch is durable | +| `error` | `(error)` with `error.op`, `error.rejected` | A peer's op refused by the register/counter rule | +| `close` | โ€” | After `close()` sealed the log | + +Hooks (`open({ hooks })`): `beforeWrite`, `afterWrite`, `beforeRead`, +`afterRead`, `beforeApply`, `afterApply` โ€” see +[Hooks and events](../guide/hooks). ### `Record` @@ -88,7 +99,24 @@ Extends `Emitter`. See [Sync](../guide/sync). | `peers()` | `Array<{ url, node, connected, subscribed, cursor, lastSync, pending }>` | Live view | | `unit({ access, authorize, tracked, pageSize, pageBytes }?)` | router unit | The procedures a host mounts: `hello`, `push`, `snapshot`, `changes` | -Properties: `storage`, `started`. Events: `peer`, `sync`, `bootstrap`, `error`. +Properties: `storage`, `started`. + +Events (contained, like the storage's): `peer` `({ url, connected, node?, +reconnected? })`, `sync` `({ url, pushed })` after a push and `({ url, +received, applied, skipped, duplicates, gaps })` after a received page, +`bootstrap` `({ url, records })`, `error` (with `.url`), `start`, `stop`, +`node` `({ url, action: 'add' | 'remove' | 'enable' | 'disable' })`, `connect` +`({ url })` once a link is accepted (before hello), and on the hosting side +`hello` `({ node, scopes })` once a peer is authorized. + +Hooks (`open({ sync: { hooks } })`): `beforeConnect`, `beforeBootstrap` โ€” see +[Hooks and events](../guide/hooks#sync-hooks). + +### `HookError` + +What an operation rejects with when one of its hooks threw: `name` +`'HookError'`, `code` `'HOOK_FAILED'`, `hook` (the hook's name), `cause` (the +original error). Exported from the package root. ### `isSyncLink(value)` From da4d614d268af22d3249cb194942c1882f0c58a8 Mon Sep 17 00:00:00 2001 From: Alex Dolid Date: Sun, 18 Oct 2026 12:07:22 +0300 Subject: [PATCH 15/67] chore: add Claude skills from metaschema Six of metaschema's seven metaskills v1.0.5 skills. data-structures, error-handling, js-data-structures and js-gof are verbatim copies. js-conventions (120 columns, 'use strict', no runtime dependencies, the engine's error and hot-path rules) and npm-publish (CI gates, hand-synced docs nav version, first-release notes) are adapted to syncom. metautil-data-structures is left out: syncom takes no runtime dependencies. CLAUDE.md gains a Skills section. Co-Authored-By: Claude Sonnet 5.5 --- .claude/skills/data-structures/SKILL.md | 515 ++++++++++++ .claude/skills/error-handling/SKILL.md | 127 +++ .claude/skills/js-conventions/SKILL.md | 109 +++ .claude/skills/js-data-structures/SKILL.md | 209 +++++ .claude/skills/js-gof/SKILL.md | 873 +++++++++++++++++++++ .claude/skills/npm-publish/SKILL.md | 219 ++++++ CLAUDE.md | 19 + 7 files changed, 2071 insertions(+) create mode 100644 .claude/skills/data-structures/SKILL.md create mode 100644 .claude/skills/error-handling/SKILL.md create mode 100644 .claude/skills/js-conventions/SKILL.md create mode 100644 .claude/skills/js-data-structures/SKILL.md create mode 100644 .claude/skills/js-gof/SKILL.md create mode 100644 .claude/skills/npm-publish/SKILL.md diff --git a/.claude/skills/data-structures/SKILL.md b/.claude/skills/data-structures/SKILL.md new file mode 100644 index 0000000..af30b8c --- /dev/null +++ b/.claude/skills/data-structures/SKILL.md @@ -0,0 +1,515 @@ +--- +name: data-structures +description: Implements custom JavaScript data structures: queues, deques, stacks, linked lists, cons lists, circular buffers, unrolled lists, tries, heaps, graphs, LRU caches, CRDTs, pools, structs. Use when building or choosing non-native collections, optimizing enqueue/dequeue, designing persistent lists, or when the user asks for data structures beyond Map/Set/Array/Object. +--- + +# Custom Data Structures (JavaScript) + +## Complexity and code characteristics + +Choose a structure for the property you need to control โ€” not only for speed. + +**Big-O (typical; n = size, k = key/word length):** + +- Singly-linked push/pop at head: O(1); search / index: O(n) +- Doubly-linked append/prepend / splice at known node: O(1); index: O(n) (nearer-end walk helps) +- Cons `prepend` / `uncons`: O(1); `reverse` / `map` / random access: O(n); tails share structure +- CircularBuffer / Queue / Deque / Stack ends: amortized O(1); grow rare O(n); never hot `Array.shift` +- UnrolledList enqueue/dequeue: amortized O(1); better locality than per-item nodes; pool cuts allocs +- BST insert/search: O(log n) balanced, O(n) skewed; in-order: O(n) +- Binary heap push/pop: O(log n); peek: O(1) +- Trie insert/has: O(k); autocomplete: O(matches ร— k) +- Adjacency-list addEdge: O(1); BFS/DFS: O(V + E) +- LRU (Map) get/set/evict: O(1); G-Counter inc O(1), merge O(replicas); Pool capture O(1) amortized +- Prefer the structure whose hot operation is O(1) or O(log n); measure before optimizing rare paths + +**Readability and semantics:** + +- Name by role: `pending` (Queue), `frontier` (Deque), `undo` (Stack) โ€” not `buffer1` +- Type states intent: ring/unrolled = throughput; cons = persistent; heap = priority; trie = prefix +- Small public API (`enqueue`/`dequeue`) โ€” do not expose nodes or buffers + +**Stability and contracts:** + +- Return copies or iterators; document live vs snapshot and iteration order +- Cap growth (capacity, pool size, LRU max); unbounded queues/caches are operability bugs +- Immutable Cons/Struct: updates are new values (`prepend`, `fork`); mutable lists document aliasing + +**Testability:** + +- Assert on contents (`[...q]`, `toArray()`, size), not private fields; keep `Symbol.iterator` +- Deterministic fixtures; inject clocks/timeouts for Pool waiters + +**Encapsulation and cost:** + +- Hide representation so callers survive swaps (array โ†’ ring โ†’ unrolled) +- Amortized grow and node pools trade memory for latency; clear slots on dequeue for GC +- Power-of-2 ring capacity; index with `& (len - 1)` instead of `%` + +## Lists + +Singly-linked (push/pop at head): + +```javascript +class LinkedList { + #head = null; + #length = 0; + + push(data) { + this.#head = { data, next: this.#head }; + this.#length++; + } + + pop() { + if (!this.#head) return undefined; + const { data } = this.#head; + this.#head = this.#head.next; + this.#length--; + return data; + } + + *[Symbol.iterator]() { + let node = this.#head; + while (node) { + yield node.data; + node = node.next; + } + } +} +``` + +Doubly-linked list ideas: + +- `#head` / `#tail` / `#size`; nodes `{ value, prev, next }` (fixed key order) +- Resolve index from nearer end; splice ranges by relinking, not rebuild +- Ops worth adding: `append`/`prepend`/`insert`/`delete`, `rotate`, `move`, `slice`/`take`/`drop`, `reverse`, `groupBy` โ†’ `Map` of lists +- `Symbol.iterator`; `toArray` only at boundaries + +Immutable cons list ideas: + +- Cell `{ value, next, size }` + singleton empty; `prepend` returns a new cell (share the old list) +- Build from arrays backwards; merge by prepending earlier lists onto the last +- Prefer for persistent pipelines; use mutable doubly-linked for mid-list edits + +## Circular Buffer + +Growable ring, power-of-2 capacity, bitmask index โ€” backing for Queue / Deque / Stack: + +```javascript +class CircularBuffer { + #buffer = new Array(16); + #head = 0; + #size = 0; + + get size() { + return this.#size; + } + + #grow() { + const cap = this.#buffer.length; + const mask = cap - 1; + const next = new Array(cap * 2); + for (let i = 0; i < this.#size; i++) { + next[i] = this.#buffer[(this.#head + i) & mask]; + } + this.#buffer = next; + this.#head = 0; + } + + push(value) { + if (this.#size === this.#buffer.length) this.#grow(); + const m = this.#buffer.length - 1; + this.#buffer[(this.#head + this.#size) & m] = value; + this.#size++; + } + + unshift(value) { + if (this.#size === this.#buffer.length) this.#grow(); + const m = this.#buffer.length - 1; + this.#head = (this.#head - 1) & m; + this.#buffer[this.#head] = value; + this.#size++; + } + + shift() { + if (!this.#size) return undefined; + const m = this.#buffer.length - 1; + const v = this.#buffer[this.#head]; + this.#buffer[this.#head] = undefined; + this.#head = (this.#head + 1) & m; + this.#size--; + return v; + } + + pop() { + if (!this.#size) return undefined; + const m = this.#buffer.length - 1; + const i = (this.#head + this.#size - 1) & m; + const v = this.#buffer[i]; + this.#buffer[i] = undefined; + this.#size--; + return v; + } + + at(index) { + const i = index < 0 ? this.#size + index : index; + if (i < 0 || i >= this.#size) return undefined; + return this.#buffer[(this.#head + i) & (this.#buffer.length - 1)]; + } +} +``` + +Ideas: clear slots on remove (GC); `fromArray` with capacity `2^k > n`; iterate `(head + i) & mask`. + +## Queue / Deque / Stack + +Compose on CircularBuffer โ€” never hot `Array.shift()`: + +```javascript +class Queue { + #buf = new CircularBuffer(); + enqueue(v) { + this.#buf.push(v); + } + dequeue() { + return this.#buf.shift(); + } + peek() { + return this.#buf.at(0); + } + get size() { + return this.#buf.size; + } +} + +class Deque { + #buf = new CircularBuffer(); + push(v) { + this.#buf.push(v); + } + pop() { + return this.#buf.pop(); + } + unshift(v) { + this.#buf.unshift(v); + } + shift() { + return this.#buf.shift(); + } +} + +class Stack { + #buf = new CircularBuffer(); + push(v) { + this.#buf.push(v); + } + pop() { + return this.#buf.pop(); + } + peek() { + return this.#buf.at(-1); + } +} +``` + +## Unrolled List + +Linked fixed-size buffers + node pool (high-throughput queue): + +```javascript +class UnrolledNode { + constructor(size) { + this.buffer = new Array(size); + this.size = size; + this.readIndex = 0; + this.writeIndex = 0; + this.length = 0; + this.next = null; + } + enqueue(item) { + if (this.writeIndex >= this.size) return false; + this.buffer[this.writeIndex++] = item; + this.length++; + return true; + } + dequeue() { + if (!this.length) return undefined; + const i = this.readIndex++; + const item = this.buffer[i]; + this.buffer[i] = undefined; + this.length--; + return item; + } + reset() { + this.readIndex = this.writeIndex = this.length = 0; + this.next = null; + } +} +``` + +Ideas: pool of N nodes (`acquire`/`release`); enqueue grows a new node when full; dequeue releases empty tail nodes back to the pool; typical `nodeSize` 1024. + +## Binary Search Tree + +```javascript +class BinarySearchTree { + constructor(data) { + this.data = data; + this.left = null; + this.right = null; + } + + insert(data) { + if (data < this.data) { + if (this.left) this.left.insert(data); + else this.left = new BinarySearchTree(data); + } else { + if (this.right) this.right.insert(data); + else this.right = new BinarySearchTree(data); + } + } + + *inOrder() { + if (this.left) yield* this.left.inOrder(); + yield this.data; + if (this.right) yield* this.right.inOrder(); + } +} +``` + +## Heap / Priority Queue + +```javascript +class MinHeap { + #data = []; + + #parent(i) { + return (i - 1) >> 1; + } + #left(i) { + return 2 * i + 1; + } + #right(i) { + return 2 * i + 2; + } + #swap(i, j) { + [this.#data[i], this.#data[j]] = [this.#data[j], this.#data[i]]; + } + + push(value) { + this.#data.push(value); + let i = this.#data.length - 1; + while (i > 0 && this.#data[i] < this.#data[this.#parent(i)]) { + this.#swap(i, this.#parent(i)); + i = this.#parent(i); + } + } + + pop() { + const top = this.#data[0]; + const last = this.#data.pop(); + if (this.#data.length === 0) return top; + this.#data[0] = last; + let i = 0; + while (true) { + let smallest = i; + const l = this.#left(i), + r = this.#right(i); + if (l < this.#data.length && this.#data[l] < this.#data[smallest]) + smallest = l; + if (r < this.#data.length && this.#data[r] < this.#data[smallest]) + smallest = r; + if (smallest === i) break; + this.#swap(i, smallest); + i = smallest; + } + return top; + } + + get size() { + return this.#data.length; + } +} +``` + +## LRU Cache + +```javascript +class LRUCache { + #capacity; + #cache = new Map(); + + constructor(capacity) { + this.#capacity = capacity; + } + + get(key) { + if (!this.#cache.has(key)) return undefined; + const value = this.#cache.get(key); + this.#cache.delete(key); + this.#cache.set(key, value); + return value; + } + + set(key, value) { + this.#cache.delete(key); + this.#cache.set(key, value); + if (this.#cache.size > this.#capacity) { + this.#cache.delete(this.#cache.keys().next().value); + } + } +} +``` + +## Trie (prefix tree) + +`Object.create(null)` nodes + `Symbol` terminal (no char-key collisions). Ideas: store values, `#size`, autocomplete, prune empty branches on delete: + +```javascript +const VALUE = Symbol('value'); + +class Trie { + #root = Object.create(null); + #size = 0; + + insert(word, value = true) { + let node = this.#root; + for (const ch of word) node = node[ch] ??= Object.create(null); + if (!Object.hasOwn(node, VALUE)) this.#size++; + node[VALUE] = value; + } + + has(word) { + const n = this.#find(word); + return n !== null && Object.hasOwn(n, VALUE); + } + + complete(prefix) { + const node = this.#find(prefix); + if (!node) return []; + const out = []; + (function walk(n, path) { + if (Object.hasOwn(n, VALUE)) out.push(path); + for (const ch of Object.keys(n)) walk(n[ch], path + ch); + })(node, prefix); + return out; + } + + #find(word) { + let node = this.#root; + for (const ch of word) { + node = node[ch]; + if (!node) return null; + } + return node; + } +} +``` + +Delete idea: walk path storing parents; remove `VALUE`; walk up deleting children with no `VALUE` and no keys. + +## Graph (adjacency list) + +```javascript +class Graph { + #adj = new Map(); + + addEdge(from, to) { + (this.#adj.get(from) ?? this.#adj.set(from, []).get(from)).push(to); + } + + *bfs(start) { + const visited = new Set([start]); + const queue = new Queue(); + queue.enqueue(start); + while (queue.size) { + const node = queue.dequeue(); + yield node; + for (const n of this.#adj.get(node) ?? []) { + if (!visited.has(n)) { + visited.add(n); + queue.enqueue(n); + } + } + } + } +} +``` + +## Resource Pool + +Ideas: parallel `#items` / `#free` arrays; round-robin `#current`; `Lease` with one-shot `release`; `WeakSet` of valid leases; when empty, queue waiters (optional timeout); on release, hand the same slot to the next waiter or mark free. + +```javascript +class Lease { + #release; + #released = false; + constructor(resource, release) { + this.resource = resource; + this.#release = release; + } + release() { + if (this.#released) throw new Error('already released'); + this.#released = true; + this.#release(); + } +} +``` + +## Struct (typed record factory) + +Ideas: `Struct.immutable` / `Struct.mutable` from defaults; infer field types; `Object.freeze` vs `Object.seal`; `fork` = copy+updates; `branch` = prototype overlay for cheap variants; validate unknown fields / wrong types; deep-copy array/object defaults. + +```javascript +class Struct { + static immutable(name, defaults) { + const fields = Object.keys(defaults); + const Entity = { + [name]: class { + constructor(data = {}) { + for (const k of fields) + this[k] = Object.hasOwn(data, k) ? data[k] : defaults[k]; + Object.freeze(this); + } + fork(updates = {}) { + return new Entity({ ...this, ...updates }); + } + }, + }[name]; + return Entity; + } +} +``` + +## CRDT (G-Counter) + +```javascript +class GCounter { + #id; + #counts; + + constructor(id, size) { + this.#id = id; + this.#counts = new Array(size).fill(0); + } + + inc(x = 1) { + this.#counts[this.#id] += x; + } + + merge(remote) { + for (let i = 0; i < this.#counts.length; i++) { + this.#counts[i] = Math.max(this.#counts[i], remote[i]); + } + } + + get value() { + return this.#counts.reduce((a, b) => a + b, 0); + } +} +``` + +## Conventions + +- Prefer CircularBuffer or UnrolledList for Queue/Deque/Stack +- `Object.create(null)` for trie nodes; Symbol for reserved keys; LRU via Map order; CRDTs for distributed counters diff --git a/.claude/skills/error-handling/SKILL.md b/.claude/skills/error-handling/SKILL.md new file mode 100644 index 0000000..cb4a838 --- /dev/null +++ b/.claude/skills/error-handling/SKILL.md @@ -0,0 +1,127 @@ +--- +name: error-handling +description: Apply error handling and recovery patterns in JavaScript/TypeScript or Node.js. Use when implementing error handling, retry logic, or when the user mentions domain errors, error recovery, error escalation. +--- + +# Error Handling + +## Error classification + +- Programming errors: Bugs (TypeError, ReferenceError, assertion failures, etc.). Fix the code; do not catch and continue +- Operational errors: Expected failures (network timeout, file not found, invalid input, etc.). Handle gracefully with recovery, escalation, user notification, or logging + +## Sync error handling + +```javascript +try { + const result = parse(input); + return result; +} catch (error) { + console.error({ error }); + return defaultValue; +} +``` + +## Async error handling + +```javascript +try { + const data = await fetchData(url); + return data; +} catch (error) { + if (error.code === 'ECONNREFUSED') return fallback(); + throw err; +} +``` + +## DomainError + +Structured business errors with codes for the API layer: + +```javascript +({ + method: async ({ email }) => { + const user = await domain.user.findByEmail(email); + if (!user) return new DomainError('ENOTFOUND'); + return user; + }, + errors: { ENOTFOUND: 'User not found' }, +}); +``` + +Domain layer throws plain errors; API layer translates to DomainError when needed. + +## Error propagation across layers + +``` +domain throws Error โ†’ API catches โ†’ returns DomainError(code) โ†’ client gets structured error +``` + +Do not expose internal error details to clients; map to known error codes or general errors. + +## Conventions + +- Distinguish programmer vs operational errors; only recover from operational +- Use DomainError for business validation in API methods; throw for programming bugs +- Never swallow errors silently; always log or propagate +- Implement graceful shutdown: stop accepting connections, drain existing, release resources +- Use retry with exponential backoff for transient failures (network, DB connections) +- Always handle both `uncaughtException` and `unhandledRejection` at process level + +## Retry pattern + +```javascript +const retry = async (fn, { attempts = 3, delay = 1000 } = {}) => { + for (let i = 0; i < attempts; i++) { + try { + return await fn(); + } catch (err) { + if (i === attempts - 1) throw err; + await new Promise((r) => setTimeout(r, delay * (i + 1))); + } + } +}; +``` + +## Process-level handlers + +```javascript +process.on('uncaughtException', (error) => { + console.error('Uncaught:', error); + process.exit(1); +}); + +process.on('unhandledRejection', (reason) => { + console.error('Unhandled rejection:', reason); +}); +``` + +Log and exit on uncaughtException (state may be corrupted). Log unhandledRejection and consider it a bug to fix. + +## Graceful shutdown + +Track connections and drain before exit: + +```javascript +const connections = new Map(); + +server.on('connection', (conn) => { + const res = null; + connections.set(conn, res); + conn.on('close', () => connections.delete(conn)); +}); + +const shutdown = () => { + server.close(() => { + freeResources(); + process.exit(0); + }); + for (const [conn, res] of connections) { + if (res) res.end('Server shutting down'); + conn.destroy(); + } +}; + +process.on('SIGINT', shutdown); +process.on('SIGTERM', shutdown); +``` diff --git a/.claude/skills/js-conventions/SKILL.md b/.claude/skills/js-conventions/SKILL.md new file mode 100644 index 0000000..d906739 --- /dev/null +++ b/.claude/skills/js-conventions/SKILL.md @@ -0,0 +1,109 @@ +--- +name: js-conventions +description: Apply the Alexis JavaScript style (Metarhia-derived, linted by oxlint and formatted by oxfmt). Use when writing or editing .js, .mjs, .ts files (with certain corrections for typescript), formatting code, or when the user asks about code style or linting. +--- + + + +# JavaScript Code Style + +Use following conventions for JavaScript and TypeScript code. + +## Preparations + +- Linting is oxlint (`.oxlintrc.json`) and formatting is oxfmt (`.oxfmtrc.json`, `.editorconfig`); + never add ESLint or Prettier +- The package manager is pnpm; the scripts are already in package.json: + - `pnpm lint`: oxlint over `src`, `tests`, `scripts` and `bench` + - `pnpm format`: oxfmt rewrites those files in place + - `pnpm format:check`: oxfmt check only, as CI runs it +- Before and after code analysis with AI run `pnpm lint` and `pnpm test` +- Use `pnpm format` to fix formatting; fix lint errors in the code, and suppress a rule only for an + intentional construct with a targeted `// oxlint-disable-next-line ` + +## Formatting + +- Use single quotes and semicolons +- Keep max line length 120 (ignore URLs) +- Use trailing commas in multiline arrays, objects, and params +- Keep one empty line between semantic blocks +- Follow oxfmt and `.editorconfig` for spacing, braces, and indentation (2 spaces, LF) + +## Modules + +- Use CommonJS: `require` and `module.exports`, no `import`/`export` in `.js` sources; there is no + build step, `src/` is what gets published +- Start every `src/` file with `'use strict'` +- Use the `node:` prefix for Node.js built-in modules (`require('node:fs')`) +- Always include the `.js` extension in relative requires (`require('./ops.js')`) +- No runtime dependencies: the only bare `require` allowed in `src/` is a `node:` built-in, and + none at all in a file that reaches the browser bundle (put helpers in `src/utils.js`) +- A platform split is a `*.browser.js` file (dot, not dash); see CLAUDE.md for the three edits it needs + +## Naming + +- Use `camelCase` for variables and parameters +- Use `UpperCamelCase` for classes and types +- Use `UPPER_SNAKE_CASE` for constants +- Use `new` with capitalized constructor names +- Use `error` in catch blocks, not `err` +- Use short names (`e`, `i`) only in short one-line callbacks +- Use descriptive names in multiline callbacks (`event`, `index`, `item`) +- Use boolean prefixes: `is`, `has`, `can`, `should` +- Use singular names for entities and plural names for collections +- Use verb names for functions and handlers (`parse`, `create`, `handle`, `on`) +- Include units in numeric names (`timeoutMs`, `sizeBytes`) except options + +## Best practices + +- Prefer `const`, minimize `let`, never use `var` +- Prefer arrow functions (except prototype/class methods) +- Do not mutate input parameters; avoid `arguments` +- Keep functions small and single-purpose +- Decompose long expressions into intermediate variables +- Prefer explicit loops (`for`, `for..of`); index loops in hot paths; avoid `forEach` and `reduce` + (the code base has none); `map`, `filter` and `find` are fine for a one-shot transform off the hot path +- Use `===` and `!==`; the one loose comparison is `value == null` for "null or undefined" + (`eqeqeq` ignores `null`) +- Keep return types consistent +- Avoid nested ternaries and deep callback nesting +- Self-documented and self-descriptive code; a comment explains why (an invariant, a trap, a + decision), never what the next line does +- Throw a `TypeError` for a bad argument, prefixed with the thing being configured + (`${label}: hooks must be an object`); throw a typed error with a stable `code` when a caller + has to branch on it (`HookError`, `code: 'HOOK_FAILED'`, with the original as `cause`); never + swallow an error without saying why in a comment +- Events leave through `notify()` and never fail the operation they observe; a hook may fail it +- `await` `calculateHash`, `encrypt` and `decrypt` in shared code: they are synchronous in Node and + promise-returning in the browser + +## Optimizations + +- Keep object shapes stable: same keys, types, and key order +- Change shape only in constructors and creational patterns; avoid mix-ins +- Use shape mutations only for metaprogramming +- Use `null` for empty reference types; use `undefined` for empty primitives +- Avoid array destructuring in assignments and loops; object destructuring is ok +- Keep hot functions monomorphic: stable arg count, types, and return types +- Avoid reading the same property from many unrelated object shapes +- Prefer string or number literals for status/codes; avoid mixing primitives with objects in one variable +- Keep objects that share a property to the same shape; avoid polymorphic hotspots +- Prefer fixed property access (`obj.x`) over dynamic keys (`obj[key]`) +- Keep arrays dense, avoid holes and mixed element kinds +- Initialize object fields in constructor/factory once and in one order +- Set field to `null` or `undefined`; avoid `delete` +- Move reusable callbacks outside loops +- Keep `try/catch`, spread/rest away from hot loops +- Use `Map` for dynamic key dictionaries, plain objects for fixed schemas +- Use `Set` for big and dynamic collections, prefer arrays for short and stable +- Use `Object.create(null)` for pure dictionaries +- Use typed arrays for numeric and binary workloads +- Reduce GC pressure: reuse arrays, objects, and buffers when safe +- In the hot path (applying and merging ops, `OpLog.append`, vector clocks, `packOps`): no closure + or `RegExp` created per call, one object shape per kind of op and log entry, and `try/catch` + only around user code (hooks, listeners) and I/O +- Check `pnpm bench` before and after touching the hot path; a number moved is a finding diff --git a/.claude/skills/js-data-structures/SKILL.md b/.claude/skills/js-data-structures/SKILL.md new file mode 100644 index 0000000..959da8f --- /dev/null +++ b/.claude/skills/js-data-structures/SKILL.md @@ -0,0 +1,209 @@ +--- +name: js-data-structures +description: Choose and use native JavaScript collections for Big-O, readability, semantics, naming, stability, or testability. Use when working with Map, Object, Set, Array, WeakMap, WeakSet, TypedArray, or deciding among built-in collections. For custom queues, lists, heaps, tries, and related structures, use the data-structures skill. +--- + +# Native Data Structures (JavaScript) + +## Choosing a native collection + +- Fixed schema / DTO / JSON: use `Object` (avoid `Map`, no `JSON.stringify`) +- Dynamic string-to-value dict: use `Map` or `Object.create(null)` (avoid plain `{}` if keys are user input) +- Non-string keys, insertion order, `.size`: use `Map` (avoid `Object`) +- Ordered list, index access, transform: use `Array` (avoid `Set`, no indexes) +- Unique values, fast membership: use `Set` (avoid `Array.includes` on large/dynamic sets) +- Attach data to objects without retaining them: use `WeakMap` (avoid `Map`, leaks) +- Track object presence without retaining: use `WeakSet` (avoid `Set`, leaks) +- Numeric / binary data, I/O, crypto: use TypedArray / `ArrayBuffer` (avoid `number[]` for hot binary work) +- Hot queue / deque / stack ends: do not rely on `Array.shift`/`unshift` โ€” use a custom ring or linked structure + +## Complexity and code characteristics + +Choose a structure for the property you need to control โ€” not only for speed. + +**Big-O (typical average case; n = collection size):** + +- `Object` field access: O(1) for fixed shapes; dynamic/`delete`-heavy objects degrade toward dictionary mode +- `Array` index get/set, `push`/`pop`: O(1); `includes`/`indexOf`/`splice`/`shift`/`unshift`: O(n) +- `Map` get/set/has/delete, `Set` add/has/delete: amortized O(1); iteration: O(n) +- `WeakMap`/`WeakSet` get/set/has/delete: amortized O(1); no iteration +- TypedArray index get/set: O(1); grow/copy: O(n) +- Prefer the structure whose hot operation is O(1); measure before optimizing rare paths + +**Readability and semantics:** + +- Let the type state the intent: `Set` = unique membership; `Map` = keyed registry; `Array` = ordered sequence; plain `Object` = record/DTO +- Prefer a named collection over encoding the same idea in ad-hoc flags, parallel arrays, or stringly keys +- Use array methods (`.map`, `.filter`, `.reduce`) when they match the transform; use explicit loops when control flow or performance matters +- Avoid over-abstracting: a small static `Array` allowlist is clearer than a `Set` used once + +**Naming:** + +- Plural names for collections (`users`, `sessions`); singular for entities (`user`) +- Name by role, not by type: `seenIds` (Set), `socketToSession` (Map) โ€” not `set1` / `mapData` +- Boolean membership reads as `has`/`is`: `if (seen.has(id))`, `blocked.has(user)` +- Include units in size names (`sizeBytes`, `timeoutMs`) when the collection holds measured quantities + +**Stability and contracts:** + +- Treat inputs as immutable at API boundaries; copy or freeze when you must share +- Prefer stable iteration order when callers rely on it (`Map`/`Set`/`Array` insertion order) +- Keep record shapes and array element kinds stable over time (see per-structure V8 hints) +- Document whether a returned collection is live (shared) or a snapshot + +**Testability:** + +- Prefer pure functions over collections: same inputs, same outputs; inject maps/sets as arguments when state is required +- Favor deterministic structures in tests: plain data over WeakMap (non-enumerable), over insertion-timed WeakRef behavior +- Assert on observable contents (`[...set]`, `Object.fromEntries(map)`, length/size) rather than private fields +- Small fixtures: build with literals/`Object.freeze` when the shape should not change under test + +**Encapsulation, safety, operability:** + +- Hide representation behind a small API (`add`/`has`, getters) so internals can change without call-site churn +- Use `Object.create(null)` or `Map` for untrusted keys (prototype-pollution safety) +- Use WeakMap/WeakSet when lifetime must follow the key (leak control) +- Cap growth (max size, TTL, eviction) for long-lived Maps/Sets; unbounded collections are operability bugs +- Pick serializable structures (`Object`/`Array`) at I/O boundaries; convert `Map`/`Set` explicitly + +## Object + +Use for records with a known, stable shape (config, entities, options). + +```javascript +const user = { id: 1, name: 'Ada', active: true }; + +// Pure dictionary (no prototype pollution via __proto__) +const dict = Object.create(null); +dict[key] = value; +``` + +**Hints:** + +- Keep a stable shape (hidden class): same keys, types, and key order; init once in a factory/constructor +- Prefer fixed access (`obj.x`) over dynamic keys (`obj[key]`) when the schema is known +- Do not `delete` fields; set `null` for refs, `undefined` for primitives +- Objects that share a property should share the full shape โ€” avoid polymorphic property reads +- `for...in` is slow on hot paths; prefer `Object.keys` / `Object.entries` + +## Array + +Use for ordered sequences, stacks (`push`/`pop`), and small membership checks. + +```javascript +const ids = [10, 20, 30]; +ids.push(40); +const last = ids.pop(); + +const ALLOWED = ['read', 'write']; +if (ALLOWED.includes(role)) { + /* ... */ +} +``` + +**Hints:** + +- Keep arrays dense and monomorphic (one element kind): SMI ints, doubles, or objects โ€” do not mix +- Prefer SMI-friendly integers (31-bit signed) for indexes/counters (`PACKED_SMI_ELEMENTS`); a float, `NaN`, or hole promotes the whole array +- Avoid holey arrays (`new Array(n)`, deletes, skips); build with `push` or fill sequentially +- Avoid `shift()` / `unshift()` in hot paths; use a queue or index cursors +- Prefer `for` / `for...of` in hot loops; avoid array destructuring in hot assignments + +## Map + +Use for dynamic key-value stores with any key type and predictable iteration. + +```javascript +const sessions = new Map(); +sessions.set(socket, { userId: 42 }); +sessions.set('fallback', null); + +for (const [key, value] of sessions) { + // insertion order +} +sessions.delete(socket); +console.log(sessions.size); +``` + +**Hints:** + +- Dynamic keys on `Map` avoid dictionary-mode / shape pollution on `{}` +- Values read on hot paths should still use stable shapes +- `NaN` is a valid single key; object keys compare by reference +- No `JSON.stringify` โ€” use `Object.fromEntries(map)` only if keys are strings + +## Set + +Use for unique values and fast membership. + +```javascript +const seen = new Set(); +for (const id of incoming) { + if (seen.has(id)) continue; + seen.add(id); + process(id); +} + +const unique = [...new Set(items)]; +``` + +**Hints:** + +- Uniqueness is by reference for objects, not deep value +- Insertion-order iteration; no index access +- Array/Set conversion allocates โ€” do it at boundaries, not in hot loops + +## WeakMap / WeakSet + +Weak collections hold keys weakly so entries can be GC'd when the key is otherwise unreachable. Keys must be objects or symbols. Not iterable; no `.size`. + +```javascript +const meta = new WeakMap(); +const tracked = new WeakSet(); + +const attach = (obj, data) => { + meta.set(obj, data); + tracked.add(obj); +}; +``` + +**Hints:** + +- Prefer WeakMap side tables over adding ad-hoc fields to host objects +- Cannot enumerate โ€” keep a strong list only if you must iterate +- `WeakRef` / `FinalizationRegistry` are for lifetime hooks, not general storage + +## Typed arrays + +Use for contiguous numeric/binary data: network frames, files, crypto, image/audio, WASM. + +- `Uint8Array` โ€” bytes, I/O, hashing +- `Int16Array` / `Uint16Array` โ€” PCM, compact IDs +- `Uint32Array` / `Int32Array` โ€” indexes, bitsets, WASM i32 +- `Float32Array` / `Float64Array` โ€” geometry, signals, tensors +- `BigInt64Array` / `BigUint64Array` โ€” 64-bit IDs, timestamps +- `DataView` โ€” endian-aware protocol parsing +- `ArrayBuffer` / `SharedArrayBuffer` โ€” backing store; SAB for workers + +```javascript +const buf = new ArrayBuffer(16); +const bytes = new Uint8Array(buf); +const view = new DataView(buf); +view.setUint32(0, 0xdeadbeef, true); +bytes[4] = 0xff; +const frame = bytes.subarray(0, 8); // shared memory; `slice` copies +``` + +**Hints:** + +- Fixed length โ€” size ahead or allocate larger and copy to grow +- Prefer typed arrays over `number[]` for tight numeric loops (stable element type) +- Use `DataView` for wire endianness; multi-byte typed views are platform-endian +- Reuse/pool buffers in hot paths; Node `Buffer` is a `Uint8Array` โ€” prefer `Uint8Array` in shared code unless you need Buffer APIs + +## Conventions + +- Keep hot code monomorphic: stable argument count, types, return types, and object shapes +- Prefer SMI integers for indexes/sizes; do not mix ints and floats in one variable +- Reduce GC pressure: reuse arrays, objects, and buffers when safe +- Move reusable callbacks outside loops; keep `try/catch` and spread out of hot loops diff --git a/.claude/skills/js-gof/SKILL.md b/.claude/skills/js-gof/SKILL.md new file mode 100644 index 0000000..22bb9d0 --- /dev/null +++ b/.claude/skills/js-gof/SKILL.md @@ -0,0 +1,873 @@ +--- +name: js-gof +description: Apply Gang of Four and related design patterns in JavaScript and TypeScript. Use when implementing creational, structural, or behavioral patterns, or when the user mentions factories, builder, prototype, flyweight, singleton, object pool, adapter, wrapper, decorator, proxy, bridge, composite, facade, chain of responsibility, command, interpreter, iterator, mediator, memento, EventEmitter, EventTarget, state, strategy, template method, visitor, revealing constructor, actor, service locator, or any other pattern. +--- + +# GoF Patterns + +- Use structural composition over inheritance +- Use multiparadigm code and contract-programming +- Use domain-specific languages (DSLs), prefer declarative way +- Separate and do not mix system and domain code +- Use GRASP and SOLID principles; especially reduce coupling +- Remember about referential transparency +- Prefer platform-agnostic code +- Implement isolation and layer borders with IoC & DI +- Use object/Map lookup for Strategy and polymorphic/dynamic dispatch +- Keep patterns simple +- Do not over-engineer for the sake of the pattern name + +## Creational patterns + +### Abstract factory + +Creates related objects belonging to one family without specifying their concrete classes, e.g., UI components for different platforms. + +Refs: https://github.com/HowProgrammingWorks/AbstractFactory + +```javascript +const dataAccess = { + fs: { + createDatabase: (...args) => new FileStorage(...args), + createCursor: (...args) => new FileLineCursor(...args), + }, + minio: { + // factories collection for minio + }, + // other implementations +}; + +// Usage + +const accessLayer = dataAccess.fs; +const storage = accessLayer.createDatabase('./storage.dat'); +const cursor = accessLayer.createCursor({ city: 'Roma' }, storage); +for await (const record of cursor) { + console.dir(record); +} +``` + +### Builder + +Step-by-step assembly of a complex configurable object, often using chaining, e.g., Query Builder or Form Generator. + +Refs: https://github.com/HowProgrammingWorks/Builder + +```javascript +class QueryBuilder { + #options; + + constructor(table) { + this.#options = { table, where: {}, limit: null, order: null }; + } + + where(cond) { + Object.assign(this.#options.where, cond); + return this; + } + + order(field) { + this.#options.order = field; + return this; + } + + limit(n) { + this.#options.limit = n; + return this; + } + + build() { + return { ...this.#options }; + } +} +``` + +```javascript +const query = new QueryBuilder('User') + .where({ active: true }) + .order('name') + .limit(10) + .build(); +``` + +Alternatively we can create async constructor to be invoked with `await new QueryBuilder`, or put all steps into declarative structure like: + +```javascript +const query = await new QueryBuilder({ + entity: 'User', + where: { active: true }, + order: 'name', + limit: 10, +}); +``` + +### Factory + +Function or method that creates objects using different techniques: assembling from literals and methods, mixins, `setPrototypeOf`. + +Refs: https://github.com/HowProgrammingWorks/Factory + +```javascript +const createUser = (name, role) => ({ name, role, createdAt: Date.now() }); +const createAdmin = (name) => createUser(name, 'admin'); +``` + +Or following: + +```javascript +class Connection { + constructor(url) { + // implementation + } + // implementation +} + +const factory = (() => { + let index = 0; + return () => new Connection(`http://10.0.0.1/${index++}`); +})(); +``` + +### Factory Method + +Chooses the correct abstraction to create an instance; in JavaScript, this can be implemented using `if`, `switch`, or selection from a collection (dictionary). + +Refs: https://github.com/HowProgrammingWorks/FactoryMethod + +```javascript +class Person { + constructor(name) { + this.name = name; + } + + static factory(name) { + return new Person(name); + } +} +``` + +```javascript +class Product { + constructor(value) { + this.field = value; + } +} + +class Creator { + factoryMethod(...args) { + return new Product(...args); + } +} +``` + +### Prototype + +Creates objects by cloning a prepared instance to save resources (not to be confused with [Prototype-programming](https://github.com/HowProgrammingWorks/Prototype), which is closer to Flyweight). + +Refs: https://github.com/HowProgrammingWorks/PrototypePattern + +```javascript +const proto = { type: 'widget', color: 'blue' }; +const clone = () => ({ ...proto }); +``` + +```javascript +class Point { + #x; + #y; + + constructor(x, y) { + this.#x = x; + this.#y = y; + } + + move(x, y) { + this.#x += x; + this.#y += y; + } + + clone() { + return new Point(this.#x, this.#y); + } +} + +class Line { + #start; + #end; + + constructor(start, end) { + this.#start = start; + this.#end = end; + } + + move(x, y) { + this.#start.move(x, y); + this.#end.move(x, y); + } + + clone() { + const start = this.#start.clone(); + const end = this.#end.clone(); + return new Line(start, end); + } +} + +// Usage + +const p1 = new Point(0, 0); +const p2 = new Point(10, 20); +const line = new Line(p1, p2); +const cloned = line.clone(); +cloned.move(2, 3); +``` + +```javascript +const point = (x, y) => { + const move = (dx, dy) => { + x += dx; + y += dy; + }; + const clone = () => point(x, y); + return { move, clone }; +}; + +// Usage + +const { move, clone } = point(10, 20); +const c1 = clone(); +move(-5, 10); +``` + +### Flyweight + +Saves memory allocation by sharing common state among multiple instances. + +Refs: https://github.com/HowProgrammingWorks/Flyweight + +```javascript +const flyweightPool = new Map(); + +const getFlyweight = (shared) => { + const { char, font, size } = shared; + const key = `${char}:${font}:${size}`; + let flyweight = flyweightPool.get(key); + if (!flyweight) { + flyweight = Object.freeze({ ...shared }); + flyweightPool.set(key, flyweight); + } + return flyweight; +}; + +const createChar = (char, font, size, row, col) => { + const intrinsic = getFlyweight({ char, font, size }); + return { intrinsic, row, col }; +}; + +const a1 = createChar('A', 'Arial', 12, 0, 0); +const a2 = createChar('A', 'Arial', 12, 0, 5); +console.log(a1.intrinsic === a2.intrinsic); // true +``` + +### Singleton + +Provides global access to a single instance; often considered an anti-pattern, easiest implemented via ESM/CJS module caching exported refs. + +Refs: https://github.com/HowProgrammingWorks/Singleton + +Everywhere we import this state will be the same collection + +```javascript +const connections = new Map(); // shared state +const nop = () => {}; // pure function with no state +module.exports = { connections, nop }; +``` + +```javascript +class Singleton { + static #instance; + + constructor() { + const instance = Singleton.#instance; + if (instance) return instance; + Singleton.#instance = this; + } +} +``` + +```javascript +function Singleton() { + const { instance } = Singleton; + if (instance) return instance; + Singleton.instance = this; +} +``` + +```javascript +const Singleton = new (function () { + const single = this; + return function () { + return single; + }; +})(); +``` + +```javascript +const Singleton = (() => { + let instance; + + class Singleton { + constructor() { + if (instance) return instance; + instance = this; + } + } + + return Singleton; +})(); +``` + +```javascript +const singleton = (() => { + const instance = {}; + return () => instance; +})(); +``` + +```javascript +const singleton = ( + (instance) => () => + instance +)({}); +``` + +### Object Pool + +Reuses pre-created objects to save resources during frequent creation and destruction. + +Refs: https://github.com/HowProgrammingWorks/Pool + +```javascript +class Pool { + #available = []; + #factory = null; + + constructor(factory, size) { + this.#factory = factory; + for (let i = 0; i < size; i++) this.#available.push(factory()); + } + + acquire() { + return this.#available.pop() || this.#factory(); + } + + release(obj) { + this.#available.push(obj); + } +} +``` + +## Structural patterns + +### Adapter + +Converts an incompatible interface into a compatible one, enabling third-party component usage without altering its code; can even transform a function contract into an object or vice versa. + +Refs: https://github.com/HowProgrammingWorks/Adapter + +```javascript +const promisify = + (fn) => + (...args) => + new Promise((resolve, reject) => { + fn(...args, (err, data) => (err ? reject(err) : resolve(data))); + }); +``` + +```javascript +const timer = new Timer(1000); // wraps setInterval +for await (const step of timer) { + // gives [Symbol.asyncIterator] contract + console.log({ step }); +} +``` + +```javascript +class ArrayToQueueAdapter { + #array = null; + + constructor(array) { + if (!Array.isArray(array)) { + throw new Error('Array instance expected'); + } + this.#array = array; + } + + enqueue(data) { + this.#array.push(data); + } + + dequeue() { + return this.#array.pop(); + } + + get count() { + return this.#array.length; + } +} +``` + +### Wrapper + +Function wrapper that delegates calls and adds behavior; a specialized case of Adapter. + +Refs: https://github.com/HowProgrammingWorks/Wrapper + +```javascript +const withLogging = + (fn, label) => + (...args) => { + console.log(`${label} called`, args); + const result = fn(...args); + console.log(`${label} returned`, result); + return result; + }; +``` + +```javascript +const wrapInterface = (anInterface) => { + const wrapped = {}; + for (const key in anInterface) { + const fn = anInterface[key]; + wrapped[key] = wrapFunction(fn); + } + return wrapped; +}; +``` + +```javascript +class Timeout { + constructor(fn, msec) { + this.function = fn; + this.timer = setTimeout(() => { + this.timer = null; + }, msec); + } + + execute(...args) { + let result = undefined; + if (!this.timer) return result; + clearTimeout(this.timer); + this.timer = null; + result = this.function(...args); + return result; + } +} +``` + +### Boxing + +Wraps primitives into object types to add methods or unify interfaces, e.g., narrowing `String` to `AddressString`. + +Refs: https://github.com/HowProgrammingWorks/ADT + +Box, Container or Value-Object examples: + +```javascript +class AddressString { + #value; + + constructor(value) { + if (typeof value !== 'string') { + throw new TypeError('Address must be a string'); + } + const str = value.trim(); + if (str === '') throw new TypeError('Address must be a non-empty'); + this.#value = str; + } + + get city() { + const index = this.#value.indexOf(','); + return this.#value.slice(0, index).trim(); + } + + toString() { + return this.#value; + } + + valueOf() { + return this.#value; + } +} + +// Usage + +const address = new AddressString('London, 221B Baker Street'); +console.log(address.city); // 'London' +console.log(`Delivery to: ${address}`); +``` + +Value-Object examples: + +```javascript +const KMH_TO_MPH = 0.621371; + +class SpeedValue { + #kmh; + + constructor(value, unit = 'kmh') { + if (typeof value !== 'number') { + throw new TypeError('Speed must be a number'); + } + if (value < 0) throw new TypeError('Speed must be non-negative'); + if (unit !== 'kmh' && unit !== 'mph') { + throw new TypeError('Unit must be kmh or mph'); + } + this.#kmh = unit === 'mph' ? value / KMH_TO_MPH : value; + } + + get kmh() { + return this.#kmh; + } + + get mph() { + return this.#kmh * KMH_TO_MPH; + } + + toString() { + return `${this.#kmh} km/h`; + } + + valueOf() { + return this.#kmh; + } +} +``` + +### Decorator + +Dynamically extends behavior without inheritance, typically via composition and declarative syntax, effectively adding metadata. + +Refs: https://github.com/HowProgrammingWorks/Decorator + +```javascript +const add = (a, b) => a + b; +const loggedAdd = withLogging(add, 'add'); +``` + +Not a good idea to use classical decorator in JavaScript/TypeScript + +```javascript +const validator = new MinLengthValidator( + new EmailValidator(new RequiredValidator(new BaseValidator())), + 20, +); + +const input = 'timur.shemsedinov@gmail.com'; +const result = validator.validate(input); +``` + +We can add metadata to an instance in several JS-idiomatic ways: + +Symbol property on instance: + +```javascript +const VALIDATOR_META = Symbol('validatorMeta'); + +class EmailValidator { + constructor(next) { + this.next = next; + this[VALIDATOR_META] = { type: 'email' }; + } +} +``` + +Wrapper that attaches metadata as own property. + +JSDoc annotation (zero runtime cost): + +```javascript +/** + * @typedef {{ type: string, label: string }} ValidatorMeta + * @typedef {{ validate(v: string): boolean, meta: ValidatorMeta }} Validator + */ + +/** @type {Validator} */ +const validator = Object.assign( + new EmailValidator(new RequiredValidator(new BaseValidator())), + { meta: { type: 'email', label: 'Email field' } }, +); +``` + +Module-local WeakMap (fully private, no shape pollution): + +```javascript +const metadata = new WeakMap(); + +const createValidator = ({ args, meta }) => { + const instance = new Validator(...args); + metadata.set(instance, meta); + return instance; +}; + +module.exports = { createValidator }; +``` + +### Proxy + +Controls access to an object by intercepting calls, reads, and writes; useful for lazy initialization, caching, and security; can be implemented via GoF or native JavaScript Proxy. + +Refs: https://github.com/HowProgrammingWorks/Proxy + +Do not use the built-in JS `Proxy` class unless a developer asks +you directly because of deopts. Use GoF Proxy instead as a container +with additional behavior. + +```javascript +const fs = require('node:fs'); + +const statistics = { bytes: 0, chunks: 0, events: {} }; + +class StatReadStream extends fs.ReadStream { + emit(name, data) { + if (name === 'data') { + statistics.bytes += data.length; + statistics.chunks++; + } + const counter = statistics.events[name] || 0; + statistics.events[name] = counter + 1; + super.emit(name, data); + } +} + +const getStatistics = () => structuredClone(statistics); + +const createReadStream = (path, options) => new StatReadStream(path, options); + +module.exports = { ...fs, createReadStream, getStatistics }; +``` + +### Bridge + +Separates two or more abstraction hierarchies via composition or aggregation, allowing them to evolve independently. + +Refs: https://github.com/HowProgrammingWorks/Bridge + +```js +class CommunicationProtocol { + sendCommand(device, command) { + console.log({ device, command }); + throw new Error('sendCommand() must be implemented'); + } +} + +class MQTTProtocol extends CommunicationProtocol { + sendCommand(device, command) { + console.log(`[MQTT] Sending '${command}' to ${device}`); + } +} + +class HTTPProtocol extends CommunicationProtocol { + sendCommand(device, command) { + console.log(`[HTTP] Sending '${command}' to ${device}`); + } +} + +class IoTDevice { + constructor(name, protocol) { + this.name = name; + this.protocol = protocol; + } + + operate(command) { + this.protocol.sendCommand(this.name, command); + } +} + +class SmartLight extends IoTDevice { + turnOn() { + this.operate('Turn On Light'); + } + + turnOff() { + this.operate('Turn Off Light'); + } +} + +class SmartThermostat extends IoTDevice { + setTemperature(temp) { + this.operate(`Set Temperature to ${temp}ยฐC`); + } +} +``` + +### Composite + +Implements a common interface to uniformly handle individual objects and their tree structures, e.g., DOM or file systems. + +Refs: https://github.com/HowProgrammingWorks/Composite + +Recursive composite with reduce: + +```js +const calculateTotal = (order) => { + const items = Array.isArray(order) ? order : Object.values(order); + return items.reduce((sum, item) => { + if (typeof item.price === 'number') return sum + item.price; + else return sum + calculateTotal(item); + }, 0); +}; +``` + +JSON as a nested composite structure: + +```javascript +const json = `{ + "electronics": { + "laptop": { "price": 1200 }, + "accessories": { + "mouse": { "price": 25 }, + "keyboard": { "price": 75 } + } + }, + "books": { + "fiction": { "price": 15 }, + "technical": { "price": 60 } + } +}`; + +const order = JSON.parse(json); +console.log(calculateTotal(order)); +``` + +### Facade + +Simplifies access to a complex system, providing a unified and clear interface, hiding and protecting internal complexity. + +Refs: https://github.com/HowProgrammingWorks/Facade + +```javascript +const createApi = (db, cache, logger) => ({ + async getUser(id) { + const cached = cache.get(id); + if (cached) return cached; + const user = await db.row('User', ['*'], { id }); + cache.set(id, user); + logger.info(`User ${id} loaded`); + return user; + }, +}); +``` + +This hides classes Timer, Logger, Task and so on: + +```js +scheduler.task('name2', '2019-03-12T14:37Z', (done) => { + setTimeout(() => { + done(new Error('task failed')); + }, 1100); +}); +``` + +### Context + +Exchanges state and dependencies between different components (abstractions, modules, layers) that do not share a common environment, without tightly coupling them. + +Refs: https://github.com/HowProgrammingWorks/Context + +```js +class AccountService { + constructor(context) { + this.context = context; + } + + getBalance(accountId) { + const { console, accessPolicy, user } = this.context; + console.log(`User ${user.name} requesting balance for ${accountId}`); + if (!accessPolicy.check(user.role, 'read:balance')) { + console.error('Access denied: insufficient permissions'); + return null; + } + const balance = 15420.5; + console.log('Access granted'); + return balance; + } +} + +class AccessPolicy { + constructor() { + this.permissions = { + admin: ['read:balance', 'read:transactions', 'write:transactions'], + user: ['read:balance'], + guest: [], + }; + } + + check(role, permission) { + return this.permissions[role]?.includes(permission); + } +} + +class User { + constructor(name, role) { + this.name = name; + this.role = role; + } +} + +const accessPolicy = new AccessPolicy(); +const user = new User('Marcus', 'admin'); +const context = { console, accessPolicy, user }; + +const accountService = new AccountService(context); +const balance = accountService.getBalance('Account-123'); +console.log(`Balance = $${balance}`); +``` + +Alternative way with closure: + +```js +const createAccountService = (context) => { + const { console, accessPolicy, user } = context; + + const getBalance = (accountId) => { + console.log(`User ${user.name} requesting balance for ${accountId}`); + if (!accessPolicy.check(user.role, 'read:balance')) { + console.error('Access denied: insufficient permissions'); + return null; + } + return 15420.5; + }; + + const getTransactions = (accountId) => { + if (!accessPolicy.check(user.role, 'read:transactions')) { + console.error('Access denied: insufficient permissions'); + return null; + } + console.log(`User ${user.name} reading transactions for ${accountId}`); + return ['TR-123', 'TR-456', 'TR-789']; + }; + + return { getBalance, getTransactions }; +}; + +const accessPolicy = { + permissions: { + admin: ['read:balance', 'read:transactions', 'write:transactions'], + user: ['read:balance'], + guest: [], + }, + check: (role, permission) => + accessPolicy.permissions[role]?.includes(permission), +}; + +const context = { + console, + accessPolicy, + user: { name: 'Marcus', role: 'admin' }, +}; + +const accountService = createAccountService(context); + +console.log('Balance:', accountService.getBalance('ACC-001')); +console.log('Transactions:', accountService.getTransactions('ACC-001')); +``` diff --git a/.claude/skills/npm-publish/SKILL.md b/.claude/skills/npm-publish/SKILL.md new file mode 100644 index 0000000..f206df5 --- /dev/null +++ b/.claude/skills/npm-publish/SKILL.md @@ -0,0 +1,219 @@ +--- +name: npm-publish +description: Prepare an @alexify package for publishing with pnpm. Handles version bump, CHANGELOG update, typings check, quality gates, README and docs refresh, package.json audit, and pnpm pack verification. Use when the user asks to prepare a release, bump version, publish a package, or do release prep. +--- + + + +# npm Publish Preparation + +Systematic workflow for preparing the package for a new release. Every step must pass before proceeding to the next. The package manager is pnpm (pinned in `packageManager`); never create `package-lock.json`. Releases are run by hand on purpose: there is no release workflow and no publish automation, and this skill never publishes. + +`@alexify/syncom` has not been published yet (`0.0.1`, no `v*` tags, an `[Unreleased]` section that says so). The first release is the first run of this skill; read the "first release" notes in Steps 2 and 3. + +## Step 0: Gather Release Intent + +Ask the user (use AskUserQuestion if available): + +1. **Version bump type**: patch / minor / major / explicit version / prerelease tag +2. **Prerelease label** (if any): e.g. `alpha`, `beta`, `rc` +3. **Confirm changelog entries**: ask if the `[Unreleased]` section in CHANGELOG is complete, or if the user wants to draft entries from recent commits + +If the user already specified intent (e.g. "prepare 0.1.0 release"), skip the question and proceed. + +## Step 1: Pre-flight Checks + +Run: + +```bash +pnpm install --frozen-lockfile # install deps exactly as locked +pnpm lint # oxlint +pnpm format:check # oxfmt +pnpm test:coverage # tests under the c8 gate (95 lines / 95 statements / 90 branches / 95 functions) +pnpm test:types # tsd against the .d.ts files +pnpm size # the bundle report; the browser budget fails the run +pnpm docs:build # the docs site; a dead link fails it +``` + +These are the three CI jobs (lint, test, docs). Do not lower a coverage threshold or raise a size budget to get green; a budget is a ratchet raised deliberately, in the change that earns the bytes, with the reason in a comment next to the number. + +Then in parallel: `pnpm outdated` and `pnpm audit` (report to the user; devDependencies only, the package has no runtime dependencies). + +If any gate fails, stop and fix issues before continuing. + +## Step 2: Version Bump + +### Read current version from `package.json` + +Compute the new version based on user intent and semver rules: + +- **patch**: `1.2.3` โ†’ `1.2.4` +- **minor**: `1.2.3` โ†’ `1.3.0` +- **major**: `1.2.3` โ†’ `2.0.0` +- **prerelease**: `1.2.3` โ†’ `1.2.4-