Repository navigation
Server rewrite: opaque-streams relay on Workers + DOs (DECISIONS P0–P5) - #82
Merged
Merged
Conversation
…-3 baseline: 7/7 baseline pass, 17 decision items fail as expected)
Cloudflare is the only host. Code excluded from the build is dead weight; git history keeps it. - Remove packages/cli, packages/server-node, Dockerfile, compose.yaml, .dockerignore, and the tests and scripts that existed only for them (tests/node-runtime, tests/headless, tests/docker, 21 tests/server suites that ran the old server on the Node storage). - Move the test harness to tests/server/helpers: a trimmed nodeSqliteStorage (sql.exec cursor, transactionSync) and a ywasm loader, so the test aliases run the Worker CRDT engine. - Drop the cli/server-node/docker scripts, chokidar, and the CI and release steps that used them. README gains "Hosting: Cloudflare only". Checks: build, server typecheck, check:deps, test:client, conformance (Wrangler), integration:worker and streams-relay (15/15) pass; run-suites keeps the same 11 pre-existing failures.
DECISIONS.md - Accepted flags folded into D2-D9; section 8 logs F# -> D# and the resolved open items O1-O8. - D3 enroll replay keyed on code + enrollmentRequestId + deviceId + token hash; a revoked device gets 409 used_code. The 200 body is exactly the six fields readEnrollment reads. - D7 gate model: device map checked first in every webSocketMessage, buffered frames of a revoked device dropped, close frame best-effort. Records the legacy revoke measurements. - D8b restore (proposed): config-DO journal and alarm, pending_restore_id marker, re-rewind on writes since boot, authority freeze, new epoch at finish. - D9 blob key v/<vaultId>/<address>. e2e/conformance - tests/revoke.ts: T-REVOKE-GATE, T-REVOKE-SILENCE, T-REVOKE-BUFFER. - lib/rawsocket.ts: minimal RFC 6455 client that keeps sending after the close frame and times the close frame and the TCP end separately. - T-REVOKE-4403 judges the error and close frames (< 1 s each); the TCP end is recorded only. - tests/flags.ts: enroll replay/conflict/body, blob key after reset, console routes, vault delete confirm, provision 404, unknown vault 401.
- Local wrangler dev of the legacy server (040c7fb) gives the same status as scratch-3 on all 50 BB tests; DECISIONS.md section 7 cites it as the second measurement and records the local timings. - guard(): `...Id` keys are record ids, not credentials; registering the pairing-code record id flagged the response that carried it. - run.ts records `<sha>+dirty` when e2e/conformance has uncommitted edits. - README: T-DEDUPE-CONFLICT reports STREAM_APPEND_REJECTED client_frame_id_conflict.
Add server/src/ports.ts (storage, socket, clock and timer ports). The streams relay and store depend only on it: StreamActor and VaultRole are local to relay.ts, the clock is a port, and the store can be built with schemaReady so the vault DO runs the DDL at init only (DECISIONS 6.1). Wire behaviour is unchanged. Port tests/server/streams-relay.ts to the ports. The legacy tickets test and the flag/route/capability assertions are dropped (the route checks move to the new router test). Upgrade wrangler to 4.147.0 and workers-types to 5.20261001.1 so the bundled workerd accepts compatibility dates >= 2026-04-07.
…eckpoint B) New Worker entry (server/src/worker.ts) with the DECISIONS §2.2 route table: method and path match, vaultId / pairing-code / blob-address formats, the 4 MiB checkpoint and 10 MiB blob caps, streamsVersion=1, CORS, the capabilities claimed cache, and 404 for every D5 legacy route with no DO call. P2/P3 behaviour answers 501 not_implemented. VaultDO (vault/) holds the §6.1 schema, vault init, the in-memory device map with the D7 gate seam, bearer auth and the streams relay through the ports. ConfigDO (config/) holds the §6.3 schema and isClaimed. wrangler.toml: main src/worker.ts, compatibility_date 2026-04-07, bindings YAOS_VAULT and YAOS_CONFIG, and migration v3 deleting the legacy classes. Tests: worker-router (T-LEGACY-404, route table, formats, caps), vault-host (schemas, init, probe, D6 constants, gate, hibernation) and hotpath (T-HOTPATH).
…o legacy-src/shared (P1 checkpoint C) DECISIONS §1/§2.3: the server is the opaque streams relay only. Deletes the legacy CRDT/recovery/settings/route server, its ywasm vendor tree, benches, spike scripts (relay2, pathology-lab, b3*, ywasm-qa), the live/conformance/ deployed drivers and every legacy-only test. Drops yjs, y-protocols and fflate from server/package.json (lib0 and qrcode-generator stay). Pure shared modules still used by legacy-src and client tests move to legacy-src/shared (no behaviour change); the @shared runner alias follows. socketLiveness and version.ts are trimmed to what the relay uses (DECISIONS-GAP comments in place). The release build stops emitting update-manifest.json (§1 removes update-metadata) and no longer pins ywasm.
Static pages per DECISIONS D5: inline CSS and JS, no external asset, a per-response nonce CSP. The console covers claim, login, vaults, create, owner code + QR, devices + revoke, reset streams and delete behind a typed vaultId, restore (D8b) and the Restore incomplete banner, and maps every documented operator error code to a plain message. White-box tests run the inline scripts in a fake DOM against the §2.2 route table.
Drop what D1 cut (members, invitations, settings sync, recovery points, the RecoveryJob binding). Get started is claim, vault, QR; Troubleshooting matches the rewritten server; engineering links point at the current spec.
… D4 tickets, D7 revoke Config DO: claim (3 rows), login with an in-memory failure limiter, logout, sessions (7 days), the vault registry, the restore-journal seam (409 restore_in_progress for revoke, owner-code and reset). Vault DO: owner and device pairing codes (1 row + pruning), enroll with the D3 replay key (2 rows; replay 0), the D3 enroll limiter, D4 HMAC tickets (0 rows) and the ticket upgrade, the devices list, D7 revoke in one synchronous turn (row, device map, pending frames, PROVISIONAL_DROPPED, 4403), O3 at the wake, and the D5 delete (sockets 1001, deleteAll). Worker: the operator routes with the D5 Origin and JSON checks, the session cookie, the R2 prefix purge; device routes never reach the config DO. reset-streams, restore, blobs and simulate-daily-limit stay 501 (P3).
…e and races, O3) and operator (claim, sessions, CSRF, delete, restore seam, T-ROWS-WB)
…asses
workerd reads every named export of the main module as an entrypoint, and
a constant there stops the runtime at startup ("Incorrect type for map
entry 'CLAIM_VAULT_NAME'" under wrangler dev). The route table and its
constants move to router.ts; worker.ts keeps the default handler, ConfigDO
and VaultDO. A worker-router test pins the entry module's exports.
…ks the D3 six fields Node's fetch sends no Origin, a browser sends it on every non-GET request; D5 refuses operator writes without a same-origin Origin. The enroll body check required principalId, which D3 removes from the 200 body (D6 constants belong to VAULT_READY only); it now requires exactly host, deviceToken, vaultId, deviceId, deviceName and vaultGeneration.
GET / and GET /mobile-setup now return consolePage() and mobileSetupPage() in place of the P1 placeholders (gap G7 closed). The router test checks both pages carry the nonce CSP.
… is VAULT_READY only
… typing at the source - StreamStore.deleteAllStreamRows(): synchronous DELETE of every row of the three stream tables; safe inside the caller's transactionSync (D8a). - StreamRelayService.discardAll(closeCode, reason): drops the pending buffer (no commit, no receipts), head cache and dedupe index, then closes every streams socket with the given code and reason (D8a 1001, D8b 1001/1013). - StreamRelayService.writesThisRuntime: committed appended rows plus checkpoint writes since boot (D8b finish). - H3: the relay types its own refusals and commit failures (VAULT_ERROR cf_daily_limit via dailyLimitControl); DailyLimitLatch.decorateControl is deleted and host sendControl sends the value as is. Feed, read and checkpoint map daily-limit errors to 503 cf_daily_limit themselves. - DailyLimitLatch.simulate(enabled)/simulating(): the debug-route hook; instrumentStorageForDailyLimit reads it by default, so simulated writes throw Cloudflare's exact error.
…tryAfterMs, H5 socket cache, H6 per-device limits, H7 retired GC, H8 min interval - H1: strict APPEND decoder (fatal UTF-8 with ignoreBOM, minimal varuints <= 2^53, lengths within bounds, full consumption); violations close 1008 malformed_frame. - H2: per-stream lazy dedupe index over the newest 4 MiB of stored rows, built by a bounded cold scan (head + <= 64 segments), DO-wide 262,144-entry LRU cap, GC'd originals deduped by seq. - H3: durability_failed carries retryAfterMs, exponential in consecutive failures. - H5: socket cache built once per runtime; revoked devices skipped on rebuild. - H6: per-device token bucket and 4-socket cap (1001 device_socket_limit) ahead of the vault-wide cap. - H7: a checkpoint at lastSeq retires the open segment (gcSeq = lastSeq). - H8: gcMinIntervalMs defaults to 1000.
The worst-case cold scan (4 MiB of 25 B rows with distinct keys, 168k rows) measured about 16 ms of parse in Node, over the 10 ms Free budget. Builds now read once (head + up to 64 segments) and parse in steps of at most 16,384 rows, one step per incoming streams message and per timer flush. A flush holds back the frames of a stream whose index is still building and re-arms itself.
… D8c epoch, H3 debug route, O9 D9: R2 blobs at v/<vaultId>/<address> (format check only, PUT overwrites, 10 MiB cap -> 413, exists <= 50), bearer checked by the vault DO with zero config calls; no bucket -> 503 attachments_unavailable and capabilities.attachments=false. T-HOTPATH covers the blob routes. D8a: reset-streams with confirmVaultId; one transactionSync deletes the stream rows and mints a new epoch, then streams sockets close 1001. Devices, codes and blobs stay. D8c: optional epoch= on feed, read and checkpoint, checked right after bearer auth (409 vault_generation_mismatch). D8b: the config DO restore journal, steps 0-4 over prepareRestore, rewind and finishRestore, the alarm that resumes a run, the 60 s restoring flag, the authority freeze (409 on revoke, owner-code and reset-streams) and the PITR port. Local workerd has no PITR -> 501 restore_unsupported. T-RESTORE-RESUME-WB runs on a fake PITR with a crash around every step. H3: POST /vault/:id/debug/simulate-daily-limit only with YAOS_DEBUG_ROUTES=1 (else 404 in the Worker). O9: a daily-limit failure on any operator RPC -> 503 cf_daily_limit.
… for D8a A full-window cold read (4 MiB) costs about one 16,384-row parse on local workerd, so a build step now charges the read one row unit per 256 bytes and a large read is a step of its own. D8a: DROP+CREATE bills a constant 7 rows on workerd, but DROP TABLE fails with SQLITE_LOCKED while an unfinished cursor on the table is live, which the read and cold-scan paths leave by design. DELETE stays.
…nder YAOS_DEBUG_ROUTES=1
… P3 done Combined local run at 15e9bf3: 49 pass, 0 fail, 1 skip (T-BLOB-UNAVAILABLE, bucket bound; it passes in the no-bucket run).
…ames vault creation Found in the P4 manual run on local dev (recorded in DECISIONS §9): at 375 px the 22em min-width pushed the confirm and restore inputs past the card, and a fresh vault's default restore point (now - 1 h) is refused by G42 with a message that did not say why.
Deploying over yaos-relay2-scratch-3 failed with 10067: the API counts a migration's new classes before its deletions against the 500-namespace cap. An account at the cap applies v3 alone first; elsewhere one deploy applies both.
relay2-protocol.md, rfc-relay-bodies.md, relay3-group-commit.md and relay3-report.md describe the relay-v2-spike body relay (YAOS_RELAY_BODIES, group commit, tail row, receipt ring) of the old VaultSyncServer. The rewrite removed that subsystem (DECISIONS.md section 1 and the "Relay v2 spike" row of section 2.3); the streams relay's group commit is specified by relay-wire.md section 5 as amended by DECISIONS.md H8. The four files linked only to each other, so no inbound link needed fixing.
No caller outside legacy-src; its comment still named the deleted RecoveryJob alarm port.
A DO RPC error logs as its local stack only, so the scratch-3 restore failure (T-RESTORE-MANUAL, 503 restore_incomplete) carried no message. describeError (ports.ts) prints name: message; the vault also logs a getBookmarkForTime rejection before rethrowing it. Lint (P3b, 44c357c): encodeDevices copies into a fresh ArrayBuffer instead of asserting the slice type, and blobs/exists maps its present flags.
…point; alarm backoff from the journal row's age Measured on scratch-3: getBookmarkForTime rejects an at 11 s after the vault's init with "Requested time is before this database existed.", which the runner treated as retryable, leaving the journal row (and the vault's authority freeze) forever. The alarm backoff was an in-memory counter, so every eviction reset it to 30 s.
… console says a restore lands on a snapshot Measured on scratch-3: a Durable Object's PITR history starts with its first snapshot 45-55 s after the vault's init, and before it getBookmarkForTime rejects with "This database has no history.". Snapshots are about a minute apart and a restore lands on the latest one at or before at.
…ileSetupQrDataUrl (O12) Encoding the QR cost claim 64 ms and owner-code up to 46 ms of Worker CPU on scratch-3, over Free's 10 ms. The console page now inlines qrcode-generator's browser build verbatim as a second nonce script (a wrangler Text rule bundles it as text; the Node tests alias the import to the same file) and encodes buildMobileSetupUrl on location.origin. The console was the only reader of mobileSetupQrDataUrl; the client never read it.
…w suites The stub proxy reads and calls host methods with Reflect, duringCrypto narrows crypto.subtle's method before wrapping it, the H2 chunk test reads the fresh store from the harness, and FakeBucket.asR2 states its partial R2 surface with @ts-expect-error, as the client fakes do.
… and §15 match the rewritten server scratch-3 version 4de93251 (d51c3ae): jsQR decodes 9 owner-code QRs drawn by the console to the exact setup URL; Worker CPU max 2.6 ms around them and 6.1 ms over the suite (48 pass, 0 fail, 2 SKIP).
kavinsood
added a commit
that referenced
this pull request
Oct 8, 2026
…into client-remake-latency Conflicts: relay.ts keeps the batched read (readBatch) and the leading-edge commit next to H6/H8; both read paths go through dailyLimitAware. The H8 min interval (1000 ms) applies to lead commits too. deploy.sh keeps the cf deploy toolchain (VaultDO/ConfigDO classes). Tests: streams-relay keeps the lead and batch tests; the H8 hardening test runs with the lead off and expects leadMs/quietMs in VAULT_READY.
kavinsood
added a commit
that referenced
this pull request
Oct 8, 2026
Legacy parity port and old-client deletion, merged onto the server rewrite (PR #82), latency and E2EE design branches. Conflicts: every modify/delete took the delete. sessionLoop keeps the batched catch-up lanes and adds the background hold. Also removed the code PR #82 had kept only for the old client and its tests: legacy-src/shared, tests/client and tests/server/helpers/recoveryStateDecode.ts. The release contract drops the old client's version-pin checks. Runner aliases, package scripts, CI and the eslint ignores drop the QA harness, server-node, Docker, ywasm and live-worker entries. The suite-discovery floor is now 10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites the YAOS server against the opaque-streams contract (
docs/client-remake/relay-wire.md).docs/server-rewrite/DECISIONS.mdis the spec and the record: decisions D1–D9, hardening H1–H8, contract delta (§5), flag log (§8) and work plan P0–P5 (§9).This PR targets
client-remakebecauseserver-remakebranched from it (2d2ab2b). The diff shows only the server work. The two branches change no files in common, and they merge cleanly.What changes
retryAfterMs;minIntervalMs.mobileSetupQrDataUrl.packages/cli,packages/server-node, Docker);legacy-src/shared.minIntervalMsexample (:163) and §15's text now match the server. The remaining client-facing deltas are listed in DECISIONS §5. The broader shared docs are left for the merge intomain.server/wrangler.tomlmigration v3 deletes the legacy DO classes, and v4 adds the new ones. Deploying over an existing YAOS Worker deletes its data: this is a fresh deploy with no migration (README: the old classes are deleted with their data, and devices re-seed the new server from their files). The two migrations must be separate steps. A single migration doing both failed with10067at the account's 500-namespace cap, because the API counts the new classes before the deletions (DECISIONS §9, P5).Verification
node tests/run-suites.mjs --only tests/server: 14 suites passed, 0 failed, on this branch and on the merged tree.tscis clean.yaos-relay2-scratch-3, current version 4de93251.e2e/relay/smoke.ts: 31/31, three runs.CI
CI is expected to fail, but not because of this PR. The same five steps fail on
client-remakealone, which I checked locally on both trees:client-remakelegacy-src/andtests/suite-discovery.ts, all already onclient-remakesrc/and 3 inherited server lines (dailyLimit.ts,relay.ts, one test)src/sync,src/runtime,src/telemetryno longer existOpen items (DECISIONS §8)
exceededCpucannot show up.scriptThrewException(noise only).