Skip to content

[FEAT] Valkey adapters for the state ports, the counters, and the Durable boot arm - #454

Open
justin13888 wants to merge 11 commits into
feat/server-binary-config-operator-commands-401from
feat/valkey-adapters-403
Open

[FEAT] Valkey adapters for the state ports, the counters, and the Durable boot arm#454
justin13888 wants to merge 11 commits into
feat/server-binary-config-operator-commands-401from
feat/valkey-adapters-403

Conversation

@justin13888

@justin13888 justin13888 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Description

Lane W-VALKEY of the unattended backlog run (issue #403), stacked on #435 (feat/server-binary-config-operator-commands-401).

Summary

  • capsule-server/src/store/valkey.rs: redis-rs adapters for all six state ports (AuthStateStore, UploadSessionStore, ChallengeStore, EnrollmentStore, ChannelStore, CohortStore) over one multiplexed ConnectionManager, plus the ValkeyStores bundle. Every multi-key mutation or decide-and-write is one Lua script (EVALSHA, SCRIPT LOAD on NOSCRIPT); every derived index heals on read; expiry is decided by the injected Clock and written into each record, with PEXPIRE as the collector.
  • capsule-server/src/counter/valkey.rs: ValkeyCounters, one hash and one script per key — open, charge or refuse in one server-side step.
  • capsule-server/src/counter/conformance.rs: the counter port's suite, extracted from counter/tests.rs and generic over the store, with a racing case on a multi-threaded runtime; the in-memory adapter now runs it case by case and in one pass.
  • capsule-server/tests/valkey.rs: the store and counter suites plus a contested finalize claim and a contested counter hit, against a valkey/valkey container (CAPSULE_TEST_VALKEY=1) or a running server (CAPSULE_TEST_VALKEY_URL); skipped with one line otherwise. .config/nextest.toml puts it in the one-thread containers group.
  • capsule-server/src/boot.rs: the Durable arm connects to and PINGs VALKEY_URL first (BootError::Valkey on failure, never quoting the URL), then refuses naming DATABASE_URL and server: Postgres adapters and a conformance suite for every durable port #402 until the Postgres half lands. .env.example, local-development.md, dependencies.md (Volatile state row) and SLICES.md (S-C29) say so.

Validation

Run inside the worktree at head eb1e8d2b (review round 1 applied), foreground, CARGO_TARGET_DIR=/var/tmp/capsule-lane-403/target unless noted (the scratch volume was full when the lane started; the earlier target symlink is removed, F4).

Command Outcome Class
cargo nextest run -p capsule-server 767 passed
DOCKER_HOST=unix:///run/user/2000/podman/podman.sock CAPSULE_TEST_CONTAINER_USERNS=keep-id CAPSULE_TEST_VALKEY=1 cargo nextest run -p capsule-server --test valkey --no-fail-fast 6 passed — the whole store conformance suite (now 34 cases incl. a_claimed_session_leaves_the_eviction_view), the whole counter suite, the 16-way finalize race, the 40-way counter race, the durable arm reaching Valkey then naming DATABASE_URL/#402, and a_logically_expired_record_is_dead_but_left_for_the_collector; each test started its own valkey/valkey:9.0.4 container through the podman socket
the same without CAPSULE_TEST_CONTAINER_USERNS (earlier head) FAIL WaitContainer(WaitLog(EndOfStream([]))) — this host's rootless podman cannot read the image entrypoint without keep-id unavailable (host runtime, without the option)
cargo nextest run -p capsule-server --test binary 21 passed
mise run doc-check-rust exit 0
mise run check-docs-truth exit 0
mise run check-md exit 0
mise run check-rust exit 1 at the 14th of 15 sub-tasks: gen-bindingsmise-tasks/gen-bindings hardcodes target/debug/… and target/bindings, so it cannot find the library under CARGO_TARGET_DIR. The 13 sub-tasks before it (format-check, lint-check, doc-check, i18n-check, i18n-guard, openapi-check-kynos, architecture-check, license-check, translate-readme-check, build-rust, build-check-wasm, build-ffi, lint-check-ffi) passed. unavailable under CARGO_TARGET_DIR; sub-tasks re-run below
mise run gen-bindings (default target dir) exit 0 — bindings written, surface check passed
mise run verify-examples (default target dir) exit 0 — 16 passed, examples present
mise run test-rust exit 0: cargo nextest run --workspace 1824 passed; -p capsule-core --features ffi 729 passed; -p capsule-sdk --features ffi 160 passed
mise run check-commits (at a959cb36) exit 0
earlier heads: mise run check-rust exit 0 at a959cb36 with the worktree target/ in place; mise run test-rust exit 0 at 0af7870c (1820 passed)

Risks and rollout

  • The in-memory adapters and Backends::Memory are untouched; the Valkey adapters are reached only through Backends::Durable, which still refuses at the end of the boot path until server: Postgres adapters and a conformance suite for every durable port #402. Nothing persists a schema; every key is under capsule: and carries a TTL except the cohort hash.
  • CohortStore on Valkey is interim (decision 3); the key has no TTL so the data migrates with one HGETALL per account.
  • Review round 1 (decisions 11–14, F1–F10) is applied in abd6a5a3, dc447549, eb1e8d2b.
  • The container-gated suite needs a container runtime the gate host did not have; see Validation.

Related Issues

Closes #403. Refs #402 (the Postgres half of the Durable arm, DATABASE_URL).

Decisions taken

Issue 403 - server: Valkey adapters for the auth-state and upload-session ports
Plan:     A-1 (planned against f433d918; executed on the head of lane #401's branch)
Branch:   feat/valkey-adapters-403
Base:     feat/server-binary-config-operator-commands-401 (head of the W-SERVE PR), stacked; the PR targets that branch
Worktree: /var/mnt/scratch/golem/dev/Capsulsaurus/Capsule.worktrees/Capsule-feat-valkey-adapters-403
Cause:    -
Touches:  capsule-server/Cargo.toml (redis; dev: testcontainers, testcontainers-modules), capsule-server/src/store/{mod.rs (pub mod valkey), valkey.rs (new)}, capsule-server/src/counter/{mod.rs, conformance.rs (new), valkey.rs (new), tests.rs}, capsule-server/src/boot.rs (Durable arm's Valkey half: ConnectionManager + PING), capsule-server/tests/valkey.rs (new, env-gated CAPSULE_TEST_VALKEY=1), .config/nextest.toml (containers group filterset), capsule-docs/src/content/docs/design/dependencies.md (redis-rs row: scope + primitives), Cargo.lock, SLICES.md (S-C29 owed note ONLY — coordinate: #402 edits S-C2/S-C37 notes), capsule-server/.env.example (Valkey half of the Backends comment), capsule-docs/src/content/docs/development/local-development.md (the sentence that says VALKEY_URL refuses, if present)
Will not: write a Postgres adapter; enable the multi-replica share limiter or revocation cache; support Redis Cluster; introduce a generic TTL/CAS type; use bb8
Lane:     serialised behind #401; parallel with #402 (shared: boot.rs Durable arm — one hunk each; .config/nextest.toml — adjacent overrides; SLICES.md S-C29 row — both lanes append notes to the same row: this lane owns the row, #402 records its S-C29 remainder in its PR body instead)
Settled:  Separate typed ports, no generic TTL/CAS (AGENTS.md). Barrels (#399). Config/boot seam (#401). Base = head of PR #418 → stacks on #399 → #401.

Decisions taken.

1. Deliverable boundary - all six store ports plus the counters in one lane.
   Taken:    AuthStateStore, UploadSessionStore, ChallengeStore, EnrollmentStore, ChannelStore, CohortStore (interim Valkey hash, no expiry) + ValkeyCounters, the container-gated suite, and the Durable boot arm. store::conformance.rs:61-76 makes Harness demand all six, so a partial set cannot run run_all.
   Rejected: the two named ports only - Backends::Durable would mix Valkey and in-memory doubles in one deployment profile, which filesystem/server.md:21 forbids.
   Reverses: delete the ceremony/cohort impls from ValkeyStores; the Harness impl stops compiling, which is the signal.
   Filed:    -

2. Connection strategy and CAS
   Taken:    One redis::aio::ConnectionManager plus one Lua script per multi-key mutation (EVALSHA with SCRIPT LOAD fallback); no bb8 (the bb8/bb8-redis workspace pins stay unused).
   Rejected: bb8 + WATCH/MULTI - WATCH needs an exclusive connection held across the optimistic loop, reintroducing the read-then-write window claim_finalize and counter hit exist to close.
   Reverses: swap ConnectionManager for bb8::Pool and each Script::invoke for a WATCH/MULTI closure; call sites unchanged.

3. Where the device-cohort map lives in the Durable arm
   Taken:    A Valkey hash capsule:cohorts:{user_id} with no expiry, documented as interim until #402's remainder lands a Postgres CohortStore (note: #402 ships PostgresCohorts in this run — the lane wires ValkeyStores::cohorts to satisfy Harness and boot.rs prefers the Postgres adapter when DATABASE_URL is set; record which was wired).
   Rejected: InMemoryCohorts in Durable (authentication.md:211: durable); deferring (Harness has no optional member).
   Reverses: point ValkeyStores::cohorts at the Postgres adapter and delete ValkeyCohorts; data migrates with one HGETALL.

4. How the Valkey harness moves time
   Taken:    Real tokio::time::sleep in Harness::advance against stores built with a 250 ms uniform TTL; the containers group is max-threads = 1.
   Rejected: a fresh container per expiry case with production TTLs.
   Reverses: change with_uniform_ttl's argument and the sleep.

Decisions taken inside the manifest.

5. How expiry is decided - the injected Clock, with PEXPIRE as the collector (reverses 4)
   Taken:    Every record hash carries an adapter-internal expires_at (microseconds) written from the injected Clock when the record is opened; every script that reads a record checks it against a `now` argument and deletes an expired one; PEXPIRE is set on the same key with the same lifetime and only collects keys nothing reads again. The Valkey harness in tests/valkey.rs therefore advances a ManualClock exactly as InMemoryStores does, with a one-minute uniform TTL and no sleeps. The counter window is likewise measured from the port's own `at` argument (opened_at in the hash), so counter::conformance passes absolute instants and drives both adapters unchanged.
   Rejected: Sleeping (decision 4). store::conformance's touching_a_session_records_activity_without_extending_its_life advances to ttl - 1 ns and expects the session live, then 1 ns more and expects it gone; no real-clock harness can land a round trip inside the last nanosecond of a window, so a sleeping harness needs a heuristic that under-sleeps sub-TTL advances and repays them on the next - a claim about the suite rather than the adapter, and one the counter suite's own sub-window steps break. Also rejected: INCR plus first-hit EXPIRE for counters, which measures the window on the server's clock and cannot answer the suite's `at`.
   Reverses: delete the expires_at/opened_at fields and the `live`/`expired` Lua helpers; the harness must then sleep, and decision 4's caveat returns.
   Filed:    -

6. What the Durable arm does once Valkey answers
   Taken:    boot::valkey connects, PINGs and builds ValkeyStores first; boot::durable then returns BootError::AdapterUnavailable { key: "DATABASE_URL", issue: "#402 (Postgres)" }. A new BootError::Valkey names an unreachable server without the URL. #402's merge fills durable's body; the hunk is one function each.
   Rejected: assembling the App over Valkey stores plus in-memory doubles for the index, accounts, albums and the rest - the mixed profile store/mod.rs forbids. Also rejected: leaving the old VALKEY_URL refusal in place - the Valkey half would be unreachable from any boot path.
   Reverses: replace durable's body with the assembly; delete nothing.
   Filed:    -

7. The eviction view excludes a claimed session
   Taken:    capsule:upload:progress holds only Pending and Uploading sessions; claim_finalize and set_status(WaitingForProcessing) ZREM the id, per the port's claim_finalize doc ("the winner leaves the progress view rather than being evicted out from under itself").
   Note:     the divergence from the in-memory double this originally recorded was closed by decision 11 in review round 1; both adapters now apply UploadSessionStatus::is_evictable and the conformance suite asserts it.
   Rejected: mirroring the double.
   Reverses: make is_evictable equal to is_active.
   Filed:    -

8. Test-side environment surface
   Taken:    CAPSULE_TEST_VALKEY=1 (container via testcontainers), CAPSULE_TEST_VALKEY_URL (an already-running server, takes precedence), CAPSULE_TEST_VALKEY_TAG (image tag; default 9.0.4, the compose.yaml image), CAPSULE_TEST_CONTAINER_USERNS (the container's userns mode, e.g. keep-id — what this host's rootless podman needs; the name is shared with lane #402's suite). The plan named CAPSULE_TEST_VALKEY_IMAGE; testcontainers-modules' Valkey image fixes the repository and varies only the tag, so the variable is named for what it sets.
   Rejected: no URL override - the gate host's rootless podman could not run the image without keep-id, and a URL override is also the developer path against `mise run serve-deps`.
   Reverses: delete the branch in tests/valkey.rs::server.
   Filed:    -

10. Manifest widened by the orchestrator for the test that encodes the old behaviour
   Taken:    capsule-server/tests/binary.rs joins the manifest for one case: a_durable_backend_refuses_with_the_issue_that_will_honour_it asserted the refusal this PR replaces (VALKEY_URL set, stderr naming #403 as unimplemented), so it is part of this change. It now points at redis://127.0.0.1:1 and asserts a non-zero exit, VALKEY_URL named, and the address never echoed; the reachable case stays in tests/valkey.rs. The hunk is one test so #447, which also edits binary.rs, merges mechanically after #454.
   Rejected: leaving test-rust red for #447 to fix.
   Reverses: restore the old case; it fails again on the first commit that connects to VALKEY_URL.
   Filed:    -

11. The in-memory double is the defect (review round 1, Q1 -> a, F1)
   Taken:    Manifest widened by capsule-server/src/store/{memory.rs, conformance.rs, upload.rs}. UploadSessionStatus::is_evictable (Pending | Uploading) is the one predicate least_recently_progressed applies on both adapters; memory.rs used is_active and kept a claimed session a pressure-eviction candidate (upload-protocol.md, the finalization claim). New conformance case a_claimed_session_leaves_the_eviction_view (open -> claim_finalize -> least_recently_progressed no longer returns it) in run_all and the in-memory case list; the set_status and least_recently_progressed docs name the real predicate.
   Rejected: b (a red suite until a follow-up), c (leaving a live pressure-eviction hazard in the only bootable profile).
   Reverses: revert abd6a5a3.
   Filed:    -

12. The read gate answers, the collector deletes (review round 1, Q2 -> a, F3)
   Taken:    live() is non-destructive: it returns false on logical expiry and leaves the key for PEXPIRE. One replica's clock is never allowed to delete state for every replica; every derived index already heals on a miss. tests/valkey.rs::a_logically_expired_record_is_dead_but_left_for_the_collector asserts a record past its expires_at is absent to a reader while EXISTS is 1 and PTTL is positive. The module doc's "two enforcers" paragraph is rewritten to one collector and a read gate.
   Rejected: b (an NTP requirement), c (a monotonic guard). Accepted cost of rejecting b: the index-listing scripts (SESSIONS_FOR_USER, UPLOADS_FOR_UPLOADER, PENDING_FOR_ADDRESS, IN_FLIGHT_FOR_ALBUM, LEAST_RECENTLY_PROGRESSED) SREM/ZREM a member they judge dead from a shared index, so a fast clock on one replica hides a still-live record from every replica's listings — not from direct reads — until a state change re-indexes it or it expires for real; the exposure is bounded by the skew and the record is never deleted.
   Reverses: put the DEL back in live(); decision 12's test fails, which is the signal. To remove the listing residual instead, take option c: a monotonic guard that never judges a record dead earlier than the previous verdict on this replica, or an NTP requirement (b).
   Filed:    -

13. NOSCRIPT is retryable (review round 1, Q3 -> a, F2)
   Taken:    ServerErrorKind::NoScript joins never_sent -> StoreError::Unavailable; driver_errors_are_classified_by_whether_the_command_could_have_run unit-asserts the classification of every kind named, and that a corrupt reply's text never reaches the error.
   Rejected: leaving it Rejected.
   Reverses: drop the variant from the match.
   Filed:    -

14. Stale lines about what reads VALKEY_URL (review round 1, Q4 -> a, F5)
   Taken:    Manifest widened by capsule-server/compose.yaml (the header comment), mise.toml (the serve-deps and serve task notes) and capsule-server/src/config.rs (the valkey_url field doc), each fixed to the behaviour that ships: Valkey is read; serve on the durable path refuses naming DATABASE_URL/#402 until #447 lands. One hunk per line so #447 merges mechanically.
   Rejected: leaving them for #447.
   Reverses: revert eb1e8d2b.
   Filed:    -

9. redis 1.2.2, not the 1.6.0 the caret pin resolves to
   Taken:    Cargo.lock pins redis at 1.2.2, the version the workspace manifest declares, via `cargo update -p redis --precise 1.2.2`; features tokio-rustls-comp and script are added in capsule-server/Cargo.toml (the root manifest is outside the manifest).
   Rejected: accepting 1.6.0 - the adapter was written against the 1.2.2 API and the declared pin should be the locked one.
   Reverses: `cargo update -p redis`.
   Filed:    -

Unresolved review notes

  • (F10, out of manifest) capsule-server's rustdoc is not gated by doc-check-rust, and capsule-server/src/lib.rs:72 and :98 carry two pre-existing ambiguous intra-doc links ([openapi]); for the gates lane.
  • mise-tasks/gen-bindings hardcodes target/debug/… and target/bindings, so mise run check-rust cannot pass under a CARGO_TARGET_DIR; out of manifest, recorded for the gates lane.
  • Bearer secrets are the key names (capsule:challenge:{token}, capsule:enroll:code:{code}), as in the retired server, so they appear in SLOWLOG/MONITOR. The record values carry the same secrets, so hashing the key names alone would not remove them from MONITOR; a follow-up if the operational exposure matters.

`redis` 1.2.2 (the version the workspace declares; the caret pin resolved to
1.6.0 and is locked back with `--precise`) with `tokio-rustls-comp` so a
`rediss://` URL terminates TLS in rustls, and `script` for the Lua scripts
every multi-key mutation becomes. `testcontainers` and
`testcontainers-modules` (`valkey`) as dev-dependencies for the env-gated
live suite. No `bb8`: one multiplexed `ConnectionManager` is the whole of
what a server talking to one Valkey needs.

The Volatile state row in design/dependencies.md records the scope, the
primitives and the two rejections (a pool, Redis Cluster).

Refs #403
One adapter per port in `store::valkey` — sessions, upload sessions, the
three ceremonies and the device-cohort map — and `counter::valkey` for the
counter port, all over one multiplexed, self-reconnecting
`ConnectionManager`. Every multi-key mutation or decide-and-write is one
Lua script (`EVALSHA`, `SCRIPT LOAD` on `NOSCRIPT`): the finalize claim,
the challenge consume, the enrollment redeem and the counter hit cannot be
read-then-written because there is no read a caller performs separately.

Every derived index — the per-user and per-uploader sets, the per-album and
pending-address sets, the global progress sorted-set — resolves each member
through its record inside the script and drops a stale one, so an expired
record leaves no listing entry behind without a second lifetime on the
index. Expiry is decided by the injected `Clock`, written into each record
as `expires_at`; `PEXPIRE` on the same key is only the collector. That is
what lets the shared conformance suite drive this adapter with a manual
clock exactly as it drives the in-memory double, one nanosecond either side
of a boundary, with no sleeps.

The counter port's cases move from `counter/tests.rs` into
`counter::conformance`, generic over the store and with a racing case on a
multi-threaded runtime; the in-memory adapter runs them case by case and in
one pass. `tests/valkey.rs` runs both suites, a contested finalize claim
and a contested counter hit against a `valkey/valkey` container
(`CAPSULE_TEST_VALKEY=1`) or a running server (`CAPSULE_TEST_VALKEY_URL`),
and passes as skipped otherwise; `.config/nextest.toml` places it in the
one-thread `containers` group.

Refs #403
`assemble` on `Backends::Durable` now connects to `VALKEY_URL`, proves it
answers `PING`, and builds every Valkey store before anything else is
assembled; a server that cannot be reached is `BootError::Valkey`, whose
detail never carries the URL. The Postgres half is not written, so
`durable` then refuses as `AdapterUnavailable` naming `DATABASE_URL` and
in-memory doubles — #402 fills that one function's body.

`.env.example`, the local-development page and the S-C29 slice row say
what `serve` does with each variable now.

Refs #403
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 2, 2026

Copy link
Copy Markdown

Deploying capsule with  Cloudflare Pages  Cloudflare Pages

Latest commit: e05c633
Status: ✅  Deploy successful!
Preview URL: https://204df76f.capsule-22k.pages.dev
Branch Preview URL: https://feat-valkey-adapters-403.capsule-22k.pages.dev

View logs

- A failure the driver cannot place before the command was sent — a
  response timeout, a connection dropped mid-flight — is `Rejected`, not
  `Unavailable`: the script may already have burned the challenge or won
  the claim, and the port defines `Unavailable` as "certainly did not
  happen". Only a refused connection and a server that declined to execute
  (`LOADING`, `TRYAGAIN`, `MASTERDOWN`, `CLUSTERDOWN`) stay `Unavailable`.
- A reply of the wrong shape is `Corrupt` with the driver's error *kind*
  only: redis-rs quotes the offending value in a type error, and for the
  ceremony stores that value is the record carrying the bearer secret.
- A derived index set's TTL is only ever raised (`extend` in Lua), so one
  member's remaining life cannot shorten another's; and re-opening an
  upload under a different uploader, owner, hash or album unindexes the
  previous record, as `OPEN_SESSION` already did for a previous user.
- Listings report expired members (`debug`) apart from mismatched ones
  (`warn`), so the routine heal is not the drift signal.
- The eviction query over-fetches one page so the members sharing the
  horizon's microsecond do not cost a candidate; `peek` on an unopened
  window answers `Admitted` for every limit, as the double does; the
  counter adapter reuses the store's `Lua` type.

Refs #403
`CAPSULE_TEST_CONTAINER_USERNS` (for instance `keep-id`) is passed to
testcontainers as the container's userns mode, the knob a rootless podman
may need to run the image at all.

Refs #403
The binary smoke asserted the refusal the Durable boot arm replaced — a
set `VALKEY_URL` naming #403 as unimplemented. The arm now connects to it,
so the case points at a port nothing listens on and asserts what the new
refusal promises: a non-zero exit, `VALKEY_URL` named, and the address
never echoed. The reachable case lives in `tests/valkey.rs`.

Refs #403
…apter

`UploadSessionStatus::is_evictable` — `Pending` or `Uploading` — is now the
one predicate `least_recently_progressed` applies. The in-memory double used
`is_active`, which kept a `WaitingForProcessing` session a pressure-eviction
candidate after a finalizer had claimed it; the port's `claim_finalize` doc
promised the opposite, and the upload protocol's finalization claim is that
promise. The conformance suite gains
`a_claimed_session_leaves_the_eviction_view` so both adapters are held to it,
and the `set_status`/`least_recently_progressed` docs name the real predicate.

Refs #403
`live()` answers whether a record has passed its logical `expires_at` and
leaves the key for `PEXPIRE` to collect: a replica whose clock ran ahead
must not delete, for every other replica, state that is still live by the
store's own lifetime, and the derived indexes already heal on a miss. The
live suite asserts a logically expired record is absent to a reader and
still present with a collector TTL.

Also: `PEXPIRE` lifetimes round up to whole milliseconds so the collector
never runs ahead of the logical lifetime; `NOSCRIPT` is classified
`Unavailable` (nothing executed, retryable) and the classification is
unit-tested, including that a corrupt reply's text never reaches the error;
the pending-address key asserts its two variable segments carry no `:`;
the module doc records the microsecond floor and that the progress view
heals only when the pressure sweep runs; the adapter uses the shared
`UploadSessionStatus::is_evictable`.

Refs #403
`compose.yaml`, the `serve-deps`/`serve` task notes and the `valkey_url`
field doc said nothing read either URL yet. Valkey is read (#403); `serve`
on the durable path connects to it and then refuses naming `DATABASE_URL`
until the Postgres adapters (#402) land.

Refs #403
…ect reads

A false "not live" verdict from a fast clock costs only that replica a miss
on the direct-read scripts. The index-listing scripts remove the member from
a shared index, so it hides a still-live record from every replica's
listings until a state change or real expiry: the residual accepted in
decision 12 for not requiring synchronised clocks, now stated where the
gate is documented.

Refs #403
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant