Skip to content

Server P2b: compile a live configuration from books and start live from it - #141

Merged
sujitn merged 1 commit into
mainfrom
server-p2b-compiled-plan
Oct 22, 2026
Merged

sujitn merged 1 commit into
mainfrom
server-p2b-compiled-plan

Conversation

@sujitn

@sujitn sujitn commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Third of the smaller P2 PRs, stacked on #139. A live runtime can now be planned from stored books and inputs instead of a hand-written plan file.

Owner decisions (2026-10-22):

  • D7: compile → validate → stage → activate runs offline with convex-config against a stopped database; the server plans the active configuration at boot. In-process restart moves to the admin activation route (P8).
  • D8: the seed's SHOWCASE-USD uses MarketQuote{Mid}.
  • D9: each compiled version is its own runtime, <pricing config>-v<version>.

What it adds

configuration::assemble_from_repository (engine, configuration/compile.rs). It reads at one cutoff (as_of, known_at):

  • the universe: positions of 1 to 64 books that are effective at as_of and recorded by known_at, at most 256 instruments, deduplicated;
  • reference pins: each instrument's version at the cutoff. A tombstone is refused;
  • pricing: the latest version of the named stored config. It must be active, recorded by the cutoff, and its applies_to must match every instrument. Its id and version are the configuration identity, as the runtime already requires (configs.get_version);
  • curves: every configured role (discount, government, swap, FRN reference) through resolve_curves, so recipe revisions and their dependencies are pinned and must be fresh at the cutoff;
  • quotes: one requirement per instrument from the selected source, which must be a provider the server composes for the mode.

Everything else comes from a PlanSelection JSON with no defaults. The output is a ConfigurationDocument that the existing validate/stage/activate use cases take unchanged.

convex-config DB compile SELECTION prints that document. validate, stage and activate now check a document against the capabilities the server would compose for it: its mode and its quote sources. Before, they always used replay capabilities, so a live document could not pass.

live profile from the active configuration. CONVEX_CONFIGURATION_ID (or configuration_id in TOML) is the alternative to runtime_plan, and exactly one is required. Composition opens storage, reads the active configuration, plans it, and applies the same mode and build checks as a plan file. The stored assembly does not change, so a restart of the same activation plans the same digest and restores its checkpoint.

Engine fix: a new Live checkpoint starts at the current time. It used to start at the plan's as_of/known_at. For a compiled plan that is the compile cutoff, which is earlier than the activation, so ActiveConfiguration::capture refused it ("active configuration is later than information cutoff") and startup failed with RecoveryConflict. Live already values every tick at wall-clock time, so the start moves forward to the latest of the plan cutoff, the current time and the selected activation's activated_at. Live reads the wall clock on every tick, and a checkpoint cutoff must never go backwards. So each tick and each data expiry now also clamps to the checkpoint cutoff, as Replay already does with source and receipt times. Before, a tick after the clock stepped back was refused, and on the stream path that stopped the runtime. Replay and Captured are unchanged.

Seed: SHOWCASE-USD v1 uses {"MarketQuote": {"side": "Mid"}} in seed.json and make_seed.py. Price-by-ID never reads pricing_spec, and the existing price-by-ID test still gets SHOWCASE-USD@1, D.OIS@1,P.SOFR@1 and its price.

Tests

  • application_path::compiled_live_configuration_prices_posted_quotes_and_restarts_fresh replaces the ignored live test. It runs over the real composition, the real convex-config binary and HTTP:
    1. import the seed, PUT the book, post fresh Valid revisions of D.OIS and P.SOFR (the seed's dated Demo curves cannot pass fixture validation);
    2. compile, then check the runtime id SHOWCASE-USD-v1, the instrument and the curve versions;
    3. stage with a mid fixture (expected clean mid = quoted mid, tolerance 0), then activate;
    4. start live from configuration_id, post an image (202), wait for ready, and check results carry the runtime id;
    5. restart: not ready, revision higher; a fresh image brings it back to ready;
    6. check no snapshot or results archive was written.
  • convex-engine/tests/configuration_compile.rs:
    • exact pins, and the same digest for the same cutoff;
    • refusals: book recorded after the cutoff, book not yet effective, stale curves, unknown pricing config, a source that is not a composed provider, no books.
  • config.rs: Live needs exactly one plan source; configuration_id is refused outside Live.

Known limits

  • Universe changes: a staged version is immutable, so changing the universe needs a new pricing config version first. Pricing configs have no HTTP write route yet (P8); convex-core-inputs can import one.
  • Curve age: pinned recipes age from their recording time. After max_age_ms (at most a day) live valuations report the curve stale until a new configuration is activated. The showcase slice (P2b-2) has to handle this.
  • No WebSocket assertion yet: WebSocket delivery in the application-path test is P3.
  • Wall-clock steps (pre-existing, not fixed here): under WSL the realtime clock stepped back by 0.36 to 1.2 s about every 30 s during a stress run, measured with a monitor running alongside it. Reference and book writes are stamped at server now, and the next lookup at now can miss a record written just before such a step. In 1 of 30 runs this made the P6 test book_snapshot_ingest_resolves_external_identifiers return 422. The new live test uses a monotonic timestamp floor that includes the server-stamped book time. Monotonic recorded_at in the reference repository is a follow-up.

Verification (WSL)

  • cargo test --locked -p convex-engine --no-fail-fast: all binaries pass, including configuration_compile and runtime.
  • cargo test --locked -p convex-server --no-fail-fast: all binaries pass, 0 failed.
  • cargo clippy --locked -p convex-engine -p convex-server --all-targets -- -D warnings: clean.
  • cargo fmt --all --check: clean.
  • python3 tools/assurance/check.py: "status": "passed".
  • application_path stress, 30 runs in a row with a realtime-clock monitor alongside: 30/30 pass.
    • Earlier runs logged backward clock steps of 0.36 to 1.3 s. They exposed the Live start and tick bugs fixed here, one failure in about 30 runs each.
    • The final run logged steps of about 10 ms.
    • The pre-existing P6 test can still fail during large steps; see Known limits.
  • CI Test is red on main until Bump redb to 4.4: fix integrity failure after compaction (main CI red since #136) #140 (redb 4.4) merges.

Next

P2b-2, the showcase on the live profile:

  • the entrypoint compiles and activates from a seeded book with curves stamped at boot;
  • a local feeder posts quotes;
  • no deploy without approval.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2686f982-0331-4ee1-a858-15da5ce01f25
📥 Commits

Reviewing files that changed from the base of the PR and between 84f1899 and ccdc525.

📒 Files selected for processing: 14
  • crates/convex-engine/CONFIGURATION.md
  • crates/convex-engine/RUNTIME.md
  • crates/convex-engine/src/configuration.rs
  • crates/convex-engine/src/configuration/compile.rs
  • crates/convex-engine/src/curve_resolution.rs
  • crates/convex-engine/src/runtime/actor.rs
  • crates/convex-engine/tests/configuration_compile.rs
  • crates/convex-server/README.md
  • crates/convex-server/src/bin/convex-config.rs
  • crates/convex-server/src/composition.rs
  • crates/convex-server/src/config.rs
  • crates/convex-server/tests/application_path.rs
  • deploy/showcase/make_seed.py
  • deploy/showcase/seed.json
+------------------------------------------------------+
| Weeks of programming can save you hours of planning. |
+------------------------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…om it

- configuration::assemble_from_repository reads books, reference pins, the
  latest stored pricing config and fresh curve recipes at one cutoff and
  builds a ConfigurationDocument; convex-config gains `compile`.
- convex-config validates, stages and activates against the capabilities
  the server composes for the document's mode and quote sources.
- The live profile can plan the stored active configuration
  (CONVEX_CONFIGURATION_ID) instead of a plan file.
- Live runtime: a new checkpoint starts no earlier than now and the
  selected activation; tick and expiry cutoffs never move backwards when
  the wall clock steps back.
- Seed SHOWCASE-USD uses MarketQuote Mid.
- application_path: compiled live configuration prices posted quotes and
  restarts fresh; engine tests for compile pins and refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sujitn
sujitn force-pushed the server-p2b-compiled-plan branch from dd35e09 to ccdc525 Compare October 22, 2026 20:42
@sujitn
sujitn changed the base branch from server-p2a-quote-ingress to main October 22, 2026 20:42
@sujitn
sujitn merged commit f7084db into main Oct 22, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant