Skip to content

feat(observability): per-module fuel and memory gauges - #379

Merged
mfw78 merged 2 commits into
mainfrom
feat/fuel-and-memory-gauges
Aug 26, 2026
Merged

mfw78 merged 2 commits into
mainfrom
feat/fuel-and-memory-gauges

Conversation

@mfw78

@mfw78 mfw78 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What

Two per-module gauges, both holding the last dispatch.

nexum_runtime_module_fuel_consumed is the seed's fuel ceiling minus Store::get_fuel() after the call, read beside the existing latency histogram so every outcome that entered the guest contributes. It changes no behaviour.

nexum_runtime_module_memory_bytes comes from ObservedLimits, a new ResourceLimiter in crates/nexum-runtime-wasm/src/limits.rs that holds a real wasmtime::StoreLimits, forwards all seven trait methods to it, and records the last allowed desired in memory_growing. Since wasm memory never shrinks, that value is the current size rather than an estimate.

Why

Closes #370

A module approaching fuel exhaustion was invisible until it trapped, and capacity planning had no memory signal to read.

Enforcement stays in wasmtime

The wrapper decides nothing. Every refusal is still StoreLimits' own logic, and StoreLimitsBuilder in supervisor/store.rs is unchanged apart from being wrapped. Replacing it outright was considered and rejected in the issue: owning the refusal path means a mistake lets a module exceed a ceiling the operator set, silently, and the signal never required it.

The omission hazard, and the two tests that catch it

StoreLimits overrides seven ResourceLimiter methods and the trait supplies defaults for five of them. A wrapper forwarding only memory_growing therefore compiles clean, reads fine, and silently replaces the operator's table and instance caps with wasmtime's built-in 10,000 defaults, with nothing warning and no existing test failing.

Two tests close that, and neither is about memory:

  • the_table_ceiling_still_refuses_past_its_limit proves the forwarded table_growing still bites.
  • the_instance_table_and_memory_counts_are_the_wrapped_ones proves instances, tables and memories return the configured values rather than the trait defaults.

the_memory_ceiling_refuses_what_it_refused_before covers the Done-when directly: the ceiling refuses what it refused, not merely that the gauge moved.

Gauge semantics, stated

Both report only after a dispatch, so a module that has never run has no series. That is deliberate and differs from module_poisoned and module_unverified, which are bad-state flags where absence means healthy. These are levels, and a level for a module that has not run is not zero, it is unknown.

Memory is observed from store construction, so it includes the instance's initial allocation and a module that never grows past instantiation still reports a non-zero size. That is what the >= 64 KiB assertion encodes.

One series per module per signal, labelled by the operator-written module id and nothing a guest can influence.

Two things for a reviewer

HostState.limits changed type from wasmtime::StoreLimits to ObservedLimits, so nexum-runtime-wasm's state.rs and lib.rs are in the diff beyond the files the charter named. That is the only place the limiter object can live, since the supervisor depends on nexum-runtime-wasm and not the reverse. One construction site exists workspace-wide.

The gauges are also read after a deadline-cancelled dispatch, where the call future was dropped mid-await. Store::get_fuel and Store::data are plain accessors and do not re-enter the guest, and the deadline test passes with the emit in place, but it is worth agreeing that sampling a store the trap arm is about to mark dead is what you want.

No alert rule was added, because neither ceiling is exported as a series, so an expression would have to hardcode the operator's configured number. The metric rows say so instead.

Testing

just build, then the full just test set: 649 passed. just test-e2e: 16 passed. cargo nextest run --workspace --all-features: 952 passed, 0 skipped. just ci end to end, including cargo deny check, workspace clippy and rustdoc under -D warnings. Cargo.lock untouched; no new dependencies.

AI Assistance

Implementation: claude-opus-5. Red-team review: claude-opus-5. Verification: claude-opus-5. PR description: claude-opus-5.

mfw78 added 2 commits August 26, 2026 09:02
A module approaching its fuel or memory ceiling was invisible until it trapped. Two gauges now hold the last dispatch, matching the shape of the poisoned and unverified gauges: nexum_runtime_module_fuel_consumed, read from Store::get_fuel against the seed's ceiling, and nexum_runtime_module_memory_bytes.

Memory is observed through ObservedLimits, a ResourceLimiter holding a real wasmtime::StoreLimits and forwarding every decision to it, recording the last allowed desired on memory_growing. Enforcement stays in wasmtime, and because linear memory never shrinks that value is the current size, so the reading is exact rather than an estimate.

Forwarding all seven trait methods is what the unit tests pin. The trait supplies defaults for five, so a wrapper forwarding only memory_growing compiles clean and silently replaces the operator's table and instance caps with wasmtime's built-in 10,000.

Closes #370.

AI Assistance: Claude Code used for implementation, tests and docs.
`memory_growing` reports one memory at a time, and `current` restarts at
zero for each memory a store creates. Recording the last allowed `desired`
therefore made the gauge bounce between memories on a component holding
more than one, reporting neither the total nor the size that matters. The
wrapped ceiling is applied per memory, so the largest allowed `desired` is
the reading that compares against it.

The dispatch test allowed the gauge to equal the memory ceiling, which is
the one wrong value its own comment says it is checking against; it now
requires strictly less.

AI Assistance: Claude Code used for the red-team review and this fix.
@mfw78
mfw78 merged commit 662d500 into main Aug 26, 2026
7 checks passed
@mfw78
mfw78 deleted the feat/fuel-and-memory-gauges branch August 26, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

observability: per-module fuel and memory gauges, observing the limiter rather than replacing it

1 participant