feat(observability): per-module fuel and memory gauges - #379
Merged
Merged
Conversation
A module approaching its fuel or memory ceiling was invisible until it trapped. Two gauges now hold the last dispatch, matching the shape of the poisoned and unverified gauges: nexum_runtime_module_fuel_consumed, read from Store::get_fuel against the seed's ceiling, and nexum_runtime_module_memory_bytes. Memory is observed through ObservedLimits, a ResourceLimiter holding a real wasmtime::StoreLimits and forwarding every decision to it, recording the last allowed desired on memory_growing. Enforcement stays in wasmtime, and because linear memory never shrinks that value is the current size, so the reading is exact rather than an estimate. Forwarding all seven trait methods is what the unit tests pin. The trait supplies defaults for five, so a wrapper forwarding only memory_growing compiles clean and silently replaces the operator's table and instance caps with wasmtime's built-in 10,000. Closes #370. AI Assistance: Claude Code used for implementation, tests and docs.
`memory_growing` reports one memory at a time, and `current` restarts at zero for each memory a store creates. Recording the last allowed `desired` therefore made the gauge bounce between memories on a component holding more than one, reporting neither the total nor the size that matters. The wrapped ceiling is applied per memory, so the largest allowed `desired` is the reading that compares against it. The dispatch test allowed the gauge to equal the memory ceiling, which is the one wrong value its own comment says it is checking against; it now requires strictly less. AI Assistance: Claude Code used for the red-team review and this fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two per-module gauges, both holding the last dispatch.
nexum_runtime_module_fuel_consumedis the seed's fuel ceiling minusStore::get_fuel()after the call, read beside the existing latency histogram so every outcome that entered the guest contributes. It changes no behaviour.nexum_runtime_module_memory_bytescomes fromObservedLimits, a newResourceLimiterincrates/nexum-runtime-wasm/src/limits.rsthat holds a realwasmtime::StoreLimits, forwards all seven trait methods to it, and records the last alloweddesiredinmemory_growing. Since wasm memory never shrinks, that value is the current size rather than an estimate.Why
Closes #370
A module approaching fuel exhaustion was invisible until it trapped, and capacity planning had no memory signal to read.
Enforcement stays in wasmtime
The wrapper decides nothing. Every refusal is still
StoreLimits' own logic, andStoreLimitsBuilderinsupervisor/store.rsis unchanged apart from being wrapped. Replacing it outright was considered and rejected in the issue: owning the refusal path means a mistake lets a module exceed a ceiling the operator set, silently, and the signal never required it.The omission hazard, and the two tests that catch it
StoreLimitsoverrides sevenResourceLimitermethods and the trait supplies defaults for five of them. A wrapper forwarding onlymemory_growingtherefore compiles clean, reads fine, and silently replaces the operator's table and instance caps with wasmtime's built-in 10,000 defaults, with nothing warning and no existing test failing.Two tests close that, and neither is about memory:
the_table_ceiling_still_refuses_past_its_limitproves the forwardedtable_growingstill bites.the_instance_table_and_memory_counts_are_the_wrapped_onesprovesinstances,tablesandmemoriesreturn the configured values rather than the trait defaults.the_memory_ceiling_refuses_what_it_refused_beforecovers the Done-when directly: the ceiling refuses what it refused, not merely that the gauge moved.Gauge semantics, stated
Both report only after a dispatch, so a module that has never run has no series. That is deliberate and differs from
module_poisonedandmodule_unverified, which are bad-state flags where absence means healthy. These are levels, and a level for a module that has not run is not zero, it is unknown.Memory is observed from store construction, so it includes the instance's initial allocation and a module that never grows past instantiation still reports a non-zero size. That is what the
>= 64 KiBassertion encodes.One series per module per signal, labelled by the operator-written module id and nothing a guest can influence.
Two things for a reviewer
HostState.limitschanged type fromwasmtime::StoreLimitstoObservedLimits, sonexum-runtime-wasm'sstate.rsandlib.rsare in the diff beyond the files the charter named. That is the only place the limiter object can live, since the supervisor depends onnexum-runtime-wasmand not the reverse. One construction site exists workspace-wide.The gauges are also read after a deadline-cancelled dispatch, where the call future was dropped mid-await.
Store::get_fuelandStore::dataare plain accessors and do not re-enter the guest, and the deadline test passes with the emit in place, but it is worth agreeing that sampling a store the trap arm is about to mark dead is what you want.No alert rule was added, because neither ceiling is exported as a series, so an expression would have to hardcode the operator's configured number. The metric rows say so instead.
Testing
just build, then the fulljust testset: 649 passed.just test-e2e: 16 passed.cargo nextest run --workspace --all-features: 952 passed, 0 skipped.just ciend to end, includingcargo deny check, workspace clippy and rustdoc under-D warnings.Cargo.lockuntouched; no new dependencies.AI Assistance
Implementation: claude-opus-5. Red-team review: claude-opus-5. Verification: claude-opus-5. PR description: claude-opus-5.