feat(supervisor): count fuel-set drops and sample latency on every outcome

[mfw78]

What

nexum_runtime_dispatch_latency_seconds was recorded in the success arm of dispatch_to and nowhere else, so a fault, a trap or a deadline hit contributed no sample. It is now recorded once after the match, for every dispatch that reached the guest, carrying an outcome label: ok, fault, trap or deadline.

The deadline case was previously folded into the trap arm and invisible. with_dispatch_deadline is now bound before the error is converted, and the trap arm returns a new DispatchOutcome::Deadline. Health handling is byte-identical to Trapped, module death, poison verdict and panic record included; only the label differs.

The fuel-set step moved into a refuel helper that keeps the existing error! and increments nexum_runtime_dispatch_dropped_total with reason = "fuel_set_failed", beside the shutdown and rate_limited values it already carried. The two pre-guest drops record no latency, since they never reached the guest, and carry their label as the drop reason instead, which single-sources both strings.

No new metric name, so nexum-runtime-metrics needs no table entry and the name guard stays quiet.

Why

Closes #368

p95 was biased low exactly when things went wrong, which is the reading an operator trusts least and needs most. A FuelSetFailed drop was logged and never counted.

Cardinality, decided

Acceptable. Eleven buckets per (module, outcome) pair over four fixed outcome values, so the series count grows by a constant factor of at most four on one histogram, and the outcome set cannot grow with operator or guest input. Recorded in the metrics table row rather than left implicit.

Constraints confirmed rather than assumed

The exporter's bucket list uses Matcher::Full on the bare metric name, so a label cannot defeat it. That confirmation is executable rather than asserted: the_latency_histogram_renders_bucket_series in crates/nexum-runtime/src/addons.rs now records an outcome label and still asserts the _bucket series at le="5".

NexumDispatchLatency sums by (module, le), so it aggregates the new label away exactly as it already does trigger_kind. No alert change needed.

Testing

Every outcome is driven for real, using the adversarial fixtures rather than asserted in the abstract.

• a_trap_and_a_success_each_record_a_labelled_latency_sample: fuel-bomb on chain 1, example on chain 100.
• a_deadline_hit_records_a_latency_sample_labelled_deadline: slow-host parked an hour past a one-second deadline, clock paused after boot so neither is waited out.
• a_fault_records_a_latency_sample_labelled_fault: max_state_bytes = 0 makes flaky-bomb's first store write return a typed WIT fault, and the module stays alive.
• a_failed_fuel_set_is_counted_as_a_drop: refuel over a Store on a fuel-less engine, which is the only way set_fuel fails, asserting the reason, module and trigger kind.

Gate: 257 tests across the three touched crates, all passing; just build then just test-e2e 16 passed; clippy, rustdoc and doctests clean under -D warnings; no Cargo.lock churn.

The two metrics now agree about one event

The error counter hardcoded error_kind = "trap" in the arm that also handles a dispatch cut off at [limits.dispatch] deadline_secs, while this histogram calls that same event outcome = "deadline". A dashboard joining the two on what it took to be one failure got two answers.

Both now read the same DispatchOutcome variant, so they cannot drift again.

A deadline is a limit doing its job rather than a module bug, so NexumModuleTraps deliberately stops counting one. The alert carries a comment saying so and pointing at error_kind="deadline" for anyone who wants to alert on a module that should never reach its deadline. The metric row in docs/production.md names both values and says they are the same strings the histogram uses.

a_deadline_hit_records_a_latency_sample_labelled_deadline asserts both halves: the counter carries deadline, and no sample carries trap.

The label comes from a derive, not a match

DispatchOutcome derives strum::IntoStaticStr with serialize_all = "snake_case", and one #[strum(serialize = "trap")] override on Trapped. strum is already a workspace dependency this crate inherits, and nexum-runtime-wasm already derives it for the same purpose.

The first revision hand-wrote a match from variant to string. It compiled and read fine, and it gave a variant a second place to be added without its label, which is exactly the drift the paragraph above describes.

The block_on helper is shared, not local

The first revision added a local block_on_current_thread to the dispatch tests. The same runtime-builder incantation was already open-coded eight times across four files, so that would have made nine.

It now lives beside capture_metrics in nexum-runtime-testing, which is what makes it necessary in the first place: the recorder is thread-local, so work driven on another runtime's worker records into a different one and the capture comes back empty. nexum-runtime-testing gains the rt tokio feature it needs.

Five of the eight call sites now use it. The three in event_loop.rs add start_paused(true) and are deliberately left, because #373 is in that file and a sweep there would conflict for no gain. They are the obvious follow-on for whichever of the two lands second.

It keeps the longer name rather than block_on, because the supervisor tests already have a block_on that builds a chain block.

Merge note

#371 touches supervisor/dispatch.rs and the test module too. The two conflict only on adjacent test additions and a re-export list. Landing this one first leaves that one a trivial rebase.

AI Assistance

Implementation: claude-opus-5. Red-team review: claude-opus-5. Verification: claude-opus-5. PR description: claude-opus-5.