Skip to content

feat(state): export twin, MTI coverage, TAK output and picture recovery metrics - #2258

Open
c310j wants to merge 2 commits into
Maps-Messaging:developmentfrom
c310j:feat/kpi-metrics-export
Open

c310j wants to merge 2 commits into
Maps-Messaging:developmentfrom
c310j:feat/kpi-metrics-export

Conversation

@c310j

@c310j c310j commented Sep 29, 2026

Copy link
Copy Markdown

Summary

Adds JMX metrics so operational KPIs (MTI coverage, unknown/stale assets, TAK output health, recovery time) can be computed in Prometheus/Grafana. Everything is additive: no behaviour changes, only counters and observers.

  • TwinStatusJMX (type=Integration,name=Twins,twinType=..,lifecycle=.., managed by TwinMetricsManager): twin counts per type × lifecycle, MTI state counts (none/go/mitigate/hold/unknown/other), cumulative MTI-age buckets, report-age buckets (10/30/60/120 s) and a joint "reported recently and MTI-known" count. The joint count is needed because "stale or unknown" can't be derived from separate stale and unknown counts without double-counting.
  • MtiStatusRegistry.snapshot(): a side-effect-free read of an asset's MTI status for the metrics above, plus a StatusListener hook that MtiStatusAdapter fires on status changes.
  • TakOutputJMX / TakOutputStats (name=TakOutput): TAK socket connects, connect failures, disconnects, connected state, last-write age, socket and publisher drop counts, and a receipt-to-CoT latency histogram. TakSocketConnection still doesn't log its failures, but they are now counted.
  • PictureRecoveryTracker / PictureRecoveryJMX (name=PictureRecovery,failureType=..,stage=..): per-asset restoration delay, from valid data arriving back at MAPS to a fresh, classified CoT handed to TAK. Recovery events are opened on feed loss, restart or MTI status loss, and closed in two stages: PICTURE (position back in TAK) and STATUS (MTI status back). Each stage records count, sum, last value and a cumulative histogram, plus a not-restored count for events that time out or are superseded. Hooked into TakTwinObserver and the MTI status listener.
  • MAVLink: last-message age on MavlinkIntegrationJMX, for detecting feed gaps.

Metric definitions (what counts as active, covered or stale) are left to the dashboard on purpose, so they can change without code changes.

Test plan

  • TwinStatusJMXTest, TakOutputStatsTest, PictureRecoveryTrackerTest: bucket boundaries, the joint fresh-and-known count, recovery open/close per failure type and stage, supersede/timeout handling, histogram cumulativity
  • All io.mapsmessaging.state.** tests pass on this branch (858 run, 0 failures)
  • Live-fleet verification: deployed to a 7-node exercise fleet with the JMX Prometheus exporter; all new series present in Prometheus, and restart and feed-loss recoveries recorded with plausible delays (ms for picture, ~2 s for MTI status)

🤖 Generated with Claude Code

c310j and others added 2 commits September 29, 2026 16:03
Adds the raw data behind the requested MTI coverage, unknown-asset,
per-asset staleness and TAK-output KPIs, kept definition-free so the
thresholds can be decided in the dashboard:

- TwinStatusJMX (name=Twins, per twin type and lifecycle): twin counts
  broken down by MTI state and cumulative MTI-age buckets. Reads the MTI
  cache through a new side-effect-free MtiStatusRegistry snapshot, so
  scrapes don't skew the existing lookup hit/miss counters.
- TakOutputJMX (name=TakOutput): TAK socket connects, connect failures,
  disconnects, open connections, last write age, socket and publisher
  drop counts, and a receipt-to-CoT latency histogram. TakSocketConnection
  previously swallowed every connection failure without any trace.
- MAVLink last message age, for feed-gap detection.

NO-ISSUE

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds report-age buckets and joint "reported and MTI-known" counts to
TwinStatusJMX, and a PictureRecoveryTracker that times each asset's
restoration from valid data back at MAPS to a fresh classified CoT handed
to TAK, by failure type (feed loss, restart, MTI status loss) and stage.

NO-ISSUE

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 29, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 8a1eef24-3944-4280-bf4a-3a13da1edb11

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant