diff --git a/doc/telemetry-hardening-report.md b/doc/telemetry-hardening-report.md new file mode 100644 index 00000000..7fb759cc --- /dev/null +++ b/doc/telemetry-hardening-report.md @@ -0,0 +1,320 @@ +# OnAIR Telemetry Ingestion Hardening — Final Report + +Repo root for all commands: `OnAIR-main/`. The root `conftest.py` was used for every +test invocation and every mutmut run (it injects `pytest.gen`, mocks `simdkalman` / +`sbn_client` / `message_headers`); it was never bypassed, modified, or removed. + +Environment note: the host had no `python` launcher, no pip, and no `venv`/`ensurepip`. +A virtual environment was bootstrapped at `OnAIR-main/.venv` (Python 3.12.3, pip +bootstrapped via `get-pip.py`), and `requirements.txt` (superset: run + unittest) plus +`mutmut 2.5.1` and `hypothesis 6.169.1` were installed into it. All `python` / +`mutmut` commands below ran from `.venv` (`PATH="$PWD/.venv/bin:$PATH"`). Because +mutmut executes its runner via `subprocess` **without a shell**, the Pass-2 runner's +leading `MUTMUT_HYPOTHESIS_FAST=1` was supplied by exporting the variable in the +invoking shell (semantically identical: the pytest child process inherits it). + +## 1. Baseline evidence (Step 0) + +``` +python -m pip install -r requirements-unittest.txt +python -m pip install "mutmut>=2.4,<3" hypothesis # mutmut 2.5.1, hypothesis 6.169.1 +python -m pytest test/ -q +→ 371 passed in 4.02s # 100% green before any change (conftest seed printed) +``` + +## 2. Pre-patch mutation proof (Step 1) — the blind-spot catalog + +Runner (both scoped passes): `python -m pytest -x -q +test/onair/data_handling/test_parser_util.py test/plugins/kalman/test_kalman_plugin.py`. + +### 2a. `onair/data_handling/parser_util.py` — 52 mutants, 51 killed, 1 survivor + +| Mutant id | Function | Diff | Blind spot demonstrated | +|---|---|---|---| +| 40 | `floatify_input` | trailing `continue` → `break` after the 0.0-masking fallback | No legacy test feeds ≥2 junk items, so early loop exit (silent truncation of the rest of the frame) is invisible. All other floatify/convert mutants were killed pre-patch — the masking bug itself (`0.0`) is *encoded* by the legacy tests, hence unkillable there. | + +### 2b. `plugins/kalman/kalman_plugin.py` — 55 mutants, 42 killed, 13 survivors + +| Mutant id | Function | Diff | Classification | +|---|---|---|---| +| 22 | `update()` | None-warning message text `XX…XX` | **in scope** — survived pre-patch (legacy test only checked `call_count`) | +| 23 | `update()` | `["FAIL"]` → `["XXFAILXX"]` | **in scope** — survived pre-patch (clrs never asserted) | +| 3, 4 | `__init__` | `window_size < 3` → `<= 3` / `< 4` | out of scope (catalogued) | +| 5 | `__init__` | error-message string | out of scope (catalogued) | +| 15–18 | `__init__` | `process_noise` diag / `observation_model` literals | out of scope; values feed only the conftest-mocked `simdkalman.KalmanFilter` | +| 28 | `render_reasoning` | `>` → `>=` threshold | out of scope (catalogued) | +| 33 | `render_reasoning` | `"TIME"` → `"XXTIMEXX"` | out of scope (catalogued) | +| 36, 38 | `_generate_residuals` | `frames[0]`→`frames[1]`; `>2`→`>3` | **documented-excluded** (`_predict`/`_generate_residuals` run against a conftest `MagicMock` simdkalman; behavior cannot be genuinely asserted) | + +## 3. Pre-patch property falsifying examples (Step 2) + +Files: `test/onair/data_handling/test_parser_util_properties.py`, +`test/plugins/kalman/test_kalman_properties.py`. Both were written **before** the +production patch and executed against the unpatched code +(`python -m pytest … -v --tb=long`): **11 failed, 3 passed**. + +| Property | Result pre-patch | Falsifying example (verbatim from hypothesis) | +|---|---|---| +| P1 never-raises (json scalars) | FAIL | `items=[None]` → `TypeError: float() argument must be a string or a real number, not 'NoneType'` (leaks past `except ValueError`) | +| P2 length preserved (`remove_str=False`) | FAIL | `items=[None]` → same TypeError leak | +| P3 identity on finite floats | **PASS** (already held pre-patch, as predicted) | — | +| P4 `"-"` never 0.0 | FAIL | `items=['-']` → result `[0.0]`, `assert 0.0 != 0.0` (the 0.0-masking bug itself) | +| P5 non-finite spellings (strings) | FAIL | `s='inf'` → `math.isnan(inf)` is False (`inf` passed straight through) | +| P5 non-finite floats | FAIL | `v=inf` → passthrough | +| P6 valid `MM:SS` exact epoch | **PASS** (held pre-patch) | — | +| P6 valid `%Y-%j-%H:%M:%S.%f` exact epoch | **PASS** (held pre-patch) | — | +| P6 malformed → ValueError/nan | FAIL | `s='12:34:56'` → returned `946671154.0` (`:56` silently truncated) instead of ValueError/nan | +| P6 non-string input | FAIL | `v=None` → TypeError leak | +| Kalman short-frame rejection (@given) | FAIL | `num_headers=1, window_size=3, short_by=1` → frame stored / desync (no guard) | +| Kalman long-frame rejection (@given) | FAIL | `num_headers=1, extra=1` → `IndexError` at `kalman_plugin.py:114` (`self.frames[i].append`) | +| Kalman None-frame (@given) | **PASS** (None handling existed pre-patch) | — | +| Stateful machine (`runTest`) | FAIL | (a) `junk_frame(vals=[None,…])` → TypeError at `parser_util.py:55`; (b) `nan_spike(spike=nan)` → I4 violated (non-finite stored in window); (c) `long_frame(extra=1)` → frames mutated | + +## 4. The hardening patches (Step 3) + +Only the three target functions changed; no signatures changed; stdlib `math` only. + +### 4a. `floatify_input` — policy ladder + +``` +try: x = float(i); if math.isfinite(x): append; continue +except (ValueError, TypeError): pass # D1 dead: TypeError can no longer leak +# non-finite float() results (nan/inf strings) fall through to the timestamp +# attempt, which rejects them -> junk policy below +try: x = convert_str_to_timestamp(i); append; continue +except Exception: pass +if remove_str: continue # drop entirely +print_msg(f"floatify_input: could not parse {i!r} to a finite float; substituting math.nan", ["WARNING"]) +floatified.append(math.nan) # no 0.0 anywhere (D2 dead) +``` + +`import math` and `from onair.src.util.print_io import print_msg` were added (no +import cycle: `print_io` imports nothing). Timestamp results are appended without an +isfinite gate (strptime epochs are inherently finite; keeps the legacy mock test at +former lines 478–503 green). Full policy is documented in the docstring. + +### 4b. `convert_str_to_timestamp` — guarded `MM:SS` fallback + +`except Exception` kept around `strptime` (legacy `Fake_Datetime` test raises plain +`Exception`). In the handler: `isinstance(time_str, str)` else `raise ValueError`; +`split(":")` must yield exactly 2 parts; both parts `int()`-parseable; minute and +second each in 0..59; else `ValueError` with a clear, distinct message per failure +mode. `datetime(2000, 1, 1, 1, minute, second, 0).timestamp()` kept (`12:34:56`, +`99:99`, `12:60`, `None` now raise — D4 dead; no IndexError/AttributeError leaks). + +### 4c. `Plugin.update` — length guard + finite-only windows + +`import math` added. New branch: +`elif len(low_level_data) != len(self.headers):` → +`print_msg(f"Kalman plugin update: frame length {len(low_level_data)} does not match +header length {len(self.headers)}; frame rejected.", ["WARNING"]); return` +(no IndexError, no frame/window desync — D5 dead). After `floatify_input`, a +per-attribute `if not math.isfinite(value): continue` keeps only finite floats in +the windows; skipped samples advance no window. + +## 5. Property inventory (locked post-patch) + +`test_parser_util_properties.py` — `json_scalar = +st.one_of(st.none(), st.booleans(), st.integers(), st.floats(allow_nan=True, +allow_infinity=True), st.text(max_size=8))`; sentinels `-`, `nan`, `inf`, `-inf`, +`Infinity`, `""`, `12`, `12:34`, `12:34:56`, `2020-001-12:34:56.789`; +settings constant = `settings(max_examples=1000, deadline=None)` normally, +`settings(max_examples=25, deadline=None, derandomize=True)` when +`MUTMUT_HYPOTHESIS_FAST` is set; autouse fixture records `parser_util.print_msg` +(`raising=False`, pre-patch-safe). + +- **P1** `floatify_input` never raises for any list of json scalars (both `remove_str` modes). +- **P2** `len(out) == len(in)` for `remove_str=False`, all inputs. +- **P3** identity on finite floats. +- **P4** `"-"` never yields `0.0`; post-patch `math.isnan` + `["WARNING"]` asserted + `remove_str=True` drops. +- **P5** `nan`/`inf`/`NaN`/`Infinity` strings and NaN/Inf floats → `math.isnan(out[0])`, never 0.0/crash. +- **P6** valid `MM:SS` (m,s ∈ 0..59) and `%Y-%j-%H:%M:%S.%f` → exact epoch equality vs + `datetime` computed in the test; malformed (`12:34:56`, `99:99`, `-`, `None`, …) → + `ValueError` or `nan`, never IndexError/AttributeError. + +`test_kalman_properties.py` — real `Plugin` instances (conftest-mocked `simdkalman`; +`update()` never touches `kf`; `ai_plugin.ServiceManager` stubbed exactly like the +existing csv_output plugin tests); autouse fixture records `kalman_plugin.print_msg` +(`raising=False`). + +- Stateful `RuleBasedStateMachine` (`@initialize` header count 1..6, `window_size` 3..8): + rules `ValidFrame`, `JunkFrame` (json scalars incl. `None`/`-`/`nan`), `ShortFrame`, + `LongFrame`, `NanSpike` (finite frame + injected NaN/Inf/`nan`/`-`). +- **I1** no rule ever raises; **I2** `len(frames) == len(headers)`; **I3** every window + `<= window_size`; **I4** every stored value finite; **I5** all window lengths equal + while windows are in lockstep and only valid frames arrive (the machine tracks + factual equality per rule: skipping rules desync by design — the spiked attribute's + window advances not — and saturation re-equalizes, at which point the strict + equality assertion re-arms); **growth** — after `>= window_size` consecutive valid + frames every window is exactly `window_size`. +- Extra `@given` properties: short/long frames leave the `frames` snapshot unchanged + and the rejection warning contains both lengths; `None` frame warns exact text with + `["FAIL"]` and stores nothing. + +## 6. Legacy-test change log (Step 3d — only the tabulated edits) + +| Test (original lines) | Change | Rationale | +|---|---|---| +| `test_parser_util.py` 435–452 | none (verified green) | plain-`Exception` float mock still propagates through the new `except (ValueError, TypeError)` gap → `Exception` leaks as before | +| `test_parser_util.py` 455–475 | float mock return `MagicMock()` → `7.5` | new `math.isfinite` gate would treat a mock as unparseable (mock-only variant comment) | +| `test_parser_util.py 478–503` | none (verified green) | timestamp results appended without isfinite gate | +| `test_parser_util.py` 506–531 | `result == [0.0]` → `len==1 and math.isnan(result[0])` | encodes removed 0.0-masking behavior | +| `test_parser_util.py` 534–550 | same → isnan | same | +| `test_parser_util.py` 553–576 | none (verified green) | `remove_str=True` still drops | +| `test_parser_util.py` 579–598 | float mock return → `3.25` | isfinite gate (mock-only variant) | +| `test_parser_util.py` 601–653 | mock float/convert returns → `pytest.gen.uniform(-10,10)` | isfinite gate (mock-only variant) | +| `test_parser_util.py` 656–709 | same + `expected_result.append(0.0)` → `math.nan` | encodes removed 0.0-masking behavior | +| `test_kalman_plugin.py` 152–165 | none (verified green) | None-warning path unchanged | +| `test_kalman_plugin.py` 168–190, 193–222, 225–253 | added `cut.headers = [...]` | new length guard reads `self.headers` (mock-only variant) | + +No assertion was weakened anywhere. Added (additions only) deterministic kill-tests in +both legacy files: exact epochs (`12:34`, `00:00`, `59:59`, `2020-001-12:34:56.789`), +`^`-anchored exact ValueError message asserts, exact `print_msg` text + +`["WARNING"]`/`["FAIL"]` clrs, short/long rejection with unchanged frames, junk frame +stores nothing (3 floatify warnings), `remove_str=True` drops (incl. junk-then-valid +ordering), NaN input stores nothing, nan-spiked frame skips only the spiked +attribute, window saturates to exactly `window_size` with oldest-value eviction. + +## 7. Mutant kill table (Step 4) + +| Scoped pass | Mutants | Killed | Survivors | Survivors in target functions | +|---|---|---|---|---| +| parser_util pre-patch | 52 | 51 | 1 (#40) | 1 — `floatify_input` continue→break | +| kalman pre-patch | 55 | 42 | 13 | 2 — `update()` #22 (message), #23 (clrs) | +| parser_util post **Pass 1** (exact Step-1 runner) ×2 runs | 71 | **71** | **0** | **0** | +| parser_util post **Pass 2** (4-file runner, `MUTMUT_HYPOTHESIS_FAST=1`) ×2 runs | 71 | **71** | **0** | **0** | +| kalman post **Pass 1** (×2 runs) | 60 | 48–50 | 10–12 | **0** — every `update()` mutant dead in every run | +| kalman post **Pass 2** (×2 runs) | 60 | 50–51 | 8–10 (+1 "suspicious") | **0** | + +Post-patch kalman survivors, itemized (all outside `update()`): +`__init__` #5 (error-message string); #15–#18 (kf config literals — conftest-mocked +simdkalman domain); `render_reasoning` #33 (`>`→`>=`), #38 (`"TIME"` literal); +`_generate_residuals` #41–#43 (documented-excluded, conftest MagicMock). Pass-1 also +survives #3/#4 (`__init__` window boundary — the two-file runner never constructs a +real `Plugin`; the Pass-2 property machine does construct real Plugins and kills both), +and #42/#43 fluctuate with the `--conftest-seed` (legacy `_generate_residuals` tests +draw random frame lengths). These fluctuations are confined to excluded/out-of-scope +functions; the in-scope survivor set was empty in **all four** post-patch kalman runs +and both post-patch parser runs (reproducible where the DoD demands it). + +Mutant #24 (`update()` guard `!=` → `==`), flagged "suspicious" (slow-kill) in one +Pass-2 run, was proven deterministically killed out-of-band: `mutmut apply 24` → +legacy runner exit code 1 (`test_kill_update_valid_frame_stored_in_lockstep_windows` +fails) → source restored. In the repeat Pass-2 run it was killed outright. + +Equivalent-mutant check: none of the zero-survivor target functions required an +"equivalent mutant" exemption; every in-scope mutant is killed by a deterministic +example test (no kill depends on hypothesis sampling — the Pass-1 runner, which +excludes the property files, achieves zero on its own). + +## 8. Full verification (Step 5) + +1. `python -m pytest test/ -q` → **401 passed** in 160.53s (371 baseline + 30 new: + 10 parser properties + 4 kalman properties/machine + 8 + 8 deterministic + kill-tests). Zero failures, zero errors, zero skips. Full stdout read. +2. `python -m pytest test/onair/data_handling/test_parser_util_properties.py + test/plugins/kalman/test_kalman_properties.py -v` (no env var → 1000 examples) → + **14 passed** in 147.53s. Full stdout read. +3. `mutmut results` for every scoped pass recorded in §7. + +## 9. Honest limits + +- **mutmut operator set**: mutmut 2.x mutates strings, numbers, comparison operators, + `continue`/`break`, and indices; it does not remove statements wholesale or mutate + every logical fault (e.g. swapped independent arguments of the same type). Zero + survivors means "zero survivors under this operator set", not absolute correctness. +- **Excluded domain**: `_predict` / `_generate_residuals` execute against a + conftest-injected `MagicMock` simdkalman, so their mutants (#41–#43 here) are + documented-excluded rather than force-killed; likewise the kf construction kwargs + (#15–#18) only ever reach a mock. +- **Seed-dependent out-of-scope kills**: legacy `_generate_residuals` tests draw + `pytest.gen` random frame lengths (seeded from wall clock), so #42/#43 flip between + killed/survived across runs; #3/#4 are killed only by Pass 2 (property machine + constructs real `Plugin`s). All fluctuation is outside the three target functions. +- **Hypothesis sampling**: properties are randomized (derandomized only in the + 25-example mutmut fast mode); 1000-example runs are strong evidence, not proof, of + property validity. No in-scope mutant kill depends on hypothesis — the deterministic + kill-tests alone reach zero, and Pass 2 verifies the properties are themselves + killers of every reachable in-scope mutant. +- **I5 semantics**: unconditional window-length equality is false by design after a + partial-skip frame (that is the finite-only-windows policy); I5 therefore asserts + equality while windows are factually in lockstep (re-armed after every rule from + actual state), and the growth invariant pins the re-synchronization point. + +## 10. Out-of-scope guarantee + +`telemetry_test_suite.py`, `redis_adapter.py`, `sbn_adapter.py`, `tlm_json_parser.py`, +`agent.py`, `vehicle_rep.py`, planners/complex-reasoning interfaces, and root +`conftest.py` were not touched. Changes were confined to +`onair/data_handling/parser_util.py`, `plugins/kalman/kalman_plugin.py`, and the two +legacy test files (tabulated updates + additive kill-tests), plus the two new property +files. + +## 11. Post-hardening refinement (external review round) + +An external review of the patched `convert_str_to_timestamp` raised three +refinements. Disposition of each: + +### 11.1 ACCEPTED — type contract hoisted to the function boundary + +The `isinstance(time_str, str)` guard moved out of the `except` handler to the +top of the function, so the input contract ("must be a `str`") is enforced +before any parsing path runs, rather than relying on `strptime` failing first. +Observable behavior for all tested inputs is unchanged (non-strings still get +the same `ValueError`, now raised earlier). + +### 11.2 ACCEPTED — `strptime` handler narrowed from `except Exception` to `except ValueError` + +Only the expected format-mismatch error falls through to the `MM:SS` fallback; +any other unexpected error (the review's hypothetical `.timestamp()` platform +failure) now propagates loudly instead of being masked as a format retry. +This required one legacy mock-fidelity fix: the `Fake_Datetime.strptime` in +`test_parser_util.py` raised plain `Exception` where the real +`datetime.strptime` raises `ValueError`; the fake now matches the real +dependency's contract (annotated as a mock-only update). The "both raise" +legacy test was left untouched — under the narrowed handler its plain +`Exception` propagates, which is precisely the new fail-loud semantics and +satisfies its `pytest.raises(Exception)` assertion. +`floatify_input`'s broad `except Exception` around its +`convert_str_to_timestamp` call was intentionally KEPT — that call site is the +pipeline's input-normalization boundary and must convert *any* parser failure +into the nan/drop policy (P1 never-raises). + +### 11.3 REJECTED (documented instead) — timezone semantics of the `MM:SS` epoch + +The review correctly notes `datetime(2000, 1, 1, 1, M, S).timestamp()` on a +naive datetime interprets LOCAL time, so the epoch varies by host timezone. +This is **pre-existing upstream behavior** (the original implementation called +`.timestamp()` identically); changing it (e.g. to UTC) would silently shift +every produced epoch, altering residuals pipeline-wide — outside this +hardening's no-redesign mandate. Documented in the function docstring +instead. Values remain self-consistent within any single machine, so +residuals and relative comparisons are unaffected. Flagged as a candidate +decision for upstream maintainers. + +### 11.4 Review-driven test additions + +- `"1:2"` pinned as ACCEPTED (== epoch of 1m 2s), matching `int()` semantics + per the mandate ("both parse as int via try, minute/second in 0..59") — no + two-digit requirement exists in the spec; decision now documented and + regression-locked. +- `"12:-1"`, `"-1:30"` (negative components → range check) with anchored + message asserts; `"12:ab"` (non-integer seconds → int-parse assert); + `"1:2:3:4"` (4-part → part-count guard). +- `"2024-032-14:30:45.123456"` exact-epoch assert (post-2020 full format, + leap-year day-of-year). +- Cases already covered from the original round: `"12:60"`, `"60:00"`, + `"99:99"`, `"aa:bb"`, `"-"`, `""`, `"12:"`, `":34"`, non-strings + (`None`, `5`, `5.0`, `True`, `[]`, `{}`). + +### 11.5 Re-verification after refinement + +| Check | Result | +|---|---| +| Full suite `pytest test/ -q` | **401 passed**, zero failures/errors/skips | +| Property files (dedicated run, `hypothesis profile 'default'` = 1000 examples) | **14/14 passed** (P1–P6, machine, guard properties) | +| mutmut Pass 1 (exact Step-1 runner, 2 legacy files) | **71/71 killed, 0 survived, 0 suspicious** | +| mutmut Pass 2 (4-file runner, `MUTMUT_HYPOTHESIS_FAST=1` derandomized cap) | **0 survivors, 0 suspicious** (`mutmut results` empty) | + +`plugins/kalman/kalman_plugin.py` was not modified in this round; its prior +zero-survivor result for `Plugin.update()` stands. diff --git a/onair/data_handling/parser_util.py b/onair/data_handling/parser_util.py index 620b34e7..26ded972 100644 --- a/onair/data_handling/parser_util.py +++ b/onair/data_handling/parser_util.py @@ -8,7 +8,9 @@ # See "NOSA GSC-19165-1 OnAIR.pdf" from .tlm_json_parser import parseTlmConfJson, str2lst +from onair.src.util.print_io import print_msg import datetime +import math def extract_meta_data_handle_ss_breakdown(meta_data_file, ss_breakdown): @@ -47,30 +49,107 @@ def extract_meta_data(meta_data_file): def floatify_input(_input, remove_str=False): + """Convert each item of ``_input`` to a finite float, preserving length. + + Parsing policy (hardened): + + - Values that ``float()`` parses to a FINITE float are appended as-is + (finite floats pass through unchanged). + - Values that ``convert_str_to_timestamp`` parses (a ``%Y-%j-%H:%M:%S.%f`` + telemetry timestamp, or a strict ``MM:SS`` pair with minute and second + each in 0..59) are appended as their epoch float. + - Everything else -- ``"-"``, ``None``, junk text, and non-finite + spellings like ``"nan"``/``"inf"``/``"Infinity"`` (whose ``float()`` + result is not finite) -- is either dropped entirely when ``remove_str`` + is True, or substituted with ``math.nan`` plus a WARNING via + ``print_msg`` when ``remove_str`` is False. A ``0.0`` fallback is + never used: masking junk as a real measurement value hides telemetry + faults. + """ floatified = [] for i in _input: try: x = float(i) - floatified.append(x) - except ValueError: - try: - x = convert_str_to_timestamp(i) + if math.isfinite(x): floatified.append(x) - except: - if remove_str == False: - floatified.append(0.0) - else: - continue continue + except (ValueError, TypeError): + pass + # non-finite float() results (nan/inf strings and floats) fall through + # to the timestamp attempt, which rejects them -> junk policy below + try: + x = convert_str_to_timestamp(i) + floatified.append(x) + continue + except Exception: + pass + if remove_str: + continue + print_msg( + f"floatify_input: could not parse {i!r} to a finite float; substituting math.nan", + ["WARNING"], + ) + floatified.append(math.nan) return floatified def convert_str_to_timestamp(time_str): + """Convert a telemetry time string to an epoch float. + + Contract: ``time_str`` must be a ``str``; any other type raises + ``ValueError`` immediately at the function boundary. + + Accepted formats: + + - ``%Y-%j-%H:%M:%S.%f`` (e.g. ``"2020-001-12:34:56.789"``), parsed with + ``datetime.strptime``; or + - a strict ``MM:SS`` pair (e.g. ``"12:34"``) with exactly two colon + separated parts, both integer-parseable via ``int()``, minute and + second each in 0..59 (single-digit components such as ``"1:2"`` are + accepted, matching ``int()`` semantics). The date is fixed to 1am + on Jan 1st, 2000. + + Anything else -- non-strings (``None``, numbers, objects), ``"12:34:56"``, + ``"99:99"``, ``"-"``, empty strings -- raises ``ValueError`` instead of + silently truncating extra parts or leaking ``IndexError`` / + ``AttributeError``. + + Only ``ValueError`` from ``strptime`` falls through to the ``MM:SS`` + fallback; any other unexpected error propagates loudly instead of + being masked as a format retry. + + Note: the ``MM:SS`` branch builds a naive ``datetime`` and calls + ``.timestamp()``, which interprets it in the host's LOCAL timezone. + This is the pre-existing upstream semantics (identical to the original + implementation) and is intentionally preserved here: values are + consistent within any single machine, so residuals and relative + comparisons are unaffected. + """ + if not isinstance(time_str, str): + raise ValueError( + f"convert_str_to_timestamp: expected a time string, got {type(time_str).__name__}: {time_str!r}" + ) try: t = datetime.datetime.strptime(time_str, "%Y-%j-%H:%M:%S.%f") return t.timestamp() - except: - min_sec = time_str.split(":") - # Use 1 am on Jan 1st, 2000 as the date if only minutes and seconds are specified - t = datetime.datetime(2000, 1, 1, 1, int(min_sec[0]), int(min_sec[1]), 0) - return t.timestamp() + except ValueError: + pass + min_sec = time_str.split(":") + if len(min_sec) != 2: + raise ValueError( + f"convert_str_to_timestamp: expected MM:SS, got {time_str!r}" + ) + try: + minute = int(min_sec[0]) + second = int(min_sec[1]) + except ValueError: + raise ValueError( + f"convert_str_to_timestamp: expected MM:SS with integer minute and second, got {time_str!r}" + ) + if not (0 <= minute <= 59 and 0 <= second <= 59): + raise ValueError( + f"convert_str_to_timestamp: minute and second must be in 0..59, got {time_str!r}" + ) + # Use 1 am on Jan 1st, 2000 as the date if only minutes and seconds are specified + t = datetime.datetime(2000, 1, 1, 1, minute, second, 0) + return t.timestamp() diff --git a/plugins/kalman/kalman_plugin.py b/plugins/kalman/kalman_plugin.py index 6350a623..5faa92e6 100644 --- a/plugins/kalman/kalman_plugin.py +++ b/plugins/kalman/kalman_plugin.py @@ -16,6 +16,8 @@ The module is part of NASA's On-Board Artificial Intelligence Research (OnAIR) Platform. """ +import math + import simdkalman import numpy as np from onair.data_handling.parser_util import floatify_input @@ -94,6 +96,16 @@ def update(self, low_level_data=None, _high_level_data=None): corresponding frames, and maintains the window size by removing older data points if necessary. + Hardened ingestion policy: + + - ``low_level_data`` of ``None`` triggers a FAIL message and no store. + - Frames whose length does not match the header length are REJECTED + with a WARNING (both lengths reported) and no data is stored; this + prevents frame/window desynchronization. + - After floatification, non-finite values (``math.nan`` substituted by + ``floatify_input`` for ``"-"``/junk, or genuine NaN/Inf readings) + are skipped, so windows only ever contain finite floats. + Parameters ---------- low_level_data : list @@ -107,10 +119,18 @@ def update(self, low_level_data=None, _high_level_data=None): print_msg( "Kalman plugin requires low_level_data but received None.", ["FAIL"] ) + elif len(low_level_data) != len(self.headers): + print_msg( + f"Kalman plugin update: frame length {len(low_level_data)} does not match header length {len(self.headers)}; frame rejected.", + ["WARNING"], + ) + return else: frame = floatify_input(low_level_data) for i, value in enumerate(frame): + if not math.isfinite(value): + continue # windows hold only finite floats self.frames[i].append(value) if len(self.frames[i]) > self.window_size: self.frames[i].pop(0) diff --git a/test/onair/data_handling/test_parser_util.py b/test/onair/data_handling/test_parser_util.py index a700cc2b..a9dcb3b5 100644 --- a/test/onair/data_handling/test_parser_util.py +++ b/test/onair/data_handling/test_parser_util.py @@ -9,6 +9,9 @@ """Test Parser Util Functionality""" +import datetime +import math + import pytest from unittest.mock import MagicMock @@ -462,7 +465,9 @@ def test_parser_util_flotify_input_returns_list_of_size_one_that_contains_the_ca fake_item = str(MagicMock()) arg__input.append(fake_item) # list of single str, one iteration - expected_result = MagicMock() + # updated: mock-only variant -- mock returns a finite float because + # floatify_input now gates appends on math.isfinite (see policy in floatify_input) + expected_result = 7.5 mocker.patch("builtins.float", return_value=expected_result) @@ -528,7 +533,9 @@ def test_parser_util_flotify_input_returns_list_of_size_one_that_contains_0_dot_ assert parser_util.convert_str_to_timestamp.call_args_list[0].args == ( arg__input[0], ) - assert result == [0.0] + # updated: encodes removed 0.0-masking behavior, see policy in floatify_input + assert len(result) == 1 + assert math.isnan(result[0]) def test_parser_util_flotify_input_default_arg_remove_str_is_False(mocker): @@ -547,7 +554,9 @@ def test_parser_util_flotify_input_default_arg_remove_str_is_False(mocker): result = parser_util.floatify_input(arg__input) # Assert - assert result == [0.0] # shows flow was correct for remove_str being False + # updated: encodes removed 0.0-masking behavior, see policy in floatify_input + assert len(result) == 1 + assert math.isnan(result[0]) # shows flow was correct for remove_str being False def test_parser_util_flotify_input_returns_empty_list_when_two_Exceptions_are_thrown_and_remove_str_is_True( @@ -585,7 +594,9 @@ def test_parser_util_flotify_input_returns_call_to_float_that_was_given___input_ fake_item = MagicMock() arg__input.append(fake_item) # list of one item, one iteration - expected_result = MagicMock() + # updated: mock-only variant -- mock returns a finite float because + # floatify_input now gates appends on math.isfinite (see policy in floatify_input) + expected_result = 3.25 mocker.patch(parser_util.__name__ + ".float", return_value=expected_result) @@ -616,27 +627,29 @@ def test_parser_util_flotify_input_returns_expected_values_for_given__input_that ["str", "str_need_replace", "str_fail_replace", "other"], 1 )[0] + # updated: mock-only variant -- mocked float/convert results are + # finite floats because floatify_input now gates appends on + # math.isfinite (see policy in floatify_input) if rand_type_of_item == "str": arg__input.append(MagicMock()) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(resultant_float) expected_result.append(resultant_float) elif rand_type_of_item == "str_need_replace": fake_input = MagicMock() arg__input.append(fake_input) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(ValueError) side_effects_for_convert_str.append(resultant_float) expected_result.append(resultant_float) elif rand_type_of_item == "str_fail_replace": fake_input = MagicMock() arg__input.append(fake_input) - resultant_float = MagicMock() side_effects_for_float.append(ValueError) side_effects_for_convert_str.append(Exception) else: arg__input.append(MagicMock()) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(resultant_float) expected_result.append(resultant_float) @@ -671,28 +684,31 @@ def test_parser_util_flotify_input_returns_expected_values_for_given__input_that ["str", "str_need_replace", "str_fail_replace", "other"], 1 )[0] + # updated: mock-only variant -- mocked float/convert results are + # finite floats because floatify_input now gates appends on + # math.isfinite (see policy in floatify_input) if rand_type_of_item == "str": arg__input.append(MagicMock()) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(resultant_float) expected_result.append(resultant_float) elif rand_type_of_item == "str_need_replace": fake_input = MagicMock() arg__input.append(fake_input) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(ValueError) side_effects_for_convert_str.append(resultant_float) expected_result.append(resultant_float) elif rand_type_of_item == "str_fail_replace": fake_input = MagicMock() arg__input.append(fake_input) - resultant_float = MagicMock() side_effects_for_float.append(ValueError) side_effects_for_convert_str.append(Exception) - expected_result.append(0.0) + # updated: encodes removed 0.0-masking behavior, see policy in floatify_input + expected_result.append(math.nan) else: # other arg__input.append(MagicMock()) - resultant_float = MagicMock() + resultant_float = pytest.gen.uniform(-10, 10) side_effects_for_float.append(resultant_float) expected_result.append(resultant_float) @@ -760,7 +776,11 @@ def __init__(self, year, month, day, hour, minute, second, subsecond): assert subsecond == 0 def strptime(arg1, arg2): - raise Exception + # updated: encodes removed mock-only behavior; real + # datetime.strptime raises ValueError on format mismatch, so the + # fake must match the real dependency's contract for the narrowed + # handler (see policy in convert_str_to_timestamp) + raise ValueError def timestamp(self): Fake_Datetime.timestamp_call_count = self.timestamp_call_count + 1 @@ -807,3 +827,149 @@ def timestamp(): # Assert assert e_info.match("") + + +# --------------------------------------------------------------------------- +# Deterministic kill-tests for the hardened floatify_input / +# convert_str_to_timestamp policy (see the policy docstring on floatify_input). +# These example-based asserts exist so the scoped mutation-testing runner -- +# which does NOT include the hypothesis property files -- reaches zero +# survivors in the two target functions. +# --------------------------------------------------------------------------- +def test_kill_convert_str_to_timestamp_exact_epochs_for_boundaries_and_doy(): + # exact-epoch asserts: kill datetime-literal, index-swap and 0/59 boundary mutants + assert parser_util.convert_str_to_timestamp("12:34") == datetime.datetime( + 2000, 1, 1, 1, 12, 34, 0 + ).timestamp() + assert parser_util.convert_str_to_timestamp("00:00") == datetime.datetime( + 2000, 1, 1, 1, 0, 0, 0 + ).timestamp() + assert parser_util.convert_str_to_timestamp("59:59") == datetime.datetime( + 2000, 1, 1, 1, 59, 59, 0 + ).timestamp() + assert parser_util.convert_str_to_timestamp( + "2020-001-12:34:56.789" + ) == datetime.datetime(2020, 1, 1, 12, 34, 56, 789000).timestamp() + # updated: pin single-digit MM:SS acceptance ("1:2" == 1m 2s), matching + # int() semantics documented in convert_str_to_timestamp + assert parser_util.convert_str_to_timestamp("1:2") == datetime.datetime( + 2000, 1, 1, 1, 1, 2, 0 + ).timestamp() + # updated: post-2020 full-format timestamp (2024 is a leap year; doy 032 = Feb 1) + assert parser_util.convert_str_to_timestamp( + "2024-032-14:30:45.123456" + ) == datetime.datetime(2024, 2, 1, 14, 30, 45, 123456).timestamp() + + +def test_kill_convert_str_to_timestamp_malformed_inputs_raise_ValueError(): + # updated: extended per external review -- negative components ("12:-1", + # "-1:30") are caught by the 0..59 range check, non-integer seconds + # ("12:ab") by the int() parse, and 4-part strings by the part-count guard + for bad in [ + "12:34:56", + "1:2:3", + "99:99", + "12:60", + "60:00", + "-", + "", + "12:", + ":34", + "12:-1", + "-1:30", + "12:ab", + "1:2:3:4", + ]: + # ^-anchored: the message must START with the production prefix, which + # kills any string mutation of the message text + with pytest.raises(ValueError, match=r"^convert_str_to_timestamp: "): + parser_util.convert_str_to_timestamp(bad) + with pytest.raises( + ValueError, match=r"^convert_str_to_timestamp: expected MM:SS, got '12:34:56'$" + ): + parser_util.convert_str_to_timestamp("12:34:56") + with pytest.raises( + ValueError, + match=r"^convert_str_to_timestamp: minute and second must be in 0\.\.59, got '99:99'$", + ): + parser_util.convert_str_to_timestamp("99:99") + with pytest.raises( + ValueError, + match=r"^convert_str_to_timestamp: minute and second must be in 0\.\.59, got '12:60'$", + ): + parser_util.convert_str_to_timestamp("12:60") + with pytest.raises( + ValueError, + match=r"^convert_str_to_timestamp: expected MM:SS with integer minute and second, got 'aa:bb'$", + ): + parser_util.convert_str_to_timestamp("aa:bb") + # updated: anchored asserts for review-driven cases -- negative second + with pytest.raises( + ValueError, + match=r"^convert_str_to_timestamp: minute and second must be in 0\.\.59, got '12:-1'$", + ): + parser_util.convert_str_to_timestamp("12:-1") + # updated: non-integer second component + with pytest.raises( + ValueError, + match=r"^convert_str_to_timestamp: expected MM:SS with integer minute and second, got '12:ab'$", + ): + parser_util.convert_str_to_timestamp("12:ab") + + +def test_kill_convert_str_to_timestamp_non_string_raises_ValueError_not_AttributeError(): + for non_str in [None, 5, 5.0, True, [], {}]: + with pytest.raises( + ValueError, match=r"^convert_str_to_timestamp: expected a time string, got" + ): + parser_util.convert_str_to_timestamp(non_str) + + +def test_kill_floatify_input_finite_values_pass_through_identically(mocker): + mocker.patch(parser_util.__name__ + ".print_msg") + out = parser_util.floatify_input([12.5, -0.0, 1e308, 0, "2.75"]) + assert out == [12.5, -0.0, 1e308, 0.0, 2.75] + + +def test_kill_floatify_input_mixed_frame_keeps_order_length_and_exact_values(mocker): + mocker.patch(parser_util.__name__ + ".print_msg") + out = parser_util.floatify_input(["1.5", "12:34", 2, "-", "nan", None]) + assert len(out) == 6 # length + ordering: kills continue->break mutants + assert out[0] == 1.5 + assert out[1] == datetime.datetime(2000, 1, 1, 1, 12, 34, 0).timestamp() + assert out[2] == 2.0 + assert math.isnan(out[3]) + assert math.isnan(out[4]) + assert math.isnan(out[5]) + + +def test_kill_floatify_input_junk_substitutes_nan_with_exact_warning(mocker): + mock_print = mocker.patch(parser_util.__name__ + ".print_msg") + out = parser_util.floatify_input(["-", "-"]) + assert len(out) == 2 + assert math.isnan(out[0]) and math.isnan(out[1]) + assert mock_print.call_count == 2 + for call in mock_print.call_args_list: + assert call.args == ( + "floatify_input: could not parse '-' to a finite float; substituting math.nan", + ["WARNING"], + ) + + +def test_kill_floatify_input_remove_str_true_drops_junk_without_warning(mocker): + mock_print = mocker.patch(parser_util.__name__ + ".print_msg") + assert parser_util.floatify_input(["-", "abc", None], remove_str=True) == [] + # junk followed by a valid value: the loop must CONTINUE past junk, not break + assert parser_util.floatify_input(["-", "abc", 2.5], remove_str=True) == [2.5] + assert mock_print.call_count == 0 # dropped silently; no substitution warning + + +def test_kill_floatify_input_nonfinite_spellings_become_nan(mocker): + mocker.patch(parser_util.__name__ + ".print_msg") + for s in ["inf", "-inf", "Infinity", "-Infinity", "nan", "NaN", "1e309", "INF"]: + out = parser_util.floatify_input([s]) + assert len(out) == 1 + assert math.isnan(out[0]), s + for v in [float("nan"), float("inf"), float("-inf")]: + out = parser_util.floatify_input([v]) + assert math.isnan(out[0]) diff --git a/test/onair/data_handling/test_parser_util_properties.py b/test/onair/data_handling/test_parser_util_properties.py new file mode 100644 index 00000000..1bc1b044 --- /dev/null +++ b/test/onair/data_handling/test_parser_util_properties.py @@ -0,0 +1,230 @@ +# GSC-19165-1, "The On-Board Artificial Intelligence Research (OnAIR) Platform" +# +# Copyright © 2023 United States Government as represented by the Administrator of +# the National Aeronautics and Space Administration. No copyright is claimed in +# the United States under Title 17, U.S. Code. All Other Rights Reserved. +# +# Licensed under the NASA Open Source Agreement version 1.3 +# See "NOSA GSC-19165-1 OnAIR.pdf" + +"""Property-based tests for parser_util.floatify_input / convert_str_to_timestamp. + +Properties P1-P6 lock the hardened parsing policy: + finite-parseable -> float, timestamps -> epoch float, + "-" / None / junk / non-finite -> math.nan + WARNING (or dropped when remove_str=True). + +These tests are ALSO runnable pre-patch (the print_msg recorder is installed with +raising=False), in which case the falsifying examples document the pre-patch +blind spots (0.0-masking, TypeError leaks, silent ":56" truncation, inf passthrough). +""" + +import datetime +import math +import os + +import pytest +from hypothesis import given, settings, strategies as st + +import onair.data_handling.parser_util as parser_util + + +# --------------------------------------------------------------------------- +# Hypothesis settings: full 1000-example runs normally; a derandomized 25-example +# cap when MUTMUT_HYPOTHESIS_FAST is set (used by the mutation-testing runner). +# --------------------------------------------------------------------------- +if "MUTMUT_HYPOTHESIS_FAST" in os.environ: + HYPO_SETTINGS = settings(max_examples=25, deadline=None, derandomize=True) +else: + HYPO_SETTINGS = settings(max_examples=1000, deadline=None) + +# Any scalar a JSON telemetry field could carry. +json_scalar = st.one_of( + st.none(), + st.booleans(), + st.integers(), + st.floats(allow_nan=True, allow_infinity=True), + st.text(max_size=8), +) + +# Sentinel strings the telemetry pipeline is known to emit. +SENTINEL_STRINGS = [ + "-", + "nan", + "inf", + "-inf", + "Infinity", + "", + "12", + "12:34", + "12:34:56", + "2020-001-12:34:56.789", +] + +finite_float = st.floats(allow_nan=False, allow_infinity=False) + +# Recorded print_msg traffic: list of (msg, clrs) tuples. +print_msg_calls = [] + + +def _record_print_msg(msg, clrs=None): + print_msg_calls.append((msg, clrs)) + + +@pytest.fixture(autouse=True) +def _silence_and_record_print_msg(monkeypatch): + """Silence AND record parser_util.print_msg. + + raising=False keeps this file runnable pre-patch (the module gains a + print_msg attribute only after the hardening patch) and lets P4 assert + the WARNING post-patch. + """ + print_msg_calls.clear() + monkeypatch.setattr(parser_util, "print_msg", _record_print_msg, raising=False) + yield + print_msg_calls.clear() + + +# --------------------------------------------------------------------------- +# P1: floatify_input never raises, whatever the list of JSON scalars contains. +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given(st.lists(json_scalar, max_size=20)) +def test_P1_floatify_input_never_raises_on_json_scalars(items): + parser_util.floatify_input(items) # must not raise + parser_util.floatify_input(items, remove_str=True) # must not raise either + + +# --------------------------------------------------------------------------- +# P2: with remove_str=False the output length is preserved for ALL inputs. +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given(st.lists(json_scalar, max_size=20)) +def test_P2_len_preserved_when_remove_str_is_False(items): + out = parser_util.floatify_input(items, remove_str=False) + assert len(out) == len(items) + + +# --------------------------------------------------------------------------- +# P3: identity on finite floats (holds pre-patch as well). +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given(st.lists(finite_float, max_size=20)) +def test_P3_identity_on_finite_floats(items): + out = parser_util.floatify_input(items) + assert out == items + + +# --------------------------------------------------------------------------- +# P4: "-" must never degrade to 0.0. Post-patch: math.nan + WARNING, and +# remove_str=True drops it entirely. +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given(st.lists(st.sampled_from(["-"]), min_size=1, max_size=5)) +def test_P4_dash_is_never_zero_but_nan_with_warning(items): + print_msg_calls.clear() + out = parser_util.floatify_input(items, remove_str=False) + assert len(out) == len(items) + for value in out: + assert value != 0.0 # the 0.0-masking failure mode is forbidden + assert math.isnan(value) + dropped = parser_util.floatify_input(items, remove_str=True) + assert dropped == [] + assert any(clrs == ["WARNING"] for _, clrs in print_msg_calls) + + +# --------------------------------------------------------------------------- +# P5: non-finite spellings (strings and floats) become math.nan, never 0.0, +# never a crash. +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given( + st.sampled_from( + ["nan", "inf", "-inf", "NaN", "Infinity", "-Infinity", "INF", "-INF"] + ) +) +def test_P5_nonfinite_strings_become_nan(s): + out = parser_util.floatify_input([s]) + assert len(out) == 1 + assert math.isnan(out[0]) + assert out[0] != 0.0 + + +@HYPO_SETTINGS +@given(st.sampled_from([float("nan"), float("inf"), float("-inf")])) +def test_P5_nonfinite_floats_become_nan(v): + out = parser_util.floatify_input([v]) + assert len(out) == 1 + assert math.isnan(out[0]) + + +# --------------------------------------------------------------------------- +# P6: valid MM:SS and %Y-%j-%H:%M:%S.%f timestamps convert to the EXACT epoch +# float computed independently via datetime; malformed inputs raise ValueError +# or become nan -- never IndexError / AttributeError. +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given(st.integers(min_value=0, max_value=59), st.integers(min_value=0, max_value=59)) +def test_P6_valid_mm_ss_converts_to_exact_epoch(minute, second): + out = parser_util.floatify_input([f"{minute:02d}:{second:02d}"]) + expected = datetime.datetime(2000, 1, 1, 1, minute, second, 0).timestamp() + assert out == [expected] + + +@HYPO_SETTINGS +@given( + st.integers(min_value=1900, max_value=2100), + st.integers(min_value=1, max_value=365), + st.integers(min_value=0, max_value=23), + st.integers(min_value=0, max_value=59), + st.integers(min_value=0, max_value=59), + st.integers(min_value=0, max_value=999999), +) +def test_P6_valid_doy_timestamp_converts_to_exact_epoch( + year, doy, hour, minute, second, micro +): + time_str = ( + f"{year:04d}-{doy:03d}-{hour:02d}:" + f"{minute:02d}:{second:02d}.{micro:06d}" + ) + out = parser_util.floatify_input([time_str]) + expected = ( + datetime.datetime(year, 1, 1, hour, minute, second, micro) + + datetime.timedelta(days=doy - 1) + ).timestamp() + assert out == [expected] + + +@HYPO_SETTINGS +@given( + st.sampled_from( + [ + "12:34:56", + "99:99", + "-", + "", + "12:", + ":34", + "1:2:3", + "aa:bb", + "12:34:56.789", + ] + ) +) +def test_P6_malformed_timestamps_raise_ValueError_or_become_nan(s): + try: + out = parser_util.floatify_input([s], remove_str=False) + except ValueError: + return # acceptable outcome for malformed input + assert len(out) == 1 + assert math.isnan(out[0]) # never a silently-truncated finite value + + +@HYPO_SETTINGS +@given(st.sampled_from([None, 5, 5.0, True])) +def test_P6_non_string_to_convert_str_to_timestamp_raises_ValueError_only(v): + try: + out = parser_util.floatify_input([v], remove_str=False) + except ValueError: + return + assert len(out) == 1 + assert math.isnan(out[0]) or math.isfinite(out[0]) diff --git a/test/plugins/kalman/test_kalman_plugin.py b/test/plugins/kalman/test_kalman_plugin.py index a77a86e9..49f8855d 100644 --- a/test/plugins/kalman/test_kalman_plugin.py +++ b/test/plugins/kalman/test_kalman_plugin.py @@ -169,6 +169,9 @@ def test_Kalman_update_with_initially_empty_frames(mocker): # Arrange cut = Kalman.__new__(Kalman) num_fake_data_points = pytest.gen.randint(1, 10) + # updated: mock-only variant -- arranges cut.headers for the frame-length + # guard added to Plugin.update (see policy in floatify_input) + cut.headers = [f"header{i}" for i in range(num_fake_data_points)] cut.frames = [[] for _ in range(num_fake_data_points)] cut.window_size = 1 # smallest value without popping starting at none @@ -201,6 +204,10 @@ def test_Kalman_update_with_existing_data_in_frames_but_less_than_full_window_si existing_data_points + 1, existing_data_points + 5 ) # arbitrary 1-5 extra points left in sliding window + # updated: mock-only variant -- arranges cut.headers for the frame-length + # guard added to Plugin.update (see policy in floatify_input) + cut.headers = [f"header{i}" for i in range(num_headers)] + cut.frames = [ [pytest.gen.uniform(-10, 10) for _ in range(existing_data_points)] for _ in range(num_headers) @@ -229,6 +236,10 @@ def test_Kalman_update_with_full_window_size(mocker): cut.window_size = 5 existing_data_points = cut.window_size # Fill the window + # updated: mock-only variant -- arranges cut.headers for the frame-length + # guard added to Plugin.update (see policy in floatify_input) + cut.headers = [f"header{i}" for i in range(num_headers)] + original_frames = [ [pytest.gen.uniform(-10, 10) for _ in range(existing_data_points)] for _ in range(num_headers) @@ -444,3 +455,145 @@ def test_Kalman__generate_residuals_with_sufficient_frame_data_length(mocker): cut._predict.call_args[0][2], expected_initial_value ) np.testing.assert_array_almost_equal(result, expected_residuals) + + +# --------------------------------------------------------------------------- +# Deterministic kill-tests for the hardened Plugin.update ingestion policy +# (length guard + finite-only windows; see the Plugin.update docstring and the +# policy in floatify_input). These example-based asserts exist so the scoped +# mutation-testing runner -- which does NOT include the hypothesis property +# files -- reaches zero survivors in update(). +# --------------------------------------------------------------------------- +def _make_cut_for_update_kill_tests(num_headers, window_size, existing=0): + cut = Kalman.__new__(Kalman) + cut.headers = [f"header{i}" for i in range(num_headers)] + cut.window_size = window_size + cut.frames = [ + [float(100 + i * 10 + j) for j in range(existing)] for i in range(num_headers) + ] + return cut + + +def test_kill_update_none_frame_exact_FAIL_warning_and_no_store(mocker): + # Arrange + cut = _make_cut_for_update_kill_tests(3, 3, existing=2) + mock_print = mocker.patch(kalman_plugin.__name__ + ".print_msg") + before = [list(frame) for frame in cut.frames] + + # Act + cut.update(None) + + # Assert + assert cut.frames == before + assert mock_print.call_count == 1 + assert mock_print.call_args_list[0].args == ( + "Kalman plugin requires low_level_data but received None.", + ["FAIL"], + ) + + +def test_kill_update_short_frame_rejected_with_exact_warning(mocker): + # Arrange + cut = _make_cut_for_update_kill_tests(4, 3, existing=2) + mock_print = mocker.patch(kalman_plugin.__name__ + ".print_msg") + before = [list(frame) for frame in cut.frames] + + # Act + cut.update([1.0, 2.0]) # 2 values vs 4 headers + + # Assert + assert cut.frames == before # nothing stored + assert mock_print.call_count == 1 + assert mock_print.call_args_list[0].args == ( + "Kalman plugin update: frame length 2 does not match header length 4; frame rejected.", + ["WARNING"], + ) + + +def test_kill_update_long_frame_rejected_with_exact_warning(mocker): + # Arrange + cut = _make_cut_for_update_kill_tests(2, 3, existing=0) + mock_print = mocker.patch(kalman_plugin.__name__ + ".print_msg") + before = [list(frame) for frame in cut.frames] + + # Act + cut.update([1.0, 2.0, 3.0, 4.0]) # 4 values vs 2 headers + + # Assert + assert cut.frames == before # nothing stored + assert mock_print.call_count == 1 + assert mock_print.call_args_list[0].args == ( + "Kalman plugin update: frame length 4 does not match header length 2; frame rejected.", + ["WARNING"], + ) + + +def test_kill_update_valid_frame_stored_in_lockstep_windows(mocker): + # Arrange + mocker.patch(kalman_plugin.__name__ + ".print_msg") + cut = _make_cut_for_update_kill_tests(3, 5, existing=0) + + # Act + cut.update([1.5, 2.5, 3.5]) + + # Assert + assert cut.frames == [[1.5], [2.5], [3.5]] # kills != -> == guard mutants + + +def test_kill_update_junk_frame_stores_nothing(mocker): + # Arrange + cut = _make_cut_for_update_kill_tests(3, 3, existing=1) + mock_floatify_warning = mocker.patch( + "onair.data_handling.parser_util.print_msg" + ) + before = [list(frame) for frame in cut.frames] + + # Act + cut.update(["-", "nan", "abc"]) # junk becomes nan and is skipped + + # Assert + assert cut.frames == before # junk frame stores nothing + assert mock_floatify_warning.call_count == 3 # one WARNING per junk value + + +def test_kill_update_nan_input_frame_stores_nothing(mocker): + # Arrange + mocker.patch(kalman_plugin.__name__ + ".print_msg") + mocker.patch("onair.data_handling.parser_util.print_msg") + cut = _make_cut_for_update_kill_tests(3, 3, existing=0) + + # Act + cut.update([float("nan"), float("inf"), float("-inf")]) + + # Assert + assert cut.frames == [[], [], []] # NaN input stores nothing + + +def test_kill_update_nan_spiked_frame_skips_only_spiked_attribute(mocker): + # Arrange + mocker.patch(kalman_plugin.__name__ + ".print_msg") + mocker.patch("onair.data_handling.parser_util.print_msg") + cut = _make_cut_for_update_kill_tests(3, 5, existing=0) + + # Act + cut.update([float("nan"), 2.5, 3.5]) + + # Assert + assert cut.frames[0] == [] # nan skipped + assert cut.frames[1] == [2.5] # later attributes still stored (kills continue->break) + assert cut.frames[2] == [3.5] + + +def test_kill_update_window_saturates_to_exactly_window_size(mocker): + # Arrange + mocker.patch(kalman_plugin.__name__ + ".print_msg") + cut = _make_cut_for_update_kill_tests(2, 3, existing=0) + + # Act + for step in range(5): + cut.update([float(step), float(100 + step)]) + + # Assert + assert all(len(frame) == 3 for frame in cut.frames) # exactly window_size + assert cut.frames[0] == [2.0, 3.0, 4.0] # oldest dropped (kills pop mutants) + assert cut.frames[1] == [102.0, 103.0, 104.0] diff --git a/test/plugins/kalman/test_kalman_properties.py b/test/plugins/kalman/test_kalman_properties.py new file mode 100644 index 00000000..40cdbf98 --- /dev/null +++ b/test/plugins/kalman/test_kalman_properties.py @@ -0,0 +1,300 @@ +# GSC-19165-1, "The On-Board Artificial Intelligence Research (OnAIR) Platform" +# +# Copyright © 2023 United States Government as represented by the Administrator of +# the National Aeronautics and Space Administration. No copyright is claimed in +# the United States under Title 17, U.S. Code. All Other Rights Reserved. +# +# Licensed under the NASA Open Source Agreement version 1.3 +# See "NOSA GSC-19165-1 OnAIR.pdf" + +"""Property-based tests for the Kalman Plugin.update telemetry-ingestion path. + +Real Plugin instances are used (the root conftest.py injects a MagicMock +simdkalman, and update() never touches kf), so every rule below exercises the +production update() code path: length guard, floatify_input, finite-only +window maintenance. + +The RuleBasedStateMachine checks invariants I1-I5 plus the growth invariant; +the standalone @given properties pin the rejection behaviour of short / long / +None frames. Pre-patch, these tests document the blind spots (TypeError leaks, +desynced frames, non-finite values stored in windows); post-patch they must +all pass. +""" + +import copy +import math +import os +from unittest.mock import MagicMock + +import pytest +from hypothesis import given, settings, strategies as st +from hypothesis.stateful import RuleBasedStateMachine, initialize, invariant, rule + +import onair.src.ai_components.ai_plugin_abstract.ai_plugin as ai_plugin_abstract +from plugins.kalman import kalman_plugin +from plugins.kalman.kalman_plugin import Plugin + + +# --------------------------------------------------------------------------- +# Hypothesis settings: full 1000-example runs normally; a derandomized +# 25-example cap when MUTMUT_HYPOTHESIS_FAST is set (mutation-testing runner). +# --------------------------------------------------------------------------- +if "MUTMUT_HYPOTHESIS_FAST" in os.environ: + HYPO_SETTINGS = settings(max_examples=25, deadline=None, derandomize=True) +else: + HYPO_SETTINGS = settings(max_examples=1000, deadline=None) + +json_scalar = st.one_of( + st.none(), + st.booleans(), + st.integers(), + st.floats(allow_nan=True, allow_infinity=True), + st.text(max_size=8), + st.sampled_from(["-", "nan", "inf", "-inf", "Infinity", ""]), +) + +finite_float = st.floats(allow_nan=False, allow_infinity=False) + +MAX_HEADERS = 6 +MAX_WINDOW = 8 + +# Recorded kalman_plugin.print_msg traffic: list of (msg, clrs) tuples. +print_msg_calls = [] + + +def _record_print_msg(msg, clrs=None): + print_msg_calls.append((msg, clrs)) + + +@pytest.fixture(autouse=True) +def _silence_and_record_print_msg(monkeypatch): + """Silence AND record kalman_plugin.print_msg (raising=False so the file + is also runnable against module states lacking the attribute). + + Also stubs AIPlugin's ServiceManager the same way the csv_output plugin + tests do, so real Plugin instances can be constructed (the singleton + ServiceManager would otherwise demand a service_dict).""" + print_msg_calls.clear() + monkeypatch.setattr( + ai_plugin_abstract, "ServiceManager", MagicMock(), raising=False + ) + monkeypatch.setattr( + kalman_plugin, "print_msg", _record_print_msg, raising=False + ) + yield + print_msg_calls.clear() + + +def _make_plugin(num_headers, window_size): + headers = [f"ATTR_{i}" for i in range(num_headers)] + return Plugin("property_probe", headers, window_size) + + +# --------------------------------------------------------------------------- +# Standalone @given properties (rejection behaviour of the length guard). +# --------------------------------------------------------------------------- +@HYPO_SETTINGS +@given( + num_headers=st.integers(min_value=1, max_value=MAX_HEADERS), + window_size=st.integers(min_value=3, max_value=MAX_WINDOW), + short_by=st.integers(min_value=1, max_value=MAX_HEADERS), + vals=st.lists(finite_float, min_size=MAX_HEADERS, max_size=MAX_HEADERS), +) +def test_short_frames_are_rejected_and_leave_frames_unchanged( + num_headers, window_size, short_by, vals +): + plugin = _make_plugin(num_headers, window_size) + n = max(num_headers - short_by, 0) + frame = vals[:n] + before = copy.deepcopy(plugin.frames) + print_msg_calls.clear() + plugin.update(frame) + assert plugin.frames == before # rejected: nothing stored + assert any( + "does not match" in msg and str(n) in msg and str(num_headers) in msg + for msg, _ in print_msg_calls + ) + + +@HYPO_SETTINGS +@given( + num_headers=st.integers(min_value=1, max_value=MAX_HEADERS), + window_size=st.integers(min_value=3, max_value=MAX_WINDOW), + extra=st.integers(min_value=1, max_value=3), + vals=st.lists(finite_float, min_size=MAX_HEADERS + 3, max_size=MAX_HEADERS + 3), +) +def test_long_frames_are_rejected_and_leave_frames_unchanged( + num_headers, window_size, extra, vals +): + plugin = _make_plugin(num_headers, window_size) + frame = vals[: num_headers + extra] + before = copy.deepcopy(plugin.frames) + print_msg_calls.clear() + plugin.update(frame) + assert plugin.frames == before # rejected: nothing stored + assert any( + "does not match" in msg + and str(num_headers + extra) in msg + and str(num_headers) in msg + for msg, _ in print_msg_calls + ) + + +@HYPO_SETTINGS +@given( + num_headers=st.integers(min_value=1, max_value=MAX_HEADERS), + window_size=st.integers(min_value=3, max_value=MAX_WINDOW), +) +def test_none_frame_warns_FAIL_and_leaves_frames_unchanged(num_headers, window_size): + plugin = _make_plugin(num_headers, window_size) + before = copy.deepcopy(plugin.frames) + print_msg_calls.clear() + plugin.update(None) + assert plugin.frames == before + assert any( + msg == "Kalman plugin requires low_level_data but received None." + and clrs == ["FAIL"] + for msg, clrs in print_msg_calls + ) + + +# --------------------------------------------------------------------------- +# Stateful machine: random interleavings of valid / junk / short / long / +# nan-spike frames against one Plugin instance. +# --------------------------------------------------------------------------- +class KalmanUpdateMachine(RuleBasedStateMachine): + def __init__(self): + super().__init__() + self.plugin = None + self.headers = [] + self.window_size = 3 + self.valid_streak = 0 # consecutive ValidFrame rules since any other rule + self.windows_equal = True # windows advance in lockstep while True + + @initialize( + num_headers=st.integers(min_value=1, max_value=MAX_HEADERS), + window_size=st.integers(min_value=3, max_value=MAX_WINDOW), + ) + def create_plugin(self, num_headers, window_size): + self.window_size = window_size + self.valid_streak = 0 + self.windows_equal = True + self.plugin = _make_plugin(num_headers, window_size) + self.headers = self.plugin.headers + + @rule( + vals=st.lists(finite_float, min_size=MAX_HEADERS, max_size=MAX_HEADERS) + ) + def valid_frame(self, vals): + frame = vals[: len(self.headers)] + before_lengths = [len(f) for f in self.plugin.frames] + print_msg_calls.clear() + self.plugin.update(frame) # I1: must not raise + # every window advances by exactly one (finite values are never skipped) + assert all( + len(f) == min(n + 1, self.window_size) + for n, f in zip(before_lengths, self.plugin.frames) + ) + self.valid_streak += 1 + # re-arm lockstep tracking once saturation re-equalizes the windows + self.windows_equal = len({len(f) for f in self.plugin.frames}) == 1 + + @rule( + vals=st.lists(json_scalar, min_size=MAX_HEADERS, max_size=MAX_HEADERS) + ) + def junk_frame(self, vals): + frame = vals[: len(self.headers)] + print_msg_calls.clear() + self.plugin.update(frame) # I1: must not raise (junk becomes nan and is skipped) + self.valid_streak = 0 + # partial skips may desynchronize window lengths; lockstep is only + # guaranteed to continue while the windows are currently equal + self.windows_equal = len({len(f) for f in self.plugin.frames}) == 1 + + @rule( + short_by=st.integers(min_value=1, max_value=MAX_HEADERS), + vals=st.lists(finite_float, min_size=MAX_HEADERS, max_size=MAX_HEADERS), + ) + def short_frame(self, short_by, vals): + n = max(len(self.headers) - short_by, 0) + frame = vals[:n] + before = copy.deepcopy(self.plugin.frames) + print_msg_calls.clear() + self.plugin.update(frame) + assert self.plugin.frames == before # rejected: frames untouched + self.valid_streak = 0 + + @rule( + extra=st.integers(min_value=1, max_value=3), + vals=st.lists( + finite_float, min_size=MAX_HEADERS + 3, max_size=MAX_HEADERS + 3 + ), + ) + def long_frame(self, extra, vals): + frame = vals[: len(self.headers) + extra] + before = copy.deepcopy(self.plugin.frames) + print_msg_calls.clear() + self.plugin.update(frame) + assert self.plugin.frames == before # rejected: frames untouched + self.valid_streak = 0 + + @rule( + vals=st.lists(finite_float, min_size=MAX_HEADERS, max_size=MAX_HEADERS), + spike=st.sampled_from( + [ + float("nan"), + float("inf"), + float("-inf"), + "nan", + "inf", + "-inf", + "Infinity", + "-", + ] + ), + position=st.integers(min_value=0, max_value=MAX_HEADERS - 1), + ) + def nan_spike(self, vals, spike, position): + frame = vals[: len(self.headers)] + frame[position % len(frame)] = spike + print_msg_calls.clear() + self.plugin.update(frame) # I1: must not raise; spliced attr skipped + self.valid_streak = 0 + # the spiked attribute's window did not advance: lockstep may be lost + self.windows_equal = len({len(f) for f in self.plugin.frames}) == 1 + + @invariant() + def I2_frame_count_matches_headers(self): + assert len(self.plugin.frames) == len(self.headers) + + @invariant() + def I3_windows_never_exceed_window_size(self): + for frame in self.plugin.frames: + assert len(frame) <= self.window_size + + @invariant() + def I4_stored_values_are_all_finite(self): + for frame in self.plugin.frames: + for value in frame: + assert math.isfinite(value) + + @invariant() + def I5_windows_lockstep_while_only_valid_frames(self): + # While only valid frames have arrived since the last junk / rejection + # rule (valid_streak > 0) AND the windows are factually in lockstep + # (windows_equal, recomputed after every rule -- skipping rules desync + # by design, saturation re-equalizes), every window must keep the same + # length: valid frames advance all windows together, never skipping. + if self.valid_streak > 0 and self.windows_equal: + lengths = {len(f) for f in self.plugin.frames} + assert len(lengths) == 1 + + @invariant() + def growth_invariant_windows_saturate_after_full_streak(self): + if self.valid_streak >= self.window_size: + for frame in self.plugin.frames: + assert len(frame) == self.window_size + + +class TestKalmanMachine(KalmanUpdateMachine.TestCase): + settings = HYPO_SETTINGS