Skip to content

docs: explain autonomous-science metric definitions and exclusions #19

Description

@fraware

Why this helps

sm metrics autonomous-science already emits numerator/denominator/included_ids/exclusions for each assurance slice, but docs/metrics.md only has a one-paragraph summary. New contributors (and reviewers of Gate 6 assurance failures) need a readable definition of what each metric counts, what exclusion reasons mean, and what the numbers do not claim.

Scope

Add a dedicated subsection (or short companion page under docs/assurance/) that documents every key in METRIC_KEYS from pipeline/src/sm_pipeline/assurance/metrics/__init__.py, matching actual slice construction in that module. Cross-link from docs/metrics.md and docs/assurance/README.md.

Suggested starting points

  • pipeline/src/sm_pipeline/assurance/metrics/__init__.py (METRIC_KEYS, _slice, compute_autonomous_science_metrics)
  • docs/metrics.md (Autonomous science paragraph)
  • docs/assurance/README.md, docs/assurance/pilot-interpretation.md
  • benchmarks/assurance/README.md, benchmarks/baseline_thresholds.json (floors context)
  • examples/assurance-pilot/metrics.json (example output shape)

Acceptance criteria

  • Each metric key has: plain-language definition, what enters the numerator, population/denominator, and typical exclusion reason strings
  • Explicit non-claims block (not live authorization rates; does not rewrite claim status; presence of a record is not acceptance)
  • Links from docs/metrics.md and docs/assurance/README.md
  • Example walkthrough using pilot fixtures / examples/assurance-pilot/metrics.json
  • No schema or CLI behavior changes required for this issue

Non-goals

  • Changing metric formulas, floors, or Gate 6 scorers
  • Building a portal metrics UI (separate issue if desired)
  • Rewriting PCS or corpus claim semantics

How to test

uv run --project pipeline sm metrics autonomous-science --out /tmp/assurance-metrics.json
# Manually confirm the new docs match keys and exclusion reasons in the JSON
# Spot-check against examples/assurance-pilot/metrics.json after regenerating if needed:
uv run --project pipeline sm import-assurance-release tests/assurance/fixtures/pilot-retrospective/
uv run --project pipeline sm metrics autonomous-science --out examples/assurance-pilot/metrics.json

If labels good first issue / help wanted are unavailable in your fork, ask a maintainer to apply them; they exist on fraware/scientific-memory.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationgood first issueGood for newcomershelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions