Skip to content

feat(decision): keep a trace when any check flags it, with per-question thresholds and severity - #1

Merged
tecnosam merged 2 commits into
mainfrom
experiment/logarithmic-pool-multiple-questions
Oct 29, 2026
Merged

tecnosam merged 2 commits into
mainfrom
experiment/logarithmic-pool-multiple-questions

Conversation

@tecnosam

@tecnosam tecnosam commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Replaces the logarithmic pool with an any-of rule. Averaging the checks let the ones that said "no" pull down a trace that had one real failure, which lowered recall.

Changes

  • Any-of decisions: a trace is kept when answer_wrong, or any question in the manifest's decision.any_of, reaches its threshold.
  • Per-question thresholds: decision.thresholds sets a question's own threshold; questions without one use policy.keep_threshold.
  • Priority: answer_wrong ranks first, then any_of in the order written. Ranks become rank-order centroid weights that sum to 1.
  • Severity: the sum of weight × answer over a trace's flags. It ranks kept traces for review and does not change whether a trace is kept.
  • Flags: the questions that reached their threshold, listed in priority order.
  • Output: answers, flags and severity are in the decision log and on exported traces (eval.answers.<question>, eval.flags, eval.severity), and they travel through the export queue.
  • Validation: a manifest fails to load if a question is unknown, isn't a yes/no question, is listed twice, or has a threshold outside (0, 1].
  • Removed: pipeline.Pool and the decision.intercept / decision.weights manifest keys.
  • Docs: README section "Any question can keep a trace"; issues/export-by-severity.md proposes exporting in severity order.

Notes for reviewers

  • Behaviour change: six questions can now keep a trace, so expect a higher keep rate. Tune with decision.thresholds or a shorter any_of.
  • Backward compatible: a manifest without a decision block decides exactly as before.

Testing

  • make lint and make test pass. New tests cover the weights, per-check decisions and severity, manifest validation, the log fields, the export attributes and the queue round trip.
  • Not yet run against the live Jev.

@tecnosam tecnosam self-assigned this Oct 28, 2026
@tecnosam tecnosam changed the title Score on 5 key questions feat(decision): keep a trace when any check flags it, with per-question thresholds and severity Oct 29, 2026
@tecnosam
tecnosam merged commit 7e9eec3 into main Oct 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant