feat(decision): keep a trace when any check flags it, with per-question thresholds and severity

README.md

@@ -262,18 +262,83 @@ A manifest holds the questions the model is asked:
 ```
 
 - `questions` is sent to the model unchanged, all questions, in the
-  manifest's order. The decision question's answer is `p_keep`, and the
-  reason question's chosen option is the `reason`.
+  manifest's order, in one call. The reason question's chosen option is the
+  `reason`.
+- `decision`, optional, lists more yes/no (`noul`) questions that can keep a
+  trace (see [Any question can keep a trace](#any-question-can-keep-a-trace)).
+  Without it, the decision question decides alone. It stays in the engine
+  and is not sent to the model.
 - A relative `scorer.manifest` path resolves against the config file's
   directory.
 
 A manifest is optional, for every System One model. Without one the model
-is asked the built-in questions (`answer_wrong`, and `why_wrong` for the
-reason), and the engine logs a warning at start-up. To word the questions
+is asked the built-in questions
+([`builtin_manifest.json`](internal/config/builtin_manifest.json)), and the
+engine logs a warning at start-up. They are:
+
+| Question | Asks |
+| --- | --- |
+| `answer_wrong` | Is the final answer wrong or only partially correct? |
+| `unsupported_answer` | Does it make claims the evidence does not support? |
+| `missing_tool_calls` | Did the agent skip lookups the task needed? |
+| `answered_on_failure` | Did it answer despite a failed or partial tool call? |
+| `ignored_instructions` | Does it go against the user's instructions? |
+| `contradicts_knowledge` | Does it contradict general knowledge? |
+| `why_wrong` | Which [reason](#reasons) explains what is wrong? |
+
+Each yes/no answer is the probability that the agent failed in that way, and
+any of them can keep the trace. To word the questions
 for your agent, start from
 [`examples/manifests/my-agent.json`](examples/manifests/my-agent.json) and
 set `scorer.manifest` (or `TAILWISE_MANIFEST`).
 
+### Any question can keep a trace
+
+A manifest's `decision` block lists the yes/no questions that can keep a
+trace besides the decision question, in priority order, and can give any of
+them its own threshold:
+
+```json
+"decision": {
+  "any_of": ["unsupported_answer", "missing_tool_calls", "answered_on_failure",
+             "ignored_instructions", "contradicts_knowledge"],
+  "thresholds": {"contradicts_knowledge": 0.7}
+}
+```
+
+- **Any question keeps.** A trace is kept when the decision question, or any
+  question in `any_of`, reaches its threshold. Each question checks a
+  different way an agent fails, and a wrong answer usually fails in one way
+  only: a check that says "no" is not evidence the answer is right, so it
+  never counts against keeping.
+- **Thresholds.** `thresholds` sets a question's own threshold, in (0, 1];
+  a question without one uses `policy.keep_threshold`. Raise it for a check
+  that flags too often. The decision question can have one too.
+- **Flags.** The questions that reached their threshold are the trace's
+  flags: the problems the model found, in priority order. They are in the
+  decision log line (`flags`) and on the exported trace (`eval.flags`), next
+  to every answer (`eval.answers.<question>`). A reviewer can filter on them,
+  for example every kept trace flagged `answered_on_failure`.
+- **Priority and severity.** The decision question always ranks first, then
+  `any_of` in the order written. Ranks become weights that sum to 1, falling
+  steeply from the top ([rank-order centroid](https://doi.org/10.1287/mnsc.42.11.1515)
+  weights: with all six built-in questions, 0.41, 0.24, 0.16, 0.10, 0.06 and
+  0.03). A trace's **severity** is the sum of weight × answer over its flags:
+  higher for more problems, and for higher-priority ones. It does not change
+  whether a trace is kept; it ranks kept traces for review (`severity` in the
+  log, `eval.severity` on the export).
+- `p_keep` is the highest answer among the deciding questions.
+- Every listed question must be a `noul` question of the same manifest,
+  listed once, and the model must answer all of them: a missing answer is a
+  scoring error, like a missing decision answer. Questions not listed are
+  still asked, logged and exported, but do not keep a trace.
+- Each listed question is one more way to be kept, so listing more keeps more
+  traces. Recheck the share kept after changing the list, a threshold or a
+  question's wording.
+
+Without a `decision` block, the decision question decides alone, against
+`policy.keep_threshold`.
+
 ### One model per agent
 
 Each agent can have its own model: its own questions, and
@@ -566,12 +631,18 @@ so it does not take the engine out of a load balancer; watch
    queue for redelivery. On the synchronous endpoint, which has no queue,
    the engine retries it with `scorer.max_retries` and jittered
    `scorer.backoff`. The two kinds of retry never multiply.
-4. **Decide.** `p_keep` is the model's answer to the decision question.
-   - `p_keep >= keep_threshold`: keep, rule `above_threshold`.
+4. **Decide.** The trace is kept when any question that decides (the
+   decision question, and those in the manifest's `decision.any_of`)
+   reaches its threshold; those questions are its flags (see
+   [Any question can keep a trace](#any-question-can-keep-a-trace)).
+   - A question reached its threshold: keep, rule `above_threshold`.
    - Otherwise keep with probability `random_floor`, rule `random_floor`.
    - Otherwise drop, rule `below_threshold`.
 5. **Export.** Kept traces are queued to every exporter. Each decision produces one JSON log
-   line with `trace_id`, `agent`, `decision`, `rule`, `reason`, `cause`, `p_keep`, `model` and `latency_ms`.
+   line with `trace_id`, `agent`, `decision`, `rule`, `reason`, `cause`, `p_keep`, `model`, `latency_ms`,
+   `answers`, every yes/no answer by question, `flags`, the deciding
+   questions that reached their threshold, and `severity`, as in
+   `"answers": {"answer_wrong": 0.91, "unsupported_answer": 0.84, ...}, "flags": ["answer_wrong", "unsupported_answer"], "severity": 0.57`.
 
 ### Reasons
 
@@ -844,7 +915,10 @@ Root span attributes:
 | `eval.cause` | string | both | The model's short name for the reason, e.g. `contradicts_evidence`; omitted when it gave none |
 | `eval.model` | string | both | The scorer model that decided |
 | `eval.latency_ms` | double | both | Scoring time in milliseconds |
-| `eval.p_keep` | double | both | The model's probability that the answer is wrong; omitted on a fallback keep |
+| `eval.p_keep` | double | both | The highest answer among the questions that decide; omitted on a fallback keep |
+| `eval.flags` | string | both | The deciding questions that reached their threshold, comma-separated in priority order, e.g. `answer_wrong,unsupported_answer`; omitted when none did |
+| `eval.severity` | double | both | Sum of priority weight × answer over the flags; omitted when there are none |
+| `eval.answers.<question>` | double | both | Each yes/no answer, e.g. `eval.answers.missing_tool_calls`; omitted on a fallback keep |
 
 Tool span attributes (rebuilt traces only; forwarded tool spans are not
 changed):
@@ -950,8 +1024,12 @@ These are open problems in the current version, not design choices to rely on.
   98% dropped with the default `random_floor`, and only `WARN` log lines show
   it.
 - **Decisions use the model's probabilities as they come.** Nothing tunes
-  them to your agent, so `keep_threshold` may keep too much or too little at
-  first. Watch the share of traces kept and adjust the threshold.
+  them to your agent, and any of six questions can keep a trace, so the
+  defaults may keep too much at first. Watch the share of traces kept, and
+  which flags keep them, and adjust `decision.thresholds` or the `any_of`
+  list.
+- **Severity ranks traces but nothing acts on it yet.** Exporters send kept
+  traces in the order they are decided, not by severity.
 - **Every scored trace is a billed call to Jev, and leaves your network.**
   Nothing caps the spend or redacts fields before sending; only
   `server.workers` limits the rate.

architecture/sequence-diagrams.md

@@ -152,7 +152,7 @@ sequenceDiagram
     EW->>EW: decodeRecord(msg.Data()) → pipeline.Record
     EW->>T: Send(ctx, pipeline.Record)
     T->>B: Send(ctx, pipeline.Record)
-    Note over B: evalAttrs(Record) → eval.decision, rule, reason,<br/>cause, model, p_keep, latency
+    Note over B: evalAttrs(Record) → eval.decision, rule, reason,<br/>cause, model, p_keep, latency,<br/>flags, answers.*
     alt Record.Trace.OTLP set (came in as spans)
         B->>O: Annotate(OTLP bytes, Annotation)
     else from the JSON API

examples/manifests/my-agent.json

@@ -1,11 +1,31 @@
 {
   "model": "jev-latest",
-  "note": "A manifest for your agent: the questions it is asked. answer_wrong decides keep or drop, and why_wrong picks the plain-English reason; word both for your agent.",
+  "note": "A manifest for your agent: the questions it is asked, all in one call. A trace is kept when answer_wrong, or any question in decision.any_of, reaches its threshold (decision.thresholds, else policy.keep_threshold); any_of is in priority order after answer_wrong, which ranks how severe a trace's problems are. why_wrong picks the plain-English reason. Word them all for your agent.",
   "questions": {
     "answer_wrong": {
       "type": "noul",
       "instructions": "Is the final answer wrong or only partially correct for the user's question?"
     },
+    "unsupported_answer": {
+      "type": "noul",
+      "instructions": "Does the final answer make claims that the evidence the agent gathered does not support?"
+    },
+    "missing_tool_calls": {
+      "type": "noul",
+      "instructions": "Did the agent leave out tool calls the task needed, so it answered without information it should have looked up?"
+    },
+    "answered_on_failure": {
+      "type": "noul",
+      "instructions": "Did the agent answer even though a tool call failed or returned only partial results?"
+    },
+    "ignored_instructions": {
+      "type": "noul",
+      "instructions": "Does the final answer ignore or go against instructions given in the user's question?"
+    },
+    "contradicts_knowledge": {
+      "type": "noul",
+      "instructions": "Does the final answer contradict well-established general knowledge?"
+    },
     "why_wrong": {
       "type": "choice",
       "instructions": "Which best explains what is wrong with the final answer? Choose \"correct\" if nothing is wrong.",
@@ -19,5 +39,17 @@
         "no_answer": "The agent did not give an answer."
       }
     }
+  },
+  "decision": {
+    "any_of": [
+      "unsupported_answer",
+      "missing_tool_calls",
+      "answered_on_failure",
+      "ignored_instructions",
+      "contradicts_knowledge"
+    ],
+    "thresholds": {
+      "contradicts_knowledge": 0.7
+    }
   }
 }

internal/config/builtin_manifest.json

@@ -1,11 +1,31 @@
 {
   "model": "built-in",
-  "note": "The questions the engine asks when a model has no manifest of its own: answer_wrong decides keep or drop, and why_wrong picks the plain-English reason.",
+  "note": "The questions the engine asks when a model has no manifest of its own, all in one call. A trace is kept when answer_wrong, or any question in decision.any_of, reaches the keep threshold; any_of is in priority order after answer_wrong. why_wrong picks the plain-English reason.",
   "questions": {
     "answer_wrong": {
       "type": "noul",
       "instructions": "Is the final answer wrong or only partially correct for the user's question?"
     },
+    "unsupported_answer": {
+      "type": "noul",
+      "instructions": "Does the final answer make claims that the evidence the agent gathered does not support?"
+    },
+    "missing_tool_calls": {
+      "type": "noul",
+      "instructions": "Did the agent leave out tool calls the task needed, so it answered without information it should have looked up?"
+    },
+    "answered_on_failure": {
+      "type": "noul",
+      "instructions": "Did the agent answer even though a tool call failed or returned only partial results?"
+    },
+    "ignored_instructions": {
+      "type": "noul",
+      "instructions": "Does the final answer ignore or go against instructions given in the user's question?"
+    },
+    "contradicts_knowledge": {
+      "type": "noul",
+      "instructions": "Does the final answer contradict well-established general knowledge?"
+    },
     "why_wrong": {
       "type": "choice",
       "instructions": "Which best explains what is wrong with the final answer? Choose \"correct\" if nothing is wrong.",
@@ -19,5 +39,14 @@
         "no_answer": "The agent did not give an answer."
       }
     }
+  },
+  "decision": {
+    "any_of": [
+      "unsupported_answer",
+      "missing_tool_calls",
+      "answered_on_failure",
+      "ignored_instructions",
+      "contradicts_knowledge"
+    ]
   }
 }

internal/config/config.go

@@ -181,10 +181,23 @@ type Manifest struct {
 	Model string `json:"model"`
 	// Questions is kept as written (compacted), so keys reach the model in order.
 	Questions json.RawMessage `json:"questions"`
+	// Decision, when set, lists more questions that decide; it stays in the
+	// engine and is not sent to the model.
+	Decision *ManifestDecision `json:"decision"`
 
 	Reasons map[string]string `json:"-"`
 }
 
+// ManifestDecision lists the yes/no questions that decide besides the
+// decision question, in priority order after it: a trace is kept when any of
+// their answers reaches its threshold.
+type ManifestDecision struct {
+	AnyOf []string `json:"any_of"`
+	// Thresholds overrides policy.keep_threshold for some of the deciding
+	// questions, the decision question included.
+	Thresholds map[string]float64 `json:"thresholds"`
+}
+
 // DefaultModel is what an agent's model starts from: the built-in scorer
 // settings, minus the endpoint and model it must set itself.
 func DefaultModel() Scorer {

internal/config/config_test.go

@@ -8,6 +8,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/tecnosam/tailwise/internal/pipeline"
 	"github.com/tecnosam/tailwise/internal/scorer"
 )
 
@@ -277,6 +278,10 @@ func TestLoadWithoutAFile(t *testing.T) {
 	if m.Client.ReasonQuestion != "why_wrong" || m.Client.Reasons["contradicts_evidence"] != "The final answer contradicts the evidence the agent gathered." {
 		t.Errorf("reason question %q, reasons %v", m.Client.ReasonQuestion, m.Client.Reasons)
 	}
+	// Any of the five checks can keep a trace, as well as answer_wrong.
+	if a := m.Client.AnyOf; len(a) != 5 || a[0] != "unsupported_answer" || a[4] != "contradicts_knowledge" {
+		t.Errorf("any_of = %q, want the five built-in checks", a)
+	}
 	t.Setenv(DefaultJevAPIKeyEnv, "")
 	if _, err := Load(""); err == nil || !strings.Contains(err.Error(), "$JEV_API_KEY is not set") {
 		t.Errorf("no key: error = %v", err)
@@ -380,3 +385,72 @@ func TestNATSNamesFromThePrefix(t *testing.T) {
 		t.Errorf("names = %q\nwant    %q", got, want)
 	}
 }
+
+// A manifest's decision block lists more questions that decide; it may list
+// only its own yes/no questions.
+func TestManifestDecisionAnyOf(t *testing.T) {
+	cfg, err := Load(write(t, "scorer:\n  manifest: manifest.json\n", testManifest))
+	if err != nil || cfg.Models.Default.Client.AnyOf != nil {
+		t.Fatalf("no decision block: any_of %q, error %v", cfg.Models.Default.Client.AnyOf, err)
+	}
+
+	with := func(decision string) string {
+		return `{"questions": {"answer_wrong": {"type": "noul"}, "checks": {"type": "noul"},` +
+			` "quality": {"type": "score", "criteria": ["a", "b"]}}, "decision": ` + decision + `}`
+	}
+	cfg, err = Load(write(t, "scorer:\n  manifest: manifest.json\n", with(`{"any_of": ["checks"]}`)))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if a := cfg.Models.Default.Client.AnyOf; len(a) != 1 || a[0] != "checks" {
+		t.Errorf("any_of = %q", a)
+	}
+	if strings.Contains(string(cfg.Models.Default.Client.Questions), "any_of") {
+		t.Errorf("the decision block reached the questions sent to the model: %s", cfg.Models.Default.Client.Questions)
+	}
+
+	for decision, wantErr := range map[string]string{
+		`{"any_of": []}`:                        "must list questions in any_of or set thresholds",
+		`{"any_of": ["checks", "checks"]}`:      `"checks" is listed twice`,
+		`{"any_of": ["answer_wrong"]}`:          `"answer_wrong" is listed twice, or is the decision question`,
+		`{"thresholds": {"checks": 0.6}}`:       `"checks" is neither the decision question nor in any_of`,
+		`{"thresholds": {"answer_wrong": 0}}`:   "want a number in (0, 1]",
+		`{"thresholds": {"answer_wrong": 1.5}}`: "want a number in (0, 1]",
+		`{"any_of": ["tone"]}`:                  `question "tone" not found`,
+		`{"any_of": ["quality"]}`:               `question "quality" has type "score", want "noul"`,
+	} {
+		if _, err := Load(write(t, "scorer:\n  manifest: manifest.json\n", with(decision))); err == nil || !strings.Contains(err.Error(), wantErr) {
+			t.Errorf("%s: error = %v, want %q", decision, err, wantErr)
+		}
+	}
+}
+
+// The deciding questions become checks in priority order, the decision
+// question first, weighted by rank, each with its own threshold or none.
+func TestManifestDecisionChecks(t *testing.T) {
+	manifest := `{"questions": {"answer_wrong": {"type": "noul"}, "a": {"type": "noul"}, "b": {"type": "noul"}},` +
+		` "decision": {"any_of": ["b", "a"], "thresholds": {"answer_wrong": 0.8, "a": 0.6}}}`
+	cfg, err := Load(write(t, "scorer:\n  manifest: manifest.json\n", manifest))
+	if err != nil {
+		t.Fatal(err)
+	}
+	checks := cfg.Models.Default.Checks
+	if len(checks) != 3 || checks[0].Question != "answer_wrong" || checks[1].Question != "b" || checks[2].Question != "a" {
+		t.Fatalf("checks = %+v, want answer_wrong, then any_of in order", checks)
+	}
+	if checks[0].Threshold != 0.8 || checks[1].Threshold != 0 || checks[2].Threshold != 0.6 {
+		t.Errorf("thresholds = %v %v %v, want 0.8, the policy's (0) and 0.6", checks[0].Threshold, checks[1].Threshold, checks[2].Threshold)
+	}
+	if !(checks[0].Weight > checks[1].Weight && checks[1].Weight > checks[2].Weight) {
+		t.Errorf("weights = %v %v %v, want them falling with priority", checks[0].Weight, checks[1].Weight, checks[2].Weight)
+	}
+
+	// Without a decision block, the decision question is the one check.
+	cfg, err = Load(write(t, "scorer:\n  manifest: manifest.json\n", testManifest))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if c := cfg.Models.Default.Checks; len(c) != 1 || c[0] != (pipeline.Check{Question: "answer_wrong", Weight: 1}) {
+		t.Errorf("checks = %+v, want answer_wrong alone with weight 1", c)
+	}
+}

internal/config/engine.go

@@ -140,12 +140,14 @@ func model(s Scorer, apiKey string, m *Manifest) scorer.ModelConfig {
 			Model:            s.Model,
 			Questions:        m.Questions,
 			DecisionQuestion: s.DecisionQuestion,
+			AnyOf:            m.anyOf(),
 			ReasonQuestion:   reasonQuestion(s, m),
 			Reasons:          m.Reasons,
 			APIKey:           apiKey,
 			Timeout:          s.Timeout,
 		},
-		Retry: retry.Policy{Attempts: s.MaxRetries + 1, Backoff: s.Backoff, Jitter: true},
+		Retry:  retry.Policy{Attempts: s.MaxRetries + 1, Backoff: s.Backoff, Jitter: true},
+		Checks: m.checks(s.DecisionQuestion),
 	}
 }