Feat: Add PRB-level correctness suite for AIAC eval framework

aiac/CLAUDE.md

@@ -75,7 +75,7 @@ ls src/aiac/<subsystem>/               # drill into any layer
 
 `pyproject.toml`'s `addopts` defaults `-m` to excluding every live-infra marker
 (`integration`, `eval_extended`, `eval_consistency`,
-`eval_robustness`), so a bare invocation never makes a real LLM/Keycloak
+`eval_robustness`, `eval_correctness_prb`), so a bare invocation never makes a real LLM/Keycloak
 call. The whole `test/` tree collects and runs green — no `--ignore` flags are
 needed. (This wasn't always true: the Policy Computation Engine was migrated to
 the SPM store surface in Wave 3, which resolved the earlier PCE-chain collection
@@ -124,13 +124,14 @@ set -a; . test/integration/.env; set +a
 When the cluster is not wired or the env is unset, the suite **skips cleanly** (it never false-passes).
 
 A passed `-m` always overrides the default, so this opts back into exactly
-`integration` (not the heavier markers below). Three heavier, narrower-infra
+`integration` (not the heavier markers below). Four heavier, narrower-infra
 markers exist alongside it — `eval_extended` (same live infra as
-`integration`, many more PRB/LLM calls), `eval_consistency` and
-`eval_robustness` (LLM only, no Keycloak/`opa`) — each invoked the same
+`integration`, many more PRB/LLM calls), `eval_consistency`,
+`eval_robustness`, and `eval_correctness_prb` (LLM only, no Keycloak/`opa`) — each invoked the same
 way, e.g. `pytest eval/ -m eval_extended`. See
-`docs/specs/eval/policy-eval-scenarios.md` and
-`docs/specs/eval/policy-eval-robustness-consistency.md` for their
+`docs/specs/eval/policy-eval-scenarios.md`,
+`docs/specs/eval/policy-eval-robustness-consistency.md`, and
+`docs/specs/eval/policy-eval-correctness-prb.md` for their
 runbooks.
 
 **Smoke test** (requires live service at `AIAC_PDP_CONFIG_URL`, default `http://127.0.0.1:7071`):

aiac/docs/specs/PRD.md

@@ -599,6 +599,7 @@ Beyond the marker-gated pytest tests above, individual integration tests are spe
 | `uc1-onboarding-pipeline` — a **ladder** of UC-1 onboarding tests | Discovery-driven sibling of `policy-pipeline` validating the **phase-1** deliverable against **one** in-cluster AIAC stack (OPA filesystem-stub writer, single abstract `policy.md`): with `github-agent` + a simplified `github-tool` **already deployed and registered** as Keycloak clients, three gradual rungs drive **real UC-1 onboarding** (`POST /apply/service/{id}`) — agent-only, agent→tool, tool→agent — and assert the generated Rego with `opa eval` (verdicts from `scenario_uc1.py`). Rungs 2/3 assert onboarding-**order-independence**. A fourth two-policy rung is **deferred** (two-stack topology discarded). Same scenario facts/tables as `policy-pipeline`; Rego semantically similar (not byte-identical). `@pytest.mark.integration`. | [integration-test/uc1-onboarding-pipeline.md](integration-test/uc1-onboarding-pipeline.md) |
 | `policy-eval-scenarios` — `test_policy_pipeline_eval.py` + guardrail tests | Generalized evaluation suite extending `policy-pipeline`'s single-agent/single-tool proof to ten scenarios: baseline-scale (many entities, names decoupled from roles, one agent→agent delegation grant), missing-details (emergent unreachability/zero-access under deny-by-default, a broad-sounding clause narrowed by an explicit qualifier, wildcard-grant expansion), adversarial-authoring (misleading names/descriptions, an identity/boundary-confusion probe, empty descriptions), and ambiguous-and-contradictory / adversarial-injection-and-edge-cases (whole-document `xfail` checks against the PRB directly, no Keycloak or `opa`). The eight heavy scenarios (`@pytest.mark.eval_extended`, scenario modules under `eval/scenarios/` except `agent_delegation`) assert full per-cell `opa eval` truth tables; the two light scenarios (`@pytest.mark.integration`) assert PRB-level rejection. | [eval/policy-eval-scenarios.md](eval/policy-eval-scenarios.md) |
 | `policy-eval-robustness-consistency` — `test_policy_pipeline_consistency.py` + `test_policy_pipeline_robustness.py` | Companion to `policy-eval-scenarios`, reusing its 8-scenario corpus to check the PRB's raw grant decisions (no OPA/PCE/k8s) for **consistency** (`@pytest.mark.eval_consistency`: N repeated runs on the same input, exact grant-set equality) and **robustness** (`@pytest.mark.eval_robustness`: mechanical text/order perturbation + a hand-reworded semantic-sibling corpus under `eval/scenarios_perturbed/`, both checked against the truth-table oracle). No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [eval/policy-eval-robustness-consistency.md](eval/policy-eval-robustness-consistency.md) |
+| `policy-eval-correctness-prb` — `test_policy_pipeline_correctness_prb.py` | Companion to `policy-eval-scenarios`/`policy-eval-robustness-consistency`, reusing the same 8-scenario corpus to score the PRB's raw grant/deny output (no OPA/PCE/k8s) against each scenario's truth table via a reusable, effect-aware scorer (`eval/correctness_scorer.py`): precision and recall tracked separately per gate and aggregated, plus a non-gating denial-precision figure for explicit `Deny` rules. `@pytest.mark.eval_correctness_prb`, zero-tolerance over-grant gate; under-grants/incorrect denials reported only. No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [eval/policy-eval-correctness-prb.md](eval/policy-eval-correctness-prb.md) |
 
 Tracking issues: the live-Keycloak pytest integration tests in `testing/5.1-integration-tests.md`; the PDP Policy Writer integration test in `testing/5.2-pdp-writer-integration-test.md`; the policy-pipeline integration test in `testing/5.3-policy-pipeline-integration-test.md`; the UC-1 onboarding pipeline integration-test ladder in `testing/5.4-uc1-onboarding-integration-test.md` (epic) with rungs `testing/5.4.1`/`5.4.2`/`5.4.3` and the deferred two-policy `testing/5.4.4`.