Skip to content

feat(findings): optional counterevidence and confidence_rationale - #193

Merged
BaaRai merged 1 commit into
mainfrom
feat/finding-evidence-schema
Nov 11, 2026
Merged

BaaRai merged 1 commit into
mainfrom
feat/finding-evidence-schema

Conversation

@BaaRai

@BaaRai BaaRai commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What

Adds two optional free-text fields to findings (Tier 1 item 3, finding evidence schema):

  • counterevidence: what was tested and did not work (blocked variants, negative paths).
  • confidence_rationale: why the conclusion holds.

Changes:

  • db/schema.sql: ALTER TABLE findings ADD COLUMN IF NOT EXISTS ... TEXT NOT NULL DEFAULT '' for both. Idempotent, no CHECK. Existing rows read back as ''.
  • db/finding_traffic.go: RecordFindingInput gets both fields, and the INSERT writes them.
  • db/findings.go: DBFinding gets both fields. They are added to the shared findingSelectCols and to all three scan sites (scanFindings, ListFindingsForExport, GetFinding).
  • agent/tools.go: report_finding accepts counterevidence and confidence_rationale as optional string properties and passes them into RecordFindingInput.
  • agent/worker.go, agent/promptcatalog.go (pentest): one sentence on the report_finding bullet. Fill both fields when confidence is not high. They are optional and never block reporting.
  • server/dto.go: FindingDTO carries both (omitempty). It is used by the list endpoints and by the detail endpoint GET /api/exploration/findings/{id}.
  • server/finding_verification.go + server/orchestration.go: seedFindingEvidenceSchema() adds the two properties to the stored report_finding tool schema on existing databases. Without it, the model would never see them, because SeedTool is first-insert only and the DB schema overrides the code schema. It follows the same pattern as seedFindingVerificationSchema: a property is added only if it is missing, it is idempotent, and it preserves user edits to the description, other properties, bindings and the enabled flag.
  • web/src/lib/types.ts and web/src/app/(main)/function/findings/detail/page.tsx: the detail page shows a "Counterevidence" card and a "Confidence rationale" card, each only when its value is non-empty.

Why

A confirmed finding currently records the positive proof only. Reviewers cannot see what the reporter already ruled out, or why the reporter believes the conclusion. That matters most when confidence is not high. These two fields give that reasoning a fixed place, outside the free-text evidence.

Optional fields, and why

Both fields are optional. A missing field never refuses a confirmed finding. This is the same philosophy as traffic_refs: a valid finding must not be lost over a missing auxiliary field. The tool descriptions and the prompts ask for them when confidence is not high, but the tool does not enforce it.

Limit

Other tools apply a rule like "required when confidence is not high". That rule cannot be enforced here, because findings have no confidence field. report_finding cannot tell a low-confidence report from a high-confidence one, so the requirement stays as prompt guidance only. A structured confidence field, and enforcement based on it, can come in 0.7.0 if needed.

Prompt seeds are first-insert only. Existing databases keep their stored worker/pentest prompt, so the new prompt sentence reaches new installs only. The tool-schema descriptions, which also carry the guidance, do reach existing databases through seedFindingEvidenceSchema.

Tests

  • Pure: agent/TestReportFindingSchemaReasoningFieldsOptional. Both properties exist, are type: string, are described as optional, and are not in required.
  • Pure: server/TestFindingDTOReasoningFields. findingFromDB maps both fields, and the JSON omits them when they are empty.
  • PG: db/TestFindingReasoningFieldsRoundTrip. Stores a finding with and without the fields, then reads back via GetFinding, ListFindingsPage and ListFindingsForExport. Empty values default to ''.
  • PG: agent/TestReportFindingStoresReasoningFields. The report_finding tool passes both fields into storage, and a call without them is still recorded.
  • PG: server/TestFindingReasoningDetailAndSchemaUpgrade. The detail endpoint returns both fields. The schema upgrade adds both properties to a customized report_finding row (run twice) and does not overwrite the custom description, properties, required or enabled flag.

Local verification (WSL Kali, Go 1.26.7, throwaway postgres:16-alpine container on 127.0.0.1):

  • go build ./...: OK
  • go vet ./...: OK
  • go test -count=1 ./agent ./server ./db on a fresh database: ok for all three packages. 642 top-level tests passed, 0 failed, 3 skipped: 2 in agent (known fresh-database skips that already exist on main) and 1 in server (live model config not set). All 5 new tests ran and passed.
  • gofmt -l (CR stripped) on the changed files: only agent/tools.go is reported, for two hunks that already exist on main (detectStagnation struct alignment and the stagnation_warning map). Neither is touched by this PR.
  • web: npx tsc --noEmit exit 0. npm run build exit 0. NEXT_EXPORT=1 next build (the build:static command used in CI) exit 0.

Not verified

  • The detail page was not rendered in a browser. Only the typecheck and build were run.
  • No end-to-end agent run with a real model was done. Nothing checks that models actually fill the fields.
  • The schema upgrade was tested in the PG unit test only, not against a real upgraded deployment.

Add two optional free-text fields to findings so a reporter can record
what was tested and did not work (counterevidence) and why the
conclusion holds (confidence_rationale).

- db: idempotent ADD COLUMN IF NOT EXISTS, TEXT NOT NULL DEFAULT '',
  no CHECK; RecordFindingInput, the INSERT, the shared select list and
  all three scan sites carry them.
- agent: report_finding accepts both as optional strings; the worker
  and pentest prompts say to fill them when confidence is not high.
- server: FindingDTO (list and detail) carries both, omitted when empty;
  existing databases get the two schema properties added to the stored
  report_finding row once, without overwriting user edits.
- web: the finding detail page shows each section only when non-empty.

Both fields stay optional: a missing field never refuses a finding.
@BaaRai
BaaRai merged commit 9355a37 into main Nov 11, 2026
4 checks passed
BaaRai added a commit that referenced this pull request Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant