Skip to content

Audit October feature coverage and surface recorded evidence in Lab - #723

Open
Shayne Boyer (spboyer) with Copilot wants to merge 4 commits into
mainfrom
copilot/audit-october-features-and-fixes
Open

Shayne Boyer (spboyer) with Copilot wants to merge 4 commits into
mainfrom
copilot/audit-october-features-and-fixes

Conversation

Copilot AI commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Audit the October 5–19 changes against repository docs, the published site, and the opt-in Lab dashboard. The audit reconciles 50 merged PRs with default-branch commits and records each feature’s documentation, UI/API, and regression coverage.

  • Lab evidence: Add a Recorded evidence tab for saved reliability estimates, portable evidence, scoped requirement observations, and independent prompt-judging results. Distinguish missing or invalid metadata, explicit zero credits, cached provenance, and unverified evidence; do not imply evidence contents were independently verified.
  • Audit and docs: Add a linked coverage inventory and update Lab guidance, CLI/runtime wording, and October version boundaries. Add a screenshot generated from a mock-provenance fixture with a matching site mirror.
  • Known gap: Malformed runtime reliability interval and pass^k values still need validation in the Lab inspector.

Copilot AI self-assigned this Oct 19, 2026
Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

Co-authored-by: spboyer <7681382+spboyer@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Malformed judging metadata can crash the inspector, and regression fixtures contain provenance that production cannot emit.

3 open issues

Review effort: Balanced

Comment thread web/src/lab/RunInspector.tsx
Comment on lines +1045 to +1046
evidenceRuns: [evidenceRun()],
promptJudgingRuns: promptJudging.tasks[0]!.promptJudgingRuns,
Comment thread web/src/lab/RunInspector.tsx Outdated
Co-authored-by: spboyer <7681382+spboyer@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Malformed object-shaped evidence records can still crash the inspector instead of displaying a withheld state.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Validate malformed rows before rendering header fields

web/​src/​lab/​RunInspector.tsx:74

Object-shaped malformed rows still reach the header. For example, { runNumber: {}, attempts: 1, cached: false, assessment: "invalid" } passes this check; the manifest guard returns false, but React then tries to render runNumber as a child and crashes the inspector. Validate the envelope fields before rendering them and return the existing withheld placeholder on failure. Add a regression with a malformed object beside a valid record.

2 resolved since last review

Review effort: Balanced

Co-authored-by: spboyer <7681382+spboyer@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:56
Copilot AI changed the title [WIP] Audit October 5–19 features and fixes for documentation and dashboard Audit October feature coverage and surface recorded evidence in Lab Oct 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Malformed object-shaped evidence rows can still crash the inspector instead of displaying a withheld state.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Validate malformed evidence envelopes before rendering

web/​src/​lab/​RunInspector.tsx:114

Object-shaped malformed rows still pass this guard. For example, { runNumber: {}, attempts: 1, cached: false, assessment: "invalid" } reaches Run {evidence.runNumber}, where React throws instead of showing the withheld placeholder. Validate the envelope fields before rendering them, and add a regression with a malformed object alongside a valid sibling so one bad row cannot hide the remaining evidence.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audit October 5–19 features and fixes for documentation, site, and new dashboard coverage

3 participants