Publish skills CI eval runs to evalboard as unlisted runs, linked from the GH checks

[bai-uipath]

Problem

Skills CI eval runs are write-only today. smoke-skills.yml (per-PR) and run-coder-eval.yml (ad-hoc dispatch) both produce a full coder-eval run directory, then bury it in a GitHub Actions artifact zip. Reading the result means downloading the artifact and poking at JSON by hand, while the exact same data rendered on evalboard already gives per-task status, transcripts, cost, and artifact browsing.

At the same time we do not want these runs on the evalboard front page. They are partial suites on unmerged branches, and mixing them into the recent-runs table, trends, watchlist, or maturity streaks would corrupt the nightly signal.

Proposal

Upload skills CI runs to the evalboard blob container as unlisted runs, reachable only by direct link, and surface that link in the GitHub Actions job summary for both workflows.

Four independent workstreams, listed by dependency order rather than value:

1. Upload from skills CI (coder_eval_uipath + skills)

eval-runner upload --adhoc already does the core of this: it writes the meta.json sidecar and, because a CI run id is date-shaped, uploads under an adhoc-<id> prefix so it can never clobber a nightly. The adhoc flag also already excludes the run from the recent-runs aggregate, trends, watchlist, and the maturity history fetch.

What is missing is the ability to run it from a GitHub Actions job in the skills repo:

• eval-runner is unpublished and lives in this repo, so the skills workflow needs a private checkout plus a local install (preferred over an inline az storage blob upload-batch, which would lose the exclude patterns that keep uploads slim).
• Blob write credentials have to reach the skills repo. Preferred: azure/login with OIDC federated credentials and Storage Blob Data Contributor on the storage account. Fallback: an account key in repo secrets. This is the long pole and needs an explicit security decision, because smoke-skills.yml is pull_request-triggered.
• A retention/lifecycle policy on the adhoc- prefix, decided before this is switched on rather than after. Every PR uploading a full artifact tree with no GC will grow the container steadily.

2. Unlisted runs on evalboard (coder_eval, plus a flag in coder_eval_uipath)

The adhoc flag keeps a run out of every aggregate, but the main page still renders an ad-hoc section that lists all non-date-shaped run ids, so CI uploads would appear there.

Add an unlisted (or source: "ci") field to the meta sidecar, thread it through the evalboard run-metadata type, and skip those rows in the ad-hoc row builder. The run detail page fetches straight from blob and needs no change, so the direct link keeps working. Add a matching --unlisted flag to eval-runner upload.

Roughly 15 lines in evalboard plus a CLI flag.

3. Link back in CI (skills)

One step after the upload that echoes the evalboard run URL into $GITHUB_STEP_SUMMARY, in both workflows. No upstream change required.

Worth pairing with JUnit output so failed tasks also land as PR annotations. Skills produces no JUnit today; coder-eval run --junit-xml plus a JUnit report action covers it.

4. Adopt the published composite action in skills (coder_eval + skills, optional)

Lowest value of the four, and explicitly not a prerequisite for the rest. The UiPath/coder_eval composite action currently has no real consumer beyond its own dogfood job. Adopting it in the skills Linux jobs would standardize install, JUnit, job summary, and the per-task score floor.

Two small upstream gaps first: the action cannot install extras (skills needs coder-eval[codex] and [antigravity]), and it has no working-directory input (both skills jobs run from tests/). Public PyPI carries the pinned version with the needed extras, so the package index is not a blocker.

The Windows job in run-coder-eval.yml is out of scope. It is a per-task loop with a content-filter retry classifier, which a single-invocation action cannot express.

Acceptance criteria

• A skills CI eval run (both workflows) lands in the evalboard blob container under an adhoc- prefix.
• That run does not appear on the evalboard main page, in trends, in the watchlist, or in maturity streak history.
• The run is fully viewable at its direct URL.
• Both workflows print the evalboard link in the job summary.
• Nightly metrics are provably unchanged after the first CI upload.

Open questions

• Unlisted means undiscoverable. Once the GH checks entry ages out, the run cannot be found again. Do we accept that, keep the ad-hoc listing, or add an opt-in "show unlisted" toggle on the main page?
• Should per-PR smoke runs upload on every PR, or only on demand (label, or dispatch only)? Cost and container growth scale directly with this choice.
• OIDC federated credential vs. storage account key for the skills repo.

Repo split

• coder_eval_uipath (internal): upload path, --unlisted flag, Azure credential and retention decisions.
• coder_eval (public): evalboard unlisted filtering, and the action extras / working-directory inputs if workstream 4 goes ahead.
• skills (public): workflow wiring, upload step, summary link.