Skip to content

evetest-pr: compute the PR diff with git, not the API - #7012

Merged
eriknordmark merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-pr-large-diff
Nov 3, 2026
Merged

eriknordmark merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-pr-large-diff

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Description

gh pr diff fails with HTTP 406 ("the diff exceeded the maximum number of files (300)") for a PR touching more than 300 files, which a vendor bump easily exceeds. The "Fetch the PR diff" step of discover-suites in evetest-pr.yml then failed before any suite was scheduled, so evetest never ran for such PRs even though "Ready for evetest" was reported. This is what happened to #5644, both after its approval and after a /evetest comment.

This PR:

  • Checks out the PR head (refs/pull/<n>/head) from the upstream repo with actions/checkout, blobless (filter: blob:none), with the full history of every branch (about 20 MB) and a sparse working tree. The PR is diffed against its merge-base with the base branch.
  • Leaves vendored code (**/vendor/**) out of pr.diff. The vendored modules added, removed or bumped are read from vendor/modules.txt and listed at the end of pr-context.txt, and the Claude prompt says so.
  • Checks that the checked-out commit is still the head SHA the run was triggered for.
  • Removes the clone right after the diff, so it is not part of the tree Claude reads.
  • Falls back to the full suite if fetching or diffing fails: both steps are continue-on-error, and Claude is only asked when the diff step succeeded. Previously the whole job failed and nothing ran.

How to test and validate this PR

Not covered by an automated test. Validated locally by reproducing what the checkout action fetches for #5644 (all branches plus refs/pull/5644/head, blobless, sparse) and running the new diff step on it:

  • the fetch took about 3.5 s
  • pr.diff has 47 files (go.mod/go.sum included) and no vendored file, where the API refused the diff
  • pr-context.txt holds the PR title, description, commit messages and the vendored module changes of edgeview, kube-init, newlog and wwan/mmagent with their versions
  • a wrong head SHA makes the step fail, which leads to the full-suite fallback

actionlint passes.

To validate in CI: once merged, comment /evetest on #5644 (or any PR touching more than 300 files). discover-suites should get past "Fetch the PR diff", Claude should post its skip decision, and the suites should be scheduled.

Changelog notes

None. CI-only change.

PR Backports

  • 17.0-stable: Backported in [17.0-stable] evetest: add the evetest CI workflows and the fixes made to them on master #7111.
  • 16.0-stable: No, not as a standalone PR. All evetest workflow files will be backported together once they work properly.
  • 14.5-stable: No, not as a standalone PR. All evetest workflow files will be backported together once they work properly.
  • 13.4-stable: No, not as a standalone PR. All evetest workflow files will be backported together once they work properly.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't check them. This is a CI workflow change: no documentation or device testing applies. The diff step was tested locally against pubsub: send messages as frames over unix stream sockets #5644.

🤖 Generated with Claude Code

`gh pr diff` fails with HTTP 406 for a PR touching more than 300 files,
which a vendor bump easily exceeds (e.g. PR lf-edge#5644). discover-suites
then failed before any suite was scheduled, so evetest never ran for
such PRs even though "Ready for evetest" was reported.

Check out the PR head from the upstream repo with actions/checkout,
blobless and with the full history of every branch (about 20 MB), and
diff it against its merge-base with the base branch. Vendored code is
left out of the diff; instead, the vendored modules added, removed or
bumped are read from vendor/modules.txt and listed in pr-context.txt.

If fetching or diffing fails, Claude is not asked for a skip decision
and the run falls back to the full suite instead of failing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added the testing New test or test framework changes label Nov 2, 2026
@github-actions
github-actions Bot requested a review from europaul November 2, 2026 18:35

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eriknordmark

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @eriknordmark's task in 3m 20s —— View job


Review

I reviewed the switch from gh pr diff to a git-computed diff. The change is well-reasoned and correct.

  • Read the PR diff
  • Read the surrounding workflow for context
  • Analyze the diff computation logic
  • Post review

Verified correct:

  • actions/checkout with fetch-depth: 0 + ref: refs/pull/<n>/head uses the all-history refspec (+refs/heads/*:refs/remotes/origin/* plus the PR head), so origin/$BASE resolves and merge-base "origin/$BASE" HEAD gives the right diff base — equivalent to origin/$BASE...HEAD.
  • continue-on-error + gating the next steps on steps.pr-src.outcome/steps.pr-diff.outcome == 'success' is the correct pattern: a fetch/diff failure falls through to the else branch of decide (full suite) instead of aborting the job. outcome (pre-continue-on-error) is the right field to test here.
  • blob:none + sparse-checkout: .github keeps the clone small; git diff works on commit objects (lazily fetching blobs anonymously, fine for this public repo), so the sparse working tree doesn't matter.
  • The trap 'rm -rf "$src"' EXIT reliably removes the clone on every exit path (including the SHA-mismatch exit 1), keeping it out of the tree Claude reads.
  • set -euo pipefail with the grep -c ... || true guard is handled correctly (grep exits 1 on no matches).
  • The vendor split is sound: pr.diff excludes **/vendor/** (incl. modules.txt), while the summary reads **/vendor/modules.txt separately — no double-counting, go.mod/go.sum stay in the diff. The sed correctly excludes ## explicit lines (requires a space after #) and captures only +#/-# module-version headers.
  • The stale-head guard (rev-parse HEAD == $SHA) correctly protects against a force-push between the gate and this run.

One non-blocking nit (posted inline): the per-module-tree header is derived from the +++ b/... line, so a PR that deletes an entire vendored tree (+++ /dev/null) would mis-attribute its removed module lines in the advisory summary. Cosmetic only — it feeds the skip heuristic, which is backed by the nightly full-suite re-run.

No correctness, security, or performance blockers. LGTM.
· pr-7012

git -C "$src" diff "$mb" HEAD -- . ":(glob,exclude)**/vendor/**" > "$out/pr.diff"
# Vendored modules added, removed or bumped, from vendor/modules.txt.
vendored=$(git -C "$src" diff --unified=0 "$mb" HEAD -- ':(glob)**/vendor/modules.txt' \
| sed -nE -e 's#^\+\+\+ b/(.*)/modules\.txt$#\n\1:#p' -e 's#^([-+])\# (.*)$# \1 \2#p')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor / non-blocking: the dir header is derived only from the +++ b/.../modules.txt line. If an entire vendored tree is removed in the PR, that line is +++ /dev/null, so no dir: header is emitted and the - github.com/... module lines get attributed to the previous module's section (or to none). Newly-added and bumped modules.txt are fine — only the whole-dir-deletion case is affected. Since this only feeds an advisory summary (and the skip heuristic is backed by the nightly full-suite net), it's cosmetic, not a correctness problem for the gating.

@eriknordmark
eriknordmark merged commit e6f1a74 into lf-edge:master Nov 3, 2026
41 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants