Skip to content

evetest: reduce PR noise from CI comments - #7045

Merged
rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-stop-spamming
Nov 5, 2026
Merged

rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-stop-spamming

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

The evetest CI jobs post a lot of comments on PRs (see #7037 for an example):
a failure analysis after every failed suite run, and a long comment listing
every skipped test with its reasoning. This reduces the noise.

  • The failure analysis is now posted as a PR comment only if Claude judges
    at least one failure of the suite as related to the PR (RELATED: yes or
    likely). The judgment is made from the PR diff and the nightly ledger,
    and Claude is told to default to unlikely unless it can name the changed
    code that explains the failure. The comment lists only the related
    failures. Unrelated failures stay visible through the red check, the job
    summary and the claude-analysis-<suite> artifact, which still contain
    the full analysis. Collapsing of stale analysis comments is unchanged.
  • The skipped-tests comment is now a short note with the number of skipped
    names, a link to the run summary and the /evetest usage hint. The full
    list of skipped tests and the reasoning moved to the job summary of the
    run. The comment is still re-created at the end of the conversation on
    each run.

How to test and validate this PR

Not tested end-to-end yet; the workflow changes only run on a PR with the
evetest gate enabled. Locally checked that the YAML parses and that
live-investigation-watcher.sh passes bash -n. To validate, observe the
evetest runs of this PR and of the next PRs:

  • A skipped-tests comment with a count and a link, and the reasoning in the
    run summary.
  • A failing suite caused by the PR gets an analysis comment; a failure
    unrelated to the PR gets none.

Changelog notes

None. CI-only change, no user-facing changes.

PR Backports

This PR is not going to be backported on its own. All evetest workflow files
will be backported together once they work reliably.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

This is a CI-only change: there is no device-facing code to test on amd64 or
arm64, and no documentation covers these PR comments.

🤖 Generated with Claude Code

@milan-zededa milan-zededa added the testing New test or test framework changes label Nov 5, 2026
@github-actions
github-actions Bot requested a review from europaul November 5, 2026 14:30
Post the failure analysis on a PR only for failures Claude judges
related to the PR (RELATED: yes|likely), based on the PR diff and the
nightly ledger. Unrelated failures stay visible through the red check
and the job summary, which still hold the full analysis.

Shorten the skipped-tests comment to a count and a link; the full list
of skipped tests and the reasoning moves to the run's job summary.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa force-pushed the evetest-stop-spamming branch from 1b6d808 to b15a6a1 Compare November 5, 2026 14:33
@rene
rene merged commit 86fb231 into lf-edge:master Nov 5, 2026
40 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants