Skip to content

evetest: show failure analysis in the Allure report - #7002

Merged
rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:allure-failure-analysis
Oct 31, 2026
Merged

rene merged 1 commit into
lf-edge:masterfrom
milan-zededa:allure-failure-analysis

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

Clicking a failed test in the nightly Allure report showed only gotestsum's
generic "Failed" message plus the raw log trace; the real failure message and
Claude's conclusion ended up only in the nightly ledger and the job's step
summary, so the report itself carried no information about why a test failed.

Inject both into the JUnit XML before it is uploaded for the report: the new
inject-junit-analysis.py rewrites each failing testcase's failure message
with the failure message and conclusion that live-investigation-watcher.sh
recorded in live-investigation.md, so Allure's test overview page displays
them. The suite-level testcase fails whenever any of its subtests do, so it
gets every recorded analysis, each prefixed with the subtest's name.

PR dependencies

None.

How to test and validate this PR

Tested on my fork: a nightly run restricted to a single suite
(TestTelemetrySuite) with two intentionally broken assertions. The published
Allure report shows, for each failed test (and aggregated on the suite-level
entry), the real assertion failure plus Claude's conclusion on the test
overview page:
https://milan-zededa.github.io/eve/test/master/runs/57/

To validate locally, run the injector on a gotestsum-generated JUnit XML and a
live-investigation.md from a failed suite run:

python3 .github/actions/evetest-run-suite/inject-junit-analysis.py \
  <suite>.xml live-investigation.md <suite-name>

and inspect the rewritten <failure message="..."> attributes, or feed the
result to allure generate.

Changelog notes

None. CI and test tooling only.

PR Backports

Not to be backported as an individual PR: the evetest CI workflow files will
be backported as a whole once everything works well.

Checklist

  • I've provided a proper description

  • I've added the proper documentation

  • I've tested my PR on amd64 device

  • I've tested my PR on arm64 device

  • I've written the test verification instructions

  • I've set the proper labels to this PR

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Unchecked items do not apply: CI and test tooling only, no device impact.

🤖 Generated with Claude Code

Clicking a failed test in the nightly Allure report showed only
gotestsum's generic "Failed" message plus the raw log trace; the real
failure message and Claude's conclusion ended up only in the nightly
ledger and the job's step summary.

Inject both into the JUnit XML before it is uploaded for the report:
the new inject-junit-analysis.py rewrites each failing testcase's
failure message with the failure message and conclusion that
live-investigation-watcher.sh recorded in live-investigation.md, so
Allure's test overview page displays them. The suite-level testcase
fails whenever any of its subtests do, so it gets every recorded
analysis, each prefixed with the subtest's name.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added the testing New test or test framework changes label Oct 31, 2026
@github-actions
github-actions Bot requested a review from europaul October 31, 2026 16:44
@rene
rene merged commit 2240189 into lf-edge:master Oct 31, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants