Skip to content

evetest-pr: retry status posting, collapse stale evetest comments - #6975

Merged
milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pr-status-retries
Oct 30, 2026
Merged

milan-zededa merged 3 commits into
lf-edge:masterfrom
milan-zededa:evetest-pr-status-retries

Conversation

@milan-zededa

@milan-zededa milan-zededa commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

Three fixes to the evetest-pr workflow:

  1. Retry commit status posting. The "Report suite statuses" step posts one
    commit status per suite in a quick burst. A single transient 5xx from the
    GitHub status API aborted the whole discover-suites job and, with it, the
    evetest run. Statuses are now posted with retries and backoff on 5xx, 429
    and 403.
  2. Keep a single, current skipped-tests PR comment. Claude used to post a
    new PR comment on every evetest run that skipped tests, so reruns piled up
    stale comments. Claude now writes the comment body to
    .evetest-skip/comment.md and the workflow posts it. Any previous
    evetest skip comment is deleted and the new one is posted at the end of the
    conversation. If the latest run skips nothing (e.g. after /evetest all),
    no comment is left on the PR.
  3. Collapse stale failure analysis comments. Every failed suite posted a
    new Claude analysis comment, so reruns piled up stale ones. Each comment is
    now tagged with a per-suite marker; at the end of a run the suite's
    previous comment is collapsed (as resolved if the run passed, as outdated
    if it failed again) and a new one is posted only if the run failed.

PR dependencies

Based on the evetest-pr dedupe change that is already in master (389299ac7d).

How to test and validate this PR

CI workflow change; it can only be exercised on a real PR:

  • Trigger evetest on a PR where Claude skips some suites, then rerun it:
    there must be exactly one skip comment, at the end of the conversation,
    describing the latest run.
  • Comment /evetest all: the previous skip comment must be removed.
  • Retry: a transient failure of the status API (seen as
    HttpError: Unexpected end of JSON input, status 500, in run 40941552841)
    must no longer fail discover-suites; a warning is logged per retry.
  • Failure analysis: let a suite fail and then rerun it. The first analysis
    comment must be collapsed as outdated when it fails again (with a new
    comment posted) and collapsed as resolved when the rerun passes.

Changelog notes

None. No user-facing changes (CI only).

PR Backports

This PR will not be backported on its own. Instead, all evetest workflow files
will be submitted to all LTS branches together once they are confirmed to work
well.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

@milan-zededa milan-zededa added the testing New test or test framework changes label Oct 29, 2026
@github-actions
github-actions Bot requested a review from europaul October 29, 2026 20:18
@milan-zededa milan-zededa changed the title evetest-pr: retry status posting, keep a single skipped-tests comment evetest-pr: retry status posting, collapse stale evetest comments Oct 29, 2026
@milan-zededa
milan-zededa force-pushed the evetest-pr-status-retries branch from c3860eb to 9ca3217 Compare October 29, 2026 20:31
@eriknordmark

Copy link
Copy Markdown
Contributor

@claude

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @eriknordmark's task in 2m 36s —— View job


Review complete

I reviewed the three changes (status retry, single skip comment, collapsing analysis comments). Posted 3 inline comments:

  • Read the diff
  • Analyze the three changes
  • Post review feedback

Summary of findings (details inline):

  1. Skip-comment step lacks the resilience this PR adds elsewhere (evetest-pr.yml ~348) — the new "Update the skipped-tests PR comment" step runs in the same discover-suites job but makes listComments/deleteComment/createComment calls with no retry and no continue-on-error. A transient 5xx there aborts the whole run — the exact failure this PR fixes two steps below — and can even leave the PR with the old comment deleted and no replacement. (main concern)

  2. job.status inside a composite action (action.yml 383) is not a reliable mid-job failure signal; the classifier (RESOLVED vs OUTDATED) can be wrong on a repeated failure. Cosmetic only, but worth aligning with the failure() signal the post step already uses.

  3. Retrying 403 (evetest-pr.yml 400) — minor: retries genuine permission errors and ignores Retry-After in favor of fixed backoff.

The core retry logic, delete-then-repost approach, and per-suite markers otherwise look correct and match the stated intent.

Comment thread .github/workflows/evetest-pr.yml Resolved
Comment thread .github/workflows/evetest-pr.yml Resolved

@eriknordmark eriknordmark left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@milan-zededa
milan-zededa force-pushed the evetest-pr-status-retries branch from 9ca3217 to 296b9df Compare October 30, 2026 07:45
@github-actions
github-actions Bot requested a review from eriknordmark October 30, 2026 07:45
milan-zededa and others added 3 commits October 30, 2026 08:48
The "Report suite statuses" step posts one commit status per suite in a
quick burst. A single 5xx from the GitHub status API aborted the whole
discover-suites job and, with it, the evetest run. Retry with backoff on
5xx, 429 and 403 before giving up.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Claude used to post a new PR comment on every evetest run that skipped
tests, so reruns piled up stale comments. Have the workflow post the
summary instead: a marker-tagged comment is updated in place for the
latest run, and removed when the latest run skips nothing (e.g. after
"/evetest all").

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
Every failed run posted a new analysis comment, so reruns piled up stale
ones. Tag the comment with a per-suite marker and, at the end of each
run (unless cancelled), collapse the suite's previous comment: as
resolved if the run passed, as outdated if it failed again. A new
comment is posted only when the run failed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa
milan-zededa force-pushed the evetest-pr-status-retries branch from 296b9df to 8ce8572 Compare October 30, 2026 07:48
@milan-zededa
milan-zededa merged commit 6f6cd80 into lf-edge:master Oct 30, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants