Skip to content

[13.4-stable] Fix bugs in unit tests - Race when accessing stats of drona request - #119

Merged
rene merged 3 commits into
lf-edge:13.4-stablefrom
christoph-zededa:backport-51-13.4-stable
Oct 1, 2026
Merged

rene merged 3 commits into
lf-edge:13.4-stablefrom
christoph-zededa:backport-51-13.4-stable

Conversation

@christoph-zededa

@christoph-zededa christoph-zededa commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

Backport of #51 to 13.4-stable: the one commit of that PR, cherry-picked with
-x, and on top of it the two nettrace test-skip commits described further
down. The #51 commit takes the request's read lock in DronaRequest.GetDoneParts
and locks the stats updates in the S3 and Azure transports, so that a reader of
the transfer statistics no longer races the stats goroutine that writes them.

Why now: EVE's backport of its downloader fixes to this branch (lf-edge/eve#6687)
adds unit tests that run a real HTTP transfer through the transport under the
race detector, and they fail on this branch with GetDoneParts reading
doneParts while the stats goroutine writes it. 16.0-stable and main have
had this fix since August 2025; this branch and 14.5-stable never received it.
The race existed here all along, it was just never exercised under -race.

One conflict in zedUpload/datastore.go: the lock around the write of
doneParts that the commit adds to reqPostSize is already present on this
branch inside recordStats, from the backport of #103, so this branch's
version is kept there; everything else applied cleanly.

On top of that, the two commits of #84 and #82 ("nettrace: skip TLSCertErrors
if host unreachable"), which 16.0-stable received the same way in #106:
badssl.com intermittently resets the TLS handshake, or never answers it,
before it presents its certificate, which failed TestTLSCertErrors on this
branch with an empty TLSTunnels list while running the tests for this PR.
With them each endpoint is its own subtest and a case whose handshake never
reached the certificate is skipped instead of failed. The first is a clean
cherry-pick; the second conflicted on the Let's Encrypt issuer pattern with
this branch's earlier intermediate-name fix from #108 and was resolved to
main's current text, so the helper and TestTLSCertErrors now match main's
apart from the session-UUID argument that NewHTTPClient does not take on
this branch. 17.0-stable and 14.5-stable do not carry these two commits yet
(#118 does not include them).

How to test and validate this PR

go test -race ./zedUpload/ passes on this branch. With the EVE backport's
drain_test.go vendored against this change the race report disappears.

go test ./nettrace/ -run TestTLSCertErrors passes as three subtests; when
badssl.com does not answer, the affected case is skipped rather than failed
(seen locally with the untrusted-root endpoint while the other two passed).

PR Backports

Checklist

  • I've provided a proper description
  • I've added the proper documentation — none needed
  • I've tested my PR on amd64 device — library change, unit tests with the race detector
  • I've tested my PR on arm64 device — not applicable
  • I've written the test verification instructions
  • I've set the proper labels to this PR — left to the maintainers
  • I've checked the boxes above, or I've provided a good reason why I didn't check them

For backport PRs:

  • I've added a reference link to the original PR
  • PR's title follows the template

🤖 Generated with Claude Code

- Adding calculation of md5checksum if not present in resp.ContentMD5 for Azure metadata

Signed-off-by: Ioannis Sfakianakis <jsfakas@gmail.com>
(cherry picked from commit de07a00)
eriknordmark and others added 2 commits October 1, 2026 13:12
TestTLSCertErrors validates that the trace captures the offending peer
certificate when a TLS handshake fails against the badssl.com endpoints.
Those endpoints are only reachable over the public internet, so on a
runner that cannot complete the handshake (timeout, reset, unreachable)
no certificate is captured and the test failed for a reason unrelated to
the behavior under test.

Factor the three repeated request blocks into a helper that still asserts
the full cert detail when the handshake reaches the certificate exchange,
but skips the test when no peer certificate could be observed. The flake
is environmental, not a defect, so a skip is the correct outcome there.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 3227cfe)
TestTLSCertErrors validates that the trace captures the offending peer
certificate when a TLS handshake fails against the badssl.com endpoints.
Those endpoints are only reachable over the public internet and
intermittently reset the handshake before presenting their certificate;
when that happens no certificate is captured and the test failed for a
reason unrelated to the behavior under test.

Exercise each endpoint as its own subtest so an intermittent reset only
skips that case while the others still run, and skip a case (rather than
fail it) whenever no peer certificate could be observed. The flake is
environmental, not a defect, so a skip is the correct outcome there.

Signed-off-by: eriknordmark <erik@zededa.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit f808748)
@christoph-zededa
christoph-zededa marked this pull request as ready for review October 1, 2026 11:26
@rene
rene merged commit eddbd83 into lf-edge:13.4-stable Oct 1, 2026
4 checks passed
rene pushed a commit to lf-edge/eve that referenced this pull request Oct 5, 2026
Update eve-libs from 06fd0b63af43 to eddbd838d583, the tip of its
13.4-stable branch, which adds the three commits of
lf-edge/eve-libs#119: the backport of lf-edge/eve-libs#51, followed by
the two nettrace test-skip commits of lf-edge/eve-libs#84 and
lf-edge/eve-libs#82.

The race fix makes GetDoneParts take the request's read lock, which the
stats updater already holds for writing when it records the parts, and
has the AWS and Azure transports update the request's size and status
under the lock as well. Pillar runs its unit tests under the race
detector, and TestTransportSendersFinishAfterDownload, which #6687
brought to this branch, drives a request through a real transport whose
stats updater keeps writing those fields while the downloader reads them
from the progress messages; the CI run of that PR reported the race.

The two nettrace commits only change its tests, which are not vendored.
Pillar is the only module on this branch that pins eve-libs; it moves to
the new version, and only the four zedUpload files that the backport
touches are re-vendored. No transitive dependency moves.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants