fix(ci): repair the weekly trusted-packages download - #540
Open
presendapp wants to merge 2 commits into
Open
presendapp wants to merge 2 commits into
presendapp wants to merge 2 commits into
Conversation
Collaborator
|
Hello @presendapp! Thanks for taking the time to look at it and create this PR. Would you mind adding to this PR the two changes that you suggest for a follow-up? They're rather small and could as well fit here. |
…ad script A source that moves (as top-pypi-packages did) is now followed, and any other non-success status fails with httpx's own message (status and URL) instead of falling through to response.json() on an empty body. Both new tests fail without the change.
Author
|
Done in f5a2c4f:
Two tests added: a 404 raises after a single request without reading the body, and a 301 is followed end to end through |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclosure: I'm the developer of Presend; this PR has nothing to do with it. While measuring twyn against current package rankings, I noticed that the trusted-package lists are stale (
dependencies/pypi.jsondated 2026-03-02,npm.json2025-12-08), although the "Weekly download" workflow shows a green run every Monday.Why
download_packages.pyandutils.pyimportInvalidJSONErrorfromrequests, but the workflow installs only thedownloadgroup (click,httpx,stamina), so the script dies at import withModuleNotFoundError: No module named 'requests'. The dependency tests don't catch it because that CI job also installs the project, which depends onrequests.scripts/exceptions.pyalready defines its ownInvalidJSONError, which is clearly the one meant.https://hugovk.github.io/top-pypi-packages/...now answers301tohttps://hugovk.dev/..., andhttpx.Clientdoesn't follow redirects by default. Since only 5xx errors are re-raised, the code goes on toresponse.json()on the redirect's empty body and ends inInvalidJSONError.continue-on-error: trueon the download step turns both failures into a green job; the push step then finds no changes.Changes
InvalidJSONErrorfromscripts.exceptions, so the scripts no longer needrequests.TOP_PYPI_SOURCEtohttps://hugovk.dev/...(and the test that checks it).continue-on-errorfrom the download step.fail-fast: falsealready keeps one ecosystem's failure from cancelling the others; the failure is simply visible now.Tested in an environment with only the
downloadgroup plus pytest, pytest-cov and freezegun, i.e. withoutrequests: before, test collection fails with theModuleNotFoundError; after,dependencies/testspasses (11 tests), anddownload_packages.py download pypi/download npmboth save 15,000 packages. The lists themselves are left out of this PR; the scheduled run will refresh them.Effect: over today's top 15,000 PyPI packages,
twyn runflags 407 of them with the committed (March) list, all absent from it (e.g.httpx2,httpcore2,fastspec); with a freshly downloaded list, none, by construction since the reference is that ranking, and detection of 12 known PyPI typosquats is unchanged (11/12). On npm, the refresh alone doesn't reduce noise against the npm-high-impact list (218 → 284 findings), as ecosyste.ms ranks packages differently; I haven't looked into that part.Added after review:
follow_redirects=Trueon the client, and re-raising non-5xx HTTP errors instead of falling through toresponse.json(), so a source that moves is followed, and any other HTTP error fails with a clear message.