Skip to content

fix(ci): repair the weekly trusted-packages download - #540

Open
presendapp wants to merge 2 commits into
elementsinteractive:mainfrom
presendapp:fix/weekly-download
Open

presendapp wants to merge 2 commits into
elementsinteractive:mainfrom
presendapp:fix/weekly-download

Conversation

@presendapp

@presendapp presendapp commented Sep 27, 2026 •

Copy link
Copy Markdown

Disclosure: I'm the developer of Presend; this PR has nothing to do with it. While measuring twyn against current package rankings, I noticed that the trusted-package lists are stale (dependencies/pypi.json dated 2026-03-02, npm.json 2025-12-08), although the "Weekly download" workflow shows a green run every Monday.

Why

  • download_packages.py and utils.py import InvalidJSONError from requests, but the workflow installs only the download group (click, httpx, stamina), so the script dies at import with ModuleNotFoundError: No module named 'requests'. The dependency tests don't catch it because that CI job also installs the project, which depends on requests. scripts/exceptions.py already defines its own InvalidJSONError, which is clearly the one meant.
  • With the import fixed, PyPI still fails: https://hugovk.github.io/top-pypi-packages/... now answers 301 to https://hugovk.dev/..., and httpx.Client doesn't follow redirects by default. Since only 5xx errors are re-raised, the code goes on to response.json() on the redirect's empty body and ends in InvalidJSONError.
  • continue-on-error: true on the download step turns both failures into a green job; the push step then finds no changes.

Changes

  • Import InvalidJSONError from scripts.exceptions, so the scripts no longer need requests.
  • Point TOP_PYPI_SOURCE to https://hugovk.dev/... (and the test that checks it).
  • Remove continue-on-error from the download step. fail-fast: false already keeps one ecosystem's failure from cancelling the others; the failure is simply visible now.

Tested in an environment with only the download group plus pytest, pytest-cov and freezegun, i.e. without requests: before, test collection fails with the ModuleNotFoundError; after, dependencies/tests passes (11 tests), and download_packages.py download pypi / download npm both save 15,000 packages. The lists themselves are left out of this PR; the scheduled run will refresh them.

Effect: over today's top 15,000 PyPI packages, twyn run flags 407 of them with the committed (March) list, all absent from it (e.g. httpx2, httpcore2, fastspec); with a freshly downloaded list, none, by construction since the reference is that ranking, and detection of 12 known PyPI typosquats is unchanged (11/12). On npm, the refresh alone doesn't reduce noise against the npm-high-impact list (218 → 284 findings), as ecosyste.ms ranks packages differently; I haven't looked into that part.

Added after review: follow_redirects=True on the client, and re-raising non-5xx HTTP errors instead of falling through to response.json(), so a source that moves is followed, and any other HTTP error fails with a clear message.

@sdn4z

sdn4z commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Hello @presendapp! Thanks for taking the time to look at it and create this PR.

Would you mind adding to this PR the two changes that you suggest for a follow-up? They're rather small and could as well fit here.

…ad script

A source that moves (as top-pypi-packages did) is now followed, and any other non-success status fails with httpx's own message (status and URL) instead of falling through to response.json() on an empty body. Both new tests fail without the change.
@github-actions github-actions Bot added the fix label Oct 1, 2026
@presendapp

Copy link
Copy Markdown
Author

Done in f5a2c4f:

  • follow_redirects=True on the client, so a source that moves is followed.
  • Non-5xx HTTP errors are re-raised as is (httpx.HTTPStatusError, whose message gives the status and URL) instead of falling through to response.json(). They are not in RETRY_ON, so they fail on the first attempt.

Two tests added: a 404 raises after a single request without reading the body, and a 301 is followed end to end through _run (mocked at the transport level, so the client's real redirect handling runs). Both fail without the change. dependencies/tests: 13 passed, coverage 99.28%, with the same setup as test.yml (--group download --group dev, Python 3.14). I also updated the PR description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants