diff --git a/plan/tickets/76/plan.md b/plan/tickets/76/plan.md new file mode 100644 index 0000000..0e47acf --- /dev/null +++ b/plan/tickets/76/plan.md @@ -0,0 +1,258 @@ +# Issue #76 — Publish releases to PyPI + +**Status:** triaged 2026-10-05; open questions resolved the same day. Implementation has not started. + +## Overview + +Enable the `publish-pypi` job in `release.yml` so that a `rel-X.Y.Z` tag publishes the same wheel and +sdist to PyPI that it already attaches, signed, to the GitHub Release. Users (a SLAC user asked for +this) can then `pip install "dp-python-lib[analysis]"` instead of downloading a wheel by hand. The +GitHub Release, its Sigstore bundles, and `SHA256SUMS` are unchanged; PyPI becomes a second channel +for byte-identical files, and that identity is checked after every publish. + +The work is part account setup (done by hand on pypi.org, test.pypi.org, and the repo's settings) and +part repo change: the workflow, a TestPyPI rehearsal path, the install docs, and the release checklist. + +## Background / triage findings + +- **The draft's premise holds: the job is nearly complete.** `publish-pypi` already uses Trusted + Publishing (`id-token: write`, environment `pypi`), strips `SHA256SUMS` / `RELEASE_NOTES.md` / + `*.sigstore.json` before upload, sets `attestations: true`, and pins `pypa/gh-action-pypi-publish` + by SHA. It has been gated `if: false` since the workflows were added (114c12f, #29). The repo is + public, so environment protection rules are available. +- **The name is free.** `https://pypi.org/pypi/dp-python-lib/json` and its test.pypi.org counterpart + both return 404 (checked 2026-10-05). +- **Correction to the draft: a name cannot be claimed on PyPI ahead of an upload.** The draft's first + task, "check that `dp-python-lib` is available on PyPI and claim it", has no direct equivalent. The + mechanism is a *pending* trusted publisher, registered from the PyPI account page before the + project exists. It does **not** reserve the name; the project is created by the first upload made + through it. PyPI's similar-name check also only runs at upload time. The risk of losing the name + before the next release is low, and the TestPyPI rehearsal (D3) exercises the same path. +- **Correction to the draft: a TestPyPI rehearsal from `workflow_dispatch` would be rejected as + things stand.** A dispatch build is versioned by setuptools-scm with a local segment + (`1.16.1.dev61+g496f0e0` at `496f0e0`), and PyPI and TestPyPI both refuse local versions. + `SETUPTOOLS_SCM_OVERRIDES_FOR_DP_PYTHON_LIB='{local_scheme="no-local-version"}'` yields + `1.16.1.dev61`, which they accept (verified locally with setuptools-scm 10.3.4). +- **Missing from the draft: the README is PyPI's project page and has relative links.** + `readme = "README.md"` becomes the long description, and its relative links (`doc/cookbook/*.md`, + `README.env`) 404 on pypi.org. `twine check --strict` does not catch this, since it validates + markup, not targets. +- **Missing from the draft: the cookbook already claims a PyPI install.** `doc/cookbook/conventions.md` + ("Optional dependencies") and `doc/cookbook/query.md` ("Getting results into pandas and NumPy") both + say `pip install dp-python-lib[analysis]`, which fails today. Unquoted, it also fails in zsh, the + macOS default shell, because of the brackets. `README.md` (Installation, and the roadmap bullet + "Publishing to PyPI"), `README.env` (Notes: "Publishing to PyPI is not enabled yet"), the + `## Installing` section of `doc/release-notes/NEXT.md`, and `CLAUDE.md` (Continuous Integration and + Releases) all describe the current state and need updating. +- **Missing from the draft: the PyPI job runs in parallel with the GitHub Release.** It has + `needs: build` only, so the one upload that can never be replaced may land before the GitHub Release + has been published and its body verified, or after it has failed. +- **Missing from the draft: a job naming an environment that does not exist creates it, unprotected.** + So the `pypi` environment and its reviewer must exist **before** the workflow change merges; otherwise + the first tag push creates `pypi` with no approval gate. +- **The sdist contains every tracked file** (setuptools-scm's file finder): `plan/`, `.github/`, + `CLAUDE.md`, `specifications/`, `tests/`, and the one tracked `.dev/` file, about 500 KB. That is the + same sdist the GitHub Release already ships and signs, so it is not changed here (see Out of scope). +- **No open ticket folds in.** #68 (paging for `queryRequestStatus()`) waits on osprey-dcs/dp-grpc#165 + and osprey-dcs/dp-service#302. #61's `py.typed` marker becomes more useful once the package is on + PyPI, but it still waits on a stub sync carrying `.pyi` files, so it stays there. No other osprey-dcs + repo has a PyPI ticket. + +## Design decisions + +**D1. Ownership: two personal PyPI owners now, an organization later if wanted.** The project is +created under Craig's account through the pending publisher. A second owner, with 2FA (which PyPI +requires), is added as soon as the project exists, which is right after the first upload. Rejected: +waiting for a PyPI organization, because PyPI's admins review organization requests and the SLAC user +would wait on that. A project can be moved into an organization later without republishing. +*(Decided 2026-10-05.)* + +**D2. One required reviewer on the `pypi` environment: Craig.** The environment also gets a deployment +rule allowing only `rel-*` tags, as a second gate besides the job's `if:`. Since the same person pushes +the tag and approves the deployment, GitHub's "prevent self-review" option must stay **off**. The +approval is a deliberate pause before an upload that can never be replaced, not a second pair of eyes. +That bus-factor-of-one is accepted: if the approver is unavailable, the GitHub Release still ships and +PyPI follows later by approving the pending deployment or re-running the job. That fallback lasts only +as long as the `release-dist` artifact, which `publish-pypi` downloads and which is uploaded with +`retention-days: 7`, while GitHub holds a deployment for approval for up to 30 days. Retention +therefore goes to 30 days to match the approval window. A full workflow re-run is no substitute once +the artifact has expired: it rebuilds and re-signs, so the files would no longer match the +`SHA256SUMS` and bundles already on the GitHub Release, and it re-runs the release step against a +release that already exists. *(Decided 2026-10-05; the retention window was added in review of PR #78.)* + +**D3. A permanent TestPyPI rehearsal behind an opt-in dispatch input.** `workflow_dispatch` gains a +boolean input `testpypi` (default `false`). When it is set, a `publish-testpypi` job uploads the +rehearsal build to test.pypi.org through its own trusted publisher and a `testpypi` environment (no +reviewer). The dispatch build drops the local version segment using the setuptools-scm override above, +which is set on the build step only for `workflow_dispatch`, so tag builds are unaffected. This changes +the workflow's stated invariant from "a dispatch never publishes" to "a dispatch never publishes to PyPI +or a GitHub Release", and the header comment says so. Rejected: a one-off twine upload with an API +token, which checks rendering but not the OIDC/trusted-publisher configuration that is the actual risk; +and skipping TestPyPI, which makes the first real release the first test. TestPyPI is a sandbox, but +its versions cannot be re-uploaded either. *(Decided 2026-10-05.)* + +**D3a. Both upload steps set `skip-existing: true`, and D5 is what keeps a conflict loud.** The upload +sends files one at a time, so a wheel can land and the sdist fail. Without `skip-existing`, every +"Re-run failed jobs" after that fails on the wheel with "File already exists", and the release can never +be completed on PyPI. A re-run downloads the same `release-dist` artifact rather than rebuilding, so a +skipped file is the same file; D5's digest check then compares every file on the index, skipped ones +included, against `SHA256SUMS`. A genuine conflict (a rebuilt sdist at a version already on the index, +from a fresh dispatch at the same commit, say) still fails, at the digest check, with both digests +printed. Rejected: leaving `skip-existing` off so the upload itself fails loudly, which makes a partial +upload unrecoverable without a new version. *(Added in review of PR #78.)* + +**D4. PyPI publishes after the GitHub Release, not beside it.** `publish-pypi` gets +`needs: [build, publish-github-release]`. The GitHub Release can be edited or deleted; a PyPI file +cannot. Publishing PyPI last means a failure anywhere earlier, including the body check, stops before +the irreversible step. If the PyPI job itself fails, "Re-run failed jobs" retries it with the same tag +and the same OIDC identity. Rejected: keeping the jobs parallel, which saves about a minute. + +**D5. After publishing, check that PyPI serves the same bytes as the GitHub Release.** A final step +polls the index's JSON API (`/pypi/dp-python-lib//json`, retrying for a few minutes while the +CDN catches up) and compares each file's `digests.sha256` with `SHA256SUMS`. The existing step that +deletes `SHA256SUMS` from `dist/` before upload therefore moves a copy aside first. The same step runs +after the TestPyPI upload against test.pypi.org. This closes the loop the same way the body check does +for the GitHub Release, and it is what makes "the PyPI files are the signed files" a checked statement, +so `README.env` can tell a user to verify a `pip download` against `SHA256SUMS` from the GitHub +Release. That recipe must be spelled out, because the obvious one fails: `SHA256SUMS` lists both the +wheel and the sdist, so a bare `sha256sum -c` over a directory holding only the wheel reports the sdist +missing and exits non-zero, and `pip download` without `--no-deps` also fetches every dependency. The +documented form is: +```bash +pip download --no-deps "dp-python-lib==X.Y.Z" +sha256sum --ignore-missing -c SHA256SUMS +``` +Rejected: a post-publish `pip install` smoke test, which the build job already does +against the same wheel; it would add resolver and CDN flakiness without checking anything new. + +**D6. Only the next release goes to PyPI.** rel-1.16.0 and earlier are not backfilled: that would be a +hand upload with an API token and no PEP 740 attestations, of a release carrying the #19 +config-precedence bug. The first PyPI version is whatever the next tag is. *(Decided 2026-10-05.)* + +**D7. README links become absolute `blob/main` URLs.** They then work on GitHub, on pypi.org, and in +the sdist alike. This includes the fragment-only links (`[current state](#current-state)`, +`[TODO](#todo)`), which become `blob/main/README.md#...`: whether PyPI's renderer gives headings `id`s +was not confirmed, and an absolute link works either way. The TestPyPI rehearsal's link check covers +them. A version's PyPI page will link to the docs as they are on `main`, not as they were +at that release; that drift is accepted, because the alternative (rewriting links at build time) adds +moving parts to the release build for a landing page. `[project.urls]` gains `Documentation` +(`doc/cookbook/README.md` on `main`) and `Changelog` (the GitHub Releases page) for the PyPI sidebar. + +**D8. Docs lead with PyPI and keep the verified-download path second.** The install commands are +quoted (`pip install "dp-python-lib[analysis]"`) so they work in zsh. Editable installs stay +documented for development. + +## Implementation tasks + +### Manual setup (before the workflow PR merges; see Dependencies) + +1. **PyPI:** under Craig's account, add a pending trusted publisher: project `dp-python-lib`, owner + `osprey-dcs`, repository `dp-python-lib`, workflow `release.yml`, environment `pypi`. +2. **TestPyPI:** a separate account on test.pypi.org; the same pending publisher with environment + `testpypi`. +3. **GitHub environments** (Settings → Environments): + - `pypi`: required reviewer Craig, "prevent self-review" off, deployment branches and tags limited + to the tag pattern `rel-*`. + - `testpypi`: no reviewer and no ref restriction, so a rehearsal can run from a PR branch. +4. **After the first real upload:** add the second PyPI owner (D1) and confirm the project shows the + attestations and the trusted publisher (no API tokens). + +### `.github/workflows/release.yml` + +- Header comment: state the revised dispatch invariant (D3). +- `workflow_dispatch.inputs.testpypi`: boolean, default `false`, description naming test.pypi.org. +- Build step: set `SETUPTOOLS_SCM_OVERRIDES_FOR_DP_PYTHON_LIB: '{local_scheme="no-local-version"}'` + only when `github.event_name == 'workflow_dispatch'`, with a comment explaining why (PyPI rejects + local versions). +- `publish-pypi`: `if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/rel-')`, + `needs: [build, publish-github-release]`; replace the "WIRED UP BUT INTENTIONALLY DISABLED" block + with a short comment that names the one-time setup (pending publisher, environment) and points to + this plan. Keep `attestations: true` explicit. +- New `publish-testpypi` job: `needs: build`, + `if: github.event_name == 'workflow_dispatch' && inputs.testpypi`, environment `testpypi` (url + `https://test.pypi.org/p/dp-python-lib`), `repository-url: https://test.pypi.org/legacy/`, otherwise + the same steps as `publish-pypi`. The shared steps may stay duplicated; two copies of three steps is + clearer than a composite action. +- `build`: raise the `release-dist` upload's `retention-days` from 7 to 30, with a comment tying it to + the approval window (D2). +- Both jobs: `skip-existing: true` on the upload step, with a comment pointing at D3a. +- Both jobs: move `SHA256SUMS` aside before cleaning `dist/`, then the D5 digest check after upload. + The version comes from the wheel filename, which covers both tag and dispatch builds. Retry the JSON + fetch for about five minutes; fail on a missing file, an extra file, or any digest mismatch, and + print both sides. +- Every new `uses:` is SHA-pinned with a `# vX.Y.Z` comment (the CLAUDE.md grep must stay empty). + +### `pyproject.toml` + +- `[project.urls]`: add `Documentation` and `Changelog` (D7). + +### Docs + +- `README.md`: Installation leads with `pip install dp-python-lib` / `pip install "dp-python-lib[analysis]"`, + then the editable and `[dev]` installs for development, then the GitHub Release / `README.env` path for + verified downloads. Remove the roadmap bullet "Publishing to PyPI". Make every relative link absolute, + fragment-only ones included (D7). Quote the editable installs' extras too (`pip install -e ".[analysis]"`, + `pip install -e ".[dev]"`): unquoted, they fail in zsh just as the PyPI form does. +- `README.env`: replace the Notes line about PyPI not being enabled. Add a short section explaining that + the PyPI files are byte-identical to the release's (checked by the workflow), how to verify a + `pip download` against the release's `SHA256SUMS` and Sigstore bundles using exactly D5's commands + (`--no-deps`, `--ignore-missing`), and that PyPI's PEP 740 attestations are a second, independent + provenance record. +- `doc/cookbook/conventions.md`, `doc/cookbook/query.md`: quote the extra. `doc/cookbook/README.md`: quote + `pip install -e ".[dev]"` in its setup section, and check that section for anything that assumes a source + checkout for ordinary use. +- `doc/release-notes/NEXT.md`: a new section, "Install from PyPI (#76)", plus a Contents entry. Update + `## Installing` to lead with `pip install dp-python-lib`, keeping the wheel form. No version may be + named (the checker enforces this). Add a step to "Cutting the release" after the tag push: approve + the `pypi` deployment once the GitHub Release job is green, then confirm the digest check passed. +- `CLAUDE.md`, Continuous Integration and Releases: replace "A PyPI publish job is wired up but disabled" + with the enabled flow (order, approval, digest check, TestPyPI input), and add the approval step to + "Cutting a release", including the warning that the environment must exist before a job names it. + +### Verification for the PR + +- `grep` for the pinning check, `ruff`, `mypy src/`, the cookbook checker, and + `.github/scripts/check-release-notes.py` all pass. +- `actionlint` (if available) over `release.yml`. +- Run the workflow by dispatch from the PR branch with `testpypi=true`: the build has no local version, + the upload succeeds, the digest check passes, and the test.pypi.org page renders the README with + working links. Then, in a clean venv, install the rehearsal build and import `MldpClient`: + ```bash + pip install --index-url https://test.pypi.org/simple/ \ + --extra-index-url https://pypi.org/simple/ "dp-python-lib==" + ``` + If GitHub rejects the input because `main`'s `release.yml` lacks it, run the rehearsal + immediately after merge instead. +- A dispatch with `testpypi=false` still stops after build/sign, and `publish-pypi` shows as skipped. + +## Out of scope + +- How GitHub Releases are built, signed, or verified (unchanged, as the draft says). +- Trimming the sdist (`plan/`, `.github/`, `CLAUDE.md`, and so on). It would make the PyPI sdist differ + from the one already shipped and signed on GitHub Releases, so it is its own ticket if wanted. +- Backfilling rel-1.16.0 or earlier (D6). +- A PyPI organization (D1); it can be adopted later without republishing. +- `py.typed`: #61. +- Publishing dp-grpc's Python stubs as their own package: not proposed anywhere, and dp-python-lib + vendors them. + +## Dependencies and sequencing + +1. Manual setup steps 1–3, **before** the workflow PR merges. Step 3 is the hard ordering constraint: + without it, the first tag push auto-creates an unprotected `pypi` environment. +2. The workflow and docs PR (`Closes #76`), rehearsed on TestPyPI from its branch before merge. +3. The next release cut, following the updated checklist; manual step 4 right after. + +The release train order (dp-grpc → dp-service → dp-desktop-app → dp-python-lib → data-platform) is +unchanged, and so is the rule that the release-time `grpc-sync-*` PR merges before dp-python-lib is +tagged. Nothing here depends on #61, #68, or any upstream repo. + +## Open questions + +All resolved 2026-10-05: + +1. **Who owns the project on PyPI?** Two personal owners now (D1). *Remaining action:* name the second + owner before the first release. +2. **Who approves a publish?** Craig alone (D2). +3. **Rehearse on TestPyPI?** Yes, as a permanent opt-in dispatch input (D3). +4. **Which release goes up first?** The next one; no backfill (D6).