Skip to content

ci(ffi): cross-platform Python wheels + PyPI publishing - #229

Merged
Nic-dorman merged 9 commits into
mainfrom
feat/python-ffi-wheels
Sep 3, 2026
Merged

Nic-dorman merged 9 commits into
mainfrom
feat/python-ffi-wheels

Conversation

@Nic-dorman

@Nic-dorman Nic-dorman commented Aug 6, 2026

Copy link
Copy Markdown
Member

Unstacked 2026-09-03: #228 merged; this PR now targets main.

What

The release pipeline for the Python bindings added in #228: reproducible per-platform wheel builds + publishing via PyPI Trusted Publishing (OIDC, no stored token).

Build scripts

CI runs the same script a developer runs locally — one source of truth:

  • build-wheel-manylinux.sh — compiles the native lib inside a manylinux_2_28 container (glibc 2.28) so the wheel installs on RHEL8 / Ubuntu 20.04+ / Debian 10+; auditwheel verifies + retags. Arg: x86_64 | aarch64.
  • build-wheel-macos.sh — builds arm64 + x86_64, lipo-fuses to a universal2 wheel (deployment target 11.0); delocate verifies self-contained.
  • build-wheel-windows.ps1 — native x86_64 DLL; delvewheel bundles non-system DLLs.

Workflow (publish-python.yml)

  • Matrix: linux x86_64 (ubuntu-latest), linux aarch64 (ubuntu-24.04-arm, native — no QEMU), macOS universal2 (macos-latest), windows amd64.
  • Triggers: push tag python-v<ver> → build + publish to PyPI; or manual dispatch with publish = none | testpypi | pypi (none = matrix smoke test, no upload).
  • Publishes via OIDC into a GitHub environment (pypi / testpypi).

Proven

  • linux-x86_64 and macOS-universal2 wheels build, are auditwheel/delocate-honest, and install + run in clean environments (incl. debian:11 / glibc 2.31).
  • linux-aarch64 and Windows validate on first CI dispatch (no local hardware for either).

Before first publish (maintainer action)

Both steps completed 2026-09-02pypi/testpypi environments exist and the pending publishers are registered for project ant-sdk on PyPI and TestPyPI.

  1. Create GitHub environments pypi and testpypi.
  2. Register a PyPI and TestPyPI pending publisher: project ant-sdk, owner WithAutonomi, repo ant-sdk, workflow publish-python.yml, environment pypi / testpypi.

The workflow is inert until then (only runs on a python-v* tag or manual dispatch). Suggested first run: dispatch with publish=testpypi to exercise the full matrix end-to-end.

🤖 Generated with Claude Code

Nic and others added 6 commits August 6, 2026 12:49
Adds the release pipeline for the ant-ffi Python bindings: reproducible
per-platform wheel builds and a publish workflow using PyPI Trusted Publishing
(OIDC, no stored token).

Build scripts (single source of truth — CI runs the same script a developer
runs locally):
- build-wheel-manylinux.sh  compiles the native lib inside a manylinux_2_28
  container (glibc 2.28) so the wheel installs on RHEL8 / Ubuntu 20.04+ /
  Debian 10+; auditwheel verifies + retags. Arg: x86_64 | aarch64.
- build-wheel-macos.sh  builds arm64 + x86_64, lipo-fuses to a universal2
  wheel (deployment target 11.0); delocate verifies self-contained.
- build-wheel-windows.ps1  native x86_64 DLL; delvewheel bundles non-system
  DLLs.

Workflow (publish-python.yml):
- Matrix: linux x86_64 (ubuntu-latest), linux aarch64 (ubuntu-24.04-arm,
  native — no QEMU), macOS universal2 (macos-latest), windows amd64.
- Triggers: push tag `python-v<ver>` -> build+publish to PyPI; or manual
  dispatch with publish=none|testpypi|pypi (none = matrix smoke test).
- Publishes via OIDC (id-token) into a GitHub environment (pypi/testpypi).

Proven locally: linux-x86_64 and macOS-universal2 wheels build, are
auditwheel/delocate-honest, and install + run in clean environments (incl.
debian:11 / glibc 2.31). linux-aarch64 and Windows validate on first CI
dispatch (no local hardware for either).

Before the first publish, a maintainer must create the GitHub environments and
register the PyPI/TestPyPI pending publisher (project ant-ffi, owner
WithAutonomi, repo ant-sdk, workflow publish-python.yml). The workflow is inert
until then (only runs on a python-v* tag or manual dispatch).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follows the distribution-name decision on the base branch; wheel globs
are name-agnostic so only the workflow title and the pending-publisher
setup comment change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each platform script now ends by installing its repaired wheel into a
throwaway venv and importing it from outside the source tree (via
ffi/scripts/check-python-wheel.sh from the base branch; inline
PowerShell on Windows). Catches the wheel that installs cleanly but
cannot import. The manylinux script skips the check when cross-building
(pip refuses a foreign-arch wheel); both CI legs build natively.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Nic-dorman
Nic-dorman deleted the branch main September 3, 2026 07:55
@Nic-dorman Nic-dorman closed this Sep 3, 2026
@Nic-dorman Nic-dorman reopened this Sep 3, 2026
@Nic-dorman
Nic-dorman changed the base branch from feat/python-ffi to main September 3, 2026 07:55
@Nic-dorman
Nic-dorman marked this pull request as ready for review September 3, 2026 07:59

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at f674f57d77358d016910280a1167277754768de2.

Requesting changes for two release-safety gaps:

  1. publish=none can publish to production PyPI. In .github/workflows/publish-python.yml:102-104,126-127, a manual dispatch whose selected ref is a python-v* tag satisfies the tag clause regardless of event_name. Thus workflow_dispatch(ref=python-v0.0.8, publish=none) runs the publish job in the pypi environment and executes the PyPI step, contrary to the documented no-upload behaviour. A testpypi dispatch on such a ref also enables both publish steps. Please restrict tag clauses to tag push events, e.g. require github.event_name == 'push' alongside startsWith(...).

  2. No release-version invariant is enforced. The tag suffix, ffi/python/pyproject.toml version, and native ant_ffi_version()/Cargo package version can disagree while every current check passes. I reproduced this in a scratch copy: a wheel with metadata 9.9.9 containing native FFI 0.0.8 built, installed and imported successfully. Please fail before upload unless the wheel metadata and native version agree, and on tag pushes require both to equal the python-v* suffix.

Verified here: current versions are all 0.0.8; current FFI CI is green; actionlint and git diff --check pass; the macOS universal2 wheel built successfully, contains both x86_64 and arm64, installs in a fresh venv, and reports metadata/native 0.0.8.

Nonblocking follow-ups: the new four-platform matrix has no pre-merge run (Windows and Linux ARM remain unexecuted); clean wheelhouse/ and foreign native libraries before builds (I reproduced a stale .dll being packaged); use Cargo --locked and consider pinning release tooling/actions.

Review panel: five completed independent seats considered the PR merge-safe with caveats; one seat timed out. The panel agreed on the underlying version/matrix/staleness gaps but differed on severity. I am blocking on the two cases above because they can cause an unintended or permanently mis-versioned production PyPI release.

Nic-dorman and others added 3 commits September 3, 2026 10:53
…riant

Review findings on this PR:

- A workflow_dispatch whose selected ref is a python-v* tag satisfied the
  tag clauses regardless of the publish input, so publish=none (or
  testpypi) on a tag ref could upload to production PyPI. Tag clauses now
  also require the push event.
- Nothing tied the wheel metadata version, the native ant_ffi_version(),
  and the release tag together. The install/import check now fails any
  wheel whose metadata and native versions disagree, and on tag pushes CI
  exports EXPECTED_VERSION from the python-v* suffix and both must match
  it. Verified locally: matching stub passes; metadata!=native and
  tag!=version stubs both fail.

Nonblocking hardening from the same review: build scripts clean stale
wheelhouse/ and foreign native libs before staging (a leftover .dll/.so
from another platform's build would be packaged by the package-data
globs), and cargo builds run --locked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@dirvine dirvine left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at c8b5821a2c0084fdec636b760930d3c68596e56b. The two requested changes are resolved:

  • Production publishing now requires a push event for the python-v* tag path, so manual publish=none remains no-upload and testpypi cannot also invoke PyPI, including when dispatching from a tag ref.
  • Each installed wheel now asserts metadata version equals ant_ffi_version(), and tag pushes additionally assert both equal the python-v* suffix.

I reran the adversarial event/input matrix and mismatch tests: both metadata/native and tag/version mismatches fail as intended. The current macOS universal2 wheel built, installed and imported successfully with matching 0.0.8 versions and both x86_64/arm64 slices. Current-head FFI CI and actionlint are green. The added stale-native cleanup and Cargo --locked hardening are sound.

No remaining blockers found.

@Nic-dorman
Nic-dorman merged commit 9df6752 into main Sep 3, 2026
3 checks passed
@Nic-dorman
Nic-dorman deleted the feat/python-ffi-wheels branch September 3, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants