Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 191 additions & 0 deletions docs/proposals/offline-build-parallel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# Offline wheel builds from prepared artifacts

- Author: Sean Pryor
- Created: 2026-10-04
- Status: Draft

## What

Add an opt-in `--offline` mode to `build-parallel`. It consumes a dependency
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
graph and the prepared artifacts from an earlier bootstrap run to build wheels
without repeating source acquisition or source-preparation work.

Comment thread
Xaenalt marked this conversation as resolved.
Resolved
## Why

Preparing source distributions and their build environments can require network
access and package-specific hooks. Repeating those steps in every wheel-build
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
environment adds work and prevents builds from running in environments with no
network access.

Bootstrap already produces the graph and working files needed to separate
preparation from wheel building. An offline build mode lets operators prepare
those inputs once, then reuse them in a restricted build environment.

Separating preparation from building is the intended split between bootstrap
and the build commands
([#995](https://github.com/python-wheel-build/fromager/issues/995)); the
build steps doing any preparation work at all is leftover coupling. The mode
is opt-in to preserve the existing one-step behavior, but it is meant to
become the default, and eventually the only, behavior of the build commands.

## Goals

- Reuse an existing dependency graph, prepared source tree, build metadata,
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
source distribution, and build environment.
- Reuse local wheels for packages that are already built or configured as
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
pre-built.
- Skip `download_source`, `prepare_source`, `prepare_build_environment`, and
`build_sdist` when `--offline` is enabled.
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
- Fail with a package-specific error when a required local artifact is missing,
without falling back to a remote server.
- Preserve dependency ordering and the normal wheel-building outputs.

## Non-goals

- Creating or exporting a portable bootstrap bundle. This proposal covers
consuming an existing prepared workspace.
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
- Resolving package versions or regenerating the dependency graph during the
offline build.
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
- Guaranteeing that arbitrary package build scripts cannot access the
network. Network isolation remains a property of the build environment.
- Defining artifact digests, provenance, or a cross-host transfer format.

## Related issues

This proposal is partial work toward
[#995](https://github.com/python-wheel-build/fromager/issues/995) and
[#797](https://github.com/python-wheel-build/fromager/issues/797), the
broader goal of fully separating dependency discovery from building, and
does not close either issue.

- **Two-stage build** ([#995](https://github.com/python-wheel-build/fromager/issues/995),
[#797](https://github.com/python-wheel-build/fromager/issues/797)):
offline mode gives the build commands a way to avoid preparation work,
but they still perform that work by default, and neither a dedicated
`prepare` command nor a defined transferable workspace layout is part
of this proposal.
- **Discovery fallback** ([#996](https://github.com/python-wheel-build/fromager/issues/996)):
offline mode never reaches the fallback to PEP 517 discovery hooks
because it reuses the prepared build environments. Eliminating that
fallback for online builds remains open.
- **Sdist as build input** ([#1311](https://github.com/python-wheel-build/fromager/issues/1311),
[#696](https://github.com/python-wheel-build/fromager/issues/696)):
offline mode keeps building from the retained source tree, with the
prepared sdist as a required artifact. The expectation that every
package is buildable from its own sdist relies on the packaging best
practices described in #696. Building the wheel from the unpacked
downstream sdist instead remains follow-up work; the workspace already
retains the sdist either way.
- **Git source resolvers** ([#1205](https://github.com/python-wheel-build/fromager/issues/1205),
[#1206](https://github.com/python-wheel-build/fromager/issues/1206)):
offline mode depends on bootstrap producing source trees and sdists, so
packages using the git-clone source resolvers need that preparation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Both cited issues (#1205, #1206) were closed as completed on 2026-10-23; the work sits on the new-resolver-config branch pending merge to main (open PR #1414). "Need that preparation work to land first" reads as pending against closed issues, so it is worth rewording to where the work actually stands.

work to land first. Offline mode does not change how sources are
prepared.
- **Build-tag acceptance** ([#1360](https://github.com/python-wheel-build/fromager/issues/1360)):
offline mode applies the same wheel acceptance rules as an online
build, so it takes no position on what those rules should be.
Consolidating them remains follow-up work.

## How

A workflow prepares a package graph and workspace by running

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: bootstrap-parallel already chains a sdist-only bootstrap with build-parallel and manages cleanup state between the phases (it toggles cleanup_buildenv around the build phase), and the doc never says whether it gains or forwards --offline. Worth a note, since this workflow is a manual version of that command.

`fromager --no-cleanup bootstrap --sdist-only`, then runs
`build-parallel --offline` against that graph and workspace. For each graph
node, Fromager reuses a suitable local wheel when available. Otherwise, it
builds the wheel from the prepared source tree using the existing build
environment. Packages configured as pre-built require a local pre-built
wheel. Packages whose wheels were already built during preparation, such as
build requirements, are satisfied from the local wheel repository.

Every package is expected to be buildable from its own sdist; a package that
cannot be is a packaging bug. Offline mode reuses the prepared source tree to
avoid repeating the preparation work, and the prepared sdist remains a
required artifact for each node.

Offline mode skips the four source and environment preparation phases listed
above and does not use external source or wheel servers. It still runs the
wheel build and its applicable post-build hooks. The workspace therefore needs
to contain the prepared source, metadata, source distribution, and build
environment for each package that must be built from source.

The graph and workspace should come from the same preparation run and build
context. Fromager checks package and version metadata where available and
stops with a clear error if a required artifact is missing or inconsistent; it
does not retry the skipped preparation steps online.

The prepared sdist for a node is located by its expected filename, including

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Per-node discovery by expected filename with a fallback scan rides find_sdist()'s per-lookup globs, up to 8 directory scans per node when no destination_filename setting exists, which is O(nodes × files) where today's build-parallel fetch is O(1) via node.download_url. Consider specifying a one-time index instead: record each node's prepared sdist filename and source path in graph.json during preparation, or do one startup scan of the downloads and work directories.

any package naming hooks, falling back to the name and version parsed from

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: As worded, the fallback parses names and versions out of the files present, which can select an sdist of the wrong version, the exact same-name/different-run ambiguity the next sentence dismisses. The existing find_sdist matches the expected name and version (under naming-convention variants) against the files present; saying that instead would keep the implementation on the safe path.

the files present. The prepared source tree is validated against its build
metadata. Sdist filenames carry no build tag, so prepared sdists from
different runs with the same name and version are not distinguished from each
other; verifying content digests is a non-goal.

## Preparing the workspace

The preparation run is `fromager --no-cleanup bootstrap --sdist-only`,
started from a clean output directory so that the offline inputs are exactly
what the preparation produces and nothing depends on leftover state. Both
options are required:

- `--sdist-only` prepares sdists and build environments without building the
remaining wheels, so there is work left for the offline build.
- `--no-cleanup` keeps the prepared source trees and build environments in
the work directory. By default Fromager removes each package's source tree
as soon as its build metadata is recorded. A workspace prepared without
Comment on lines +134 to +135

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: The timing is off. Build metadata is written during prepare_source, but the source tree is removed only after the package's build completes (the bootstrapper's complete phase, or after the wheel build in the build commands). As written, the sentence implies the tree is gone before the sdist and build environment exist.

Suggested change
the work directory. By default Fromager removes each package's source tree
as soon as its build metadata is recorded. A workspace prepared without
the work directory. By default Fromager removes each package's source tree
and build environment after its build completes. A workspace prepared without

`--no-cleanup` has no source trees to reuse, so every from-source package
fails with an "offline mode requires prepared source" error.

The preparation run and the offline build are coupled in three ways:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: The coupling list omits --variant. Pre-built configuration, build tags, patches, and env vars are all variant-scoped, so preparing with one variant and building offline with another hard-fails: a package pre-built for the preparation variant is from-source under the build variant, with no prepared source tree. Add a fourth bullet requiring the same --variant in both phases.


- **Package settings.** Expected wheel build tags are computed from the
package settings changelogs and hooks in both phases, so both phases must
use the same settings directory and overrides. A local wheel whose build
tag no longer matches the settings is ignored, and the build then fails if
no prepared source exists for the package.
- **Constraints.** Version resolution happens during preparation and the
resolved versions are recorded in the graph, so the offline build does not
need constraints and does not consult them. Fromager reads every

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: The offline build does consult constraints. The local-wheel reuse check this proposal preserves filters candidates by the startup-loaded constraints (_is_wheel_built → resolve_all_cached_wheels → PyPICacheProvider(constraints=ctx.constraints)), so a constraints file that differs from the preparation run suppresses reuse of a wheel that matches the graph's pinned version and forces a needless rebuild or a hard failure. The bullet should require the offline invocation to pass either no constraints or the same constraints files as the preparation run.

`--constraints-file` argument, including remote URLs, when it starts and
before any command runs, so the offline invocation must reference local
files only, or pass no constraints at all.
- **Build image.** Build environment directory names embed the interpreter
version, and the `python` inside each environment is a symlink to the base
interpreter that created it. The offline build must run in the same build
image (or an image with the same interpreter at the same path) as the
preparation run. The offline phase also invokes the `uv` and `wheel`
command line tools.

## Limitations

`--offline` prevents Fromager from fetching sources, preparing them, creating

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: --offline still fetches remote --constraints-file URLs at startup before any command runs, and the proposal never mentions the existing --network-isolation flag, so the name reads as a hermeticity guarantee the design explicitly disclaims. Rejecting non-local constraints URLs when --offline is set, and documenting composition with --network-isolation (or renaming the flag to describe skipping preparation), would close that gap.

build environments, or building source distributions during `build-parallel`.
It does not prevent a package's wheel-build hook or build tool from attempting
Comment thread
Xaenalt marked this conversation as resolved.
Resolved
network access. Plugins configured with `network_isolation: false` may still
attempt remote access, and tools such as Bazel may fetch dependencies during
Comment on lines +163 to +165

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: There is no per-plugin network_isolation setting; it is a global CLI flag and a per-call argument of external_commands.run().

Suggested change
It does not prevent a package's wheel-build hook or build tool from attempting
network access. Plugins configured with `network_isolation: false` may still
attempt remote access, and tools such as Bazel may fetch dependencies during
It does not prevent a package's wheel-build hook or build tool from attempting
network access. Plugin hooks and package build tools can invoke commands that
bypass Fromager's network-isolation wrapper (Fromager itself disables it for
git operations and Rust vendoring), and tools such as Bazel may fetch
dependencies during the wheel build.

the wheel build. Strict network isolation must be applied by the container or
build environment.

The offline build does not consult remote wheel caches or package indexes.
Each graph node is still checked against the local wheel repository, with the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: "Same reuse rules as an online build" cannot deliver the rule that pre-built packages require a local pre-built wheel. Online reuse treats a package's configured wheel_server_url as exclusive and never checks the local wheel repository for pre-built packages (get_wheel_server_urls), so as written every pre-built package resolves against its remote index offline and fails. The doc needs to define the local-first lookup for pre-built packages rather than deferring to the online rules.

same reuse rules as an online build, to decide whether it needs to be built.
A wheel that exists only remotely, including in the index the built wheels
will be published to, can only be reused by populating the local wheel
repository during preparation.

Offline mode does not invoke the package hooks for the phases it skips
(source download and preparation, build environment setup, and sdist
building). The `build_wheel` hook and the prebuilt-wheel and post-build hooks
still run. The plugin interface for distinguishing bootstrap behavior from
build behavior is unchanged; refining it is follow-up work.

## Example

```bash
# Preparation, with network access: resolves versions, prepares sdists,
# source trees, and build environments.
fromager --no-cleanup bootstrap --sdist-only 'stevedore==5.2.0'

# Offline wheel build: no external network access required.
fromager build-parallel --offline work-dir/graph.json

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: The offline command omits --no-cleanup, and with the default cleanup the build deletes each package's prepared source tree and build environment as its wheel finishes (clean_build_dirs runs after the post-build hooks). The workspace becomes single-use, and the retry scenario at lines 143-145 (a build-tag mismatch needs the prepared source) fails. Pass --no-cleanup here and spell out the offline build's cleanup semantics, including reruns after a mid-run failure.

```
Loading