diff --git a/peps/pep-0780.rst b/peps/pep-0780.rst index b488e3872cc..59467c757a8 100644 --- a/peps/pep-0780.rst +++ b/peps/pep-0780.rst @@ -8,29 +8,34 @@ Status: Draft Type: Standards Track Topic: Packaging Created: 21-Mar-2025 -Python-Version: 3.14 Post-History: `05-Aug-2024 `__, `26-Mar-2025 `__, + `06-Nov-2026 `__, .. If accepted, the contents of this PEP should be integrated into the canonical documentation in the Python Packaging User Guide and the following directive should be added here: - .. canonical-pypa-spec:: :ref:`packaging:dependency-specifiers` + .. canonical-pypa-spec:: :ref:`packaging:dependency-specifiers` and + :ref:`packaging:core-metadata` Abstract ======== -This PEP defines using ABI features as environment markers for project -dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508` -(later moved to :ref:`packaging:dependency-specifiers`) introduced environment -markers to specify dependencies based on rules that describe when the -dependency should be used. This PEP extends the environment markers to allow -specifying dependencies based on specific ABI features of the Python -interpreter. For this, it defines a set of `ABI Features`_ and specifies how -they are made available for `environment markers `_ -as a new marker variable, ``sys_abi_features``. +This PEP defines the use of information about the interpreter ABI as +environment markers for project dependencies through a new ``sys_abi_info`` +environment marker variable. :pep:`508` (later moved to +:ref:`packaging:dependency-specifiers`) introduced environment markers to +specify dependencies based on rules that describe when the dependency should +be used. This PEP extends environment markers to allow specifying dependencies +based on specific information about the ABI of the Python interpreter. For +this, it defines a set of `ABI Features`_ and specifies how they are made +available for `environment markers `_ via a new marker +variable, ``sys_abi_info``. + +Use of ``sys_abi_info`` in ``Requires-Dist`` metadata is introduced with Core +Metadata version 2.7. Motivation ========== @@ -49,31 +54,26 @@ precisely in metadata is currently not possible, and increasing the minimum version of a dependency also for the GIL-enabled build is usually undesirable since it unnecessarily limits compatibility between packages. -Some concrete examples of such issues have been discussed in the `Environment -marker for free-threading`_ Discourse thread: - -- Cython has (experimental) support for free-threading only in its master - branch, and is used by a lot of projects that already publish ``cp313t`` - wheels. Picking up the wrong Cython version is causing a lot of obscure build - failures and runtime crashes. It would be beneficial if the metadata could - express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_). -- CFFI has no support for free-threading yet, and `Armin Rigo, one of the - maintainers, has stated - `__ - that it may be a good idea to fork ``cffi``, implement support for - free-threading, and only come back to the CFFI project with a single large PR - that adds support after the functionality "is reasonably well-tested (either - as tests or, better in this case, tested by being in use in various other - projects)". There are a lot of projects that depend on ``cffi``. They are - likely fine to start depending on a fork for free-threading only, however - depending on a fork for >=3.13 or for all Python versions seems like a much +At the time of writing of the first version of this PEP, concrete examples of +such issues were discussed in the `Environment marker for free-threading`_ +Discourse thread: + +- Cython had experimental support for free-threading only in its master + branch, and was used by many projects already publishing ``cp313t`` + wheels. Picking up the wrong Cython version caused obscure build + failures and runtime crashes. It would have been beneficial if the metadata + could have expressed that (see `Examples`_). +- CFFI did not yet support free-threading, and for a significant period of + time, it looked likely that support would first be added in a fork. Many + projects depend on ``cffi`` and could have depended on a fork for + free-threading only; however, depending on a fork for >=3.13 or for all + Python versions seemed like a much larger ask, and more disruptive for distribution packagers. -While these concrete examples may be addressed later this year by Cython and -CFFI making compatible releases, the same issue is going to repeat further up -the stack. The free-threading rollout is expected to take several years, and an -environment marker for free-threading will make that rollout significantly -easier. +Cython 3.1.0 and CFFI 2.0.0 have since added free-threading support, but the same +principle remains relevant as other projects further up the stack add support. +The free-threading rollout is an ongoing multi-year effort, and an environment +marker for free-threading will make that rollout significantly easier. Another important ABI feature that is not yet covered by environment markers is the bitness of the interpreter. In most cases, the ``sys_platform`` or @@ -86,9 +86,25 @@ to due to the lack of a suitable 32-bit compiler toolchain with Fortran support). Those wheels lacking can be awkward especially for projects where SciPy is an optional dependency only. In that case, it would be useful to be able to specify that SciPy is required *unless* the interpreter is a 32-bit one -on Windows (c.f. `Require SciPy Unless on 32-bit win32`_), to avoid failed +on Windows (see `Examples`_), to avoid failed from-source installations due to the missing wheels. +Enabling consistent dependency metadata across a package release is another +motivation for this PEP. The :ref:`PEP 825 investigation of metadata consistency +` identified ``apache-beam``'s omission of +``pyarrow`` on 32-bit Windows as the only remaining real-world case in that +survey where existing markers could not express the required dependency +conditions. The bitness information exposed by ``sys_abi_info`` closes that +gap, allowing the condition to be expressed in the same dependency metadata in +the sdist and every wheel of the release. + +Closing this gap also enables future requirements for consistent dependency +metadata across release artifacts. Resolvers can then use one artifact's +dependency metadata for the release, evaluating markers for each target +environment, rather than inspecting every wheel separately. Tools such as +`uv already expect this consistency +`__. + Rationale ========= @@ -107,14 +123,14 @@ process over multiple releases. It is therefore important to make sure that the mechanisms in this PEP are useable for Python interpreters where either free-threading or non-free-threading could be the default or the only option. -At the time of writing, free-threaded Python is in Phase I: experimental phase. -In this phase, there is an acute need for the proposed environment markers to -help with the transition to free-threaded Python as package authors gradually -add support. +When the first version of this PEP was written, free-threaded Python was in +Phase I: experimental phase. The proposed environment markers were needed to +help with the transition as package authors gradually added support. -As the number of packages with support increases, and particularly during -Phase II: Supported-but-not-default phase, we still anticipate a strong need -for the environment markers to help with the transition. +Since Python 3.14, free-threaded Python is officially supported, entering +Phase II: Supported-but-not-default phase, as described in :pep:`779`. +As the number of packages with support increases, there remains a need for +environment markers to help with this ongoing transition. As free-threaded Python enters into Phase III: Default phase, the need for the environment markers will decrease, though at this point it is not clear that @@ -132,77 +148,93 @@ simplicity for the foreseeable future in a changing Python ecosystem. Relation to Other PEPs ---------------------- -This PEP extends environment markers with set semantics for ABI features. -:pep:`751#additions-to-marker-expression-syntax` includes a similar extension -for lock file specific environment markers; although the two have been -developed indepedently, they are compatible where they overlap in terms of the -new set semantics. +:pep:`751#additions-to-marker-expression-syntax`, which standardized +``pylock.toml``, introduced set-valued environment markers for extras and +dependency groups in lock files. This PEP applies those set semantics to ABI +features through ``sys_abi_info``, making them available in dependency +specifications as well as lock files. + +:pep:`825` also introduces new :ref:`environment markers +` as part of its Wheel Variants design. +The considerations in `Backwards Compatibility`_ and the analysis in the +:ref:`tool behavior appendix ` apply +equally to those markers when encountered by tools that do not support them. Specification ============= -The keywords "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**", -"**SHALL NOT**", "**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**", "**MAY**", -and "**OPTIONAL**"" in this document are to be interpreted as described in -:rfc:`RFC 2119 <2119>`. - ABI Features ------------ -ABI features are intrinsic properties of the Python interpreter, expressed as -simple, understandable strings. However, not all features are equally -applicable to all Python interpreters or Python versions. For example, the -distinction between free-threaded and GIL-enabled interpreters is only relevant -for CPython 3.13 onwards, but the bitness of the interpreter is relevant for -all interpreters. - -All interpreters MUST handle the following ABI features as stated. ABI features -that are restricted to particular interpreters MUST NOT be provided by other -interpreters. The features are subdivided into groups and for each group there -MUST be exactly one feature present, except when the group is marked as -optional, in which case there MUST be at most one feature present. - -``free-threading`` or ``gil-enabled`` (only CPython) - If the Python interpreter is free-threaded, the ``free-threading`` feature - MUST be present and the ``gil-enabled`` feature MUST NOT be present. - Otherwise, the ``gil-enabled`` feature MUST be present and the - ``free-threading`` feature MUST NOT be present. - -``debug`` (only CPython, optional) - This ABI feature is reserved for the ``--with-pydebug`` build of CPython. - If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities, - the ``debug`` feature MUST be present. On POSIX systems, this corresponds - to the Python expression ``"d" in sys.abiflags``. - -``32-bit`` or ``64-bit`` (optional) - The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit - build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit, - this feature MUST NOT be present. - -.. _pep-780-sys_abi_features: - -The ``sys_abi_features`` Environment Marker -------------------------------------------- +ABI features are intrinsic properties of the Python interpreter. +However, not all features are equally applicable to all Python interpreters or +Python versions. For example, CPython introduced free-threading support in +Python 3.13. Other implementations do not currently support free-threading, +so ``free_threaded`` MUST be ``False`` for them. The bitness of the interpreter, +i.e. the number of bits in a pointer, is relevant for all interpreters. -To make ABI features available in dependency specifications, a new environment -marker variable, ``sys_abi_features``, is added to the format of dependency -specifiers. +Tools MUST provide the ABI features listed below for all Python implementations, +using :data:`sys.abi_info` where available and the rules in +`Determining ABI Features on Older Interpreters`_ otherwise: -To do this, we need to extend the grammar laid out in :pep:`508` and maintained -in the :ref:`packaging:dependency-specifiers` and document the possible values. +``pointer_bits`` + The bitness of the interpreter, that is, the number of bits in a pointer, + typically ``32`` or ``64`` [#bitness]_. + +``free_threaded`` + If the Python interpreter was built with free-threading support, + ``free_threaded`` MUST be ``True``, otherwise it MUST be ``False``. + This value reflects the build configuration, not whether the GIL is + currently enabled. It is unaffected by ``-X gil``, ``PYTHON_GIL``, or + changes to the GIL state at runtime. + +``debug`` + Indicates whether the interpreter was built in debug mode, corresponding + to the :attr:`sys.abi_info.debug` field. + +``byteorder`` + Indicates the byte order of the interpreter. The value MUST be either + ``"little"`` or ``"big"``. + +For implementations without the corresponding ``sys.abi_info`` fields, the +values of ``free_threaded`` and ``debug`` are determined by the rules in +`Determining ABI Features on Older Interpreters`_, including the ``False`` +defaults for non-CPython implementations. -The grammar is extended to include the ``sys_abi_features`` marker variable by -augmenting the definition of ``env_var`` as follows:: +.. _pep-780-sys_abi_info: + +The ``sys_abi_info`` Environment Marker +--------------------------------------- + +To make ABI features available in dependency specifications, a new environment +marker variable, ``sys_abi_info``, is added to the format of dependency +specifiers. It is built to reflect the value of :data:`sys.abi_info`, which is a +new attribute of the ``sys`` module introduced in Python 3.15 as a simple +namespace. To represent its fields in the environment marker variable, they are +expressed as a frozen set of strings of the form ``f"{name}::{value}"``, where +``name`` is the name of the field and ``value`` is the string representation of +its value. + +The ``sys_abi_info`` Environment Variable +''''''''''''''''''''''''''''''''''''''''' +To do this, we need to extend the grammar laid out in :pep:`508` and maintained +in the :ref:`packaging:dependency-specifiers` and document the possible values, +to include the ``sys_abi_info`` marker variable by augmenting the definition of +``env_var`` as follows:: env_var = ('python_version' | 'python_full_version' | 'os_name' | 'sys_platform' | 'platform_release' | 'platform_system' | 'platform_version' | 'platform_machine' | 'platform_python_implementation' | 'implementation_name' | 'implementation_version' | - 'sys_abi_features' | - 'extra' # ONLY when defined by a containing layer + 'sys_abi_info' | + 'extra' | 'extras' | 'dependency_groups' # ONLY when defined by a containing layer ) +This makes ``sys_abi_info`` available as a variable in marker expressions. + +The Values of ``sys_abi_info`` +'''''''''''''''''''''''''''''' Like the grammar, also the overview table of environment markers in :ref:`packaging:dependency-specifiers` is augmented to add the following row: @@ -212,66 +244,265 @@ Like the grammar, also the overview table of environment markers in * - Marker - Python equivalent - Sample values - * - ``sys_abi_features`` - - no direct equivalent available - - ``{'free-threading', '64-bit'}``, - ``{'gil-enabled', 'debug', '32-bit'}`` + * - ``sys_abi_info`` + - :data:`sys.abi_info` + - ``{'free_threaded::True', 'debug::False', 'pointer_bits::64', 'byteorder::little'}``, + ``{'free_threaded::False', 'debug::True', 'pointer_bits::32', 'byteorder::little'}``, + ``{'free_threaded::False', 'debug::False', 'pointer_bits::64', 'byteorder::little'}`` With these additions, ABI features can be used in dependency specifications via -the ``in`` operator to test for the presence of a feature, or the ``not in`` -operator to test for the absence of a feature. +the ``in`` operator to test for the presence of a feature-value pair, or the +``not in`` operator to test for its absence. + +This allows for marker expressions such as:: + + "free_threaded::True" in sys_abi_info + "pointer_bits::32" not in sys_abi_info and "debug::True" in sys_abi_info + +Integer-valued features support exact-value tests, not numerical comparisons. +This is sufficient for the bitness use case: excluding 32-bit Windows requires +testing for ``"pointer_bits::32"``, without enumerating other possible pointer +widths. An unfamiliar value, such as ``128``, remains a distinct string; +absence of ``"pointer_bits::32"`` does not imply a 64-bit interpreter. + +Adding new ABI features does not require changes to the marker grammar, +because the feature names and values are encoded as strings. If the tested +string is absent from ``sys_abi_info``, ``in`` evaluates to false and ``not in`` +evaluates to true. Package authors can therefore choose the +default behavior for interpreters that do not provide a new feature. For +example, ``"new_feature::True" in sys_abi_info`` requires the feature to be +explicitly true, while ``"new_feature::False" not in sys_abi_info`` also +evaluates to true when the feature is absent. + +ABI feature values MUST be unambiguously representable as strings. Currently, +all ABI features are either booleans, integers, or strings, which all satisfy +this requirement. This is expected to remain the case. When constructing +``sys_abi_info``, packaging tools MUST ignore fields whose values have other +data types. + +Core Metadata +------------- + +``Requires-Dist`` is a :ref:`Core Metadata ` field +whose value syntax is defined by the :ref:`Dependency Specifiers +` specification. Because this PEP changes what +is considered valid in that field, it updates Core Metadata to version 2.7, +following the precedent of :pep:`685`. + +Build tools serializing a ``Requires-Dist`` entry whose environment marker +references ``sys_abi_info`` MUST emit Core Metadata version 2.7 or later. +Build tools MAY continue to emit an earlier Core Metadata version when +``sys_abi_info`` is not used in ``Requires-Dist`` and no other metadata feature +requires a newer version. + +Publishing tools SHOULD report an error when a distribution uses +``sys_abi_info`` in ``Requires-Dist`` while declaring a Core Metadata version +earlier than 2.7. Index servers SHOULD reject newly uploaded distributions +with this inconsistency. + +This requirement applies only when the marker is serialized into Core +Metadata. Uses of ``sys_abi_info`` in other dependency-specifier contexts, +such as ``build-system.requires``, requirements files, or ``pylock.toml``, +do not by themselves require or imply a Core Metadata version. + +Determining ABI Features on Older Interpreters +------------------------------------------------ + +The features defined by this PEP describe properties that were already +available before :data:`sys.abi_info` was introduced in Python 3.15. Tools MUST +provide them even when that attribute, or an applicable field, is absent. +When a field is available in ``sys.abi_info``, tools MUST use its value. +Otherwise, the following rules define the values for the target interpreter; +implementations MAY use equivalent methods to obtain the same values. + +``pointer_bits`` + ``8 * struct.calcsize("P")``. + +``byteorder`` + ``sys.byteorder``. + +``free_threaded`` + On CPython, ``bool(sysconfig.get_config_var("Py_GIL_DISABLED"))``. An + undefined configuration variable is treated as false. This detects + build-time support, independently of the runtime GIL state. On other + implementations, the value MUST be ``False``. + +``debug`` + On CPython, ``bool(sysconfig.get_config_var("Py_DEBUG"))`` when the + configuration variable is defined. Otherwise, the value is true if + ``hasattr(sys, "gettotalrefcount")`` is true or ``"_d.pyd"`` is present in + ``importlib.machinery.EXTENSION_SUFFIXES``, and false otherwise. The latter + check covers Windows builds that do not expose ``Py_DEBUG``. On other + implementations, the value MUST be ``False``. + +The resulting values are encoded as ``"name::value"`` strings in the same way +as values obtained from ``sys.abi_info``; false boolean values are included, +not omitted. Tools do not need to add ``sys.abi_info`` to the interpreter. + +Future attributes exposed by ``sys.abi_info`` are included when their values +have a supported type. Inferring those attributes on interpreters where they +are absent requires additional rules in the Dependency Specifiers +specification. Unless such rules are defined, tools MUST omit the missing +attributes rather than infer a value. In particular, an absent boolean +attribute is distinct from one whose value is false. + +Handling Unsupported Marker Fields +---------------------------------- + +The :ref:`Dependency Specifiers ` specification +states that references to unknown marker fields SHOULD render a package version +ineligible for installation or inclusion in a locked dependency tree. This PEP +clarifies that rule as follows. + +Consumers do not need to successfully parse an unknown marker field to implement +this behavior. When a locking or installation tool cannot interpret published +dependency metadata because a marker field is unsupported, it SHOULD treat the +containing package version as ineligible and SHOULD continue considering other +versions where possible. Such tools MAY conservatively reject a package version +when any of its published dependency specifications cannot be parsed, without +distinguishing unsupported marker fields from malformed requirements. + +When determining dependencies for installation or inclusion in a locked +dependency tree, tools MUST NOT recover by dropping an unsupported marker, +discarding the dependency that contains it, or assuming a truth value for its +expression. Such recovery can produce an apparently successful installation +with incorrect dependencies. This does not prevent static analysis tools from +ignoring markers when collecting all potential dependencies. If no eligible +candidate remains, installation and locking tools SHOULD report a descriptive +failure explaining the unsupported metadata and, where possible, suggesting an +upgrade of the consuming tool. + +This clarification concerns interpreting distribution dependency metadata, +including metadata in wheels or successfully generated from an sdist. It does +not require tools to recover from failures while preparing an sdist or +generating its metadata, as discussed in `Backwards Compatibility`_. + Examples -======== +-------- -Require Cython Pre-release for Free-Threaded Python ----------------------------------------------------- To require a pre-release of Cython only for a free-threaded Python interpreter, the following dependency specification can be used:: - cython >3.1.0a1; "free-threading" in sys_abi_features - cython ==3.0.*; "free-threading" not in sys_abi_features + cython >3.1.0a1; "free_threaded::True" in sys_abi_info + cython ==3.0.*; "free_threaded::False" in sys_abi_info -Require SciPy Unless on 32-bit ``win32`` ----------------------------------------- To require SciPy unless on a 32-bit interpreter on Windows, the following dependency specification can be used:: - scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features + scipy; platform_system != "Windows" or "pointer_bits::32" not in sys_abi_info -Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities --------------------------------------------------------------------------- To require NumPy only for a free-threaded interpreter with debugging capabilities, the following dependency can be used:: - numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features + numpy; "free_threaded::True" in sys_abi_info and "debug::True" in sys_abi_info + +**Require Free-Threaded Fork of PyYAML** + +PyYAML initially decided not to support free-threading in its mainline +distribution before wider testing of free-threaded Python in the real world. +To facilitate this testing, a free-threaded fork, PyYAML-ft, was created. +To require this fork, for a free-threaded Python interpreter, without forcing +it also on the GIL-enabled interpreter, the following dependency specification +can be used:: + + pyyaml-ft; "free_threaded::True" in sys_abi_info + pyyaml; "free_threaded::True" not in sys_abi_info + +Note that in the absence of the features described in this PEP, early adopting +projects have had to resort to using the fork for *all* builds of a potentially +free threading Python interpreter; typical dependencies then looked like:: + + pyyaml; python_version<"3.13" + pyyaml-ft; python_version>="3.13" + +which is an undesirable side effect for GIL-enabled environments. + +.. _pep780-backwards-compatibility: Backwards Compatibility ======================= -This is a pure extension to the existing environment markers and does not -affect existing environment markers or dependency specifications, hence there -are no direct backwards compatibility concerns. - -However, the introduction of the feature has implications for a number of -ecosystem tools, especially those which attempt to support examination of data -in ``pyproject.toml`` and ``requirements.txt``. +Introducing a new environment marker can affect tools that do not understand +it. Packaging interoperability generally follows the principle of strict +validation when producing or accepting new metadata and forward-compatible +handling when consuming previously published metadata. + +Build backends, publishing tools, and package indexes should understand and +validate new metadata features before producing or accepting them. Consumers +can ignore new optional information when it is unnecessary for the operation +being performed. When new semantics are needed to determine dependencies or +compatibility, consumers should instead consider the affected package version +unsupported. + +Core Metadata version 2.7 provides an explicit producer and publisher boundary +for using ``sys_abi_info`` in ``Requires-Dist``. It does not by itself make +older consumers compatible with the new marker. Core Metadata permits +consumers to process newer minor versions, so an older consumer may continue +processing 2.7 metadata and encounter the unsupported marker. The fallback +behavior described below is therefore still needed. Beyond the metadata version +requirement, this PEP introduces no separate index activation date or waiting +period. + +Marker field names are part of a closed grammar. Tools that predate +``sys_abi_info`` may therefore report a parsing error when encountering a +dependency containing it. An open-ended marker grammar is not needed to handle +this safely: the resolver can treat the containing package version as +ineligible and try other versions. This preserves correctness without requiring +the consumer to understand the new marker. See +`Handling Unsupported Marker Fields`_ for the specification clarification. + +Not all existing tools implement this fallback behavior. For wheel dependency +metadata, pip 24.0 aborts resolution on an unsupported marker, while pip 24.1 +(Jun 2024) and later can skip the affected package version and select an older +version. Poetry versions preceding the release that resolves +`poetry#11191 `__ instead +drop the entire dependency containing an unknown marker and keep the package +version eligible. This can produce an apparently successful +installation with a missing dependency, leading to runtime failures. It is +therefore unsafe even when resolution succeeds. + +Publishing a release that uses ``sys_abi_info`` in dependency metadata therefore +drops compatibility with pip <24.1, and with Poetry (until Poetry implements a +fix). Projects adopting the marker should make this compatibility change clear +to their users. + +Pinning a version or requesting an artifact directly may leave no eligible +alternative, resulting in an installation failure. + +Source distributions have additional limitations. An unsupported marker in +``build-system.requires`` can cause the frontend to fail before running the +backend. An unsupported marker in ``project.dependencies`` can cause the backend +to fail during metadata generation. Validation may occur as early as the +backend hook that determines additional build requirements. These errors may +abort installation rather than trigger fallback to an older release, including +when installing from an index. This PEP does not require changes to that +behavior: failure is an acceptable outcome for older tools encountering such +source distributions. +Installing a usable wheel does not require inspecting the sdist's build +requirements. The :ref:`tool behavior appendix +` records these distinctions in detail. + +Projects adopting ``sys_abi_info`` should account for these compatibility +boundaries and the installation and locking tools their users rely on. Projects +decide when to adopt the marker; this PEP specifies no minimum ecosystem support +threshold. +Once tools understand ``sys_abi_info``, additional values in its set can be +handled through the membership semantics specified above. Unknown marker +*fields* must not be confused with absent or unknown ABI feature *values*. Audit and Update Tools ---------------------- -A wide range of tools understand Python dependency data as expressed in -``requirements.txt`` files. (e.g., Dependabot, Tidelift, etc) +A wide range of tools, including Dependabot and Tidelift, inspect dependency +data in ``pyproject.toml``, ``pylock.toml``, and ``requirements.txt`` files and +offer assisted or automated updates. Tools that interpret the meaning of +``sys_abi_info`` +need explicit support for it. Tools consuming published distributions can +remain correct by rejecting versions with unsupported dependency metadata, +although this can limit which releases they consider. Static analysis and +update workflows may lose functionality until their tools support the marker. -Such tools inspect dependency data and, in some cases, offer tool-assisted or -fully automated updates. It is our expectation that no such tools would support -the new environment markers at first, and broad ecosystem support could take -many months or even some number of years to arrive. - -As a result, users of the new environment markers would experience a -degradation in their workflows and tool support at the time that they start -using them. This is true of any new standard for where and how dependency data -are encoded. Security Implications ===================== @@ -310,6 +541,15 @@ the reference implementation linked above. __ https://github.com/zklaus/pip/tree/env-marker-free-threading +uv prototype +------------ + +An independent prototype by Charlie Marsh is available in +`astral-sh/uv#21840 `__. It adds +preview support for ABI feature markers during dependency resolution and wheel +coverage checks. The prototype implements an earlier draft of this PEP, using +the marker name ``sys_abi_features`` and its earlier value format. + Rejected Ideas ============== @@ -331,27 +571,11 @@ Second, the introduction of a dynamic mechanism would require a more complex implementation in the packaging library, which would be a significant departure from the current approach. -Open Issues -=========== - -Other Environment Markers -------------------------- -If other environment markers are needed right now, this PEP could be extended -to include them. - -Other Tooling -------------- -The reference implementation is based on the ``packaging`` library and ``pip``. -We have confirmed that this allows for building and installing packages with -several build backends. It is possible that other tools should be added to the -reference implementation. - - Footnotes ========= -.. [#python-free-threading] Python experimental support for free threading is - available in Python 3.13 and later. For more information, see `Python +.. [#python-free-threading] Free threading was experimental in Python 3.13 and + is officially supported since Python 3.14. For more information, see `Python experimental support for free threading`_. .. [#bitness] While there are some related environment markers available, such @@ -360,6 +584,8 @@ Footnotes particularly on platforms that allow the execution of either kind of binary. +.. _uv: https://github.com/astral-sh/uv/blob/86e7b2e97a193ce3f506804be4c41c611191fa72/crates/uv-pep508/src/marker/tree.rs#L170 +.. _pypa/packaging: https://github.com/pypa/packaging/blob/8f13a4309aa620bf643cd508304f783b456e8c49/src/packaging/_tokenizer.py#L63 .. _Python experimental support for free threading: https://docs.python.org/3/howto/free-threading-python.html .. _Python free-threading guide: https://py-free-threading.github.io/ .. _Environment marker for free-threading: https://discuss.python.org/t/environment-marker-for-free-threading/60007 @@ -374,6 +600,46 @@ Thanks to Filipe Laíns for the suggestion of the ``abi_features`` attribute and to Stephen Rosen for the Backwards Compatibility section of :pep:`735`, which served as a template for the corresponding section in this PEP. +Change History +============== + +- 05-Nov-2026 + + - Rewrote `Backwards Compatibility`_ and clarified how installation and + locking tools should reject package versions with unsupported markers + and consider alternatives. Prohibited recovery that changes dependency + semantics, while allowing static analysis to collect potential + dependencies. Documented the limitations of fallback for source builds. + - Added the :ref:`tool behavior appendix + `, documenting pip, uv, PDM, and + Poetry behavior when encountering unknown markers. + - Introduced Core Metadata version 2.7 for ``Requires-Dist`` entries using + ``sys_abi_info``, with producer, publishing tool, and index validation + rules. Other dependency-specifier contexts do not require this version. + - Specified how tools determine ABI features when ``sys.abi_info`` or its + fields are unavailable, including false defaults for ``free_threaded`` + and ``debug`` on non-CPython implementations. Defined how future missing + attributes are handled. + - Clarified that ``free_threaded`` describes the interpreter's build + configuration rather than its runtime GIL state. + - Clarified exact-value membership tests, behavior for absent values, and + support for boolean, integer, and string values. Required tools to ignore + fields with other value types. + - Added consistent dependency metadata across sdists and wheels as a + motivation, referencing the PEP 825 investigation of 32-bit Windows. + Related the compatibility guidance to PEP 825's new environment markers. + - Updated the free-threading rollout discussion and the relationship to + PEP 751, and added the independent uv prototype under Reference + Implementation. + - Removed the Open Issues section, all open issues were addressed. + + +Appendices +========== + +- :ref:`pep780-tool-behavior-unknown-markers` + + Copyright ========= diff --git a/peps/pep-0780/appendix-tool-behavior-unknown-markers.rst b/peps/pep-0780/appendix-tool-behavior-unknown-markers.rst new file mode 100644 index 00000000000..52380fee0d0 --- /dev/null +++ b/peps/pep-0780/appendix-tool-behavior-unknown-markers.rst @@ -0,0 +1,197 @@ +:orphan: + +.. _pep780-tool-behavior-unknown-markers: + +Appendix: Compatibility behavior in existing tools +================================================== + +This non-normative appendix supplements the :ref:`Backwards Compatibility +` section of :pep:`780`. It describes how +existing packaging tools handle dependencies containing markers they do not +understand. + +To test whether tools can fall back to an older release, two versions of a +package named ``marker-test`` are made available as wheels. Version 1.0 has no +dependencies. Version 2.0 adds a dependency on ``idna`` conditional on +``sys_abi_info``, which the tested tools do not recognize. Its wheel metadata +contains:: + + Requires-Dist: idna; "pointer_bits::64" in sys_abi_info + +A consumer that does not understand ``sys_abi_info`` should reject version 2.0 +and may select version 1.0. The marker expression tests whether the interpreter +has 64-bit pointers, but consumers that do not recognize the field name cannot +interpret that condition. + +The following table summarizes the wheel case. The detailed sections below +separate resolver tests from source inspection and metadata-conversion tests. + +.. list-table:: Wheel dependency metadata containing an unknown marker + :header-rows: 1 + :widths: 20 55 25 + + * - Tool + - Behavior + - Older-release fallback + * - pip + - 24.0 (Feb 2024) aborts; 24.1 (Jun 2024) and later reject the package version + - Yes from 24.1 + * - uv + - 0.7.0 (Apr 2025) and 0.13.6 (Oct 2026), without preview features, + select the older release + - Yes + * - PDM + - 2.13.0 (Mar 2024) and 2.29.5 (Oct 2026) reject candidates with invalid + dependency metadata + - Yes + * - Poetry + - 1.8.2 (Mar 2024) and 2.5.1 (Sep 2026) drop the dependency containing + the unknown marker rather than reject the package version + - No + + +For their pip-based resolver paths, Pipenv and pip-tools do not add independent +resolver behavior here: Pipenv uses an embedded/patched pip resolver, while +pip-tools invokes pip's resolver internals. Their behavior therefore follows +the corresponding pip version. + +pip +--- + +The table below shows how pip handles an unsupported marker in runtime or +build dependencies when installing a local project, a wheel, or a source +distribution. Local tests with pip 26.2.2 (Oct 2026) used ``--find-links`` +candidates, isolated setuptools builds, and dry-run installation; packages were +not uploaded to TestPyPI. The original setuptools backend did not recognize +``sys_abi_info`` and rejected ``project.dependencies`` during +``get_requires_for_build_wheel``. The source-project tests were repeated with +marker recognition patched into setuptools' build dependencies, allowing the +requirement to be preserved in generated metadata. Pip itself was unchanged. + +.. list-table:: pip behavior with an unsupported marker + :header-rows: 1 + :widths: 40 40 20 + + * - Installation case + - Outcome + - Older-release fallback + * - ``pip install .``; marker in ``project.dependencies`` + - Unmodified backend fails validation; patched backend generates metadata + that pip rejects + - Not applicable + * - ``pip install .``; marker in ``build-system.requires`` + - Frontend fails while reading ``pyproject.toml`` + - Not applicable + * - ``pip install marker-test``; marker in wheel ``Requires-Dist`` + - pip 24.1 and later reject the version as invalid metadata + - Yes + * - ``pip install marker-test``; marker in sdist ``project.dependencies`` + - Unmodified backend fails validation; patched backend generates metadata + that pip rejects + - Yes with the patched backend; no with the unmodified backend + * - ``pip install marker-test``; marker in sdist ``build-system.requires`` + - Frontend fails while preparing the candidate + - No + * - Release has a usable wheel and an sdist with an unsupported marker in + ``build-system.requires`` + - Wheel installs without inspecting the sdist's build requirements + - Not applicable + +Source builds can fail before the resolver receives dependency metadata: +build requirement parsing occurs before invoking the backend, and backend +validation can fail before metadata is generated. When the patched backend +successfully generated metadata, pip rejected the indexed sdist for version +2.0 and selected the version 1.0 wheel. Pinning version 2.0 instead failed. +Local source installs have no older index candidate to select. Metadata-level +fallback therefore applies to sdists too, but does not imply fallback for +arbitrary build or backend errors. + +The remainder of this appendix considers only the third row, the case where a +wheel's dependency metadata contains an unsupported marker. This is the key +case for fallback to an older package version, as it has by far the largest +impact on backwards compatibility. The other cases have the limitations +discussed in the :ref:`Backwards Compatibility +` section of :pep:`780`. + +For the wheel case, pip 24.0 aborts with a requirement-parsing error +instead of trying version 1.0. The behavior changed in pip 24.1 through +`pypa/pip#12300 `__, which discards +candidates with invalid dependencies and excludes other candidates of the same +version. This is package-version rejection, rather than permission to choose +another artifact with inconsistent dependency metadata. + +Local dry-run resolution with pip 24.0, 24.1, and 26.2.2 reproduced the wheel +version boundary using the current ``sys_abi_info`` example. Both newer versions +selected 1.0, and both failed when the requirement pinned version 2.0. + +The pip 24.1 and 26.2.2 diagnostic begins with a warning +that version 2.0 has invalid metadata, followed by the parser's explanation. It also suggests using +``pip<24.1``. That diagnostic addresses invalid metadata generally; +it is not appropriate upgrade advice for a newly standardized marker. An +unsupported marker needs a consumer that understands it, or rejection of the +affected version, rather than removal of the condition. + +uv +-- + +Local tests with uv 0.7.0 and 0.13.6, using ``uv pip compile --no-index`` with +``--find-links`` pointing at the two wheels, selected version 1.0. No preview +feature was enabled. + +PDM +--- + +PDM provides an independent implementation of candidate-level fallback. +`PDM 2.13.0 +`__ +has a resolver that catches requirement and specifier errors when obtaining +candidate dependencies and marks that candidate as conflicting. The same policy is present in +`PDM 2.29.5 +`__. +With wheels for both versions of ``marker-test`` available from a local index, +PDM 2.29.5 rejected version 2.0 as invalid metadata and locked version 1.0. + +Poetry +------ + +Poetry has long used permissive recovery when converting published +``Requires-Dist`` entries into dependencies. Both +`Poetry 1.8.2 +`__ +and `Poetry 2.5.1 +`__ +catch an invalid-marker exception, remove everything after the semicolon, and +parse the dependency again. This turns a conditional dependency into an +unconditional one. The exception name changed from ``InvalidMarker`` to +``InvalidMarkerError``, but the recovery policy remained. +Metadata-conversion tests with both versions confirmed that +``idna; python_version == "not a version"`` becomes unconditional ``idna``. + +Unknown marker names follow a different recovery path. Both versions parse +``idna; "pointer_bits::64" in sys_abi_info`` as an invalid requirement, log a +warning, and skip the entire dependency. Full ``poetry lock`` runs against a +local index confirmed this behavior in both versions: they locked +``marker-test`` 2.0 without ``idna``, rather than rejecting version 2.0 and +selecting version 1.0. + +Related issues document the history of this inspection path: + +* `Poetry #3629 `__ + reported malformed Kivy wheel dependencies in 2021. The reporter quoted the + marker-stripping code and questioned whether warning instead of raising could + have consequences. Closing the issue did not remove that recovery policy. +* `Poetry #7729 `__ + reported a parsing failure with malformed ``super-state-machine`` metadata, + where older Poetry versions had warned and skipped the dependency. It shows + that recovery versus failure has varied across versions and parsing errors. +* `Poetry #9154 `__ + concerns a Windows file URL in dependency metadata, with a traceback through + the same inspection code. It is a related cross-platform metadata problem, + rather than a report about unknown markers. + +None of these issues specifically addresses a future standardized marker that +an older consumer does not understand. In that case, the producer's metadata +can be valid even though the consumer rejects it. Stripping the condition can +add an unsuitable dependency; skipping the requirement can omit a needed one. +Neither preserves dependency semantics. The safe fallback is to reject the +containing package version and consider alternatives.