Redesign sdist handle to support VCS urls #776

Goal

Redesign resolve_from_provider and sdist handling to simplify default path and support VCS urls. The current API was designed and implemented before we encountered projects that need VCS download and git submodules.

current design

1. resolve_from_provider() / resolve_source() returns an URL to a tar.gz file and version
2. download_source() downloads the tar.gz file
3. prepare_source unpacks and modifies the sources (applies patches and modifiations)
4. build_sdist either repacks the sources as tar.gz or calls PEP 517 hook to build to create a tar.gz bundle

The current API has several flaws:

• Fromager unpacks and repacks all sdists. In majority of cases, there is no need to unpack the tar.gz and repack it again. Unpacking and repacking is only necessary when Fromager has patches, a non-default prepare_source hook, or project overrides.
• download_source has to return a path to a local file tar.gz and prepare_source expects a tar.gz file. This design does not work well for VCS checkouts. Some build backends expect a full git checkout, so build_sdist needs to have .git directory.
• The GitLab and GitHub tag provider return an URL to a git tar bundle. The tar bundle does not work if the project uses git submodules or the build backend (e.g. setuptools-scm) expects a .git directory.

redesign proposal

• change the resolver API to return a rich object (dataclass) instead of just an URL to a tar.gz and version. The rich object holds req: Requirement, version: Version, url: str, url type (tar, git, ...), and a VCS tag/ref/revision argument.
• change or replace download_source, prepare_source, build_sdist, so Fromager can deal with four different cases:
  • sdist tar.gz without modifications
  • sdist tar.gz with patches, project overrides, and/or modifications
  • VCS url, project supports pep517_build_sdist
  • VCS url, project does not support PEP 517 sdist (e.g. PyArrow)

Maybe we only need one or two hooks to get from resolver result to an sdist?