diff --git a/HOWTO-handle-security-issue.md b/HOWTO-handle-security-issue.md index 039cbdc2..44b1d42b 100644 --- a/HOWTO-handle-security-issue.md +++ b/HOWTO-handle-security-issue.md @@ -281,7 +281,7 @@ or doing additional hardening inspired by the original problem. ## Making a release (Moderate / High / Critical severity issues) -Most of the release mechanics are found in [HOWTO-make-a-release.md] and +Most of the release mechanics are found in [HOWTO-release.md] and will not be repeated here. For premium releases, there's nothing additional to do, simply work off the @@ -338,7 +338,7 @@ The recommended preparation is this: ``` At this point, your local repository should be properly set up to perform -the release following the instructions in [HOWTO-make-a-release.md]. When +the release following the instructions in [HOWTO-release.md]. When publishing, push to the [private openssl/openssl repository], and you may also want to push to the [private openssl/security repository] for good measure. @@ -605,7 +605,7 @@ Finish by publishing all the applicable [public openssl/openssl repository]: https://github.com/openssl/openssl [Security Policy]: https://openssl-library.org/policies/general/security-policy [GitHub Security Advisory]: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/about-repository-security-advisories -[HOWTO-make-a-release.md]: ./HOWTO-make-a-release.md +[HOWTO-release.md]: ./HOWTO-release.md [Write an early advisory text]: #write-an-early-advisory-text [Write a security advisory text]: #write-a-security-advisory-text diff --git a/HOWTO-make-a-release.md b/HOWTO-make-a-release.md deleted file mode 100644 index f77408c0..00000000 --- a/HOWTO-make-a-release.md +++ /dev/null @@ -1,170 +0,0 @@ -# HOW TO MAKE A RELEASE - -This file documents the overall OpenSSL release process. Some parts of this -process is documented in other files that go into deeper detail. - -# Table of contents - -- [Prerequisites](#prerequisites) - - [Software](#software) - - [Repositories](#repositories) - - [A method for reviewing](#a-method-for-reviewing) -- [Preparation tasks](#preparation-tasks) - - [Freeze the source repository](#freeze-the-source-repository) [three business days before release] - - [Make sure that the openssl source is up to date](#make-sure-that-the-openssl-source-is-up-to-date) -- [Stage the release](#stage-the-release) -- [Publish the release](#publish-the-release) -- [Post-releasing tasks](#post-publishing-tasks) - - [Unfreeze the source repository](#unfreeze-the-source-repository) - - [Update compatibility tests](#update-the-provider-backwards-compatibility-tests) - - [Keep in touch](#keep-in-touch) - - -# Prerequisites - -## Software - -Apart from the basic operating system utilities, you must have the following -programs in you `$PATH`: - -- ssh -- git - -(note: this may not be a complete list) - -## Repositories - -You must have access to the following repositories: - -- `git@github.openssl.org:openssl/openssl.git` - - This is the public source repository, so is only necessary to stage - a public release, which are those that haven't reached End-Of-Life - yet. - -- `git@github.com:openssl/security.git` - - This is the security source repository, where security fixes are - staged before being publically released. It is used as source - repository instead of `openssl/openssl` to stage a security - release. - -- `git@github.openssl.org:openssl/premium.git` - - This is the source repository for premium customers, used for both - security and non-security releases. - -## A method for reviewing - -In some parts of the release process, peer review may apply. The review -methods are specified in more detail in those parts. - -# Preparation tasks - -Some of the actions in this section need to be repeated for each OpenSSL -version released. - -## Prepare your repository checkouts - -- For each release to be staged, you need to checkout its source - repository, which is one of: - - - `git clone git@github.openssl.org:openssl/openssl.git` - - `git clone git@github.com:openssl/security.git` - - `git clone git@github.openssl.org:openssl/premium.git` - -## Freeze the source repository - -Three business day before the release, freeze the appropriate source -repository. - -This locks out everyone but the named user, who is doing the release, from -doing any pushes. Someone other than the person doing the release should -run the command. - -This must be done from a checkout of that source repository, so for public -as well as security releases: - - git push git@github.openssl.org:openssl/openssl.git HEAD:refs/frozen/NAME - -and for premium releases: - - git push git@github.openssl.org:openssl/premium.git HEAD:refs/frozen/NAME - -Where `NAME` is the github username of the user doing the release. - -Note: it currently doesn't matter what source branch is used when pushing, -the whole repository is frozen either way. The example above uses whatever -branch you happen to have checked out. - -Note: `git@github.openssl.org:openssl/security.git` is derived from -`git@github.openssl.org:openssl/openssl.git`, so when freezing the latter, -it's implied that the former is frozen as well. - -## Notify comitters and platform owners of the freeze - -When the tree is frozen, an email should be sent to openssl-comitters@openssl.org, as well as to the community platform owners (documented [here](https://openssl-library.org/policies/platforms/))indicating that the tree is frozen, and how long the freeze is expected to last. It should also indicate to the community platform owners that additional, more frequent testing during the freeze would be appreciated, as community platforms are not all in our CI system. This will help mitigate inadvertent breakage during the freeze period on platforms we do not consistently test against. - - -## Make sure that the openssl source is up to date - -For security releases, merge all applicable and approved security PRs. - -*NOTE: the files CHANGES.md and NEWS.md are called CHANGES and NEWS in -OpenSSL versions before version 3.0* - -For each source checkout, make sure that the CHANGES.md and NEWS.md files -have been updated and reviewed. - -The NEWS file should contain a summary of any changes for the release; -for a security release, it's often simply a list of the CVEs addressed. -You should also update NEWS.md in the master branch to include details of -all releases. Only update the bullet points - do not change the release -date, keep it as **under development**. - -# Stage the release - -See [HOWTO-stage-a-release.md](HOWTO-stage-a-release.md), which describes -this in detail. - -This may be done independently of [publishing the release](#publish-the-release). -However, if done manually, the same person should stage and publish the -release, as doing it this way depends on that person's local clones and -checkouts. - -# Publish the release - -See [HOWTO-publish-a-release.md](HOWTO-publish-a-release.md), which -describes this in detail. - -This may be done independently of [staging the release](#stage-the-release). -However, if done manually, the same person should stage and publish the -release, as doing it this way depends on that person's local clones and -checkouts. - -# Post-releasing tasks - -## Unfreeze the source repository. - -This must be done from a checkout of the appropriate source repo: - - git push --delete git@github.openssl.org:openssl/openssl.git \ - refs/frozen/NAME - -or: - - git push --delete git@github.openssl.org:openssl/premium.git \ - refs/frozen/NAME - -## Update the provider backwards compatibility tests - -In the case of a new minor release, the tags being tested by the -`.github/workflows/provider-compatibility.yml` -script need to be updated for the released version and **all** subsequent (i.e. -higher numbered versions) to include the tag for this release. - -## Keep in touch - -Check mailing lists over the next few hours for reports of any success or -failure. If necessary fix these and in the worst case make another -release. diff --git a/HOWTO-publish-a-release.md b/HOWTO-publish-a-release.md deleted file mode 100644 index b9588572..00000000 --- a/HOWTO-publish-a-release.md +++ /dev/null @@ -1,219 +0,0 @@ -# HOW TO PUBLISH A RELEASE - -This file documents how to publish an OpenSSL release. Please fix any errors -you find while doing, or just after, your next release! - -Releases are staged by another procedure, separate from this. - -# Table of contents - -- [Prerequisites](#prerequisites) - - [Software](#software) - - [Repositories](#repositories) - - [SSH access](#check-your-access) -- [Publish the release](#publish-the-release) - - [Update the source repositories](#update-the-source-repositories) - - [Publish GitHub release](#publish-github-release) - - [Update the release metadata](#update-the-release-metadata) -- [Post-publishing tasks](#post-publishing-tasks) - - [Check automations](#check-automations) - - [Check the website](#check-the-website) - - [Send the announcement mail](#send-the-announcement-mail) - - [Send out the Security Advisory](#send-out-the-security-advisory) - - [MITRE / CVE.org](#mitre-cve-org) - -# Prerequisites - -## Software - -Apart from the basic operating system utilities, you must have the following -programs in you `$PATH`: - -- ssh -- git - -(note: this may not be a complete list) - -## Repositories - -You must have access to the following repositories: - -- `git@github.com:openssl/release-metadata.git` - - This contains files to be updated as part of any release. - -- Any of: - - - `git@github.openssl.org:openssl/openssl.git` - - This is the public source repository, so is only necessary to stage - a public release, which are those that haven't reached End-Of-Life - yet. - - - `git@github.com:openssl/security.git` - - This is the security source repository, where security fixes are - staged before being publically released. It is used as source - repository instead of `openssl/openssl` to stage a security - release. - - - `git@github.openssl.org:openssl/premium.git` - - This is the source repository for premium customers, used for both - security and non-security releases. - -## SSH access - -To perform a release, you must have appropriate access to OpenSSL's -development host, dev.openssl.org. To test this, try to log in with ssh: - - ssh dev.openssl.org - -You must also check that you can perform tasks as the user 'openssl' on -dev.openssl.org. When you have successfully logged in, test your access to -that user with sudo: - - sudo -u openssl id - -# Publish the release - -## Update the source repositories - -Finish up by pushing your local changes to the appropriate source repo as -instructed by `$TOOLS/release-tools/stage-release.sh`, which was performed -when [staging the releases](HOWTO-stage-a-release.md). You may want to -sanity check the pushes by inserting the `-n` (dry-run) option. - -## Publish GitHub release - -When a tag is pushed to the GitHub repository the automation creates a draft -release in https://github.com/openssl/openssl/releases. Check the signed -announcement .asc file. Check that the tarball length and hashes match in -the .md5, .sha1, .sha256. - -For the release notes [^1], we currently use the same text as is added in the -`newsflash.md` file to announce the release. - -[^1]: The release notes field has previously been described as "description" - -If this is an alpha or beta release, check the "Set as a pre-release" -checkbox. - -If this is the latest release version, check the "Set as the latest release" -checkbox. - -Finish up by clicking "Publish release". - -## Update the release metadata - -*The changes in this section should be made in your clone of the release -data repo* - -- Newsflash *[only for public releases]* - - Update the newsflash.md file. This normally is one or two lines. Just - copy and paste existing announcements making minor changes for the date - and version number as necessary. If there is an advisory then ensure - you include a link to it. - -- Security advisory *[both public and premium releases]* - - Update the vulnerabilities.xml file if appropriate. - - If there is a Security Advisory then copy it into the secadv directory. - -Make a pull request from your changes, against the release metadata repo -(the release metadata repo being `git@github.com:openssl/release-metadata.git`). -Await approval from reviewers, then merge the pull request. - -# Post-publishing tasks - -## Check the website - -Verify that the release notes, which are built from the CHANGES.md file -in the release, have been updated. This is done automatically by OpenSSL -automation; if you see a problem, check if the web build job has been -performed yet, you may have to wait a few minutes before it kicks in. - -Wait for a while for the CDN flush to work (normally within a few minutes). - -Check the download page has updated properly: - -- - -Check the notes look sensible at: - -- - -Also check the notes here: - -- -- -- -- - -## Send the announcement mail - -Send out the announcements. Generic release announcement messages will be -created automatically by the build script and the commands you need to use -to send them were displayed when you executed `do-release.pl` above. They -should be sent from the account of the person that owns the key used for -signing the release announcement. - -## Send out the Security Advisory - -*The secadv file mentioned in this section is the Security Advisory -that you copied into the release data repo* - -*This section is only applicable if this is a security release* - -Start with signing the Security Advisory as yourself: - - gpg --clearsign secadv_FILENAME.txt - -Then copy the result to the temporary directory on dev.openssl.org: - - scp secadv_FILENAME.txt.asc dev.openssl.org:/tmp - -To finish, log in on dev.openssl.org and send the signed Security -Advisory by email as the user that signed the advisory. - -For all releases, send it to the default set of public mailing lists, -replacing `YOU@openssl.org` with your email address: - - EMAIL="YOU@openssl.org" REPLYTO="openssl@openssl.org" \ - mutt -s "OpenSSL Security Advisory" \ - openssl-project openssl-users openssl-announce \ - Before OpenSSL 3.0 these files are called `CHANGES` and `NEWS`. The +> tooling handles both, and they are not interchangeable. + +## Running the release + +Run the pipeline for the release type you are making. The parameters are +the release branch, the reviewers, and whether this is an alpha, beta or +final release. + +The pipeline will stop and wait at the review gate. Two approvals are +required, and the pipeline polls for them; it does not proceed on one. + +### What `stage-release` does + +Run from inside a source worktree, on a clean tree, on `master` or a +recognised release branch. In order: + +1. Validates the reviewers against the committer and CLA databases. This is + a **preflight** check — an unknown reviewer, one without a CLA, or one who + is not a committer aborts the run in under a second, before anything is + written or built. +2. Updates copyright years and commits, if anything changed. +3. Runs `./Configure` and `make update`, plus symbol renumbering and FIPS + checksums where the branch has those targets, and commits. +4. Writes the release version, applies the per-file `CHANGES`/`NEWS`/`README` + edits, commits, and creates an **annotated, unsigned** tag. +5. Builds the tarball and writes `.sha1` and `.sha256` beside it. +6. Writes `openssl-.dat` describing what was staged. +7. Returns the branch to development with a post-release commit, and — when + releasing from `master` at patch 0 — creates the new release branch and + moves `master` on to the next minor version. + +It signs nothing, pushes nothing and uploads nothing. It does not generate +announcement text; that was removed. + +Run `release-tools/stage-release --manual` for the full description. + +## Signing + +Release artifacts are OpenPGP-signed with a key held in a hardware security +module. Signing happens in a dedicated pipeline stage pinned to the one +agent with HSM access; the tree crosses agents as a stash, and only the +signed tag object and the detached signature come back. + +The certificate is two-tier: a Certify-only primary key protected by an +operator card quorum, and a module-protected signing subkey used unattended. +**Release artifacts and release tags are signed with the subkey, never the +primary.** + +The published certificate is fetched fresh from WKD for +`openssl@openssl.org` at signing time rather than being kept in a keystore. +The current certificate fingerprint is +`B146647E45A7B33947AB226B2A2C87D161692D40`, signing subkey key ID +`64ED7B1DCCE71CB2`. + +The release key used before 2026 — +`BA5473A2B0587B07FB27CF2D216094DFD0CB81EF` — is retired and is no longer +used by any pipeline. + +Two couplings worth knowing: + +- A release tag is rejected by the `check-tags` pre-receive hook unless the + signing certificate is in that hook's allowlist. When the release + certificate is rotated, the hook repository must be updated in the same + breath. +- The hook verifies with GnuPG and only cares *which* key signed. Tags + signed through Sequoia are ordinary v4 RSA signatures that GnuPG verifies + fine, so there is no tool incompatibility to work around. + +The key lifecycle itself — generating the primary and subkey, issuing and +rotating the certificate, issuing revocation certificates — is documented +separately in [`openpgp-tools/README.md`](openpgp-tools/README.md), which is +also where the module-specific details live. + +## After the pipeline + +### Publish the GitHub release + +The pipeline leaves a **draft** release with the signed tarball, its `.asc` +signature and its checksums attached. Check that: + +- the tarball length and the checksums match, +- the signature verifies against the published certificate, +- alpha and beta releases are marked as a pre-release, +- the newest release is marked as the latest release. + +Then publish it. The release notes are pulled automatically from the +release-metadata news endpoint. + +### Update the release metadata + +In the `release-metadata` repository: + +- Update `data.json` if the schedule, EOL or LTS status changed. +- For a security release, add the advisory to `secadv/` and the + machine-readable record to `secjson/`. + +Open a pull request, get it reviewed, and merge it. + +> `newsflash.md` in that repository is historical and no longer drives +> anything. Do not add to it. + +### Check the website + +The website renders the release table, the release notes and the advisories +from `release-metadata`, and deploys on merge. Give it a few minutes, then +check: + +- the source download page lists the new release, +- the news log shows it, +- the per-branch release notes pages are updated. + +If something is missing, check whether the site build has run before +assuming the data is wrong. + +## Security releases + +In addition to everything above: + +### Send the advisory + +Sign the advisory as yourself and send it, from your own address, to the +public project, users and announce lists. Send it separately to +`oss-security@lists.openwall.com` rather than cross-posting. + +Check the list archives to confirm the messages arrived. + +### Register the CVEs + +Inform MITRE about any CVE in the release; the procedure is at the top of +`cvepool.txt` in the security repository. + +Close the GitHub advisory without publishing it there, and delete the +private fork if one was created. + +## Post-release tasks + +### Unfreeze the source repository + +Clear the `openssl_repo_releaser` value in the Pulumi stack configuration — +set it to the empty string or remove it — and apply the stack. That removes +the Releaser collaborator and disables both freeze rulesets. + +### Update the provider compatibility tests + +For a new minor release, add the new tag to +`.github/workflows/provider-compatibility.yml` in the source repository — +for the released version **and every later branch**. + +### Keep in touch + +Watch the mailing lists for the next few hours. Fix what comes up, and in +the worst case make another release. + +## Doing it by hand + +The manual path exists for when automation fails or for a release the +pipelines are not set up for. It is the same tool, run directly: + + git clone tools # referred to below as $TOOLS + git clone openssl + cd openssl + git checkout + $TOOLS/release-tools/stage-release --reviewer=NAME --reviewer=NAME + +Staging several releases from one repository is easiest with worktrees: + + git worktree add ../openssl-3.5 openssl-3.5 + +Then check the result before pushing anything: read the commits it made, +read the `.dat` file, and confirm the tag points at the release commit. The +tag is unsigned at this point and must be signed on the signing agent before +it is pushed, because the pre-receive hook rejects unsigned release tags. + +**Do not push** until the review and signing steps have happened. diff --git a/HOWTO-stage-a-release.md b/HOWTO-stage-a-release.md deleted file mode 100644 index 26699cde..00000000 --- a/HOWTO-stage-a-release.md +++ /dev/null @@ -1,151 +0,0 @@ -# HOW TO STAGE A RELEASE - -This file documents how to make an OpenSSL release. Please fix any errors -you find while doing, or just after, your next release! - -Anyone with access to the necessary resources may stage a release. Reviews -for doing so isn't necessary, that's done "naturally" as part of publishing, -see [HOWTO-publish-a-release.md](HOWTO-publish-a-release.md). - -# Automation - -**Staging releases is becoming automated**, so this document will soon only -be interesting to know how to perform this manually, should the need arise -(automation failure, or to stage releases that automation isn't prepared -for). - -This automation is currently still undergoing tests, and isn't quite -reflected in [HOWTO-publish-a-release.md](HOWTO-publish-a-release.md). -Updates pending! - -# Table of contents - -- [Prerequisites](#prerequisites) - - [Software](#software) - - [Repositories](#repositories) - - [PGP / GnuPG key](#pgp-gnupg-key) - - [Prepare your repository checkouts](#prepare-your-repository-checkouts) -- [Staging tasks](#staging-tasks) - - [Generate the announcement text](#generating-the-tarball-and-announcement-text) - - [Remember the results](#remember-the-results) - -# Prerequisites - -## Software - -Apart from the basic operating system utilities, you must have the following -programs in you `$PATH`: - -- openssl -- gpg -- git - -(note: this may not be a complete list) - -## Repositories - -You must have access to the following repositories: - -- `git@github.openssl.org:otc/tools.git` - - This contains the release staging tool. - -- Any of: - - - `git@github.openssl.org:openssl/openssl.git` - - This is the public source repository, so is only necessary to stage - a public release, which are those that haven't reached End-Of-Life - yet. - - - `git@github.com:openssl/security.git` - - This is the security source repository, where security fixes are - staged before being publically released. It is used as source - repository instead of `openssl/openssl` to stage a security - release. - - - `git@github.openssl.org:openssl/premium.git` - - This is the source repository for premium customers, used for both - security and non-security releases. - -## PGP / GnuPG key - -You must have OpenSSL's team key: - - $ gpg --list-secret-key BA5473A2B0587B07FB27CF2D216094DFD0CB81EF - sec rsa4096 2024-04-08 [SC] [expires: 2026-04-08] - BA5473A2B0587B07FB27CF2D216094DFD0CB81EF - uid [ultimate] OpenSSL - -If you don't have it and think you should, get an export from someone on the -team that has it. - -## Prepare your repository checkouts - -- To stage a release, you need to checkout the release staging tool - - git clone git@github.openssl.org:otc/tools.git tools - - The resulting directory will be referred to as `$TOOLS` - -- For each release to be staged, you need to checkout its source - repository, which is one of: - - - `git clone git@github.openssl.org:openssl/openssl.git` - - `git clone git@github.com:openssl/security.git` - - `git clone git@github.openssl.org:openssl/premium.git` - -- If you're staging multiple releases from one repository in one go, there - are many ways to deal with it. One possibility, available since git 2.5, - is to use `git worktree`: - - (cd openssl; - git worktree add ../openssl-1.1.1 OpenSSL_1_1_1-stable) - -# Staging tasks - -## Generate the announcement text - -*The changes in this section should be made in your clone of the openssl -source repo* - -To generate and stage announcement text, there is a script -`$TOOLS/release-tools/stage-release.sh`. It's expected to be run -while standing in the worktree of an OpenSSL source repository, and the -expects the checked out branch to be the branch to stage the release from, -matching one of OpenSSL release branch patterns. - -The stage-release script has a multitude of other options that are useful -for specific cases, and is also self-documented: - -- To get a quick usage reminder: - - $TOOLS/release-tools/stage-release.sh --help - -- To get a man-page: - - $TOOLS/release-tools/stage-release.sh --manual - -It is generally called like this: - - $TOOLS/release-tools/stage-release.sh --reviewer=NAME \ - --local-user=BA5473A2B0587B07FB27CF2D216094DFD0CB81EF - -This scripts will perform a number of preparatory tasks, such as updating -the copyright year, running `make update`, update release dates, and move -the branch to the next development version. This results not only in a -staged announcement text, but also in a set of commits. - -After having run the stage-release script, verify that its results are -sensible. Check the commits that were added, using for example `git log`. -Review the announcment file. Check the data left in the metadata .dat file. - -*Do not push* the local commits to the source repo at this stage. - -## Remember the results - -*Make sure to take note of all the instructions the stage-release script gave -you at the end. They will be needed when -[publishing the release](HOWTO-publish-a-release.md).* diff --git a/README b/README deleted file mode 100644 index 14249505..00000000 --- a/README +++ /dev/null @@ -1,7 +0,0 @@ -A collection of tools and instructions useful in OpenSSL development. - -Each set of tools is in its own subdirectory and has its own manuals -and READMEs. - -More generic instructions are in this top directory, called -HOWTO-something.md diff --git a/README.md b/README.md new file mode 100644 index 00000000..94fa82eb --- /dev/null +++ b/README.md @@ -0,0 +1,25 @@ +# OpenSSL tools + +A collection of tools and instructions useful in OpenSSL development. + +Most sets of tools are in their own subdirectory with their own README: + +| Directory | What it holds | +| --- | --- | +| [`release-tools/`](release-tools/README.md) | staging an OpenSSL release | +| [`review-tools/`](review-tools/README.md) | reviewing and merging pull requests | +| [`openpgp-tools/`](openpgp-tools/README.md) | the release signing key's lifecycle | +| [`OpenSSL-Query/`](OpenSSL-Query/README.md) | Perl client for the committer and CLA database; no longer used by anything here | +| `nist-conversion/` | converting NIST DRBG test vectors for `evp_test` | +| `statistics/` | generating test data for the OpenSSL source tree | + +`lib/` is the exception: it is not a set of tools but the shared Python that +`release-tools` and `review-tools` are both built from, with one test suite +covering both. The executables stay in their own directories because +external workflows invoke them by absolute path. + +Instructions that span more than one set of tools are in this top directory: + +- [HOWTO-release.md](HOWTO-release.md) — the whole release process +- [HOWTO-handle-security-issue.md](HOWTO-handle-security-issue.md) — handling + an embargoed security issue, up to and including its release diff --git a/github-tools/stale.py b/github-tools/stale.py deleted file mode 100755 index 0ea433d4..00000000 --- a/github-tools/stale.py +++ /dev/null @@ -1,310 +0,0 @@ -#! /usr/bin/env python3 -# requires python 3 -# -# A script to run daily that looks through OpenSSL github PRs -# and creates stats, next actions, and makes comments and closes -# stale issues. -# -# note that we'd use pyGithub but we can't as it doesn't fully handle the timeline objects -# as of Feb 2020 and we might want to parse timeline if we want to ignore certain things -# from resetting 'updated' date -# -# mark@openssl.org Feb 2020 -# -import requests -import json -from datetime import datetime, timezone -from optparse import OptionParser -from statistics import median -import collections -import csv - -api_url = "https://api.github.com/repos/openssl/openssl" - -def convertdate(date): - return datetime.strptime(date.replace('Z',"+0000"), "%Y-%m-%dT%H:%M:%S%z") - -def addcommenttopr(issue,comment): - newcomment = {"body":comment} - url = api_url + "/issues/" + str(issue) + "/comments" - res = requests.post(url, data=json.dumps(newcomment), headers=headers) - if (res.status_code != 201): - print("Error adding comment", res.status_code, res.content) - return - -# Note: Closing an issue doesn't add a comment by itself - -def closepr(issue,comment): - newcomment = {"body":comment} - url = api_url + "/issues/" + str(issue) + "/comments" - res = requests.post(url, data=json.dumps(newcomment), headers=headers) - if (res.status_code != 201): - print("Error adding comment", res.status_code, res.content) - url = api_url + "/issues/" + str(issue) - res = requests.patch(url, data=json.dumps({"state":"closed"}), headers=headers) - if (res.status_code != 200): - print("Error closing pr", res.status_code, res.content) - return - -# Get all the open pull requests, filtering by approval: done label - -stale = collections.defaultdict(list) -now = datetime.now(timezone.utc) - -def parsepr(pr, days): - if (debug): - print ("Getting timeline for ",pr['number']) - url = api_url + "/issues/" + str(pr['number']) + "/timeline?per_page=100&page=1" - res = requests.get(url, headers=headers) - repos = res.json() - while 'next' in res.links.keys(): - res = requests.get(res.links['next']['url'], headers=headers) - repos.extend(res.json()) - - comments = [] - commentsall = [] - readytomerge = 0 - reviewed_state = "" - sha = "" - - for event in repos: - if (debug): - print (event['event']) - print (event) - print () - try: - eventdate = "" - if (event['event'] == "commented"): - # we need to filter out any comments from OpenSSL Machine - if "openssl-machine" in event['actor']['login']: - if (debug): - print("For stats ignoring automated comment by openssl-machine") - commentsall.append(convertdate(event["updated_at"])) - else: - eventdate = event["updated_at"] - elif (event['event'] == "committed"): - sha = event["sha"] - eventdate = event["author"]["date"] - elif (event['event'] == "labeled" or event['event'] == "unlabeled"): - eventdate = event['created_at'] - elif (event['event'] == "reviewed"): - reviewed_state = "reviewed:"+event['state'] # replace with last review - eventdate = event['submitted_at'] - elif (event['event'] == "review_requested"): - # If a review was requested after changes requested, remove changes requested label - reviewed_state = "reviewed:review pending"; - eventdate = event['created_at'] - if (eventdate != ""): - comments.append(convertdate(eventdate)) - if (debug): - print(reviewed_state) - except: - return (repos['message']) - - # We want to ignore any comments made by our automated machine when - # looking if something is stale, but keep a note of when those comments - # were made so we don't spam issues - - dayssincelastupdateall = int((now - max(comments+commentsall)).total_seconds() / (3600*24)) - dayssincelastupdate = int((now - max(comments)).total_seconds() / (3600*24)) - if (dayssincelastupdate < days): - if (debug): - print("ignoring last event was",dayssincelastupdate,"days:",max(comments+commentsall)) - return - - labellist = [] - if 'labels' in pr: - labellist=[str(x['name']) for x in pr['labels']] - if 'milestone' in pr and pr['milestone']: - labellist.append("milestone:"+pr['milestone']['title']) - labellist.append(reviewed_state) - labels = ", ".join(labellist) - - # Ignore anything "tagged" as work in progress, although we could do this earlier - # do it here as we may wish, in the future, to still ping stale WIP items - - if ('title' in pr and 'WIP' in pr['title']): - return - - data = {'pr':pr['number'],'days':dayssincelastupdate,'alldays':dayssincelastupdateall,'labels':labels} - stale["all"].append(data) - - if debug: - print (data) - - # The order of these matter, we drop out after the first one that - # matches. Try to guess which is the most important 'next action' - # for example if something is for after 1.1.1 but is waiting for a CLA - # then we've time to get the CLA later, it's deferred. - - if ('stalled: awaiting contributor response' in labels): - stale["waiting for reporter"].append(data) - return - if ('hold: need omc' in labels or 'approval: omc' in labels): - stale["waiting for OMC"].append(data) - return - if ('hold: need otc' in labels or 'approval: otc' in labels): - stale["waiting for OTC"].append(data) - return - if ('hold: cla' in labels): - stale["cla required"].append(data) - return - if ('review pending' in labels): - stale["waiting for review"].append(data) - return - if ('reviewed:changes_requested' in labels): - stale["waiting for reporter"].append(data) - return - - url = api_url + "/commits/" + sha + "/status" - res = requests.get(url, headers=headers) - if (res.status_code == 200): - ci = res.json() - if (ci['state'] != "success"): - stale["failed CI"].append(data) - return - - stale["all other"].append(data) - return - - -def getpullrequests(days): - url = api_url + "/pulls?per_page=100&page=1" # defaults to open - res = requests.get(url, headers=headers) - repos = res.json() - prs = [] - while 'next' in res.links.keys(): - res = requests.get(res.links['next']['url'], headers=headers) - repos.extend(res.json()) - - # In theory we can use the updated_at date here for filtering, but in practice - # things reset it --- like for example when we added the CLA bot, also any - # comments we make to ping the PR. So we have to actually parse the timeline - # for each event. This is much slower but more accurate for our metrics and - # we don't run this very often. - - # we can ignore anything with a created date less than the number of days we - # care about though - - try: - for pr in repos: - dayssincecreated = int((now - convertdate(pr['created_at'])).total_seconds() / (3600*24)) - if (dayssincecreated >= days): - prs.append(pr) - except: - print("failed", repos['message']) - return prs - -# main - -parser = OptionParser() -parser.add_option("-v","--debug",action="store_true",help="be noisy",dest="debug") -parser.add_option("-t","--token",help="file containing github authentication token for example 'token 18asdjada...'",dest="token") -parser.add_option("-d","--days",help="number of days for something to be stale",type=int, dest="days") -parser.add_option("-D","--closedays",help="number of days for something to be closed. Will commit and close issues even without --commit flag",type=int, dest="closedays") -parser.add_option("-c","--commit",action="store_true",help="actually add comments to issues",dest="commit") -parser.add_option("-o","--output",dest="output",help="write a csv file out") -parser.add_option("-p","--prs",dest="prs",help="instead of looking at all open prs just look at these comma separated ones") - -(options, args) = parser.parse_args() -if (options.token): - fp = open(options.token, "r") - git_token = fp.readline().strip('\n') - if not " " in git_token: - git_token = "token "+git_token -else: - print("error: you really need a token or you will hit the API limit in one run\n") - parser.print_help() - exit() -debug = options.debug -# since timeline is a preview feature we have to enable access to it with an accept header -headers = { - "Accept": "application/vnd.github.mockingbird-preview", - "Authorization": git_token -} -days = options.days or 31 -if (options.output): - outputfp = open(options.output,"a") - outputcsv = csv.writer(outputfp) - -prs = [] -if (options.prs): - for prn in (options.prs).split(","): - pr = {} - pr['number']=int(prn) - prs.append(pr) - -if (not prs): - if debug: - print("Getting list of open PRs not created within last",days,"days") - prs = getpullrequests(days) -if debug: - print("Open PRs we need to check", len(prs)) - -for pr in prs: - parsepr(pr, days) - -if ("waiting for OMC" in stale): - for item in stale["waiting for OMC"]: - if (item['alldays']>=days): - comment = "This PR is in a state where it requires action by @openssl/omc but the last update was "+str(item['days'])+" days ago" - print (" ",item['pr'],comment) - if (options.commit): - addcommenttopr(item['pr'],comment) - -if ("waiting for OTC" in stale): - for item in stale["waiting for OTC"]: - if (item['alldays']>=days): - comment = "This PR is in a state where it requires action by @openssl/otc but the last update was "+str(item['days'])+" days ago" - print (" ",item['pr'],comment) - if (options.commit): - addcommenttopr(item['pr'],comment) - -if ("waiting for review" in stale): - for item in stale["waiting for review"]: - if (item['alldays']>=days): - comment = "This PR is in a state where it requires action by @openssl/committers but the last update was "+str(item['days'])+" days ago" - print (" ",item['pr'],comment) - if (options.commit): - addcommenttopr(item['pr'],comment) - -if ("waiting for reporter" in stale): - for item in stale["waiting for reporter"]: - if (options.closedays and item['days']>=options.closedays): - comment = "This PR has been closed. It was waiting for the creator to make requested changes but it has not been updated for "+str(item['days'])+" days." - print (" ",item['pr'],comment) - if (options.commit): - closepr(item['pr'],comment) - elif (item['alldays']>=days): - comment = "This PR is waiting for the creator to make requested changes but it has not been updated for "+str(item['days'])+" days. If you have made changes or commented to the reviewer please make sure you re-request a review (see icon in the 'reviewers' section)." - print (" ",item['pr'],comment) - if (options.commit): - addcommenttopr(item['pr'],comment) - -if ("cla required" in stale): - for item in stale["cla required"]: - if (options.closedays and item['days']>=options.closedays): - comment = "This PR has been closed. It was waiting for a CLA for "+str(item['days'])+" days." - print (" ",item['pr'],comment) - if (options.commit): - closepr(item['pr'],comment) - elif (item['alldays']>=days): - comment = "This PR has the label 'hold: cla required' and is stale: it has not been updated in "+str(item['days'])+" days. Note that this PR may be automatically closed in the future if no CLA is provided. For CLA help see https://www.openssl.org/policies/cla.html" - print (" ",item['pr'],comment) - if (options.commit): - addcommenttopr(item['pr'],comment) - - -for reason in stale: - days = [] - for item in stale[reason]: - days.append(item['days']) - if options.output and reason !="all": - outputcsv.writerow([now,reason,item['pr'],item['labels'],item['days']]) - - print ("\n", reason," (", len(stale[reason]),"issues, median ",median(days)," days)\n"), - if (reason == "all" or "deferred" in reason): - print (" list of prs suppressed") - else: - for item in stale[reason]: - print (" ",item['pr'],item['labels'],"days:"+str(item['days'])) diff --git a/lib/.gitignore b/lib/.gitignore new file mode 100644 index 00000000..2cdf026c --- /dev/null +++ b/lib/.gitignore @@ -0,0 +1,6 @@ +__pycache__/ +*.pyc +.pytest_cache/ +.ruff_cache/ +.mypy_cache/ +dist/ diff --git a/lib/build-pyz b/lib/build-pyz new file mode 100755 index 00000000..ec0e2dde --- /dev/null +++ b/lib/build-pyz @@ -0,0 +1,278 @@ +#!/usr/bin/env python3 +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Build self-contained .pyz executables for the maintainer tools. + +One archive per entry point, each carrying the whole `openssl_tools` package. +They need nothing but a Python 3.10 interpreter -- no checkout, no PATH +setup, no installed packages -- so they can be dropped onto a build host or +copied into a container image. + + ./build-pyz # all four, into dist/ + ./build-pyz addrev cherry-checker # just these + ./build-pyz --output-dir /tmp/x + ./build-pyz --extension .pyz # if you want the suffix + +Archives are named exactly like the scripts they replace -- `addrev`, not +`addrev.pyz` -- so one can be dropped straight over an existing install. The +suffix is not needed on Unix: the kernel reads the shebang, and the extension +only matters to Windows file associations. + + ./build-pyz --multicall # one archive, busybox style + ./build-pyz --multicall --links # ...plus symlinks named after each tool + +A multicall archive decides what to run from the name it was invoked as, so +one file serves every tool. Unlike the scripts in release-tools/ and +review-tools/, a *copy* under a different name works as well as a symlink, +because the dispatch reads argv[0] rather than locating anything on disk. +It also answers to a subcommand, `openssl-tools addrev ...`, for when argv[0] +is not something you control. + +""" + +from __future__ import annotations + +import argparse +import shutil +import stat +import sys +import tempfile +import zipapp +from pathlib import Path + +HERE = Path(__file__).resolve().parent +PACKAGE = "openssl_tools" + +#: Executable name -> the module whose main() it runs. +ENTRY_POINTS = { + "stage-release": "openssl_tools.stagerelease.cli", + "addrev": "openssl_tools.reviewtools.addrev_cli", + "cherry-checker": "openssl_tools.reviewtools.cherry_cli", +} + +DEFAULT_INTERPRETER = "/usr/bin/env python3" + +#: Name of the single archive built by --multicall. +MULTICALL_NAME = "openssl-tools" + +#: No suffix by default, so an archive can replace a script of the same name. +DEFAULT_EXTENSION = "" + +MAIN_TEMPLATE = '''\ +# Generated by lib/build-pyz -- do not edit. +"""Entry point for {name}.""" +import sys + +if sys.version_info < (3, 10): + sys.exit( + "{name} needs Python 3.10 or later, found " + f"{{sys.version_info.major}}.{{sys.version_info.minor}}" + ) + +from {module} import main + +sys.exit(main()) +''' + + +MULTICALL_TEMPLATE = '''\ +# Generated by lib/build-pyz -- do not edit. +"""Multi-call entry point: dispatches on the name it was invoked as. + +Symlink, hardlink or copy this archive to any of the names below and it runs +that tool. Failing a recognised name, the first argument is taken as one. +""" +import sys +from pathlib import Path + +if sys.version_info < (3, 10): + sys.exit( + "openssl-tools needs Python 3.10 or later, found " + f"{{sys.version_info.major}}.{{sys.version_info.minor}}" + ) + +ENTRY_POINTS = {entry_points} + + +def _resolve(): + """Return (module, argv) for whatever we were asked to be.""" + invoked = Path(sys.argv[0]).name + for suffix in (".pyz", ".py"): + if invoked.endswith(suffix): + invoked = invoked[: -len(suffix)] + if invoked in ENTRY_POINTS: + return ENTRY_POINTS[invoked], sys.argv[1:] + + # Not installed under a tool name, so accept it as a subcommand. + if len(sys.argv) > 1 and sys.argv[1] in ENTRY_POINTS: + return ENTRY_POINTS[sys.argv[1]], sys.argv[2:] + + sys.exit( + f"{{invoked}}: not one of the tools in this archive.\\n" + "Link or copy it to one of these names, or name one as the first " + "argument:\\n " + "\\n ".join(sorted(ENTRY_POINTS)) + ) + + +def _main(): + module_name, argv = _resolve() + import importlib + + return importlib.import_module(module_name).main(argv) + + +sys.exit(_main()) +''' + + +def stage(source: Path, into: Path) -> None: + """Copy the package into `into`, leaving caches behind.""" + shutil.copytree( + source, + into / PACKAGE, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc", ".*"), + ) + + +def build( + name: str, + module: str, + output_dir: Path, + interpreter: str, + extension: str = DEFAULT_EXTENSION, +) -> Path: + target = output_dir / f"{name}{extension}" + + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + stage(HERE / PACKAGE, root) + main_py = root / "__main__.py" + main_py.write_text(MAIN_TEMPLATE.format(name=name, module=module)) + + output_dir.mkdir(parents=True, exist_ok=True) + zipapp.create_archive(root, target=target, interpreter=interpreter, compressed=True) + + target.chmod(target.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) + return target + + +def build_multicall(output_dir: Path, interpreter: str, extension: str = DEFAULT_EXTENSION) -> Path: + """Build the single archive that answers to every tool name.""" + target = output_dir / f"{MULTICALL_NAME}{extension}" + entry_points = ( + "{\n" + + "".join(f" {name!r}: {module!r},\n" for name, module in sorted(ENTRY_POINTS.items())) + + "}" + ) + + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + stage(HERE / PACKAGE, root) + main_py = root / "__main__.py" + main_py.write_text(MULTICALL_TEMPLATE.format(entry_points=entry_points)) + + output_dir.mkdir(parents=True, exist_ok=True) + zipapp.create_archive(root, target=target, interpreter=interpreter, compressed=True) + + target.chmod(target.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH) + return target + + +def make_links(archive: Path) -> list[Path]: + """Symlink the archive to each tool name, beside it.""" + links = [] + for name in sorted(ENTRY_POINTS): + link = archive.parent / name + if link.is_symlink() or link.exists(): + link.unlink() + link.symlink_to(archive.name) + links.append(link) + return links + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="build-pyz", + description="Build self-contained .pyz executables.", + ) + # Validated below rather than with choices=: for a positional with + # nargs="*", some Python versions check the empty default against the + # choices and reject a bare `build-pyz`. + parser.add_argument( + "names", + nargs="*", + default=[], + metavar="NAME", + help=f"what to build (default: all of {', '.join(sorted(ENTRY_POINTS))})", + ) + parser.add_argument( + "-o", + "--output-dir", + type=Path, + default=HERE / "dist", + help="where to write the archives (default: lib/dist)", + ) + parser.add_argument( + "-p", + "--interpreter", + default=DEFAULT_INTERPRETER, + help=f"shebang interpreter (default: {DEFAULT_INTERPRETER!r})", + ) + parser.add_argument( + "-e", + "--extension", + default=DEFAULT_EXTENSION, + metavar="EXT", + help="suffix for the archive names, e.g. .pyz (default: none, so the " + "output can replace a script of the same name)", + ) + parser.add_argument( + "--multicall", + action="store_true", + help="build one archive that dispatches on the name it is invoked as", + ) + parser.add_argument( + "--links", + action="store_true", + help="with --multicall, also symlink each tool name to the archive", + ) + args = parser.parse_args(argv) + + unknown = [name for name in args.names if name not in ENTRY_POINTS] + if unknown: + parser.error( + f"unknown tool: {', '.join(unknown)}; choose from " + ", ".join(sorted(ENTRY_POINTS)) + ) + + if args.multicall: + if args.names: + parser.error("--multicall builds every tool; do not name any") + archive = build_multicall(args.output_dir, args.interpreter, args.extension) + print(f"{archive} ({archive.stat().st_size // 1024} KiB)") + if args.links: + for link in make_links(archive): + print(f"{link} -> {archive.name}") + return 0 + + if args.links: + parser.error("--links only makes sense with --multicall") + + for name in args.names or sorted(ENTRY_POINTS): + built = build( + name, + ENTRY_POINTS[name], + args.output_dir, + args.interpreter, + args.extension, + ) + size = built.stat().st_size + print(f"{built} ({size // 1024} KiB)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/lib/openssl_tools/__init__.py b/lib/openssl_tools/__init__.py new file mode 100644 index 00000000..1a363437 --- /dev/null +++ b/lib/openssl_tools/__init__.py @@ -0,0 +1,26 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Shared Python for the OpenSSL maintainer tools. + +Two subpackages, so they can import each other with ordinary relative +imports rather than reaching across directories at runtime: + + stagerelease release staging, driven by release-tools/stage-release + reviewtools PR review helpers, driven by review-tools/addrev and + friends + +The executables stay in release-tools/ and review-tools/ because external +workflows -- Jenkins jobs, the ansible role that symlinks them into +/usr/local/bin -- invoke them by absolute path. Each one puts this +directory on sys.path and imports from here; there is no install step. + +Standard library only. +""" + +from __future__ import annotations + +__version__ = "1.0.0" diff --git a/lib/openssl_tools/reviewtools/__init__.py b/lib/openssl_tools/reviewtools/__init__.py new file mode 100644 index 00000000..8ab67301 --- /dev/null +++ b/lib/openssl_tools/reviewtools/__init__.py @@ -0,0 +1,100 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""OpenSSL PR review helpers. + +Standard library only, so these can be symlinked into /usr/local/bin on a +provisioned host without pulling in a package manager. + +The public entry point for other tools is `add_reviewers`, which resolves +reviewer names against the committer database and returns a rewritten commit +message. release-tools uses it directly, rather than shelling out to the +`addrev` script and needing it on PATH. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +from . import message as _message +from . import reviewers as _reviewers +from .errors import QueryError, ReviewError +from .policy import POLICIES, RepoPolicy, get_policy +from .query import Query +from .reviewers import PersonSource + +__version__ = "1.0.0" + +__all__ = [ + "POLICIES", + "PersonSource", + "Query", + "QueryError", + "RepoPolicy", + "ReviewError", + "add_reviewers", + "get_policy", + "resolve_reviewers", +] + + +def resolve_reviewers( + candidates: Sequence[str], + *, + author_email: str | None, + repo: str = "openssl", + release: bool = False, + query: PersonSource | None = None, +) -> list[str]: + """Validate explicitly named reviewers and return their tags. + + `candidates` are names the caller asked for, so each must belong to a + committer. `author_email` is collected automatically and is held to the + laxer standard described in reviewers.py. + + Raises ReviewError if a name is unknown, has no CLA, is not a committer, + or the repository's minimum is not met; QueryError if the database cannot + be reached. + """ + query = query or Query() + policy = get_policy(repo) + resolution = _reviewers.resolve( + query, + candidates, + author_email=author_email, + policy=policy, + release=release, + ) + _reviewers.validate(resolution, author_email=author_email, policy=policy, trivial=False) + _reviewers.require_any(resolution.reviewers) + return resolution.reviewers + + +def add_reviewers( + commit_message: str, + candidates: Sequence[str], + *, + author_email: str | None, + repo: str = "openssl", + release: bool = False, + prnum: str | None = None, + query: PersonSource | None = None, +) -> str: + """Return `commit_message` with validated Reviewed-by: trailers added.""" + tags = resolve_reviewers( + candidates, + author_email=author_email, + repo=repo, + release=release, + query=query, + ) + return _message.rewrite( + commit_message, + reviewers=tags, + repo=get_policy(repo).name, + prnum=prnum, + release=release, + ) diff --git a/lib/openssl_tools/reviewtools/addrev_cli.py b/lib/openssl_tools/reviewtools/addrev_cli.py new file mode 100644 index 00000000..4f399f0b --- /dev/null +++ b/lib/openssl_tools/reviewtools/addrev_cli.py @@ -0,0 +1,346 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""`addrev` -- add reviewer trailers to a range of commits. + +The argument grammar is positional and forgiving, because it is typed by +hand and forwarded verbatim from ghmerge. A bare number is a PR number, a +bare word is a reviewer, a word that looks like an object id is a commit +range, and `-3` means the last three commits. +""" + +from __future__ import annotations + +import re +import sys +from collections.abc import Sequence +from dataclasses import dataclass, field +from typing import TextIO + +from . import listing, message, reviewers, rewrite +from .commands import CommandRunner, run_command +from .errors import ReviewError +from .policy import POLICIES, get_policy +from .query import Query + +USAGE = """\ +usage: addrev args... + +option style arguments: + +--help Print this help and exit +--list List the known reviewers and exit (discards all other arguments) +--verbose Be a bit more verbose +--trivial Accepted for compatibility; has no effect. Put + 'CLA: Trivial' in the commit message instead. +--reviewer= A reviewer to be added on a Reviewed-by: line +--rmreviewers Remove all existing Reviewed-by: lines before adding reviewers +--commit= Only apply to commit +--myemail= Set email address. + Defaults to the result from git configuration setting user.email +--noself Do not add your own address as a possible reviewer +--nopr Do not require a PR number +--security Merge into the security repo; implies --nopr +[--prnum=]NNN Add a reference to GitHub pull request NNN +- Change the last commits. Defaults to 1 + +repository selectors (default: openssl): + +{repos} + +non-option style arguments can be: + +a string of alphanumeric or '-' characters, denoting a reviewer name. + +a string starting with @, denoting a reviewer's github ID. + +anything else will be used as a commit range. If no commit range is given, +HEAD^.. is assumed. + +Examples (all meaning the same thing): + + addrev 12345 -2 steve levitte + addrev --prnum=12345 steve @levitte HEAD^^.. + addrev 12345 --reviewer=steve --reviewer=levitte@openssl.org -2 +""" + +_PRNUM_RE = re.compile(r"^(?:--prnum=)?(\d{1,6})$") +_BARE_WORD_RE = re.compile(r"^\w[-\w]*$") +_OBJECT_ID_RE = re.compile(r"^[0-9a-f]{7,}") +_LAST_N_RE = re.compile(r"^-(\d+)$") +_REVIEWER_OPT_RE = re.compile(r"^--reviewer=(.+)$") +_MYEMAIL_RE = re.compile(r"^--myemail=(.+)$") +_COMMIT_RE = re.compile(r"^--commit=(.+)$") + + +@dataclass +class Invocation: + """What a command line asked for.""" + + reviewers: list[str] = field(default_factory=list) + prnum: str | None = None + commits: list[str] = field(default_factory=list) + repo: str = "openssl" + release: bool = False + remove_reviewers: bool = False + trivial: bool = False + verbose: bool = False + filter_args: str = "" + have_prnum: bool = False + use_self: bool = True + my_email: str | None = None + list_reviewers: bool = False + show_help: bool = False + warnings: list[str] = field(default_factory=list) + + +def parse_args(argv: Sequence[str]) -> Invocation: + result = Invocation() + + def set_filter(value: str) -> None: + if result.filter_args: + result.warnings.append(f"Warning: overriding previous filter args {result.filter_args}") + result.filter_args = value + + for arg in argv: + prnum = _PRNUM_RE.match(arg) + if prnum: + result.prnum = prnum.group(1) + result.have_prnum = True + continue + + if arg.startswith("@"): + result.reviewers.append(arg) + continue + + if _BARE_WORD_RE.match(arg): + # A long hex string is an object id, not somebody's name. + if _OBJECT_ID_RE.match(arg): + set_filter(arg) + else: + result.reviewers.append(arg) + continue + + reviewer = _REVIEWER_OPT_RE.match(arg) + if reviewer: + result.reviewers.append(reviewer.group(1)) + continue + + if arg == "--rmreviewers": + result.remove_reviewers = True + continue + + if arg == "--trivial": + result.trivial = True + continue + + if arg == "--verbose": + result.verbose = True + continue + + if arg == "--release": + result.release = True + continue + + if arg.startswith("--") and arg[2:] in POLICIES: + result.repo = arg[2:] + continue + + if arg == "--noself": + result.use_self = False + continue + + my_email = _MYEMAIL_RE.match(arg) + if my_email: + result.my_email = my_email.group(1) + continue + + if arg in ("--nopr", "--security"): + # Neither needs a PR reference: --nopr says so outright, and the + # security repo's commits do not carry one. + result.have_prnum = True + continue + + commit = _COMMIT_RE.match(arg) + if commit: + result.commits.append(commit.group(1)) + continue + + last_n = _LAST_N_RE.match(arg) + if last_n: + set_filter(f"HEAD~{last_n.group(1)}..") + continue + + if arg == "--list": + result.list_reviewers = True + break + + if arg in ("--help", "-h"): + result.show_help = True + break + + set_filter(arg) + + if not result.filter_args: + result.filter_args = "HEAD^.." + + return result + + +def _subject(commit_message: str) -> str: + """The first line of a commit message, for naming it in an error.""" + subject = commit_message.strip().split("\n", 1)[0].strip() + return subject or "(no subject)" + + +def rewrite_range( + invocation: Invocation, + *, + query: reviewers.PersonSource, + stderr: TextIO, + runner: CommandRunner = run_command, +) -> int: + """Rewrite the messages of every commit in the range. + + Reviewer resolution is per commit, because the author affects it -- the + CLA check and whether the author counts towards the total. Results are + cached by author, so a range of commits by one person costs one round of + lookups. + """ + policy = get_policy(invocation.repo) + commits = rewrite.read_range(invocation.filter_args, runner=runner) + if not commits: + print("No commits in range", file=stderr) + return 0 + + ref = rewrite.current_branch_ref(runner=runner) + old_tip = commits[-1].sha + resolutions: dict[str | None, reviewers.Resolution] = {} + + def transform(commit: rewrite.CommitInfo) -> str: + # A commit the caller did not ask for passes through untouched. + if invocation.commits and not any( + commit.sha.startswith(wanted) for wanted in invocation.commits + ): + return commit.message + + author = commit.author_email or None + if author not in resolutions: + resolutions[author] = reviewers.resolve( + query, + invocation.reviewers, + author_email=author, + self_email=invocation.my_email, + policy=policy, + release=invocation.release, + ) + resolution = resolutions[author] + + try: + reviewers.validate( + resolution, + author_email=author, + policy=policy, + trivial=message.is_trivial(commit.message), + ) + if not invocation.remove_reviewers: + reviewers.require_any(resolution.reviewers) + except ReviewError as error: + # Which commit is not obvious from a range, and the answer is + # usually the whole explanation. + raise ReviewError(f"{commit.sha[:12]} {_subject(commit.message)}: {error}") from error + + if invocation.verbose: + print( + f"{commit.sha[:12]} reviewed-by " + ", ".join(resolution.reviewers), + file=stderr, + ) + + return message.rewrite( + commit.message, + reviewers=resolution.reviewers, + repo=policy.name, + prnum=invocation.prnum, + release=invocation.release, + remove_reviewers=invocation.remove_reviewers, + ) + + mapping = rewrite.replay(commits, transform, runner=runner) + new_tip = mapping[old_tip] + if new_tip == old_tip: + print("Nothing to rewrite", file=stderr) + return 0 + + rewrite.update_branch(ref, new_tip, old_tip, runner=runner) + for name in rewrite.repoint_tags(mapping, runner=runner): + print(f"Moved tag {name}", file=stderr) + return 0 + + +def main( + argv: Sequence[str] | None = None, + *, + stdout: TextIO | None = None, + stderr: TextIO | None = None, + runner: CommandRunner = run_command, + query: listing.ListingSource | None = None, +) -> int: + argv = list(sys.argv[1:] if argv is None else argv) + stdout = stdout or sys.stdout + stderr = stderr or sys.stderr + + invocation = parse_args(argv) + for warning in invocation.warnings: + print(warning, file=stdout) + + if invocation.show_help: + print(USAGE.format(repos=_format_repos()), file=stderr) + return 0 + + try: + query = query or Query() + + if invocation.list_reviewers: + stdout.write(listing.format_listing(listing.list_reviewers(query))) + return 0 + + if not invocation.have_prnum: + raise ReviewError("Need either [--prnum=]NNN or --nopr flag") + + if invocation.use_self and not invocation.my_email: + invocation.my_email = _git_user_email(runner) + + if invocation.trivial: + print( + "Warning: --trivial has no effect; put 'CLA: Trivial' in the" + " commit message instead", + file=stderr, + ) + + return rewrite_range(invocation, query=query, runner=runner, stderr=stderr) + + except ReviewError as error: + print(str(error), file=stderr) + return 1 + + +def _git_user_email(runner: CommandRunner) -> str | None: + completed = runner( + ["git", "config", "--get", "user.email"], + capture_output=True, + text=True, + ) + if completed.returncode != 0: + return None + return (completed.stdout or "").strip() or None + + +def _format_repos() -> str: + return "\n".join( + f"--{name:<20} {policy.min_reviewers} reviewer(s)" + f"{', author counts' if policy.min_authors else ''}" + for name, policy in sorted(POLICIES.items()) + ) diff --git a/lib/openssl_tools/reviewtools/cherry.py b/lib/openssl_tools/reviewtools/cherry.py new file mode 100644 index 00000000..960d08d6 --- /dev/null +++ b/lib/openssl_tools/reviewtools/cherry.py @@ -0,0 +1,230 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Finding commits eligible for cherry-picking between two branches. + +Lists the symmetric difference of two branches, marking which commits have +an equivalent on the other side. See `--cherry-mark` in git-log(1). + +The subprocess calls are confined to `GitLog` so the parsing, sorting and +formatting below can be tested directly. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable, Iterator, Sequence +from dataclasses import dataclass +from typing import Protocol + +from .commands import CommandRunner, run_command +from .errors import ReviewError + +#: Where a commit lives: only on the left, only on the right, or both. +BRANCH_MARKERS = {"<": "<-", ">": "->", "=": "=="} + +#: The standard merge annotation, plus an older variant still in the history. +_PRNUM_RE = re.compile( + # The prose annotation, the trailer form that replaced it, and an older + # variant still present in the history. + r"\(Merged from https://github\.com/openssl/openssl/pull/(\d+)\)" + r"|Merged-from: https://github\.com/openssl/openssl/pull/(\d+)" + r"|GH: #(\d+)" +) + +_FIXES_RE = re.compile(r"Fixes:?\s+(?:#|https://github\.com/openssl/openssl/pull/)(\d+)") + +_RELEASE_BRANCH_RE = re.compile(r"^(?:.*/)?openssl-(\d+)\.(\d+)$") + +#: A separator that cannot occur in a commit subject. +_FIELD_SEP = "\x1f" + + +@dataclass(frozen=True) +class Commit: + prnum: str + fixes: str + timestamp: int + branch: str + commit: str + subject: str + + @property + def sort_key(self) -> tuple: + """Order by PR number then author date, newest first. + + The PR number is compared numerically; comparing the strings put + #9999 above #10000. Commits with no discoverable PR sort last. + """ + numeric = int(self.prnum) if self.prnum.isdigit() else -1 + return (numeric, self.timestamp) + + +def extract_prnum(message: str) -> str: + match = _PRNUM_RE.search(message) + if not match: + return "????" + # Whichever alternative matched is the only group that is set. + return next(group for group in match.groups() if group) + + +def extract_fixes(message: str) -> str: + match = _FIXES_RE.search(message) + return f"#{match.group(1)}" if match else "" + + +def shorten(subject: str, limit: int = 70) -> str: + return subject if len(subject) <= limit else subject[:limit] + "..." + + +def pick_default_right(branches: Iterable[str]) -> str | None: + """The highest openssl-N.M branch among `branches`. + + Chosen at run time rather than hardcoded, because the previous default + was OpenSSL_1_1_1-stable and had been end-of-life for years. + """ + best: tuple[tuple[int, int], str] | None = None + for branch in branches: + match = _RELEASE_BRANCH_RE.match(branch.strip()) + if not match: + continue + version = (int(match.group(1)), int(match.group(2))) + if best is None or version > best[0]: + best = (version, branch.strip()) + return best[1] if best else None + + +class GitQueries(Protocol): + """The git questions cherry-checker asks.""" + + def remotes(self) -> str: ... + def branches(self) -> list[str]: ... + def master_remote(self) -> str: ... + def symmetric_difference(self, left: str, right: str) -> list[str]: ... + def message(self, commit: str) -> str: ... + + +class GitLog: + """The git commands this tool runs.""" + + def __init__(self, runner: CommandRunner = run_command) -> None: + self._runner = runner + + def _capture(self, argv: Sequence[str]) -> str: + completed = self._runner(argv, capture_output=True, text=True) + if completed.returncode != 0: + raise ReviewError(f"{' '.join(argv)} failed: {(completed.stderr or '').strip()}") + return completed.stdout or "" + + def remotes(self) -> str: + return self._capture(["git", "remote", "-v"]) + + def branches(self) -> list[str]: + output = self._capture(["git", "for-each-ref", "--format=%(refname:short)", "refs/heads"]) + return output.splitlines() + + def master_remote(self) -> str: + """The remote master tracks, defaulting to origin. + + The Perl-era code called a non-existent `.trim()` here, so the + AttributeError was swallowed by a bare `except` and this always + returned "origin" -- making `--remote` ignore the configuration. + """ + completed = self._runner( + ["git", "config", "branch.master.remote"], + capture_output=True, + text=True, + ) + if completed.returncode != 0: + return "origin" + return (completed.stdout or "").strip() or "origin" + + def symmetric_difference(self, left: str, right: str) -> list[str]: + return self._capture( + [ + "git", + "log", + "--cherry-mark", + "--left-right", + f"{left}...{right}", + f"--pretty=%at{_FIELD_SEP}%m{_FIELD_SEP}%h{_FIELD_SEP}%s", + ] + ).splitlines() + + def message(self, commit: str) -> str: + return self._capture(["git", "show", "--no-patch", commit]) + + +def is_openssl_repo(git: GitQueries) -> bool: + """Whether the current directory is a clone of openssl.git. + + A failing `git remote -v` -- typically because this is not a repository + at all -- answers the question just as well as an empty remote list, so + it is not worth surfacing as an error of its own. + """ + try: + return "/openssl.git" in git.remotes() + except ReviewError: + return False + + +def parse_log_line(line: str) -> tuple[int, str, str, str] | None: + """(timestamp, branch marker, abbreviated id, subject) from one log line.""" + parts = line.split(_FIELD_SEP, 3) + if len(parts) != 4: + return None + timestamp, branch, commit, subject = parts + if not timestamp.isdigit() or branch not in BRANCH_MARKERS: + return None + return int(timestamp), branch, commit, subject + + +def pick_cherries( + git: GitQueries, left: str, right: str, *, include_picked: bool = False +) -> Iterator[Commit]: + """Commits in `left...right`, skipping already-picked ones by default.""" + for line in git.symmetric_difference(left, right): + parsed = parse_log_line(line) + if parsed is None: + continue + timestamp, branch, commit, subject = parsed + + if branch == "=" and not include_picked: + continue + + message = git.message(commit) + yield Commit( + prnum=extract_prnum(message), + fixes=extract_fixes(message), + timestamp=timestamp, + branch=branch, + commit=commit, + subject=shorten(subject), + ) + + +def format_table(commits: Sequence[Commit], left: str, right: str) -> str: + lines = [ + "These cherries are hanging on the git-tree:", + "", + f" <- {left}", + f" -> {right}", + " == both", + "", + " prnum | fixes | br | commit | subject", + "------- | ------ | -- | ---------- | " + "-" * 43, + ] + lines.extend( + " {:>6} | {:>6} | {} | {} | {} ".format( + f"#{entry.prnum}", + entry.fixes, + BRANCH_MARKERS[entry.branch], + entry.commit, + entry.subject, + ) + for entry in commits + ) + return "".join(f"{line}\n" for line in lines) diff --git a/lib/openssl_tools/reviewtools/cherry_cli.py b/lib/openssl_tools/reviewtools/cherry_cli.py new file mode 100644 index 00000000..76cd7fb7 --- /dev/null +++ b/lib/openssl_tools/reviewtools/cherry_cli.py @@ -0,0 +1,117 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""`cherry-checker` -- list commits eligible for cherry-picking.""" + +from __future__ import annotations + +import argparse +import subprocess +import sys +from collections.abc import Sequence +from typing import TextIO + +from .cherry import ( + GitLog, + GitQueries, + format_table, + is_openssl_repo, + pick_cherries, + pick_default_right, +) +from .errors import ReviewError + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="cherry-checker", + description=( + "Show the commits in 'left...right' eligible for cherry-picking." + " A commit counts as already picked when the other side has a" + " commit introducing an equivalent patch; see --cherry-mark in" + " git-log(1)." + ), + ) + parser.add_argument( + "left", + nargs="?", + default="master", + help="the branch to compare from (default: master)", + ) + parser.add_argument( + "right", + nargs="?", + help="the branch to compare against (default: the highest local openssl-N.M branch)", + ) + parser.add_argument( + "-a", + "--all", + action="store_true", + help="show all commits, including those already cherry-picked", + ) + parser.add_argument( + "-s", + "--sort", + action="store_true", + help="sort by pull request number and author date", + ) + parser.add_argument( + "-r", + "--remote", + action="store_true", + help="compare the remote branches instead of the local ones", + ) + return parser + + +def main( + argv: Sequence[str] | None = None, + *, + stdout: TextIO | None = None, + stderr: TextIO | None = None, + git: GitQueries | None = None, +) -> int: + args = build_parser().parse_args(argv) + stdout = stdout or sys.stdout + stderr = stderr or sys.stderr + git = git or GitLog() + + try: + if not is_openssl_repo(git): + print( + "cherry-checker: Not inside an openssl git repository.", + file=stderr, + ) + return 1 + + left = args.left + right = args.right or pick_default_right(git.branches()) + if not right: + print( + "cherry-checker: could not find a local openssl-N.M branch;" + " name the branch to compare against explicitly.", + file=stderr, + ) + return 1 + + if args.remote: + remote = git.master_remote() + left = f"{remote}/{left}" + right = f"{remote}/{right}" + + commits = list(pick_cherries(git, left, right, include_picked=args.all)) + if args.sort: + commits.sort(key=lambda entry: entry.sort_key, reverse=True) + + stdout.write(format_table(commits, left, right)) + return 0 + + except ReviewError as error: + print(f"cherry-checker: {error}", file=stderr) + return 1 + except subprocess.SubprocessError as error: + print(f"cherry-checker: {error}", file=stderr) + return 1 diff --git a/lib/openssl_tools/reviewtools/commands.py b/lib/openssl_tools/reviewtools/commands.py new file mode 100644 index 00000000..f2eea36c --- /dev/null +++ b/lib/openssl_tools/reviewtools/commands.py @@ -0,0 +1,53 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""The shape of the command runner these tools accept. + +`subprocess.run` satisfies `CommandRunner`, and so does a test double that +records invocations instead of making them. Spelling it out as a Protocol +rather than leaving the parameter untyped is what lets a substitute be +checked against the real interface. +""" + +from __future__ import annotations + +import subprocess +from collections.abc import Sequence +from typing import Any, Protocol + + +class CompletedCommand(Protocol): + """What a runner reports back. + + `stdout` is Any because subprocess.CompletedProcess is generic over it: + str under text=True, bytes otherwise. + """ + + returncode: int + stdout: Any + stderr: Any + + +class CommandRunner(Protocol): + """Runs a command and returns its outcome. + + The argv parameter is positional-only, because callers pass it + positionally and `subprocess.run` names it `args`. + """ + + def __call__(self, argv: Sequence[str], /, **kwargs: Any) -> CompletedCommand: ... + + +def run_command(argv: Sequence[str], /, **kwargs: Any) -> CompletedCommand: + """The default runner. + + A thin adapter rather than `subprocess.run` itself: that function is + overloaded, and mypy cannot show an overloaded function satisfies a + Protocol. It is also the one place to pin `check=False` -- every caller + here inspects `returncode` and reports failures itself. + """ + kwargs.setdefault("check", False) + return subprocess.run(argv, **kwargs) # noqa: S603 diff --git a/lib/openssl_tools/reviewtools/errors.py b/lib/openssl_tools/reviewtools/errors.py new file mode 100644 index 00000000..0094c30d --- /dev/null +++ b/lib/openssl_tools/reviewtools/errors.py @@ -0,0 +1,17 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Expected, operator-facing failures.""" + +from __future__ import annotations + + +class ReviewError(Exception): + """Something the caller can act on: a bad reviewer, a failed lookup.""" + + +class QueryError(ReviewError): + """api.openssl.org could not be reached, or answered with a server error.""" diff --git a/lib/openssl_tools/reviewtools/listing.py b/lib/openssl_tools/reviewtools/listing.py new file mode 100644 index 00000000..464e1399 --- /dev/null +++ b/lib/openssl_tools/reviewtools/listing.py @@ -0,0 +1,88 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Listing the reviewers a commit may be attributed to. + +Backs `addrev --list`. A person is listed when they have a 'rev' tag, a CLA +on file, and membership of the 'commit' group -- and each of their usable +identities is shown against that tag, so any of them can be typed as a +reviewer name. + +This asks the API three questions per person, so it is inherently slow; the +client caches within a run, but there is no bulk endpoint. +""" + +from __future__ import annotations + +import re +from collections.abc import Iterable +from typing import Protocol + +from .reviewers import COMMIT_GROUP, PersonSource + +#: Identities worth showing: a bare alphabetic name, or an '@handle'. The +#: handle pattern is odd -- it accepts '@ab' and '@a-b' but not '@a-b-c' -- +#: and is kept as it was so the listing does not change. +_PLAIN_NAME_RE = re.compile(r"^[A-Za-z]+$") +_HANDLE_RE = re.compile(r"^@(?:\w|\w-\w)+$") + +#: Identity keys that name a GitHub or GitHub Enterprise account, whose +#: values are shown with a leading '@'. +_HANDLE_KEYS = ("github", "ghe") + + +def usable_identities(record: Iterable) -> list[str]: + """The identities from one person record that can be typed as a reviewer.""" + flattened: list[str] = [] + for identity in record: + if isinstance(identity, dict): + for key, value in identity.items(): + flattened.append(f"@{value}" if key in _HANDLE_KEYS else str(value)) + else: + flattened.append(str(identity)) + + return sorted( + name for name in flattened if _PLAIN_NAME_RE.match(name) or _HANDLE_RE.match(name) + ) + + +def primary_email(record: Iterable) -> str | None: + """The first plain-string identity containing an '@', used for lookups.""" + for identity in record: + if isinstance(identity, str) and "@" in identity: + return identity + return None + + +class ListingSource(PersonSource, Protocol): + """PersonSource, plus the bulk listing --list walks.""" + + def list_people(self) -> list: ... + + +def list_reviewers(query: ListingSource) -> list[tuple[str, str]]: + """(identity, reviewer tag) pairs, sorted by tag then identity.""" + found: dict[str, str] = {} + + for record in query.list_people(): + email = primary_email(record) + if email is None: + continue + tag = query.find_person_tag(email, "rev") + if tag is None: + continue + if not query.has_cla(tag.lower()): + continue + if not query.is_member_of(email, COMMIT_GROUP): + continue + for name in usable_identities(record): + found[name] = tag + + return sorted(found.items(), key=lambda pair: (pair[1], pair[0])) + + +def format_listing(pairs: Iterable[tuple[str, str]]) -> str: + return "".join(f"{name:<15} ({tag})\n" for name, tag in pairs) diff --git a/lib/openssl_tools/reviewtools/message.py b/lib/openssl_tools/reviewtools/message.py new file mode 100644 index 00000000..455c36da --- /dev/null +++ b/lib/openssl_tools/reviewtools/message.py @@ -0,0 +1,151 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Rewriting a commit message's trailers. + +The lines this tool manages are stripped from the body and re-added through +`git interpret-trailers --if-exists addIfDifferent`, which is what places +them in an existing trailer block, adds the separating blank line when the +message needs one, and drops a trailer that is already present verbatim. + +Delegating that to git rather than reimplementing it is deliberate: this +replaces a script that does the same, and matching its output exactly +matters more than avoiding a subprocess. + +Trailers produced: + +- `Reviewed-by:` per reviewer, unless reviewers are being removed. +- `Merge-date:` unless the message already carries one. +- `Release: yes` on a release run. +- `Merged-from:` when a pull request number is known. +""" + +from __future__ import annotations + +import re +import subprocess +import time +from collections.abc import Callable, Sequence + +from .errors import ReviewError + +TRIVIAL_RE = re.compile(r"^CLA:\s*Trivial\s*$", re.IGNORECASE) +REVIEWED_BY_RE = re.compile(r"^Reviewed-by:\s*\S", re.IGNORECASE) +RELEASE_RE = re.compile(r"^Release:\s*yes\s*$", re.IGNORECASE) +MERGE_DATE_RE = re.compile(r"^Merge-?date:\s*\S", re.IGNORECASE) +MERGED_FROM_RE = re.compile(r"^Merged-from:\s*\S", re.IGNORECASE) + +#: The prose form of the merge reference, as commit messages carried it before +#: it became a trailer. Still recognised so that re-running over an older +#: commit replaces it rather than leaving both. +LEGACY_MERGED_FROM = "(Merged from https://github.com/openssl/{repo}/pull/" + +#: What a Merged-from: trailer points at. +MERGED_FROM_URL = "https://github.com/openssl/{repo}/pull/{prnum}" + + +def merged_from_url(repo: str, prnum: str) -> str: + return MERGED_FROM_URL.format(repo=repo, prnum=prnum) + + +def split_lines(message: str) -> list[str]: + """Split a commit message into lines, without their terminators. + + Splits on '\n' only, not str.splitlines(), which also breaks on form + feeds and several Unicode separators; rejoining those would silently + rewrite them. A trailing carriage return is dropped, as Perl's `` + plus `s|\\R$||` did. + """ + lines = message.split("\n") + if lines and lines[-1] == "": + lines.pop() + return [line.removesuffix("\r") for line in lines] + + +def is_trivial(message: str) -> bool: + """Whether the message carries a `CLA: Trivial` marker.""" + return any(TRIVIAL_RE.match(line) for line in split_lines(message)) + + +def format_merge_date(when: time.struct_time | None = None) -> str: + """The Merge-date value, in the format Perl's `scalar gmtime` produced.""" + return time.asctime(when or time.gmtime()) + + +def interpret_trailers(body: str, trailers: Sequence[str]) -> str: + """Add `trailers` to `body` using git's own trailer handling.""" + argv = ["git", "interpret-trailers", "--if-exists", "addIfDifferent"] + for trailer in trailers: + argv += ["--trailer", trailer] + + completed = subprocess.run( # noqa: S603 + argv, input=body, capture_output=True, text=True, check=False + ) + if completed.returncode != 0: + raise ReviewError( + "git interpret-trailers failed: " + + ((completed.stderr or "").strip() or f"exit {completed.returncode}") + ) + return completed.stdout + + +def rewrite( + message: str, + *, + reviewers: Sequence[str], + repo: str, + prnum: str | None = None, + release: bool = False, + remove_reviewers: bool = False, + now: time.struct_time | None = None, + add_trailers: Callable[[str, Sequence[str]], str] = interpret_trailers, +) -> str: + """Return `message` with its trailers brought up to date.""" + legacy_merged_from = LEGACY_MERGED_FROM.format(repo=repo) + + lines = split_lines(message) + # Trailing blank lines would otherwise separate the body from the trailer + # block twice over. + while lines and not lines[-1].strip(): + lines.pop() + + body: list[str] = [] + has_merge_date = False + + for line in lines: + if line.startswith(legacy_merged_from) or MERGED_FROM_RE.match(line): + # Re-added below as a trailer, if a PR number is known. + continue + + if REVIEWED_BY_RE.match(line): + if remove_reviewers: + continue + # Otherwise kept: addIfDifferent deduplicates it against the + # reviewers we are about to add. + elif RELEASE_RE.match(line): + if release: + continue + elif MERGE_DATE_RE.match(line): + has_merge_date = True + + body.append(line) + + # Dropping a managed line can leave a blank at the end of the body, which + # would separate it from the trailer block twice over. + while body and not body[-1].strip(): + body.pop() + + trailers: list[str] = [] + if not remove_reviewers: + trailers += [f"Reviewed-by: {reviewer}" for reviewer in reviewers] + if not has_merge_date: + trailers.append(f"Merge-date: {format_merge_date(now)}") + if release: + trailers.append("Release: yes") + if prnum: + trailers.append(f"Merged-from: {merged_from_url(repo, prnum)}") + + return add_trailers("\n".join(body) + "\n", trailers) diff --git a/lib/openssl_tools/reviewtools/policy.py b/lib/openssl_tools/reviewtools/policy.py new file mode 100644 index 00000000..3568b51b --- /dev/null +++ b/lib/openssl_tools/reviewtools/policy.py @@ -0,0 +1,59 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""How many reviewers each repository requires.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .errors import ReviewError + + +@dataclass(frozen=True) +class RepoPolicy: + """The review rules for one repository. + + `name` is the repository under github.com/openssl/ that a + "(Merged from ...)" line should point at. + + `min_authors` is special: 0 means an author must **not** be counted as a + reviewer at all. Any other value is a minimum. Do not collapse these + two meanings -- the difference is what stops someone approving their own + change in the main repository. + """ + + name: str + min_reviewers: int + min_authors: int + + +#: Keyed by the command line flag, without its leading dashes. +POLICIES: dict[str, RepoPolicy] = { + # The main repository: two reviewers, and the author is never one of them. + "openssl": RepoPolicy("openssl", min_reviewers=2, min_authors=0), + "tools": RepoPolicy("tools", min_reviewers=2, min_authors=1), + "perftools": RepoPolicy("perftools", min_reviewers=2, min_authors=1), + "installer": RepoPolicy("installer", min_reviewers=2, min_authors=1), + "fuzz-corpora": RepoPolicy("fuzz-corpora", min_reviewers=1, min_authors=1), + # `--technical-policies` reached addrev from ghmerge, matched none of its + # patterns, and was silently used as a commit range -- which made + # `ghmerge --technical-policies` fail on a bogus revision. + "technical-policies": RepoPolicy("technical-policies", min_reviewers=2, min_authors=0), +} + +DEFAULT_POLICY = POLICIES["openssl"] + + +def get_policy(name: str | None) -> RepoPolicy: + if not name: + return DEFAULT_POLICY + try: + return POLICIES[name] + except KeyError as error: + raise ReviewError( + f"Unknown repository {name!r}; expected one of: " + ", ".join(sorted(POLICIES)) + ) from error diff --git a/lib/openssl_tools/reviewtools/query.py b/lib/openssl_tools/reviewtools/query.py new file mode 100644 index 00000000..4678e7a9 --- /dev/null +++ b/lib/openssl_tools/reviewtools/query.py @@ -0,0 +1,195 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""A client for the OpenSSL committer and CLA database at api.openssl.org. + +The Perl equivalent is the OpenSSL-Query distribution: OpenSSL::Query with +its PersonREST and ClaREST backends, plus a registration and priority system +for plugging in alternative backends. Only the REST backend was ever used +here, and the API it speaks has seven endpoints, so this is a plain client. + + GET /0/People every known person + GET /0/Person/ one person's record + GET /0/Person//ValueOfTag/ e.g. the 'rev' reviewer tag + GET /0/Person//IsMemberOf/ e.g. the 'commit' group + GET /0/Group//Members a group's members + GET /0/HasCLA/ 200 if a CLA is on file + GET /0/CLAs whether CLAs are listable + +Status handling follows the Perl: a 5xx is an error worth reporting, while +any other non-200 means "no such thing" and yields an empty result. That +distinction matters -- a 404 for an unknown reviewer is a normal answer, and +must not be confused with the database being unreachable. + +Standard http_proxy / https_proxy / no_proxy variables are honoured, because +urllib's default opener reads them, as LWP's env_proxy did. +""" + +from __future__ import annotations + +import json +import urllib.error +import urllib.parse +import urllib.request +from collections.abc import Mapping +from typing import Any, Protocol + +from .errors import QueryError + +DEFAULT_BASE_URL = "https://api.openssl.org" +DEFAULT_TIMEOUT = 30 + + +def encode_id(identity: str | Mapping[str, str]) -> str: + """Render a person identifier for use in a URL path. + + A plain string is used as-is. A single-entry mapping is rendered as + 'tag:value', which is how the API disambiguates e.g. a GitHub handle from + an email address. + """ + if isinstance(identity, str): + return identity + if len(identity) != 1: + raise ReviewMalformedID("Malformed input ID") + ((tag, value),) = identity.items() + return f"{tag}:{value}" + + +class UrlOpener(Protocol): + """The one method this client needs from an opener. + + urllib's OpenerDirector satisfies it, and so does a stub that answers + from a table of routes. + """ + + # Positional-only, because urllib names it `fullurl` and a stub is more + # likely to name it `request`; only the shape of the call matters. + def open(self, request: Any, /, *, timeout: Any = ...) -> Any: ... + + +class ReviewMalformedID(QueryError): + """The caller passed an identifier this API cannot express.""" + + +class Query: + """Read-only access to the person and CLA databases.""" + + def __init__( + self, + base_url: str = DEFAULT_BASE_URL, + *, + timeout: int = DEFAULT_TIMEOUT, + opener: UrlOpener | None = None, + ) -> None: + scheme = urllib.parse.urlsplit(base_url).scheme + if scheme not in ("http", "https"): + raise QueryError(f"Unsupported scheme in base URL: {base_url!r}") + self.base_url = base_url.rstrip("/") + self.timeout = timeout + self._opener = opener or urllib.request.build_opener() + # find_person_tag and has_cla get asked about the same handful of + # people repeatedly within one run, and --list asks three questions + # per person. Caching keeps that to one request each. + self._cache: dict[str, tuple[int, str]] = {} + + # -- transport ---------------------------------------------------------- + + def _get(self, *path_segments: str) -> tuple[int, str]: + """GET a path, returning (status, body). Raises only on 5xx.""" + quoted = "/".join(urllib.parse.quote(segment, safe="") for segment in path_segments) + url = f"{self.base_url}/{quoted}" + + if url in self._cache: + return self._cache[url] + + # The scheme is validated in __init__, so this can only ever be + # http or https -- never file: or anything else unexpected. + request = urllib.request.Request( # noqa: S310 + url, headers={"Accept": "application/json"} + ) + try: + with self._opener.open(request, timeout=self.timeout) as response: + result = (response.status, response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + # The body is only used for context in the message; failing to + # read it must not mask the status we came here for. + try: + body = error.read().decode("utf-8", errors="replace") + except OSError: # pragma: no cover - the body is best-effort + body = "" + if error.code >= 500: + raise QueryError(f"Server error: {error.reason}") from error + result = (error.code, body) + except urllib.error.URLError as error: + raise QueryError(f"Could not reach {self.base_url}: {error.reason}") from error + + self._cache[url] = result + return result + + def _get_json(self, *path_segments: str) -> Any: + status, body = self._get(*path_segments) + if status != 200: + return None + try: + return json.loads(body) + except json.JSONDecodeError as error: + raise QueryError(f"Malformed JSON from {self.base_url}: {error}") from error + + # -- people ------------------------------------------------------------- + + def list_people(self) -> list: + """Every known person, each as a list of their identities.""" + return self._get_json("0", "People") or [] + + def find_person(self, identity: str | Mapping[str, str]) -> dict: + """One person's full record, or an empty dict if not found.""" + return self._get_json("0", "Person", encode_id(identity)) or {} + + def find_person_tag(self, identity: str | Mapping[str, str], tag: str) -> str | None: + """The value of `tag` for a person, e.g. their 'rev' reviewer name.""" + decoded = self._get_json("0", "Person", encode_id(identity), "ValueOfTag", tag) + if not decoded: + return None + return decoded[0] + + def is_member_of(self, identity: str | Mapping[str, str], group: str) -> bool: + """Whether a person belongs to `group`, e.g. 'commit'.""" + decoded = self._get_json("0", "Person", encode_id(identity), "IsMemberOf", group) + if not decoded: + return False + return bool(decoded[0]) + + def members_of(self, group: str) -> list: + return self._get_json("0", "Group", group, "Members") or [] + + # -- CLAs --------------------------------------------------------------- + + def has_cla(self, identity: str) -> bool: + """Whether a CLA is on file for an email address. + + Accepts a bare address or one wrapped in angle brackets, as it may + arrive from a git author line. + """ + address = extract_email(identity) + status, _ = self._get("0", "HasCLA", address) + return status == 200 + + def list_clas(self) -> bool: + status, _ = self._get("0", "CLAs") + return status == 200 + + +def extract_email(identity: str) -> str: + """Pull an email address out of `identity`, validating its shape.""" + start = identity.find("<") + end = identity.find(">", start + 1) + if start != -1 and end != -1: + inner = identity[start + 1 : end] + if "@" in inner and " " not in inner: + return inner + if "@" not in identity or " " in identity or identity.startswith("@"): + raise ReviewMalformedID(f"Malformed input ID: {identity!r}") + return identity diff --git a/lib/openssl_tools/reviewtools/reviewers.py b/lib/openssl_tools/reviewtools/reviewers.py new file mode 100644 index 00000000..9b1b416a --- /dev/null +++ b/lib/openssl_tools/reviewtools/reviewers.py @@ -0,0 +1,247 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Turning reviewer names into validated Reviewed-by: tags. + +The rules, carried over from gitaddrev: + +- A reviewer may be named by anything the person database recognises: a + short name, an email address, or a GitHub handle with a leading '@'. The + '@' is stripped before lookup. +- Every reviewer must resolve to a person with a 'rev' tag, that person must + have a CLA on file, and they must be a committer. +- Whether the commit's own author counts towards the reviewer total depends + on the repository policy: `min_authors == 0` means they do not. A release + run counts them regardless. +- Authors never get a Reviewed-by: trailer, even when they count. + +Explicitly named reviewers and automatically collected identities are held +to different standards, and the difference matters. Naming someone with +--reviewer asserts that they reviewed the change, so a non-committer there +is an error. The author's address and git's user.email are picked up +without being asked for, so a non-committer there simply does not count -- +erroring would make an outside contributor's patch unmergeable. + +Note that one person can arrive by both routes: `--reviewer=` on +a commit you authored is an explicit claim, and is checked as one. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from dataclasses import dataclass, field +from typing import Protocol + +from .errors import QueryError, ReviewError +from .policy import RepoPolicy + +#: The group a person must belong to before their name can appear on a +#: Reviewed-by: line. `addrev --list` has always filtered its output by this +#: same group; now the validation agrees with the listing. +COMMIT_GROUP = "commit" + + +class PersonSource(Protocol): + """The slice of the API client this module needs. + + A Protocol rather than the concrete Query, so the tests can substitute a + known database -- and so mypy checks that a substitute really does match + the interface. + """ + + def find_person_tag(self, identity: str, tag: str) -> str | None: ... + def has_cla(self, identity: str) -> bool: ... + def is_member_of(self, identity: str, group: str) -> bool: ... + + +@dataclass +class Resolution: + """The outcome of looking every candidate up.""" + + #: Reviewer tags to write as Reviewed-by: trailers, in the order given. + reviewers: list[str] = field(default_factory=list) + #: How many distinct authors were counted towards the reviewer total. + author_count: int = 0 + #: Candidates the person database does not know. + unknown: list[str] = field(default_factory=list) + #: Candidates with no CLA on file. + nocla: list[str] = field(default_factory=list) + #: Known, CLA-holding candidates who are not in the commit group. + noncommitters: list[str] = field(default_factory=list) + #: Committers named as reviewers who turned out to have authored the + #: commit, and so could not be credited. Kept only to explain a + #: too-few-reviewers failure, which is otherwise baffling. + excluded_authors: list[str] = field(default_factory=list) + + +def _strip_handle(identity: str) -> str: + """'@someone' -> 'someone'; anything else unchanged.""" + return identity.removeprefix("@") + + +def _record(bucket: list[str], candidate: str) -> None: + """Add `candidate` to `bucket` once, preserving the order given.""" + if candidate not in bucket: + bucket.append(candidate) + + +def _has_cla_quietly(source: PersonSource, identity: str) -> bool: + """has_cla, treating a malformed identifier as 'no CLA' rather than error. + + An arbitrary reviewer name is not necessarily an email address, and the + Perl relied on a regex guard before asking. Swallowing the error here + keeps that behaviour while letting genuine transport failures through. + """ + try: + return source.has_cla(identity) + except QueryError as error: + if "Malformed" in str(error): + return False + raise + + +def resolve( + source: PersonSource, + named: Iterable[str], + *, + author_email: str | None, + policy: RepoPolicy, + self_email: str | None = None, + release: bool = False, +) -> Resolution: + """Look up every candidate and sort them into reviewers, unknown, no-CLA. + + `named` are the reviewers the caller asked for explicitly. They must be + committers: naming someone is an assertion that they reviewed the change, + and only a committer can make that assertion count. + + `author_email` and `self_email` are picked up automatically -- from the + commit being rewritten and from git's user.email -- so they are treated + differently. If they are not committers they simply do not count towards + the total; that is not an error, because an outside contributor's patch + still has to be mergeable. + """ + result = Resolution() + + # One person can appear twice -- as the author and again by name -- and + # the lookup is a network round trip, so remember what we asked. + tags: dict[str, str | None] = {} + + def lookup(identity: str) -> str | None: + if identity not in tags: + tags[identity] = source.find_person_tag(identity, "rev") + return tags[identity] + + author_tag = lookup(author_email) if author_email else None + + def is_author(tag: str) -> bool: + return author_tag is not None and tag == author_tag + + # (identity, was it named explicitly) + queue: list[tuple[str, bool]] = [ + (identity, False) for identity in (author_email, self_email) if identity + ] + queue += [(identity, True) for identity in named if identity] + + # Distinct resolved tags seen so far, authors included. The Perl tracked + # this by scanning the reviewer list, which never contained authors, so an + # author named twice -- as the commit author and again via --reviewer -- + # was counted twice and lowered the effective reviewer requirement. + seen: set[str] = set() + + for candidate, explicit in queue: + identity = _strip_handle(candidate) + tag = lookup(identity) + + if tag is None: + _record(result.unknown, candidate) + # An unrecognised name might still be an email address with a CLA, + # in which case it is "unknown" but not "no CLA". + looks_like_email = "@" in candidate[1:] if candidate else False + has_cla = looks_like_email and _has_cla_quietly(source, candidate.lower()) + if not has_cla: + _record(result.nocla, candidate) + continue + + if not source.has_cla(tag.lower()): + _record(result.nocla, candidate) + continue + + author = is_author(tag) + if author and not (policy.min_authors > 0 or release): + # This repository does not let authors count as reviewers at all, + # so there is nothing further to check. + if explicit: + _record(result.excluded_authors, tag) + continue + + if not source.is_member_of(identity, COMMIT_GROUP): + if explicit: + _record(result.noncommitters, candidate) + # Otherwise: picked up automatically, so silently does not count. + continue + + if tag in seen: + continue + seen.add(tag) + + if author: + # Counted, but authors never get a Reviewed-by: trailer. + result.author_count += 1 + else: + result.reviewers.append(tag) + + return result + + +def validate( + resolution: Resolution, + *, + author_email: str | None, + policy: RepoPolicy, + trivial: bool = False, +) -> None: + """Raise ReviewError if the resolved set does not satisfy the policy.""" + # The author's own CLA is checked first, and separately: a trivial commit + # is allowed from someone who has not signed one. + if not trivial and author_email and author_email in resolution.nocla: + raise ReviewError( + f"Commit author {author_email} has no CLA, and this is a non-trivial commit" + ) + + # Now that that is settled, drop the author from both lists so they cannot + # produce a second, confusing error. + unknown = [name for name in resolution.unknown if name != author_email] + nocla = [name for name in resolution.nocla if name != author_email] + + if unknown: + raise ReviewError("Unknown reviewers: " + ", ".join(unknown)) + if nocla: + raise ReviewError("Reviewers without CLA: " + ", ".join(nocla)) + if resolution.noncommitters: + raise ReviewError( + "Reviewers who are not committers: " + + ", ".join(resolution.noncommitters) + + "\nOnly committers may be credited on a Reviewed-by: line." + " Run 'addrev --list' to see who those are." + ) + + required = policy.min_reviewers - resolution.author_count + if len(resolution.reviewers) < required: + detail = "" + if resolution.excluded_authors: + detail = ( + "\n" + + ", ".join(resolution.excluded_authors) + + (" authored this commit, so cannot be credited as a reviewer of it.") + ) + raise ReviewError(f"Too few reviewers (total must be at least {required}){detail}") + + +def require_any(reviewers: Sequence[str]) -> None: + """The final backstop: never rewrite a message with nobody credited.""" + if not reviewers: + raise ReviewError("No reviewer set!") diff --git a/lib/openssl_tools/reviewtools/rewrite.py b/lib/openssl_tools/reviewtools/rewrite.py new file mode 100644 index 00000000..44cbb4c1 --- /dev/null +++ b/lib/openssl_tools/reviewtools/rewrite.py @@ -0,0 +1,228 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Rewriting the messages of a range of commits. + +This is what `git filter-branch --msg-filter` was doing, minus the +generality: replay the range with `git commit-tree`, then move the branch +with `git update-ref`. Nothing is destroyed -- commit-tree only creates +objects, and the old tip stays in the reflog. + +Tags whose target was rewritten are re-pointed, keeping their tagger, date +and message, which is what `--tag-name-filter cat` did. Tags outside the +range are untouched, by construction: only targets present in the old-to-new +map are considered. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Sequence +from dataclasses import dataclass + +from .commands import CommandRunner, run_command +from .errors import ReviewError + +#: Record and field separators for reading commits in one pass. Neither can +#: appear in the fields git substitutes. +_RECORD = "\x1e" +_FIELD = "\x1f" + +_FIELDS = ( + "%H", # sha + "%T", # tree + "%P", # parents, space separated + "%an", + "%ae", + "%aI", + "%cn", + "%ce", + "%cI", + "%B", # raw body, must stay last: it contains newlines +) +_FORMAT = _RECORD + _FIELD.join(_FIELDS) + +_TAG_OBJECT_LINE = re.compile(r"^object [0-9a-f]{40,}$", re.MULTILINE) + + +@dataclass(frozen=True) +class CommitInfo: + """Everything needed to rebuild a commit with a different message.""" + + sha: str + tree: str + parents: tuple[str, ...] + author_name: str + author_email: str + author_date: str + committer_name: str + committer_email: str + committer_date: str + message: str + + @property + def identity_env(self) -> dict[str, str]: + """The environment `git commit-tree` reads author and committer from.""" + return { + "GIT_AUTHOR_NAME": self.author_name, + "GIT_AUTHOR_EMAIL": self.author_email, + "GIT_AUTHOR_DATE": self.author_date, + "GIT_COMMITTER_NAME": self.committer_name, + "GIT_COMMITTER_EMAIL": self.committer_email, + "GIT_COMMITTER_DATE": self.committer_date, + } + + +def _git(runner: CommandRunner, *args: str, **kwargs: object) -> str: + completed = runner(["git", *args], capture_output=True, text=True, **kwargs) + if completed.returncode != 0: + raise ReviewError( + f"git {' '.join(args)} failed: " + + ((completed.stderr or "").strip() or f"exit {completed.returncode}") + ) + return completed.stdout or "" + + +def read_range(rev_range: str, *, runner: CommandRunner = run_command) -> list[CommitInfo]: + """The commits in `rev_range`, parents before children. + + --topo-order is what makes that guarantee, and `replay` depends on it: a + commit whose parent has not been rebuilt yet keeps the original parent, + which forks the history instead of rewriting it. The default ordering is + by commit date, which is topological only by accident -- with two lines of + history it emits whichever side is newer first, so a side branch with + older dates comes out before its own parent. git filter-branch passed + --topo-order for the same reason. + """ + out = _git(runner, "log", "--topo-order", "--reverse", f"--format={_FORMAT}", rev_range) + + commits = [] + for record in out.split(_RECORD): + if not record.strip(): + continue + fields = record.split(_FIELD) + if len(fields) != len(_FIELDS): + raise ReviewError(f"could not parse commit record: {record!r}") + sha, tree, parents, an, ae, ad, cn, ce, cd, body = fields + commits.append( + CommitInfo( + sha=sha, + tree=tree, + parents=tuple(parents.split()), + author_name=an, + author_email=ae, + author_date=ad, + committer_name=cn, + committer_email=ce, + committer_date=cd, + # git appends a newline after the format; the body keeps its own. + message=body.removesuffix("\n"), + ) + ) + return commits + + +def current_branch_ref(*, runner: CommandRunner = run_command) -> str: + """The fully qualified ref HEAD points at.""" + completed = runner(["git", "symbolic-ref", "--quiet", "HEAD"], capture_output=True, text=True) + if completed.returncode != 0 or not (completed.stdout or "").strip(): + raise ReviewError("HEAD is detached; check out a branch first") + return completed.stdout.strip() + + +def replay( + commits: Sequence[CommitInfo], + transform: Callable[[CommitInfo], str], + *, + runner: CommandRunner = run_command, +) -> dict[str, str]: + """Rebuild `commits` with transformed messages, returning old -> new. + + The first commit keeps its original parents; each one after is re-parented + onto what was just built, so the range stays a chain. + """ + mapping: dict[str, str] = {} + + for commit in commits: + parents = [mapping.get(parent, parent) for parent in commit.parents] + args = ["commit-tree", commit.tree] + for parent in parents: + args += ["-p", parent] + + message = transform(commit) + new_sha = _git( + runner, *args, input=message, env={**_environ(), **commit.identity_env} + ).strip() + if not new_sha: + raise ReviewError(f"commit-tree produced nothing for {commit.sha}") + mapping[commit.sha] = new_sha + + return mapping + + +def update_branch( + ref: str, + new_tip: str, + old_tip: str, + *, + reason: str = "addrev", + runner: CommandRunner = run_command, +) -> None: + """Move `ref` to `new_tip`, failing if it no longer points at `old_tip`.""" + _git(runner, "update-ref", "-m", reason, ref, new_tip, old_tip) + + +def repoint_tags(mapping: dict[str, str], *, runner: CommandRunner = run_command) -> list[str]: + """Move any tag whose target was rewritten. Returns the names moved. + + An annotated tag is rebuilt rather than replaced, so its tagger, date and + message survive: tag objects are immutable, and `git tag -f` would stamp + whoever is running this as the tagger. + """ + if not mapping: + return [] + + # The peeled target comes from the same call: %(*objectname) is the commit + # an annotated tag points at, and is empty for a lightweight one, where + # %(objectname) is the commit already. Asking rev-parse per tag instead + # costs one subprocess for every tag in the repository -- 443 of them in + # an openssl clone, on every run, to move at most one. + listing = _git( + runner, + "for-each-ref", + "--format=%(refname)%09%(objecttype)%09%(objectname)%09%(*objectname)", + "refs/tags", + ) + + moved = [] + for line in listing.splitlines(): + if not line.strip(): + continue + refname, objecttype, objectname, peeled = line.split("\t") + + # A tag on a tree or a blob peels to something that is not a commit, + # so it is simply never in the map. + new_target = mapping.get(peeled or objectname) + if new_target is None: + continue + + if objecttype == "tag": + raw = _git(runner, "cat-file", "tag", refname) + rebuilt = _TAG_OBJECT_LINE.sub(f"object {new_target}", raw, count=1) + new_object = _git(runner, "mktag", input=rebuilt).strip() + _git(runner, "update-ref", refname, new_object, objectname) + else: + _git(runner, "update-ref", refname, new_target, objectname) + + moved.append(refname.removeprefix("refs/tags/")) + + return moved + + +def _environ() -> dict[str, str]: + import os + + return dict(os.environ) diff --git a/lib/openssl_tools/stagerelease/__init__.py b/lib/openssl_tools/stagerelease/__init__.py new file mode 100644 index 00000000..018ca5bb --- /dev/null +++ b/lib/openssl_tools/stagerelease/__init__.py @@ -0,0 +1,16 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""OpenSSL release staging. + +Replaces release-tools/stage-release.sh, its release-aux/*.sh helpers and the +release-aux/fixup-*.pl scripts. Standard library only: this runs on release +build hosts where installing packages is not always possible. +""" + +from __future__ import annotations + +__version__ = "1.0.0" diff --git a/lib/openssl_tools/stagerelease/__main__.py b/lib/openssl_tools/stagerelease/__main__.py new file mode 100644 index 00000000..c0e3df62 --- /dev/null +++ b/lib/openssl_tools/stagerelease/__main__.py @@ -0,0 +1,16 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Allow `python -m stagerelease`.""" + +from __future__ import annotations + +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/lib/openssl_tools/stagerelease/build.py b/lib/openssl_tools/stagerelease/build.py new file mode 100644 index 00000000..c8cd8811 --- /dev/null +++ b/lib/openssl_tools/stagerelease/build.py @@ -0,0 +1,67 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Driving the OpenSSL build system during staging. + +Kept behind a class so the orchestration can be tested without configuring +and building OpenSSL, which is what made the shell version untestable. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Protocol + +from .run import Runner + + +class BuildSystem(Protocol): + """The build steps a staging run drives. + + Stated as a Protocol because the point of this module is that stage.py + can be tested against a stub -- so the stub has to be a legitimate + substitute, not merely duck-typed past the checker. + """ + + def configure(self) -> None: ... + def update(self, *, is_alpha: bool) -> None: ... + + +class Build: + """The `./Configure` and `make` steps a release needs.""" + + def __init__(self, runner: Runner, source_dir: Path) -> None: + self.runner = runner + self.source_dir = source_dir + + def has_make_target(self, target: str) -> bool: + """Whether the generated Makefile defines `target`. + + Not every branch has `renumber` or `update-fips-checksums`, so their + absence is normal rather than an error. + """ + makefile = self.source_dir / "Makefile" + if not makefile.is_file(): + return False + pattern = re.compile(rf"^{re.escape(target)} *:", re.MULTILINE) + return bool(pattern.search(makefile.read_text(errors="replace"))) + + def configure(self) -> None: + self.runner.run(["./Configure", "cc"], echo_output=True) + + def update(self, *, is_alpha: bool) -> None: + """Run `make update`, plus the checks a non-alpha release requires.""" + self.runner.run(["make", "update"], echo_output=True) + + # An alpha may still have symbols without assigned ordinal numbers; + # a beta or final release may not. + if not is_alpha and self.has_make_target("renumber"): + self.runner.run(["make", "renumber"], echo_output=True) + + if self.has_make_target("update-fips-checksums"): + self.runner.run(["make", "clean"], echo_output=True) + self.runner.run(["make", "update-fips-checksums"], echo_output=True) diff --git a/lib/openssl_tools/stagerelease/cli.py b/lib/openssl_tools/stagerelease/cli.py new file mode 100644 index 00000000..262c20db --- /dev/null +++ b/lib/openssl_tools/stagerelease/cli.py @@ -0,0 +1,280 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Command line entry point.""" + +from __future__ import annotations + +import argparse +import sys +from collections.abc import Sequence +from datetime import date +from pathlib import Path + +from .errors import ReleaseError +from .git import Git +from .report import Reporter +from .run import Runner +from .stage import StageOptions, StageResult, stage_release + +#: next_method + next_method2 pairs that describe a coherent release. +VALID_METHOD_PAIRS = { + ("alpha", "alpha"), + ("alpha", "beta"), + ("beta", "beta"), + ("final", "final"), + ("", ""), + ("", "beta"), +} + +PUSH_URL = "git@github.openssl.org:openssl/openssl.git" + +MANUAL = """\ +stage-release - OpenSSL release staging + +SYNOPSIS + stage-release [--alpha | --beta | --final] [--next-beta] + [--reviewer=ID ...] [--quiet | --verbose] [--debug] + [--porcelain] + +DESCRIPTION + Stages an OpenSSL release from the current worktree. Run it from inside + an OpenSSL source checkout, with the branch to release from checked out. + It refuses to run unless that branch is master or a recognised release + branch, and unless the worktree is clean. + + If none of --alpha, --beta or --final is given, the next release is + worked out from the current state of the branch. + + Nothing is signed, pushed or uploaded. The release tag is annotated but + not signed: the signing key is held on an HSM the build host cannot + reach, so signing the tag and the tarball happens separately, where that + access exists. Shipping the artifacts is the caller's job. + +OPTIONS + --alpha, --beta + Move the branch into alpha or beta releases, or continue an ongoing + series. Both require PATCH to be zero. + + --next-beta + Use with --alpha to switch to beta releases once this one is done. + + --final + Leave the alpha or beta series and make a regular release. Only + valid when alpha or beta releases are ongoing. + + --reviewer=ID + Add ID as a reviewer of the commits this makes. May be repeated. + Without it you have to run addrev by hand afterwards, which means + re-tagging the release commit by hand as well. + + --quiet, --verbose, --debug + Control how much progress output is produced. By default each step + is announced with a "== " heading; --verbose adds the output of every + command and the files each step touched; --quiet prints only the + final result. --debug adds internal state on stderr. + + --porcelain + Print the final result as shell variable assignments instead of + instructions: orig_head and metadata. + +RELEASE BRANCHES AND TAGS + Before 3.0, release branches were named OpenSSL_-stable and tags + OpenSSL_. From 3.0 on, branches are openssl- and tags + are openssl-, with -alpha or -beta for pre-releases. + Both forms are recognised. + +FILES + Written to the parent directory of the worktree: + + openssl-.tar.gz + The source tarball. + + openssl-.tar.gz.sha1, openssl-.tar.gz.sha256 + Its checksums, in the binary-mode format sha256sum -c reads. + + openssl-.dat + Metadata for later pipeline steps, as shell variable assignments: + update_branch, release_branch (only when one was created), + release_tag, release_files and source_repo. +""" + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="stage-release", + description="Stage an OpenSSL release from the current worktree.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=( + "If none of --alpha, --beta or --final is given, the next release\n" + "is worked out from the current state of the branch." + ), + ) + + step = parser.add_mutually_exclusive_group() + step.add_argument( + "--alpha", + dest="next_method", + action="store_const", + const="alpha", + help='start or continue the "alpha" pre-release series', + ) + step.add_argument( + "--beta", + dest="next_method", + action="store_const", + const="beta", + help='start or continue the "beta" pre-release series', + ) + step.add_argument( + "--final", + dest="next_method", + action="store_const", + const="final", + help='leave "alpha" or "beta" and make a final release', + ) + parser.set_defaults(next_method="") + + parser.add_argument( + "--next-beta", + action="store_true", + help="switch to beta releases afterwards; use with --alpha", + ) + parser.add_argument( + "--reviewer", + dest="reviewers", + action="append", + default=[], + metavar="ID", + help="reviewer of the commits made (repeatable)", + ) + + noise = parser.add_mutually_exclusive_group() + noise.add_argument("--quiet", action="store_true", help="only print the final output") + noise.add_argument( + "--verbose", + action="store_true", + help="also show command output and the files each step touched", + ) + + parser.add_argument("--debug", action="store_true", help="include debug output") + parser.add_argument( + "--porcelain", + action="store_true", + help="print the result in an easy-to-parse form", + ) + parser.add_argument("--manual", action="store_true", help="print the manual and exit") + return parser + + +def resolve_methods(next_method: str, next_beta: bool) -> tuple[str, str]: + """Work out the (release, post-release) pair, rejecting incoherent ones.""" + next_method2 = "beta" if next_beta else next_method + if (next_method, next_method2) not in VALID_METHOD_PAIRS: + raise ReleaseError( + f"Invalid combination of options ({next_method or 'none'}, {next_method2 or 'none'})", + "--next-beta only goes with --alpha.", + ) + return next_method, next_method2 + + +def format_result(result: StageResult, porcelain: bool) -> str: + """The closing message: instructions, or parseable assignments.""" + if porcelain: + return f"orig_head='{result.orig_head}'\nmetadata='{result.metadata_path.name}'\n" + + lines = [ + "", + "=" * 70, + "The release is done. The release artifacts and a metadata file have", + "been written to the parent directory; pushing the commits and tag and", + "shipping the artifacts are the caller's responsibility.", + "=" * 70, + "", + "The following files were generated:", + "", + ] + lines += [f" {name}" for name in result.artifacts.names()] + lines += ["", "-" * 70, ""] + + if result.created_release_branch: + lines += [ + "A release tag and a release branch have been added to the worktree,", + "and the current branch has been updated.", + "", + f" Updated branch: {result.update_branch}", + f" Release branch: {result.release_branch}", + f" Tag: {result.release_tag}", + "", + "When pushing everything to the main repository, do it like this:", + "", + f" git push {PUSH_URL} \\", + f" {result.release_branch}", + f" git push {PUSH_URL} \\", + f" {result.update_branch}", + f" git push {PUSH_URL} \\", + f" {result.release_tag}", + ] + else: + lines += [ + "A release tag has been added to the worktree, and the current branch", + "has been updated.", + "", + f" Release/update branch: {result.update_branch}", + f" Tag: {result.release_tag}", + "", + "When pushing everything to the main repository, do it like this:", + "", + f" git push {PUSH_URL} \\", + f" {result.update_branch}", + f" git push {PUSH_URL} \\", + f" {result.release_tag}", + ] + + lines += ["", "-" * 70] + return "".join(f"{line}\n" for line in lines) + + +def main(argv: Sequence[str] | None = None, *, today: date | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + + if args.manual: + print(MANUAL, end="") + return 0 + + reporter = Reporter( + quiet=args.quiet, + verbose_enabled=args.verbose, + debug_enabled=args.debug, + ) + + try: + next_method, next_method2 = resolve_methods(args.next_method, args.next_beta) + + runner = Runner(cwd=Path.cwd(), log=reporter.verbose) + git = Git(runner) + # Everything after this runs relative to the worktree root, matching + # the shell, which cd'd there implicitly by being run from it. + runner.cwd = git.toplevel() + + result = stage_release( + StageOptions( + next_method=next_method, + next_method2=next_method2, + reviewers=tuple(args.reviewers), + ), + runner=runner, + git=git, + reporter=reporter, + today=today, + ) + except ReleaseError as error: + print(str(error), file=sys.stderr) + return 1 + + reporter.out(format_result(result, args.porcelain).rstrip("\n")) + return 0 diff --git a/lib/openssl_tools/stagerelease/copyright_year.py b/lib/openssl_tools/stagerelease/copyright_year.py new file mode 100644 index 00000000..6486650f --- /dev/null +++ b/lib/openssl_tools/stagerelease/copyright_year.py @@ -0,0 +1,106 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Bring copyright years up to date -- a port of do-copyright-year. + +Every source file touched since the start of the current year has the end +year of its OpenSSL copyright notice extended to this year. A notice that +already names a single year becomes a range, and a range that would collapse +to one year (2026-2026) is written as that year alone. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable +from dataclasses import dataclass +from datetime import date +from pathlib import Path +from typing import Protocol + +from .textutil import read_text, split_lines, write_text + +COPYRIGHT_OWNER = "The OpenSSL Project" + +_SOME_YEAR = r"[12][0-9][0-9][0-9]" +_YEAR_RANGE = re.compile(rf"({_SOME_YEAR})(-{_SOME_YEAR})?") +_REPEATED_YEAR = re.compile(rf"({_SOME_YEAR})-\1") +_COPYRIGHT = re.compile(rf"Copyright .*{_SOME_YEAR}(-{_SOME_YEAR})? .*{COPYRIGHT_OWNER}") + +#: Always refreshed, whether or not they were touched this year. +ALWAYS_CONSIDER = ("README.md", "README") + + +def update_line(line: str, this_year: int) -> str: + """Extend the copyright year range on one line, if it carries a notice.""" + if not _COPYRIGHT.search(line): + return line + line = _YEAR_RANGE.sub(lambda m: f"{m[1]}-{this_year}", line, count=1) + return _REPEATED_YEAR.sub(r"\1", line, count=1) + + +def update_text(text: str, this_year: int) -> str: + return "".join(update_line(line, this_year) for line in split_lines(text)) + + +class ChangedSince(Protocol): + """The slice of the git interface this pass needs.""" + + def changed_since(self, before: str) -> list[tuple[str, str]]: ... + def add(self, path: str) -> None: ... + + +@dataclass +class CopyrightResult: + considered: int + updated: list[str] + + +def update_copyright_years( + git: ChangedSince, + root: Path, + today: date, + log: Callable[[str], None] = lambda line: None, +) -> CopyrightResult: + """Update and stage copyright years across files touched this year. + + Returns how many files were examined and which ones changed. Files are + only written when their contents actually differ, so unchanged files keep + their mtime -- the shell achieved the same thing by rewriting a copy and + comparing before moving it back. + """ + new_year_day = f"{today.year}-01-01" + + candidates: list[str] = [path for _, path in git.changed_since(new_year_day)] + candidates.extend(name for name in ALWAYS_CONSIDER if (root / name).is_file()) + + considered = 0 + updated: list[str] = [] + seen: set[str] = set() + + for relative in candidates: + if relative in seen: + continue + seen.add(relative) + + path = root / relative + if not path.is_file(): + # Directories, submodules, and paths removed since the commit + # that mentioned them. + continue + considered += 1 + + text = read_text(path) + new_text = update_text(text, today.year) + if new_text == text: + continue + + write_text(path, new_text) + git.add(relative) + updated.append(relative) + log(f"> {relative}") + + return CopyrightResult(considered=considered, updated=updated) diff --git a/lib/openssl_tools/stagerelease/errors.py b/lib/openssl_tools/stagerelease/errors.py new file mode 100644 index 00000000..0f80eaaf --- /dev/null +++ b/lib/openssl_tools/stagerelease/errors.py @@ -0,0 +1,29 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""The one exception type this tool raises on expected failures. + +Anything that the operator could plausibly hit -- a dirty worktree, an +impossible release transition, an unrecognised branch -- is a ReleaseError. +The CLI catches it, prints the message to stderr and exits 1, so a stack +trace only ever means a genuine bug. +""" + +from __future__ import annotations + + +class ReleaseError(Exception): + """An expected, operator-facing failure.""" + + def __init__(self, message: str, hint: str | None = None) -> None: + super().__init__(message) + self.message = message + self.hint = hint + + def __str__(self) -> str: + if self.hint: + return f"{self.message}\n{self.hint}" + return self.message diff --git a/lib/openssl_tools/stagerelease/fixups.py b/lib/openssl_tools/stagerelease/fixups.py new file mode 100644 index 00000000..b3212042 --- /dev/null +++ b/lib/openssl_tools/stagerelease/fixups.py @@ -0,0 +1,263 @@ +# Copyright 2026 The OpenSSL Project Authors. All Rights Reserved. +# +# Licensed under the Apache License 2.0 (the "License"). You may not use +# this file except in compliance with the License. You can obtain a copy +# in the file LICENSE in the source distribution or at +# https://www.openssl.org/source/license.html +"""Per-file edits applied for the release and post-release commits. + +A port of the twelve `release-aux/fixup-*.pl` scripts, which were run as +`perl -pi