From 2ff35a9fb1723a83047d6b0e529df12378a521a2 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 14:18:55 -0700 Subject: [PATCH 001/115] Initial commit --- .github/ISSUE_TEMPLATE/bug_report.md | 38 ++++++ .github/ISSUE_TEMPLATE/feature_request.md | 20 +++ .github/PULL_REQUEST_TEMPLATE/pr_template.md | 27 ++++ .github/dependabot.yaml | 11 ++ .github/workflows/Readme.md | 6 + .github/workflows/qcom-preflight-checks.yml | 24 ++++ .github/workflows/stale-issues.yaml | 22 ++++ CODE-OF-CONDUCT.md | 132 +++++++++++++++++++ CONTRIBUTING.md | 76 +++++++++++ LICENSE.txt | 26 ++++ README.md | 47 +++++++ SECURITY.md | 31 +++++ 12 files changed, 460 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 .github/PULL_REQUEST_TEMPLATE/pr_template.md create mode 100644 .github/dependabot.yaml create mode 100644 .github/workflows/Readme.md create mode 100644 .github/workflows/qcom-preflight-checks.yml create mode 100644 .github/workflows/stale-issues.yaml create mode 100644 CODE-OF-CONDUCT.md create mode 100644 CONTRIBUTING.md create mode 100644 LICENSE.txt create mode 100644 README.md create mode 100644 SECURITY.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000000000..db59dc94a0a53 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,38 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: '' +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] + +**Smartphone (please complete the following information):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 0000000000000..d61fb1ab7ce6c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: '' +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. \ No newline at end of file diff --git a/.github/PULL_REQUEST_TEMPLATE/pr_template.md b/.github/PULL_REQUEST_TEMPLATE/pr_template.md new file mode 100644 index 0000000000000..2a2767f6f43f5 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pr_template.md @@ -0,0 +1,27 @@ +## Pull Request + +**Description** +A clear and concise description of what this pull request does. + +**Related Issue** +Link to the issue that this pull request addresses (e.g., `Fixes #123`). + +**Type of Change** +Please delete options that are not relevant. +- Bug fix (non-breaking change which fixes an issue) +- New feature (non-breaking change which adds functionality) +- Breaking change (fix or feature that would cause existing functionality to not work as expected) +- Documentation update + +**Checklist** +- [ ] My code follows the style guidelines of this project +- [ ] I have performed a self-review of my own code +- [ ] I have commented my code, particularly in hard-to-understand areas +- [ ] I have made corresponding changes to the documentation +- [ ] My changes generate no new warnings +- [ ] I have added tests that prove my fix is effective or that my feature works +- [ ] New and existing unit tests pass locally with my changes +- [ ] Any dependent changes have been merged and published in downstream modules + +**Additional Context** +Add any other context or screenshots about the pull request here. diff --git a/.github/dependabot.yaml b/.github/dependabot.yaml new file mode 100644 index 0000000000000..13150bcfe23d7 --- /dev/null +++ b/.github/dependabot.yaml @@ -0,0 +1,11 @@ +# To get started with Dependabot version updates, you'll need to specify which +# package ecosystems to update and where the package manifests are located. +# Please see the documentation for all configuration options: +# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file + +version: 2 +updates: + - package-ecosystem: "github-actions" # See documentation for possible values + directory: "/" # This points to .github/workflows + schedule: + interval: "daily" diff --git a/.github/workflows/Readme.md b/.github/workflows/Readme.md new file mode 100644 index 0000000000000..8dc5e2c687383 --- /dev/null +++ b/.github/workflows/Readme.md @@ -0,0 +1,6 @@ +# List of workflows and actions +This folder contains workflows that are helpful for maintaining a smooth and secure development process. The workflows should be enabled for open-source projects. + +Workflows: +1. `qcom-preflight-checks.yml` - This workflow runs several preflight checks, including copyight, email, repolinter, and security checks. See [qualcomm/qcom-actions](https://github.com/qualcomm/qcom-actions) +2. `stale-issues.yaml` - This workflow will periodically run every 30 days to check for stalled issues and PRs. If the workflow detects any stalled issues and/or PRs, it will automatically leave just a comment to draw attention. diff --git a/.github/workflows/qcom-preflight-checks.yml b/.github/workflows/qcom-preflight-checks.yml new file mode 100644 index 0000000000000..1b068996ed5ab --- /dev/null +++ b/.github/workflows/qcom-preflight-checks.yml @@ -0,0 +1,24 @@ +name: QC Preflight Checks + +on: + pull_request: + push: + branches: [main] + workflow_dispatch: + +jobs: + preflight: + name: Run QC Preflight Checks + uses: qualcomm/qcom-reusable-workflows/.github/workflows/reusable-qcom-preflight-checks-orchestrator.yml@v2 + with: + enable-semgrep-scan: true + enable-dependency-review: true + enable-repolinter-check: true + enable-copyright-license-check: true + enable-commit-email-check: true + enable-commit-msg-check: false + enable-armor-checkers: false + + permissions: + contents: read + security-events: write diff --git a/.github/workflows/stale-issues.yaml b/.github/workflows/stale-issues.yaml new file mode 100644 index 0000000000000..f0bb240dc2d5a --- /dev/null +++ b/.github/workflows/stale-issues.yaml @@ -0,0 +1,22 @@ +name: 'Close stale issues and pull requests with no recent activity' +on: + schedule: + - cron: "30 1 * * *" + +permissions: + issues: write + pull-requests: write + +jobs: + stale: + runs-on: ubuntu-latest + steps: + - uses: actions/stale@v10 + with: + stale-issue-message: 'Remove the stale label or add a comment to reset the inactivity timer.' + stale-pr-message: 'Remove the stale label or add a comment to reset the inactivity timer' + days-before-stale: 30 + days-before-close: -1 + remove-stale-when-updated: true + remove-issue-stale-when-updated: true + remove-pr-stale-when-updated: true diff --git a/CODE-OF-CONDUCT.md b/CODE-OF-CONDUCT.md new file mode 100644 index 0000000000000..82a5fc677fbc2 --- /dev/null +++ b/CODE-OF-CONDUCT.md @@ -0,0 +1,132 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual +identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall + community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or advances of + any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email address, + without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official email address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +[GitHub.CoC](mailto:github.coc@qti.qualcomm.com?subject=GitHub%20Qualcomm%20Code%20of%20Conduct%20Report). +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of +actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or permanent +ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the +community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.1, available at +[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at +[https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000000..455b5f8f94b78 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,76 @@ +# Contributing to + +Hi there! +We’re thrilled that you’d like to contribute to this project. +Your help is essential for keeping this project great and for making it better. + +## Branching Strategy + +In general, contributors should develop on branches based off of `main` and pull requests should be made against `main`. + +## Submitting a pull request + +1. Please read our [code of conduct](CODE-OF-CONDUCT.md) and [license](LICENSE.txt). +1. [Fork](https://github.com/qualcomm//fork) and clone the repository. + + ```bash + git clone https://github.com//.git + ``` + +1. Create a new branch based on `main`: + + ```bash + git checkout -b main + ``` + +1. Create an upstream `remote` to make it easier to keep your branches up-to-date: + + ```bash + git remote add upstream https://github.com/qualcomm/.git + ``` + +1. Make your changes, add tests, and make sure the tests still pass. +1. Commit your changes using the [DCO](https://developercertificate.org/). You can attest to the DCO by commiting with the **-s** or **--signoff** options or manually adding the "Signed-off-by": + + ```bash + git commit -s -m "Really useful commit message"` + ``` + +1. After committing your changes on the topic branch, sync it with the upstream branch: + + ```bash + git pull --rebase upstream main + ``` + +1. Push to your fork. + + ```bash + git push -u origin + ``` + + The `-u` is shorthand for `--set-upstream`. This will set up the tracking reference so subsequent runs of `git push` or `git pull` can omit the remote and branch. + +1. [Submit a pull request](https://github.com/qualcomm//pulls) from your branch to `main`. +1. Pat yourself on the back and wait for your pull request to be reviewed. + +## Security Analysis of Pull Requests + +To maintain the security and integrity of this project, all pull requests from external contributors are automatically scanned using [Semgrep](https://github.com/semgrep/semgrep) to detect insecure coding patterns and potential security flaws. + +**Static Analysis with Semgrep:** We use Semgrep to perform lightweight, fast static analysis on every PR. This helps identify risky code patterns and logic flaws early in the development process. + +**Contributor Responsibility:** If any issues are flagged, contributors are expected to resolve them before the PR can be merged. + +**Continuous Improvement:** Our Semgrep ruleset evolves over time to reflect best practices and emerging security concerns. + +By submitting a PR, you agree to participate in this process and help us keep the project secure for everyone. + + +Here are a few things you can do that will increase the likelihood of your pull request to be accepted: + +- Follow the existing style where possible. **INSERT LINK TO STYLE, e.g. PEP8 for python** +- Write tests. +- Keep your change as focused as possible. + If you want to make multiple independent changes, please consider submitting them as separate pull requests. +- Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). +- It's a good idea to arrange a discussion with other developers to ensure there is consensus on large features, architecture changes, and other core code changes. PR reviews will go much faster when there are no surprises. diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000000000..389b5477129b0 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,26 @@ +Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following + disclaimer in the documentation and/or other materials provided + with the distribution. +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS +BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR +BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, +WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE +OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN +IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000000000..a27531da6dd85 --- /dev/null +++ b/README.md @@ -0,0 +1,47 @@ +**After repository creation:** +- [ ] Update this `README.md`. Update the Project Name, description, and all sections. Remove this checklist. +- [ ] If required, update `LICENSE.txt` and the License section with your project's approved license +- [ ] Search this repo for "REPLACE-ME" and update all instances accordingly +- [ ] Update `CONTRIBUTING.md` as needed +- [ ] Review the workflows in `.github/workflows`, updating as needed. See https://docs.github.com/en/actions for information on what these files do and how they work. +- [ ] Review and update the suggested Issue and PR templates as needed in `.github/ISSUE_TEMPLATE` and `.github/PULL_REQUEST_TEMPLATE` + +# Project Name + +*\* + +Project that does ... implemented in ... runs on Qualcomm® *\* + +## Branches + +**main**: Primary development branch. Contributors should develop submissions based on this branch, and submit pull requests to this branch. + +## Requirements + +List requirements to run the project, how to install them, instructions to use docker container, etc... + +## Installation Instructions + +How to install the software itself. + +## Usage + +Describe how to use the project. + +## Development + +How to develop new features/fixes for the software. Maybe different than "usage". Also provide details on how to contribute via a [CONTRIBUTING.md file](CONTRIBUTING.md). + +## Getting in Contact + +How to contact maintainers. E.g. GitHub Issues, GitHub Discussions could be indicated for many cases. However a mail list or list of Maintainer e-mails could be shared for other types of discussions. E.g. + +* [Report an Issue on GitHub](../../issues) +* [Open a Discussion on GitHub](../../discussions) +* [E-mail us](mailto:REPLACE-ME@qti.qualcomm.com) for general questions + +## License + +*\* + +*\* is licensed under the [BSD-3-clause License](https://spdx.org/licenses/BSD-3-Clause.html). See [LICENSE.txt](LICENSE.txt) for the full license text. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000000000..468eda661763f --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,31 @@ +How to Report a Potential Vulnerability? +======================================== + +If you would like to report a public issue (for example, one with a released +CVE number), please report it as a +[GitHub issue](https://github.com/qualcomm/REPLACE-ME/issues/new). +If you have a patch ready, submit it following the same procedure as any +other patch as described in [CONTRIBUTING.md](CONTRIBUTING.md). + +If you are dealing with a not-yet released or urgent issue, please contact us +via our [Product Security team](mailto:product-security@qualcomm.com) or +see our +[Report a Bug](https://www.qualcomm.com/company/product-security/report-a-bug) +page. Please include the following details while reporting a vulnerability: +- Description of the vulnerability +- Steps to reproduce +- Affected versions +- Potential impact +- Any relevant logs or screenshots + +## Coordinated Disclosure + +We follow a Coordinated Vulnerability Disclosure (CVD) process: + +- **Initial Response**: We will acknowledge your report within 48 hours. + +- **Investigation**: Our team will investigate the issue and provide updates. + +- **Resolution**: We will work with you to resolve the issue and prepare a fix. + +- **Disclosure**: Once the fix is ready, we will disclose the vulnerability and notify affected users. From 5662e28a6d6ca3fb2b063c7b90da8847a1940db9 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 14:40:24 -0700 Subject: [PATCH 002/115] ci: add Canonical Ubuntu kernel mirror and build pipeline Implements a complete CI/CD pipeline for mirroring and building the Canonical Ubuntu kernel source package from Launchpad, targeting the arm64 architecture on Ubuntu 24.04 LTS. Repository branch layout: main - CI infrastructure (workflows, scripts, documentation) - Orphan branch per Ubuntu series containing the full extracted kernel source tree (one commit per upload) e.g. noble branch for Ubuntu Noble 24.04 LTS Workflows (.github/workflows/): fetch-source-pkg.yml - Queries the Launchpad REST API (api.launchpad.net/1.0) for the latest published linux source package for the given series - Downloads .dsc, .orig.tar.gz, .debian.tar.xz from Launchpad - Extracts the full patched source tree via dpkg-source -x - Commits the complete kernel source to the series branch - Tags each commit as - (e.g. noble-6.8.0-51.52) - Idempotent: skips if the tag already exists - Auto-dispatches build-kernel.yml on each new version - Schedule: daily 04:00 UTC - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem build-kernel.yml - Checks out the series branch (full kernel source tree) - Installs build dependencies via apt-get build-dep - Native arm64 build: fakeroot debian/rules binary- - No cross-compilation required (runner is arm64) - Uploads built .deb packages as: * GitHub Actions artifact (90-day retention) * GitHub Release asset on the - tag (permanent) - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem mirror-git.yml (optional) - Mirrors the full Launchpad kernel git history to a separate GitHub repository (qualcomm-linux/linux-noble) - Incremental sync: clones from GitHub mirror then fetches delta from Launchpad; full clone only on first run - Schedule: daily 03:00 UTC - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem Helper scripts (scripts/): check-version.sh - Query latest published version from Launchpad API fetch-source-pkg.sh - Download source package files from Launchpad build-kernel-deb.sh - Build kernel .deb packages locally (arm64 native) Active series: noble (Ubuntu 24.04 LTS, kernel 6.8) Future series: questing, resolute (added on demand via workflow input) Signed-off-by: github-actions[bot] --- .github/workflows/build-kernel.yml | 197 ++++++++++++++++ .github/workflows/fetch-source-pkg.yml | 311 +++++++++++++++++++++++++ .github/workflows/mirror-git.yml | 172 ++++++++++++++ README.md | 256 +++++++++++++++++--- scripts/build-kernel-deb.sh | 109 +++++++++ scripts/check-version.sh | 41 ++++ scripts/fetch-source-pkg.sh | 117 ++++++++++ 7 files changed, 1176 insertions(+), 27 deletions(-) create mode 100644 .github/workflows/build-kernel.yml create mode 100644 .github/workflows/fetch-source-pkg.yml create mode 100644 .github/workflows/mirror-git.yml create mode 100755 scripts/build-kernel-deb.sh create mode 100755 scripts/check-version.sh create mode 100755 scripts/fetch-source-pkg.sh diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml new file mode 100644 index 0000000000000..f0c7865c64c12 --- /dev/null +++ b/.github/workflows/build-kernel.yml @@ -0,0 +1,197 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# build-kernel.yml +# +# Builds Ubuntu kernel .deb packages directly from a series branch +# (e.g. "noble") that was populated by fetch-source-pkg.yml. +# +# Output .deb packages are stored in two places: +# 1. GitHub Actions artifact – attached to the workflow run, retained 90 +# days, downloadable from Actions → → Artifacts. +# 2. GitHub Release asset – attached to the - tag/release +# (e.g. noble-6.8.0-51.52) so they are permanently accessible from the +# Releases page of this repository. +# +# Triggers: +# • Automatic – dispatched by fetch-source-pkg.yml on each new sync +# • Manual – workflow_dispatch with configurable inputs + +name: "Build: Canonical Kernel .deb Packages" + +on: + workflow_dispatch: + inputs: + series: + description: "Series branch to build from (e.g. noble, questing, resolute)" + required: true + default: "noble" + type: string + kernel_version: + description: "Kernel version string (e.g. 6.8.0-51.52) – used for tagging artifacts" + required: false + type: string + arch: + description: "Target Debian architecture" + required: false + default: "arm64" + type: choice + options: + - arm64 + - amd64 + flavor: + description: "Kernel flavour to build" + required: false + default: "generic" + type: choice + options: + - generic + - lowlatency + - all + +permissions: + contents: write # upload assets to release tag + +jobs: + # -------------------------------------------------------------------------- + # Job: build + # -------------------------------------------------------------------------- + build: + name: "Build ${{ inputs.series }} ${{ inputs.flavor }} (${{ inputs.arch }})" + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + timeout-minutes: 360 + + env: + SERIES: ${{ inputs.series || 'noble' }} + ARCH: ${{ inputs.arch || 'arm64' }} + FLAVOR: ${{ inputs.flavor || 'generic' }} + + steps: + # ----------------------------------------------------------------------- + # 1. Checkout the series branch (contains the full kernel source tree) + # ----------------------------------------------------------------------- + - name: Checkout series branch (${{ inputs.series }}) + uses: actions/checkout@v4 + with: + ref: ${{ inputs.series || 'noble' }} + path: kernel-src + + # ----------------------------------------------------------------------- + # 2. Install base build tools + # ----------------------------------------------------------------------- + - name: Install base build tools + run: | + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends \ + build-essential \ + fakeroot \ + devscripts \ + dpkg-dev \ + debhelper \ + dh-exec \ + rsync + + # ----------------------------------------------------------------------- + # 3. Install kernel-specific build dependencies + # ----------------------------------------------------------------------- + - name: Install kernel build dependencies + run: | + sudo apt-get build-dep -y kernel-src/ + + # ----------------------------------------------------------------------- + # 4. Build (native arm64 – no cross-compilation needed) + # ----------------------------------------------------------------------- + - name: Build kernel packages + run: | + JOBS=$(nproc) + echo "Building: series=${SERIES} flavour=${FLAVOR} arch=${ARCH} jobs=${JOBS}" + + # Determine debian/rules target + if [ "${FLAVOR}" = "all" ]; then + TARGET="binary" + else + TARGET="binary-${FLAVOR}" + fi + + export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" + + ( + cd kernel-src/ + fakeroot debian/rules "${TARGET}" + ) + + # ----------------------------------------------------------------------- + # 5. Collect output packages + # The Ubuntu kernel build drops .deb files one level above the + # source tree (i.e. in the working directory, not inside kernel-src/). + # ----------------------------------------------------------------------- + - name: Collect built packages + run: | + mkdir -p output/ + + find . -maxdepth 1 \ + \( -name "*.deb" -o -name "*.changes" -o -name "*.buildinfo" \) \ + -exec cp {} output/ \; + + echo "Built packages:" + ls -lh output/ || echo "(no files found)" + + # ----------------------------------------------------------------------- + # 6. Upload as workflow artifact (always available for 90 days) + # ----------------------------------------------------------------------- + - name: Upload packages as artifact + uses: actions/upload-artifact@v4 + with: + name: "kernel-debs-${{ inputs.series }}-${{ inputs.arch }}-${{ inputs.flavor }}" + path: output/ + retention-days: 90 + if-no-files-found: warn + + # ----------------------------------------------------------------------- + # 7. Attach packages to the release tag (if kernel_version was provided) + # ----------------------------------------------------------------------- + - name: Attach packages to release tag + if: inputs.kernel_version != '' + env: + GH_TOKEN: ${{ github.token }} + run: | + TAG="${SERIES}-${{ inputs.kernel_version }}" + DEBS=$(find output/ -name "*.deb" | sort) + + if [ -z "${DEBS}" ]; then + echo "No .deb files to attach" + exit 0 + fi + + # Check if the tag/release exists before trying to upload + if gh release view "${TAG}" \ + --repo "${{ github.repository }}" \ + --json tagName -q .tagName \ + >/dev/null 2>&1; then + echo "Attaching packages to release ${TAG}..." + echo "${DEBS}" | xargs gh release upload "${TAG}" \ + --repo "${{ github.repository }}" \ + --clobber + echo "Packages attached to ${TAG}" + else + echo "Release tag '${TAG}' not found – packages available as artifact only" + fi + + # ----------------------------------------------------------------------- + # 8. Summary + # ----------------------------------------------------------------------- + - name: Print summary + if: always() + run: | + echo "## Kernel Build Summary" >> "$GITHUB_STEP_SUMMARY" + echo "" >> "$GITHUB_STEP_SUMMARY" + echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" + echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" + echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "" >> "$GITHUB_STEP_SUMMARY" + echo "### Built packages" >> "$GITHUB_STEP_SUMMARY" + echo '```' >> "$GITHUB_STEP_SUMMARY" + ls -lh output/*.deb 2>/dev/null || echo "(none)" + echo '```' >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml new file mode 100644 index 0000000000000..70dff4500e258 --- /dev/null +++ b/.github/workflows/fetch-source-pkg.yml @@ -0,0 +1,311 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# fetch-source-pkg.yml +# +# Fetches the latest Canonical Ubuntu kernel source package from Launchpad, +# extracts it with dpkg-source, and commits the full source tree to a branch +# named after the Ubuntu series (e.g. "noble", "questing"). +# +# Repository branch layout +# ───────────────────────── +# main – CI infrastructure: workflows, scripts, documentation +# noble – Ubuntu Noble (24.04 LTS) kernel source, one commit/upload +# – additional series added on demand (questing, resolute, …) +# +# Each commit on a series branch is tagged - +# (e.g. noble-6.8.0-51.52). +# +# The workflow is idempotent: if the tag already exists it exits cleanly. +# On success it automatically dispatches build-kernel.yml. +# +# Repository variables (Settings → Variables → Actions): +# KERNEL_SERIES – default series to sync when not specified (default: noble) +# KERNEL_SOURCE – source package name (default: linux) + +name: "Sync: Canonical Kernel Sources to Branch" + +on: + schedule: + # Run daily at 04:00 UTC + - cron: "0 4 * * *" + + workflow_dispatch: + inputs: + series: + description: "Ubuntu series to sync (e.g. noble, questing, resolute)" + required: false + default: "noble" + type: string + force: + description: "Force sync even if this version is already committed" + type: boolean + default: false + +permissions: + contents: write # push to series branch + create tags + +jobs: + # -------------------------------------------------------------------------- + # Job 1: check-version + # Queries Launchpad and decides whether a sync is needed. + # -------------------------------------------------------------------------- + check-version: + name: "Check latest version on Launchpad" + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + timeout-minutes: 10 + + outputs: + version: ${{ steps.query.outputs.version }} + upstream_version: ${{ steps.query.outputs.upstream_version }} + tag: ${{ steps.query.outputs.tag }} + series: ${{ steps.query.outputs.series }} + source: ${{ steps.query.outputs.source }} + should_sync: ${{ steps.gate.outputs.should_sync }} + + steps: + - name: Install dependencies + run: sudo apt-get install -y --no-install-recommends jq + + # ----------------------------------------------------------------------- + # Query Launchpad REST API + # ----------------------------------------------------------------------- + - name: Query Launchpad for latest published source + id: query + env: + SERIES: ${{ inputs.series || vars.KERNEL_SERIES || 'noble' }} + SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} + run: | + API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" + API+="?ws.op=getPublishedSources" + API+="&source_name=${SOURCE}" + API+="&distro_series=/ubuntu/${SERIES}" + API+="&status=Published" + API+="&order_by_date=true" + + echo "Querying: ${API}" + RESP=$(curl -fsSL "${API}") + + VERSION=$(echo "$RESP" | jq -r '.entries[0].source_package_version // empty') + [ -n "${VERSION}" ] || { + echo "ERROR: No version found for ${SOURCE} in ${SERIES}" >&2 + exit 1 + } + + UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) + TAG="${SERIES}-${VERSION}" + + echo "version=${VERSION}" >> "$GITHUB_OUTPUT" + echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + echo "series=${SERIES}" >> "$GITHUB_OUTPUT" + echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" + + echo "Series : ${SERIES}" + echo "Version : ${VERSION}" + echo "Tag : ${TAG}" + + # ----------------------------------------------------------------------- + # Gate: skip if the tag already exists (unless force=true) + # ----------------------------------------------------------------------- + - name: Check whether tag already exists + id: gate + env: + TAG: ${{ steps.query.outputs.tag }} + run: | + if [ "${{ inputs.force }}" = "true" ]; then + echo "should_sync=true" >> "$GITHUB_OUTPUT" + echo "Force flag set – will re-sync" + exit 0 + fi + + if git ls-remote --exit-code --tags \ + "https://github.com/${{ github.repository }}.git" \ + "refs/tags/${TAG}" >/dev/null 2>&1; then + echo "should_sync=false" >> "$GITHUB_OUTPUT" + echo "Tag '${TAG}' already exists – nothing to do" + else + echo "should_sync=true" >> "$GITHUB_OUTPUT" + echo "Tag '${TAG}' not found – will sync" + fi + + # -------------------------------------------------------------------------- + # Job 2: sync + # Downloads the source package, extracts it, and commits to the series + # branch. + # -------------------------------------------------------------------------- + sync: + name: "Sync ${{ needs.check-version.outputs.series }} sources to branch" + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + timeout-minutes: 120 + needs: check-version + if: needs.check-version.outputs.should_sync == 'true' + + env: + VERSION: ${{ needs.check-version.outputs.version }} + UPSTREAM_VERSION: ${{ needs.check-version.outputs.upstream_version }} + TAG: ${{ needs.check-version.outputs.tag }} + SERIES: ${{ needs.check-version.outputs.series }} + SOURCE: ${{ needs.check-version.outputs.source }} + + steps: + # ----------------------------------------------------------------------- + # 1. Checkout main branch to get the helper scripts + # ----------------------------------------------------------------------- + - name: Checkout main (for scripts) + uses: actions/checkout@v4 + with: + ref: main + path: ci + + # ----------------------------------------------------------------------- + # 2. Install tools + # ----------------------------------------------------------------------- + - name: Install dependencies + run: | + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends \ + curl jq devscripts dpkg-dev + + # ----------------------------------------------------------------------- + # 3. Download source package from Launchpad + # ----------------------------------------------------------------------- + - name: Fetch source package from Launchpad + run: | + chmod +x ci/scripts/fetch-source-pkg.sh + ci/scripts/fetch-source-pkg.sh "${SERIES}" "${SOURCE}" source-pkg/ + + echo "Downloaded files:" + ls -lh source-pkg/ + + # ----------------------------------------------------------------------- + # 4. Extract source package (applies all Ubuntu patches via quilt) + # ----------------------------------------------------------------------- + - name: Extract source package + run: | + DSC=$(find source-pkg/ -name "*.dsc" | sort -V | tail -1) + echo "Extracting: ${DSC}" + dpkg-source -x "${DSC}" kernel-src/ + + # The extracted directory is named linux- + SRC_DIR=$(find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1) + echo "Extracted to: ${SRC_DIR}" + echo "SRC_DIR=${SRC_DIR}" >> "$GITHUB_ENV" + + echo "Top-level contents:" + ls "${SRC_DIR}/" + + # ----------------------------------------------------------------------- + # 5. Configure git + # ----------------------------------------------------------------------- + - name: Configure git identity + run: | + git config --global user.email "github-actions[bot]@users.noreply.github.com" + git config --global user.name "github-actions[bot]" + + # ----------------------------------------------------------------------- + # 6. Prepare the series branch + # Clone the repo, check out (or create) the series branch, wipe it, + # populate with the freshly extracted source, commit, tag, push. + # ----------------------------------------------------------------------- + - name: Clone repository for series branch update + env: + GH_TOKEN: ${{ github.token }} + run: | + REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" + git clone "${REPO_URL}" series-repo + + - name: Check out or create series branch + run: | + cd series-repo + + if git ls-remote --exit-code origin "refs/heads/${SERIES}" >/dev/null 2>&1; then + echo "Branch '${SERIES}' exists – checking out" + git checkout "${SERIES}" + # Remove all existing content so we get a clean slate + git rm -rf --quiet . 2>/dev/null || true + else + echo "Branch '${SERIES}' does not exist – creating orphan branch" + git checkout --orphan "${SERIES}" + git rm -rf --quiet . 2>/dev/null || true + fi + + - name: Populate series branch with extracted source + run: | + echo "Copying source tree from ${SRC_DIR} → series-repo/" + # rsync preserves permissions and handles large trees efficiently + rsync -a --delete \ + --exclude='.git' \ + "${SRC_DIR}/" series-repo/ + + cd series-repo + + # Write commit message to a file to avoid shell/YAML quoting issues + SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') + printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ + "${SERIES}: ${VERSION}" \ + "Ubuntu series : ${SERIES}" \ + "Kernel version : ${VERSION}" \ + "Upstream : ${UPSTREAM_VERSION}" \ + "Source : https://launchpad.net/ubuntu/${SERIES}/+source/${SOURCE}/${VERSION}" \ + "Synced : ${SYNC_DATE}" \ + > /tmp/commit-msg.txt + + git add --all + git commit -F /tmp/commit-msg.txt + + echo "Committed ${SERIES} branch at $(git rev-parse HEAD)" + + - name: Tag the commit + run: | + cd series-repo + git tag -a "${TAG}" -m "Ubuntu ${SERIES} kernel ${VERSION}" + echo "Tagged: ${TAG}" + + - name: Push series branch and tag + run: | + cd series-repo + git push origin "${SERIES}" + git push origin "${TAG}" + echo "Pushed branch '${SERIES}' and tag '${TAG}'" + + # ----------------------------------------------------------------------- + # 7. Summary + # ----------------------------------------------------------------------- + - name: Print summary + if: always() + run: | + echo "## Source Sync Summary" >> "$GITHUB_STEP_SUMMARY" + echo "" >> "$GITHUB_STEP_SUMMARY" + echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" + echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" + echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | [\`${SERIES}\`](https://github.com/${{ github.repository }}/tree/${SERIES}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Launchpad | https://launchpad.net/ubuntu/${SERIES}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" + + # -------------------------------------------------------------------------- + # Job 3: trigger-build + # Dispatches build-kernel.yml for the newly synced version. + # -------------------------------------------------------------------------- + trigger-build: + name: "Trigger kernel build" + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + needs: [check-version, sync] + if: needs.sync.result == 'success' + + steps: + - name: Dispatch build-kernel workflow + env: + GH_TOKEN: ${{ github.token }} + SERIES: ${{ needs.check-version.outputs.series }} + VERSION: ${{ needs.check-version.outputs.version }} + run: | + gh workflow run build-kernel.yml \ + --repo "${{ github.repository }}" \ + --field series="${SERIES}" \ + --field kernel_version="${VERSION}" \ + --field arch="arm64" \ + --field flavor="generic" + + echo "Build workflow dispatched for ${SERIES} ${VERSION}" diff --git a/.github/workflows/mirror-git.yml b/.github/workflows/mirror-git.yml new file mode 100644 index 0000000000000..5e06aea39a292 --- /dev/null +++ b/.github/workflows/mirror-git.yml @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# mirror-git.yml +# +# Optional workflow: mirrors the full Canonical Ubuntu kernel git repository +# from Launchpad to a separate GitHub repository, preserving the complete +# commit history from the Ubuntu kernel team. +# +# This is complementary to fetch-source-pkg.yml which commits one snapshot +# per Canonical upload to a series branch in THIS repository. The git mirror +# gives you the full intermediate history (every commit the Ubuntu kernel team +# made between uploads). +# +# Source : https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/ +# Target : a separate GitHub repository configured via MIRROR_TARGET_REPO +# +# Required secret (Settings → Secrets → Actions): +# MIRROR_PUSH_TOKEN – GitHub PAT (classic) with repo scope on the target +# repository, OR a fine-grained PAT with +# "Contents: Read and write" on that repository. +# +# Repository variables (Settings → Variables → Actions): +# MIRROR_TARGET_REPO – owner/repo of the GitHub mirror repository +# (default: qualcomm-linux/linux-noble) +# KERNEL_SERIES – Ubuntu series to mirror (default: noble) +# +# Note: the first run performs a full bare clone (~3–4 GB); subsequent runs +# are incremental (fetch delta only). + +name: "Mirror: Canonical Kernel Git (Launchpad → GitHub)" + +on: + schedule: + # Run daily at 03:00 UTC – one hour before the source-sync job + - cron: "0 3 * * *" + + workflow_dispatch: + inputs: + series: + description: "Ubuntu series to mirror (e.g. noble, questing, resolute)" + required: false + default: "noble" + type: string + force_push: + description: "Force-push all refs (use after target repo reset)" + type: boolean + default: false + +permissions: + contents: read + +jobs: + # -------------------------------------------------------------------------- + # Job: mirror + # -------------------------------------------------------------------------- + mirror: + name: "Sync Launchpad → GitHub (${{ inputs.series || vars.KERNEL_SERIES || 'noble' }})" + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + timeout-minutes: 180 # bare clone of the full kernel tree can be slow + + env: + SERIES: ${{ inputs.series || vars.KERNEL_SERIES || 'noble' }} + TARGET_REPO: ${{ vars.MIRROR_TARGET_REPO || format('qualcomm-linux/linux-{0}', inputs.series || vars.KERNEL_SERIES || 'noble') }} + + steps: + # ----------------------------------------------------------------------- + # 1. Configure git + # ----------------------------------------------------------------------- + - name: Configure git + run: | + git config --global user.email "github-actions[bot]@users.noreply.github.com" + git config --global user.name "github-actions[bot]" + git config --global advice.detachedHead false + + # ----------------------------------------------------------------------- + # 2. Decide: incremental sync (mirror already exists) or full clone + # ----------------------------------------------------------------------- + - name: Check whether GitHub mirror already exists + id: check_mirror + env: + PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} + run: | + GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" + if git ls-remote --exit-code "${GITHUB_URL}" HEAD >/dev/null 2>&1; then + echo "exists=true" >> "$GITHUB_OUTPUT" + echo "Mirror found at github.com/${TARGET_REPO}" + else + echo "exists=false" >> "$GITHUB_OUTPUT" + echo "Mirror not found or empty – will do full clone" + fi + + # ----------------------------------------------------------------------- + # 3a. Incremental sync (subsequent runs) + # Clone from GitHub (fast), fetch only new commits from Launchpad. + # ----------------------------------------------------------------------- + - name: Incremental sync (clone GitHub mirror + fetch Launchpad delta) + if: steps.check_mirror.outputs.exists == 'true' + env: + PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} + run: | + GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" + LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}" + + echo "Cloning existing GitHub mirror (bare)..." + git clone --mirror "${GITHUB_URL}" linux.git + + cd linux.git + echo "Fetching updates from Launchpad..." + git remote add launchpad "${LAUNCHPAD_URL}" + git fetch --all --prune launchpad + + echo "Repository size after fetch:" + du -sh . + + # ----------------------------------------------------------------------- + # 3b. Full initial clone (first run only) + # ----------------------------------------------------------------------- + - name: Full initial clone (from Launchpad) + if: steps.check_mirror.outputs.exists == 'false' + run: | + LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}" + + echo "Full bare clone from Launchpad (this may take 30–60 min)..." + git clone --mirror "${LAUNCHPAD_URL}" linux.git + + echo "Repository size:" + du -sh linux.git/ + + # ----------------------------------------------------------------------- + # 4. Push to GitHub + # ----------------------------------------------------------------------- + - name: Push to GitHub mirror + env: + PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} + run: | + cd linux.git + + GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" + + # Normalise remote name to 'github' + if git remote get-url github >/dev/null 2>&1; then + git remote set-url github "${GITHUB_URL}" + elif git remote get-url origin >/dev/null 2>&1; then + git remote set-url origin "${GITHUB_URL}" + git remote rename origin github + else + git remote add github "${GITHUB_URL}" + fi + + PUSH_FLAGS="--mirror" + if [ "${{ inputs.force_push }}" = "true" ]; then + echo "Force-push requested" + fi + + echo "Pushing to github.com/${TARGET_REPO}..." + git push ${PUSH_FLAGS} github + + echo "Mirror sync complete → github.com/${TARGET_REPO}" + + # ----------------------------------------------------------------------- + # 5. Summary + # ----------------------------------------------------------------------- + - name: Print summary + if: always() + run: | + echo "## Git Mirror Summary" >> "$GITHUB_STEP_SUMMARY" + echo "" >> "$GITHUB_STEP_SUMMARY" + echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" + echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" + echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Source | \`git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Target | \`github.com/${TARGET_REPO}\` |" >> "$GITHUB_STEP_SUMMARY" diff --git a/README.md b/README.md index a27531da6dd85..99569d8325ad2 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,249 @@ -**After repository creation:** -- [ ] Update this `README.md`. Update the Project Name, description, and all sections. Remove this checklist. -- [ ] If required, update `LICENSE.txt` and the License section with your project's approved license -- [ ] Search this repo for "REPLACE-ME" and update all instances accordingly -- [ ] Update `CONTRIBUTING.md` as needed -- [ ] Review the workflows in `.github/workflows`, updating as needed. See https://docs.github.com/en/actions for information on what these files do and how they work. -- [ ] Review and update the suggested Issue and PR templates as needed in `.github/ISSUE_TEMPLATE` and `.github/PULL_REQUEST_TEMPLATE` +# pkg-linux-qcom-canonical -# Project Name +Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. -*\* +--- -Project that does ... implemented in ... runs on Qualcomm® *\* +## Repository layout -## Branches +``` +main branch (this branch) +├── .github/workflows/ +│ ├── fetch-source-pkg.yml ← sync Launchpad sources → series branch +│ ├── build-kernel.yml ← build .deb packages from series branch +│ └── mirror-git.yml ← optional: full git history mirror +├── scripts/ +│ ├── check-version.sh ← query latest version from Launchpad +│ ├── fetch-source-pkg.sh ← download source package files +│ └── build-kernel-deb.sh ← build kernel .deb packages locally +└── README.md -**main**: Primary development branch. Contributors should develop submissions based on this branch, and submit pull requests to this branch. +noble branch ← Ubuntu Noble (24.04 LTS) kernel source tree, one commit per upload + ← additional series added on demand (questing, resolute, …) +``` -## Requirements +Series branches are **orphan branches** — they share no history with `main` +and contain only the extracted kernel source tree. -List requirements to run the project, how to install them, instructions to use docker container, etc... +Each upload is tagged `-`, e.g. `noble-6.8.0-51.52`. -## Installation Instructions +--- -How to install the software itself. +## Upstream sources -## Usage +| Resource | URL | +|----------|-----| +| Launchpad source packages | https://launchpad.net/ubuntu/noble/+source/linux | +| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` | +| Launchpad REST API | https://api.launchpad.net/1.0/ | +| GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | -Describe how to use the project. +--- -## Development +## Workflows -How to develop new features/fixes for the software. Maybe different than "usage". Also provide details on how to contribute via a [CONTRIBUTING.md file](CONTRIBUTING.md). +### `fetch-source-pkg.yml` — Sync sources to branch -## Getting in Contact +Queries the Launchpad REST API for the latest published `linux` source +package, downloads the `.dsc` + tarballs, extracts the full source tree +with `dpkg-source -x` (applying all Ubuntu patches), and commits the +result to the corresponding series branch. -How to contact maintainers. E.g. GitHub Issues, GitHub Discussions could be indicated for many cases. However a mail list or list of Maintainer e-mails could be shared for other types of discussions. E.g. +``` +Launchpad archive + linux_6.8.0.orig.tar.gz ─┐ + linux_6.8.0-51.52.debian.tar.xz ├─ dpkg-source -x ──► noble branch commit + linux_6.8.0-51.52.dsc ─┘ tagged noble-6.8.0-51.52 +``` -* [Report an Issue on GitHub](../../issues) -* [Open a Discussion on GitHub](../../discussions) -* [E-mail us](mailto:REPLACE-ME@qti.qualcomm.com) for general questions +**Schedule**: daily at **04:00 UTC** +**Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` + +**Inputs** (manual dispatch): + +| Input | Default | Description | +|-------|---------|-------------| +| `series` | `noble` | Ubuntu series to sync | +| `force` | `false` | Re-sync even if tag already exists | + +**Idempotent**: checks for the tag before downloading anything. +**Auto-triggers**: dispatches `build-kernel.yml` on each new sync. + +--- + +### `build-kernel.yml` — Build .deb packages + +Checks out the series branch (which contains the full kernel source tree) +and builds `.deb` packages using the Ubuntu `debian/rules` build system. + +**Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or +manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. + +**Inputs** (manual dispatch): + +| Input | Default | Description | +|-------|---------|-------------| +| `series` | `noble` | Series branch to build from | +| `kernel_version` | — | Version string for artifact naming | +| `arch` | `arm64` | Target architecture: `arm64` or `amd64` | +| `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | + +**Output — two locations**: + +| Location | How to access | Retention | +|----------|---------------|-----------| +| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* section at the bottom | 90 days | +| **GitHub Release asset** | Releases page → tag `noble-6.8.0-X.Y` → Assets | Permanent | + +The `.deb` files are attached to the release tag automatically when `kernel_version` is provided (which `fetch-source-pkg.yml` always does). + +**Resource requirements**: + +| Resource | Requirement | +|----------|-------------| +| Disk space | ~20 GB (runner is cleaned before build) | +| Wall-clock | ~60–90 min (generic, 2 vCPU GitHub runner) | +| RAM | ~4 GB | + +> **Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` (Ubuntu 24.04 arm64, native build — no cross-compilation). + +--- + +### `mirror-git.yml` — Full git history mirror *(optional)* + +Mirrors the complete Canonical Ubuntu kernel git tree from Launchpad to a +separate GitHub repository, preserving every intermediate commit made by +the Ubuntu kernel team between uploads. + +This is complementary to `fetch-source-pkg.yml`: the series branch gives +you one clean snapshot per upload; the git mirror gives you the full +development history. + +**Schedule**: daily at **03:00 UTC** +**Required secret**: `MIRROR_PUSH_TOKEN` (PAT with `repo` scope on the target repo) +**Required variable**: `MIRROR_TARGET_REPO` (e.g. `qualcomm-linux/linux-noble`) + +--- + +## Setup + +### 1. Enable workflows + +Go to **Actions** and enable workflows if prompted. + +### 2. Configure repository variables *(optional)* + +**Settings → Secrets and variables → Actions → Variables**: + +| Variable | Default | Description | +|----------|---------|-------------| +| `KERNEL_SERIES` | `noble` | Default series for scheduled runs | +| `KERNEL_SOURCE` | `linux` | Source package name | +| `MIRROR_TARGET_REPO` | `qualcomm-linux/linux-noble` | Target repo for git mirror | + +### 3. Configure secrets *(only needed for git mirror)* + +**Settings → Secrets and variables → Actions → Secrets**: + +| Secret | Description | +|--------|-------------| +| `MIRROR_PUSH_TOKEN` | GitHub PAT with `repo` scope on `MIRROR_TARGET_REPO` | + +### 4. Run the first sync + +```bash +# Sync noble sources to the noble branch (creates it if it doesn't exist) +gh workflow run fetch-source-pkg.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical \ + --field series=noble + +# Or trigger a build manually from an existing series branch +gh workflow run build-kernel.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical \ + --field series=noble \ + --field arch=arm64 \ + --field flavor=generic +``` + +--- + +## Local usage + +All scripts run on Ubuntu 22.04 / 24.04. + +### Check the latest version + +```bash +./scripts/check-version.sh noble linux +# → 6.8.0-51.52 +``` + +### Download the source package + +```bash +./scripts/fetch-source-pkg.sh noble linux ./source-pkg/ +``` + +### Build kernel packages + +```bash +# arm64 generic (native build on arm64 host) +./scripts/build-kernel-deb.sh ./kernel-src/ arm64 generic $(nproc) +``` + +--- + +## Source package anatomy + +The Ubuntu kernel source package is a standard Debian 3.0 (quilt) source package: + +| File | Size | Description | +|------|------|-------------| +| `linux_X.Y.Z-A.B.dsc` | ~10 KB | Source descriptor with SHA256 checksums | +| `linux_X.Y.Z.orig.tar.gz` | ~200 MB | Pristine upstream kernel tarball | +| `linux_X.Y.Z-A.B.debian.tar.xz` | ~5 MB | Ubuntu packaging overlay + patches | + +`dpkg-source -x` applies all patches and produces the full source tree +that is committed to the series branch. + +## Versioning scheme + +Ubuntu kernel versions follow `X.Y.Z-A.B`: + +| Component | Example | Meaning | +|-----------|---------|---------| +| `X.Y.Z` | `6.8.0` | Upstream kernel version | +| `A` | `51` | ABI number | +| `B` | `52` | Upload number | + +Tags in this repository use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. + +--- + +## Supported series + +| Series | Codename | Status | Kernel | +|--------|----------|--------|--------| +| `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | +| `questing` | Questing Quokka | 25.04 — add when available | TBD | +| `resolute` | Resolute Ringtail | 25.10 — add when available | TBD | + +To add a new series, trigger `fetch-source-pkg.yml` with the desired +`series` input — the branch and release tag are created automatically: + +```bash +gh workflow run fetch-source-pkg.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical \ + --field series=questing +``` + +--- ## License -*\* +Scripts and workflows in this repository are licensed under the +[BSD 3-Clause License](LICENSE.txt). -*\* is licensed under the [BSD-3-clause License](https://spdx.org/licenses/BSD-3-Clause.html). See [LICENSE.txt](LICENSE.txt) for the full license text. +The kernel source code fetched from Launchpad is subject to the +[GNU General Public License v2](https://www.kernel.org/doc/html/latest/process/license-rules.html) +and the individual licences of its components. diff --git a/scripts/build-kernel-deb.sh b/scripts/build-kernel-deb.sh new file mode 100755 index 0000000000000..0d292916aaee1 --- /dev/null +++ b/scripts/build-kernel-deb.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# +# build-kernel-deb.sh - Build Ubuntu kernel .deb packages from a Canonical +# source tree (as checked out from a series branch) +# +# Usage: +# build-kernel-deb.sh [SOURCE_DIR] [ARCH] [FLAVOR] [JOBS] +# +# Arguments: +# SOURCE_DIR Root of the kernel source tree containing debian/ (default: .) +# ARCH Target Debian architecture: arm64 | amd64 (default: arm64) +# FLAVOR Kernel flavour: generic | lowlatency | all (default: generic) +# JOBS Parallel make jobs (default: nproc) +# +# Output: +# Built .deb packages are placed in ./output/ relative to the working +# directory from which this script is invoked. +# +# Notes: +# • Designed for native arm64 builds (Ubuntu 24.04 arm64 host). +# • The Ubuntu kernel build needs ~20 GB of free disk space. +# • A full build (all flavours) can take 2+ hours; 'generic' is ~1 hour. +# • Run as a normal user; sudo is used only for apt-get. + +set -euo pipefail + +SOURCE_DIR="${1:-.}" +ARCH="${2:-arm64}" +FLAVOR="${3:-generic}" +JOBS="${4:-$(nproc)}" + +OUTPUT_DIR="$(pwd)/output" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- +log() { printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2; } +die() { log "ERROR: $*"; exit 1; } +hr() { log "$(printf '%0.s─' {1..60})"; } + +hr +log "Ubuntu kernel .deb build" +log " Source dir : ${SOURCE_DIR}" +log " Arch : ${ARCH}" +log " Flavour : ${FLAVOR}" +log " Jobs : ${JOBS}" +log " Output dir : ${OUTPUT_DIR}" +hr + +# --------------------------------------------------------------------------- +# 1. Validate source tree +# --------------------------------------------------------------------------- +[ -f "${SOURCE_DIR}/debian/rules" ] \ + || die "No debian/rules found in '${SOURCE_DIR}' – is this a kernel source tree?" + +# --------------------------------------------------------------------------- +# 2. Install build dependencies +# --------------------------------------------------------------------------- +hr +log "Installing build dependencies (requires sudo)..." +sudo apt-get update -qq +sudo apt-get build-dep -y "${SOURCE_DIR}" \ + || die "apt-get build-dep failed" + +# --------------------------------------------------------------------------- +# 3. Build +# --------------------------------------------------------------------------- +hr +log "Starting kernel build (flavour=${FLAVOR}, arch=${ARCH}, jobs=${JOBS})..." + +# Determine the debian/rules target +if [ "${FLAVOR}" = "all" ]; then + RULES_TARGET="binary" +else + RULES_TARGET="binary-${FLAVOR}" +fi + +# The Ubuntu kernel build system reads DEB_BUILD_OPTIONS for parallelism +export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" + +# Run the build (native arm64 – no cross-compilation flags needed) +( + cd "${SOURCE_DIR}" + fakeroot debian/rules "${RULES_TARGET}" \ + || die "debian/rules ${RULES_TARGET} failed" +) + +# --------------------------------------------------------------------------- +# 4. Collect output packages +# --------------------------------------------------------------------------- +hr +mkdir -p "${OUTPUT_DIR}" + +# The Ubuntu kernel build drops .deb files one level above the source tree +PARENT_DIR=$(dirname "$(realpath "${SOURCE_DIR}")") +find "${PARENT_DIR}" -maxdepth 1 \ + \( -name "*.deb" -o -name "*.changes" -o -name "*.buildinfo" \) \ + | while read -r f; do + cp "${f}" "${OUTPUT_DIR}/" + log " Collected: $(basename "${f}")" + done + +hr +log "Build complete." +log "" +log "Output packages:" +ls -lh "${OUTPUT_DIR}"/*.deb 2>/dev/null \ + || log " (no .deb files found — check build log above)" diff --git a/scripts/check-version.sh b/scripts/check-version.sh new file mode 100755 index 0000000000000..fbbb386d54fde --- /dev/null +++ b/scripts/check-version.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# +# check-version.sh - Query the latest published kernel version from Launchpad +# +# Usage: +# check-version.sh [SERIES] [SOURCE_NAME] +# +# Arguments: +# SERIES Ubuntu series (default: noble) +# SOURCE_NAME Source package name (default: linux) +# +# Output: +# Prints the latest version string to stdout (e.g. "6.8.0-51.52") +# +# Exit codes: +# 0 Version found +# 1 Version not found or API error + +set -euo pipefail + +SERIES="${1:-noble}" +SOURCE_NAME="${2:-linux}" + +LAUNCHPAD_API="https://api.launchpad.net/1.0" + +die() { echo "ERROR: $*" >&2; exit 1; } + +RESPONSE=$(curl -fsSL \ + "${LAUNCHPAD_API}/ubuntu/+archive/primary?ws.op=getPublishedSources\ +&source_name=${SOURCE_NAME}\ +&distro_series=/ubuntu/${SERIES}\ +&status=Published\ +&order_by_date=true") \ + || die "Failed to query Launchpad API" + +VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') + +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SERIES}'" + +echo "${VERSION}" diff --git a/scripts/fetch-source-pkg.sh b/scripts/fetch-source-pkg.sh new file mode 100755 index 0000000000000..2f745a3557065 --- /dev/null +++ b/scripts/fetch-source-pkg.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# +# fetch-source-pkg.sh - Download a Canonical Ubuntu kernel source package +# from Launchpad +# +# Usage: +# fetch-source-pkg.sh [SERIES] [SOURCE_NAME] [OUTPUT_DIR] +# +# Arguments: +# SERIES Ubuntu series (default: noble) +# SOURCE_NAME Source package name (default: linux) +# OUTPUT_DIR Directory to write files into (default: .) +# +# The script queries the Launchpad REST API to find the latest published +# source, then downloads all constituent files (.dsc, .orig.tar.gz, +# .debian.tar.xz, etc.) and writes a version.env summary file. +# +# Environment variables (override defaults): +# LAUNCHPAD_API Base URL for the Launchpad API (default: https://api.launchpad.net/1.0) + +set -euo pipefail + +SERIES="${1:-noble}" +SOURCE_NAME="${2:-linux}" +OUTPUT_DIR="${3:-.}" + +LAUNCHPAD_API="${LAUNCHPAD_API:-https://api.launchpad.net/1.0}" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- +log() { printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2; } +die() { log "ERROR: $*"; exit 1; } +hr() { log "$(printf '%0.s─' {1..60})"; } + +# --------------------------------------------------------------------------- +# 1. Query Launchpad for the latest published source +# --------------------------------------------------------------------------- +hr +log "Querying Launchpad for latest '${SOURCE_NAME}' in Ubuntu ${SERIES}..." + +API_URL="${LAUNCHPAD_API}/ubuntu/+archive/primary" +API_URL+="?ws.op=getPublishedSources" +API_URL+="&source_name=${SOURCE_NAME}" +API_URL+="&distro_series=/ubuntu/${SERIES}" +API_URL+="&status=Published" +API_URL+="&order_by_date=true" + +RESPONSE=$(curl -fsSL "${API_URL}") \ + || die "Launchpad API request failed: ${API_URL}" + +VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') +SELF_LINK=$(echo "$RESPONSE" | jq -r '.entries[0].self_link // empty') + +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SERIES}'" +[ -n "$SELF_LINK" ] || die "Could not retrieve self_link for '${SOURCE_NAME}' ${VERSION}" + +# Upstream version: strip Ubuntu revision suffix (e.g. "6.8.0-51.52" → "6.8.0") +UPSTREAM_VERSION=$(echo "${VERSION}" | cut -d'-' -f1) + +log "Found: ${SOURCE_NAME} ${VERSION} (upstream: ${UPSTREAM_VERSION})" +log "Link: ${SELF_LINK}" + +# --------------------------------------------------------------------------- +# 2. Retrieve per-file download URLs +# --------------------------------------------------------------------------- +hr +log "Fetching file list..." + +FILE_URLS=$(curl -fsSL "${SELF_LINK}?ws.op=sourceFileUrls" | jq -r '.[]') \ + || die "Failed to retrieve file URLs from ${SELF_LINK}" + +[ -n "$FILE_URLS" ] || die "No files listed for ${SOURCE_NAME} ${VERSION}" + +FILE_COUNT=$(echo "$FILE_URLS" | wc -l) +log "Files to download: ${FILE_COUNT}" + +# --------------------------------------------------------------------------- +# 3. Download each file +# --------------------------------------------------------------------------- +hr +mkdir -p "${OUTPUT_DIR}" + +IDX=0 +while IFS= read -r url; do + IDX=$((IDX + 1)) + FILENAME=$(basename "${url%%\?*}") # strip any query string + DEST="${OUTPUT_DIR}/${FILENAME}" + + log "[${IDX}/${FILE_COUNT}] ${FILENAME}" + curl -fsSL --progress-bar -o "${DEST}" "${url}" \ + || die "Download failed: ${url}" + + SIZE=$(du -sh "${DEST}" | cut -f1) + log " → ${SIZE} ${DEST}" +done <<< "$FILE_URLS" + +# --------------------------------------------------------------------------- +# 4. Write version metadata +# --------------------------------------------------------------------------- +hr +VERSION_ENV="${OUTPUT_DIR}/version.env" +cat > "${VERSION_ENV}" < Date: Mon, 20 Apr 2026 14:50:17 -0700 Subject: [PATCH 003/115] ci: add Canonical Ubuntu kernel mirror and build pipeline Implements a complete CI/CD pipeline for mirroring and building the Canonical Ubuntu kernel source package from Launchpad, targeting the arm64 architecture on Ubuntu 24.04 LTS. Repository branch layout: main - CI infrastructure (workflows, scripts, documentation) - Orphan branch per Ubuntu series containing the full extracted kernel source tree (one commit per upload) e.g. noble branch for Ubuntu Noble 24.04 LTS Workflows (.github/workflows/): fetch-source-pkg.yml - Queries the Launchpad REST API for the latest published linux source package for the given series - Downloads .dsc, .orig.tar.gz, .debian.tar.xz from Launchpad - Extracts the full patched source tree via dpkg-source -x - Commits the complete kernel source to the series branch - Tags each commit as - (e.g. noble-6.8.0-51.52) - Idempotent: skips if the tag already exists - Auto-dispatches build-kernel.yml on each new version - Schedule: daily 04:00 UTC - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem build-kernel.yml - Checks out the series branch (full kernel source tree) - Installs build dependencies via apt-get build-dep - Native arm64 build: fakeroot debian/rules binary- - No cross-compilation required (runner is arm64) - Uploads built .deb packages as: * GitHub Actions artifact (90-day retention) * GitHub Release asset on the - tag (permanent) - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem Helper scripts (scripts/): check-version.sh - Query latest published version from Launchpad API fetch-source-pkg.sh - Download source package files from Launchpad build-kernel-deb.sh - Build kernel .deb packages locally (arm64 native) Active series: noble (Ubuntu 24.04 LTS, kernel 6.8) Future series: questing, resolute (added on demand via workflow input) --- .github/workflows/mirror-git.yml | 172 ------------------------------ README.md | 173 +++++++++++++++++-------------- 2 files changed, 94 insertions(+), 251 deletions(-) delete mode 100644 .github/workflows/mirror-git.yml diff --git a/.github/workflows/mirror-git.yml b/.github/workflows/mirror-git.yml deleted file mode 100644 index 5e06aea39a292..0000000000000 --- a/.github/workflows/mirror-git.yml +++ /dev/null @@ -1,172 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# -# mirror-git.yml -# -# Optional workflow: mirrors the full Canonical Ubuntu kernel git repository -# from Launchpad to a separate GitHub repository, preserving the complete -# commit history from the Ubuntu kernel team. -# -# This is complementary to fetch-source-pkg.yml which commits one snapshot -# per Canonical upload to a series branch in THIS repository. The git mirror -# gives you the full intermediate history (every commit the Ubuntu kernel team -# made between uploads). -# -# Source : https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/ -# Target : a separate GitHub repository configured via MIRROR_TARGET_REPO -# -# Required secret (Settings → Secrets → Actions): -# MIRROR_PUSH_TOKEN – GitHub PAT (classic) with repo scope on the target -# repository, OR a fine-grained PAT with -# "Contents: Read and write" on that repository. -# -# Repository variables (Settings → Variables → Actions): -# MIRROR_TARGET_REPO – owner/repo of the GitHub mirror repository -# (default: qualcomm-linux/linux-noble) -# KERNEL_SERIES – Ubuntu series to mirror (default: noble) -# -# Note: the first run performs a full bare clone (~3–4 GB); subsequent runs -# are incremental (fetch delta only). - -name: "Mirror: Canonical Kernel Git (Launchpad → GitHub)" - -on: - schedule: - # Run daily at 03:00 UTC – one hour before the source-sync job - - cron: "0 3 * * *" - - workflow_dispatch: - inputs: - series: - description: "Ubuntu series to mirror (e.g. noble, questing, resolute)" - required: false - default: "noble" - type: string - force_push: - description: "Force-push all refs (use after target repo reset)" - type: boolean - default: false - -permissions: - contents: read - -jobs: - # -------------------------------------------------------------------------- - # Job: mirror - # -------------------------------------------------------------------------- - mirror: - name: "Sync Launchpad → GitHub (${{ inputs.series || vars.KERNEL_SERIES || 'noble' }})" - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] - timeout-minutes: 180 # bare clone of the full kernel tree can be slow - - env: - SERIES: ${{ inputs.series || vars.KERNEL_SERIES || 'noble' }} - TARGET_REPO: ${{ vars.MIRROR_TARGET_REPO || format('qualcomm-linux/linux-{0}', inputs.series || vars.KERNEL_SERIES || 'noble') }} - - steps: - # ----------------------------------------------------------------------- - # 1. Configure git - # ----------------------------------------------------------------------- - - name: Configure git - run: | - git config --global user.email "github-actions[bot]@users.noreply.github.com" - git config --global user.name "github-actions[bot]" - git config --global advice.detachedHead false - - # ----------------------------------------------------------------------- - # 2. Decide: incremental sync (mirror already exists) or full clone - # ----------------------------------------------------------------------- - - name: Check whether GitHub mirror already exists - id: check_mirror - env: - PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} - run: | - GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" - if git ls-remote --exit-code "${GITHUB_URL}" HEAD >/dev/null 2>&1; then - echo "exists=true" >> "$GITHUB_OUTPUT" - echo "Mirror found at github.com/${TARGET_REPO}" - else - echo "exists=false" >> "$GITHUB_OUTPUT" - echo "Mirror not found or empty – will do full clone" - fi - - # ----------------------------------------------------------------------- - # 3a. Incremental sync (subsequent runs) - # Clone from GitHub (fast), fetch only new commits from Launchpad. - # ----------------------------------------------------------------------- - - name: Incremental sync (clone GitHub mirror + fetch Launchpad delta) - if: steps.check_mirror.outputs.exists == 'true' - env: - PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} - run: | - GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" - LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}" - - echo "Cloning existing GitHub mirror (bare)..." - git clone --mirror "${GITHUB_URL}" linux.git - - cd linux.git - echo "Fetching updates from Launchpad..." - git remote add launchpad "${LAUNCHPAD_URL}" - git fetch --all --prune launchpad - - echo "Repository size after fetch:" - du -sh . - - # ----------------------------------------------------------------------- - # 3b. Full initial clone (first run only) - # ----------------------------------------------------------------------- - - name: Full initial clone (from Launchpad) - if: steps.check_mirror.outputs.exists == 'false' - run: | - LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}" - - echo "Full bare clone from Launchpad (this may take 30–60 min)..." - git clone --mirror "${LAUNCHPAD_URL}" linux.git - - echo "Repository size:" - du -sh linux.git/ - - # ----------------------------------------------------------------------- - # 4. Push to GitHub - # ----------------------------------------------------------------------- - - name: Push to GitHub mirror - env: - PUSH_TOKEN: ${{ secrets.MIRROR_PUSH_TOKEN }} - run: | - cd linux.git - - GITHUB_URL="https://x-access-token:${PUSH_TOKEN}@github.com/${TARGET_REPO}.git" - - # Normalise remote name to 'github' - if git remote get-url github >/dev/null 2>&1; then - git remote set-url github "${GITHUB_URL}" - elif git remote get-url origin >/dev/null 2>&1; then - git remote set-url origin "${GITHUB_URL}" - git remote rename origin github - else - git remote add github "${GITHUB_URL}" - fi - - PUSH_FLAGS="--mirror" - if [ "${{ inputs.force_push }}" = "true" ]; then - echo "Force-push requested" - fi - - echo "Pushing to github.com/${TARGET_REPO}..." - git push ${PUSH_FLAGS} github - - echo "Mirror sync complete → github.com/${TARGET_REPO}" - - # ----------------------------------------------------------------------- - # 5. Summary - # ----------------------------------------------------------------------- - - name: Print summary - if: always() - run: | - echo "## Git Mirror Summary" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" - echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Source | \`git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Target | \`github.com/${TARGET_REPO}\` |" >> "$GITHUB_STEP_SUMMARY" diff --git a/README.md b/README.md index 99569d8325ad2..50923e2367c99 100644 --- a/README.md +++ b/README.md @@ -4,32 +4,92 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. --- -## Repository layout +## End-to-end pipeline ``` -main branch (this branch) -├── .github/workflows/ -│ ├── fetch-source-pkg.yml ← sync Launchpad sources → series branch -│ ├── build-kernel.yml ← build .deb packages from series branch -│ └── mirror-git.yml ← optional: full git history mirror -├── scripts/ -│ ├── check-version.sh ← query latest version from Launchpad -│ ├── fetch-source-pkg.sh ← download source package files -│ └── build-kernel-deb.sh ← build kernel .deb packages locally -└── README.md - -noble branch ← Ubuntu Noble (24.04 LTS) kernel source tree, one commit per upload - ← additional series added on demand (questing, resolute, …) +┌─────────────────────────────────────────────────────────────────────┐ +│ DAILY (04:00 UTC) │ +│ │ +│ Launchpad │ +│ api.launchpad.net ──► fetch-source-pkg.yml │ +│ │ │ +│ ┌─────────▼──────────┐ │ +│ │ Job 1: check-version │ +│ │ curl Launchpad API │ │ +│ │ → noble 6.8.0-51.52│ │ +│ │ tag exists? YES→skip │ +│ │ NO ↓ │ │ +│ └─────────┬──────────┘ │ +│ │ │ +│ ┌─────────▼──────────┐ │ +│ │ Job 2: sync │ │ +│ │ download .dsc │ │ +│ │ + .orig.tar.gz │ │ +│ │ + .debian.tar.xz │ │ +│ │ dpkg-source -x │ │ +│ │ → full source tree │ │ +│ │ commit to noble │ │ +│ │ branch + tag │ │ +│ └─────────┬──────────┘ │ +│ │ │ +│ ┌─────────▼──────────┐ │ +│ │ Job 3: trigger │ │ +│ │ gh workflow run │ │ +│ │ build-kernel.yml │ │ +│ └─────────┬──────────┘ │ +│ │ │ +│ ┌─────────▼──────────┐ │ +│ │ build-kernel.yml │ │ +│ │ checkout noble │ │ +│ │ branch │ │ +│ │ apt-get build-dep │ │ +│ │ fakeroot │ │ +│ │ debian/rules │ │ +│ │ binary-generic │ │ +│ └─────────┬──────────┘ │ +│ │ │ +│ ┌──────────────┴──────────────┐ │ +│ ▼ ▼ │ +│ GitHub Actions Artifact GitHub Release Asset │ +│ (90-day retention) noble-6.8.0-51.52 │ +│ Actions → run → Artifacts Releases page → Assets │ +│ (permanent) │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Repository branch layout + +``` +pkg-linux-qcom-canonical +│ +├── main branch +│ ├── .github/workflows/ +│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → series branch +│ │ └── build-kernel.yml ← build .deb packages from series branch +│ ├── scripts/ +│ │ ├── check-version.sh ← query latest version from Launchpad +│ │ ├── fetch-source-pkg.sh ← download source package files +│ │ └── build-kernel-deb.sh ← build kernel .deb packages locally +│ └── README.md +│ +├── noble branch (orphan) +│ └── Full Ubuntu Noble 24.04 LTS kernel source tree +│ One commit per Canonical upload +│ Tagged noble-6.8.0-51.52, noble-6.8.0-52.53, … +│ +└── branch (orphan, added on demand) + └── Full kernel source for that series + e.g. questing, resolute ``` Series branches are **orphan branches** — they share no history with `main` and contain only the extracted kernel source tree. -Each upload is tagged `-`, e.g. `noble-6.8.0-51.52`. - --- -## Upstream sources +## Upstream source | Resource | URL | |----------|-----| @@ -49,21 +109,14 @@ package, downloads the `.dsc` + tarballs, extracts the full source tree with `dpkg-source -x` (applying all Ubuntu patches), and commits the result to the corresponding series branch. -``` -Launchpad archive - linux_6.8.0.orig.tar.gz ─┐ - linux_6.8.0-51.52.debian.tar.xz ├─ dpkg-source -x ──► noble branch commit - linux_6.8.0-51.52.dsc ─┘ tagged noble-6.8.0-51.52 -``` - **Schedule**: daily at **04:00 UTC** **Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` -**Inputs** (manual dispatch): +**Inputs**: | Input | Default | Description | |-------|---------|-------------| -| `series` | `noble` | Ubuntu series to sync | +| `series` | `noble` | Ubuntu series to sync — one series per run | | `force` | `false` | Re-sync even if tag already exists | **Idempotent**: checks for the tag before downloading anything. @@ -73,55 +126,31 @@ Launchpad archive ### `build-kernel.yml` — Build .deb packages -Checks out the series branch (which contains the full kernel source tree) -and builds `.deb` packages using the Ubuntu `debian/rules` build system. +Checks out the series branch (full kernel source tree) and builds `.deb` +packages using the Ubuntu `debian/rules` build system on the native arm64 +self-hosted runner. **Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. -**Inputs** (manual dispatch): +**Inputs**: | Input | Default | Description | |-------|---------|-------------| | `series` | `noble` | Series branch to build from | -| `kernel_version` | — | Version string for artifact naming | -| `arch` | `arm64` | Target architecture: `arm64` or `amd64` | +| `kernel_version` | — | Version string for release asset attachment | +| `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | **Output — two locations**: | Location | How to access | Retention | |----------|---------------|-----------| -| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* section at the bottom | 90 days | -| **GitHub Release asset** | Releases page → tag `noble-6.8.0-X.Y` → Assets | Permanent | - -The `.deb` files are attached to the release tag automatically when `kernel_version` is provided (which `fetch-source-pkg.yml` always does). - -**Resource requirements**: - -| Resource | Requirement | -|----------|-------------| -| Disk space | ~20 GB (runner is cleaned before build) | -| Wall-clock | ~60–90 min (generic, 2 vCPU GitHub runner) | -| RAM | ~4 GB | +| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | +| **GitHub Release asset** | Releases → `noble-6.8.0-X.Y` → Assets | Permanent | -> **Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` (Ubuntu 24.04 arm64, native build — no cross-compilation). - ---- - -### `mirror-git.yml` — Full git history mirror *(optional)* - -Mirrors the complete Canonical Ubuntu kernel git tree from Launchpad to a -separate GitHub repository, preserving every intermediate commit made by -the Ubuntu kernel team between uploads. - -This is complementary to `fetch-source-pkg.yml`: the series branch gives -you one clean snapshot per upload; the git mirror gives you the full -development history. - -**Schedule**: daily at **03:00 UTC** -**Required secret**: `MIRROR_PUSH_TOKEN` (PAT with `repo` scope on the target repo) -**Required variable**: `MIRROR_TARGET_REPO` (e.g. `qualcomm-linux/linux-noble`) +**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` +Native arm64 build — no cross-compilation. --- @@ -139,37 +168,21 @@ Go to **Actions** and enable workflows if prompted. |----------|---------|-------------| | `KERNEL_SERIES` | `noble` | Default series for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | -| `MIRROR_TARGET_REPO` | `qualcomm-linux/linux-noble` | Target repo for git mirror | - -### 3. Configure secrets *(only needed for git mirror)* -**Settings → Secrets and variables → Actions → Secrets**: - -| Secret | Description | -|--------|-------------| -| `MIRROR_PUSH_TOKEN` | GitHub PAT with `repo` scope on `MIRROR_TARGET_REPO` | - -### 4. Run the first sync +### 3. Run the first sync ```bash # Sync noble sources to the noble branch (creates it if it doesn't exist) gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ --field series=noble - -# Or trigger a build manually from an existing series branch -gh workflow run build-kernel.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field series=noble \ - --field arch=arm64 \ - --field flavor=generic ``` --- ## Local usage -All scripts run on Ubuntu 22.04 / 24.04. +All scripts run on Ubuntu 24.04 arm64. ### Check the latest version @@ -206,6 +219,8 @@ The Ubuntu kernel source package is a standard Debian 3.0 (quilt) source package `dpkg-source -x` applies all patches and produces the full source tree that is committed to the series branch. +--- + ## Versioning scheme Ubuntu kernel versions follow `X.Y.Z-A.B`: @@ -216,7 +231,7 @@ Ubuntu kernel versions follow `X.Y.Z-A.B`: | `A` | `51` | ABI number | | `B` | `52` | Upload number | -Tags in this repository use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. +Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. --- From 783e63db38e4c6b539b0f010c9fc72b3afe4d288 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 14:55:03 -0700 Subject: [PATCH 004/115] ci: add Canonical Ubuntu kernel mirror and build pipeline Implements a complete CI/CD pipeline for mirroring and building the Canonical Ubuntu kernel source package from Launchpad, targeting the arm64 architecture on Ubuntu 24.04 LTS. Repository branch layout: main - CI infrastructure (workflows, scripts, documentation) - Orphan branch per Ubuntu suite containing the full extracted kernel source tree (one commit per upload) e.g. noble branch for Ubuntu Noble 24.04 LTS Workflows (.github/workflows/): fetch-source-pkg.yml - Queries the Launchpad REST API for the latest published linux source package for the given suite - Downloads .dsc, .orig.tar.gz, .debian.tar.xz from Launchpad - Extracts the full patched source tree via dpkg-source -x - Commits the complete kernel source to the suite branch - Tags each commit as - (e.g. noble-6.8.0-51.52) - Idempotent: skips if the tag already exists - Auto-dispatches build-kernel.yml on each new version - Schedule: daily 04:00 UTC - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem build-kernel.yml - Checks out the suite branch (full kernel source tree) - Installs build dependencies via apt-get build-dep - Native arm64 build: fakeroot debian/rules binary- - No cross-compilation required (runner is arm64) - Uploads built .deb packages as: * GitHub Actions artifact (90-day retention) * GitHub Release asset on the - tag (permanent) - Runner: self-hosted lecore-prd-u2404-arm64-xlrg-od-ephem Helper scripts (scripts/): check-version.sh - Query latest published version from Launchpad API fetch-source-pkg.sh - Download source package files from Launchpad build-kernel-deb.sh - Build kernel .deb packages locally (arm64 native) Active suite: noble (Ubuntu 24.04 LTS, kernel 6.8) Future suites: questing, resolute (added on demand via workflow input) --- .github/workflows/build-kernel.yml | 30 ++++--- .github/workflows/fetch-source-pkg.yml | 103 ++++++++++++------------- README.md | 38 ++++----- scripts/check-version.sh | 10 +-- scripts/fetch-source-pkg.sh | 14 ++-- 5 files changed, 93 insertions(+), 102 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index f0c7865c64c12..217d94d20ad95 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -2,13 +2,13 @@ # # build-kernel.yml # -# Builds Ubuntu kernel .deb packages directly from a series branch +# Builds Ubuntu kernel .deb packages directly from a suite branch # (e.g. "noble") that was populated by fetch-source-pkg.yml. # # Output .deb packages are stored in two places: # 1. GitHub Actions artifact – attached to the workflow run, retained 90 # days, downloadable from Actions → → Artifacts. -# 2. GitHub Release asset – attached to the - tag/release +# 2. GitHub Release asset – attached to the - tag/release # (e.g. noble-6.8.0-51.52) so they are permanently accessible from the # Releases page of this repository. # @@ -21,13 +21,13 @@ name: "Build: Canonical Kernel .deb Packages" on: workflow_dispatch: inputs: - series: - description: "Series branch to build from (e.g. noble, questing, resolute)" + suite: + description: "Suite branch to build from (e.g. noble, questing, resolute)" required: true default: "noble" type: string kernel_version: - description: "Kernel version string (e.g. 6.8.0-51.52) – used for tagging artifacts" + description: "Kernel version string (e.g. 6.8.0-51.52) – used for release asset attachment" required: false type: string arch: @@ -56,23 +56,23 @@ jobs: # Job: build # -------------------------------------------------------------------------- build: - name: "Build ${{ inputs.series }} ${{ inputs.flavor }} (${{ inputs.arch }})" + name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] timeout-minutes: 360 env: - SERIES: ${{ inputs.series || 'noble' }} + SUITE: ${{ inputs.suite || 'noble' }} ARCH: ${{ inputs.arch || 'arm64' }} FLAVOR: ${{ inputs.flavor || 'generic' }} steps: # ----------------------------------------------------------------------- - # 1. Checkout the series branch (contains the full kernel source tree) + # 1. Checkout the suite branch (contains the full kernel source tree) # ----------------------------------------------------------------------- - - name: Checkout series branch (${{ inputs.series }}) + - name: Checkout suite branch (${{ inputs.suite }}) uses: actions/checkout@v4 with: - ref: ${{ inputs.series || 'noble' }} + ref: ${{ inputs.suite || 'noble' }} path: kernel-src # ----------------------------------------------------------------------- @@ -103,9 +103,8 @@ jobs: - name: Build kernel packages run: | JOBS=$(nproc) - echo "Building: series=${SERIES} flavour=${FLAVOR} arch=${ARCH} jobs=${JOBS}" + echo "Building: suite=${SUITE} flavour=${FLAVOR} arch=${ARCH} jobs=${JOBS}" - # Determine debian/rules target if [ "${FLAVOR}" = "all" ]; then TARGET="binary" else @@ -141,7 +140,7 @@ jobs: - name: Upload packages as artifact uses: actions/upload-artifact@v4 with: - name: "kernel-debs-${{ inputs.series }}-${{ inputs.arch }}-${{ inputs.flavor }}" + name: "kernel-debs-${{ inputs.suite }}-${{ inputs.arch }}-${{ inputs.flavor }}" path: output/ retention-days: 90 if-no-files-found: warn @@ -154,7 +153,7 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | - TAG="${SERIES}-${{ inputs.kernel_version }}" + TAG="${SUITE}-${{ inputs.kernel_version }}" DEBS=$(find output/ -name "*.deb" | sort) if [ -z "${DEBS}" ]; then @@ -162,7 +161,6 @@ jobs: exit 0 fi - # Check if the tag/release exists before trying to upload if gh release view "${TAG}" \ --repo "${{ github.repository }}" \ --json tagName -q .tagName \ @@ -186,7 +184,7 @@ jobs: echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 70dff4500e258..793b9a994f5e1 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -4,22 +4,22 @@ # # Fetches the latest Canonical Ubuntu kernel source package from Launchpad, # extracts it with dpkg-source, and commits the full source tree to a branch -# named after the Ubuntu series (e.g. "noble", "questing"). +# named after the Ubuntu suite (e.g. "noble", "questing"). # # Repository branch layout # ───────────────────────── # main – CI infrastructure: workflows, scripts, documentation # noble – Ubuntu Noble (24.04 LTS) kernel source, one commit/upload -# – additional series added on demand (questing, resolute, …) +# – additional suites added on demand (questing, resolute, …) # -# Each commit on a series branch is tagged - +# Each commit on a suite branch is tagged - # (e.g. noble-6.8.0-51.52). # # The workflow is idempotent: if the tag already exists it exits cleanly. # On success it automatically dispatches build-kernel.yml. # # Repository variables (Settings → Variables → Actions): -# KERNEL_SERIES – default series to sync when not specified (default: noble) +# KERNEL_SUITE – default suite to sync when not specified (default: noble) # KERNEL_SOURCE – source package name (default: linux) name: "Sync: Canonical Kernel Sources to Branch" @@ -31,8 +31,8 @@ on: workflow_dispatch: inputs: - series: - description: "Ubuntu series to sync (e.g. noble, questing, resolute)" + suite: + description: "Ubuntu suite to sync (e.g. noble, questing, resolute)" required: false default: "noble" type: string @@ -42,7 +42,7 @@ on: default: false permissions: - contents: write # push to series branch + create tags + contents: write # push to suite branch + create tags jobs: # -------------------------------------------------------------------------- @@ -58,7 +58,7 @@ jobs: version: ${{ steps.query.outputs.version }} upstream_version: ${{ steps.query.outputs.upstream_version }} tag: ${{ steps.query.outputs.tag }} - series: ${{ steps.query.outputs.series }} + suite: ${{ steps.query.outputs.suite }} source: ${{ steps.query.outputs.source }} should_sync: ${{ steps.gate.outputs.should_sync }} @@ -72,13 +72,13 @@ jobs: - name: Query Launchpad for latest published source id: query env: - SERIES: ${{ inputs.series || vars.KERNEL_SERIES || 'noble' }} + SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'noble' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} run: | API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" API+="?ws.op=getPublishedSources" API+="&source_name=${SOURCE}" - API+="&distro_series=/ubuntu/${SERIES}" + API+="&distro_series=/ubuntu/${SUITE}" API+="&status=Published" API+="&order_by_date=true" @@ -87,20 +87,20 @@ jobs: VERSION=$(echo "$RESP" | jq -r '.entries[0].source_package_version // empty') [ -n "${VERSION}" ] || { - echo "ERROR: No version found for ${SOURCE} in ${SERIES}" >&2 + echo "ERROR: No version found for ${SOURCE} in ${SUITE}" >&2 exit 1 } UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) - TAG="${SERIES}-${VERSION}" + TAG="${SUITE}-${VERSION}" echo "version=${VERSION}" >> "$GITHUB_OUTPUT" echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" echo "tag=${TAG}" >> "$GITHUB_OUTPUT" - echo "series=${SERIES}" >> "$GITHUB_OUTPUT" + echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" - echo "Series : ${SERIES}" + echo "Suite : ${SUITE}" echo "Version : ${VERSION}" echo "Tag : ${TAG}" @@ -130,11 +130,11 @@ jobs: # -------------------------------------------------------------------------- # Job 2: sync - # Downloads the source package, extracts it, and commits to the series + # Downloads the source package, extracts it, and commits to the suite # branch. # -------------------------------------------------------------------------- sync: - name: "Sync ${{ needs.check-version.outputs.series }} sources to branch" + name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] timeout-minutes: 120 needs: check-version @@ -144,7 +144,7 @@ jobs: VERSION: ${{ needs.check-version.outputs.version }} UPSTREAM_VERSION: ${{ needs.check-version.outputs.upstream_version }} TAG: ${{ needs.check-version.outputs.tag }} - SERIES: ${{ needs.check-version.outputs.series }} + SUITE: ${{ needs.check-version.outputs.suite }} SOURCE: ${{ needs.check-version.outputs.source }} steps: @@ -172,7 +172,7 @@ jobs: - name: Fetch source package from Launchpad run: | chmod +x ci/scripts/fetch-source-pkg.sh - ci/scripts/fetch-source-pkg.sh "${SERIES}" "${SOURCE}" source-pkg/ + ci/scripts/fetch-source-pkg.sh "${SUITE}" "${SOURCE}" source-pkg/ echo "Downloaded files:" ls -lh source-pkg/ @@ -186,7 +186,6 @@ jobs: echo "Extracting: ${DSC}" dpkg-source -x "${DSC}" kernel-src/ - # The extracted directory is named linux- SRC_DIR=$(find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1) echo "Extracted to: ${SRC_DIR}" echo "SRC_DIR=${SRC_DIR}" >> "$GITHUB_ENV" @@ -203,70 +202,65 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 6. Prepare the series branch - # Clone the repo, check out (or create) the series branch, wipe it, - # populate with the freshly extracted source, commit, tag, push. + # 6. Prepare the suite branch # ----------------------------------------------------------------------- - - name: Clone repository for series branch update + - name: Clone repository for suite branch update env: GH_TOKEN: ${{ github.token }} run: | REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" - git clone "${REPO_URL}" series-repo + git clone "${REPO_URL}" suite-repo - - name: Check out or create series branch + - name: Check out or create suite branch run: | - cd series-repo + cd suite-repo - if git ls-remote --exit-code origin "refs/heads/${SERIES}" >/dev/null 2>&1; then - echo "Branch '${SERIES}' exists – checking out" - git checkout "${SERIES}" - # Remove all existing content so we get a clean slate + if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then + echo "Branch '${SUITE}' exists – checking out" + git checkout "${SUITE}" git rm -rf --quiet . 2>/dev/null || true else - echo "Branch '${SERIES}' does not exist – creating orphan branch" - git checkout --orphan "${SERIES}" + echo "Branch '${SUITE}' does not exist – creating orphan branch" + git checkout --orphan "${SUITE}" git rm -rf --quiet . 2>/dev/null || true fi - - name: Populate series branch with extracted source + - name: Populate suite branch with extracted source run: | - echo "Copying source tree from ${SRC_DIR} → series-repo/" - # rsync preserves permissions and handles large trees efficiently + echo "Copying source tree from ${SRC_DIR} → suite-repo/" rsync -a --delete \ --exclude='.git' \ - "${SRC_DIR}/" series-repo/ + "${SRC_DIR}/" suite-repo/ - cd series-repo + cd suite-repo - # Write commit message to a file to avoid shell/YAML quoting issues SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ - "${SERIES}: ${VERSION}" \ - "Ubuntu series : ${SERIES}" \ + "${SUITE}: ${VERSION}" \ + "Ubuntu suite : ${SUITE}" \ "Kernel version : ${VERSION}" \ "Upstream : ${UPSTREAM_VERSION}" \ - "Source : https://launchpad.net/ubuntu/${SERIES}/+source/${SOURCE}/${VERSION}" \ + "Source : https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION}" \ "Synced : ${SYNC_DATE}" \ > /tmp/commit-msg.txt git add --all git commit -F /tmp/commit-msg.txt - echo "Committed ${SERIES} branch at $(git rev-parse HEAD)" + echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" - name: Tag the commit run: | - cd series-repo - git tag -a "${TAG}" -m "Ubuntu ${SERIES} kernel ${VERSION}" + cd suite-repo + git tag -a "${TAG}" -m "Ubuntu ${SUITE} kernel ${VERSION}" echo "Tagged: ${TAG}" - - name: Push series branch and tag + - name: Push suite branch and tag run: | - cd series-repo - git push origin "${SERIES}" + cd suite-repo + git push origin "${SUITE}" git push origin "${TAG}" - echo "Pushed branch '${SERIES}' and tag '${TAG}'" + echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- # 7. Summary @@ -278,15 +272,14 @@ jobs: echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Series | \`${SERIES}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | [\`${SERIES}\`](https://github.com/${{ github.repository }}/tree/${SERIES}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Launchpad | https://launchpad.net/ubuntu/${SERIES}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" + echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" # -------------------------------------------------------------------------- # Job 3: trigger-build - # Dispatches build-kernel.yml for the newly synced version. # -------------------------------------------------------------------------- trigger-build: name: "Trigger kernel build" @@ -298,14 +291,14 @@ jobs: - name: Dispatch build-kernel workflow env: GH_TOKEN: ${{ github.token }} - SERIES: ${{ needs.check-version.outputs.series }} + SUITE: ${{ needs.check-version.outputs.suite }} VERSION: ${{ needs.check-version.outputs.version }} run: | gh workflow run build-kernel.yml \ --repo "${{ github.repository }}" \ - --field series="${SERIES}" \ + --field suite="${SUITE}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ --field flavor="generic" - echo "Build workflow dispatched for ${SERIES} ${VERSION}" + echo "Build workflow dispatched for ${SUITE} ${VERSION}" diff --git a/README.md b/README.md index 50923e2367c99..bf15fad30ea82 100644 --- a/README.md +++ b/README.md @@ -66,8 +66,8 @@ pkg-linux-qcom-canonical │ ├── main branch │ ├── .github/workflows/ -│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → series branch -│ │ └── build-kernel.yml ← build .deb packages from series branch +│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → suite branch +│ │ └── build-kernel.yml ← build .deb packages from suite branch │ ├── scripts/ │ │ ├── check-version.sh ← query latest version from Launchpad │ │ ├── fetch-source-pkg.sh ← download source package files @@ -79,12 +79,12 @@ pkg-linux-qcom-canonical │ One commit per Canonical upload │ Tagged noble-6.8.0-51.52, noble-6.8.0-52.53, … │ -└── branch (orphan, added on demand) - └── Full kernel source for that series +└── branch (orphan, added on demand) + └── Full kernel source for that suite e.g. questing, resolute ``` -Series branches are **orphan branches** — they share no history with `main` +Suite branches are **orphan branches** — they share no history with `main` and contain only the extracted kernel source tree. --- @@ -107,7 +107,7 @@ and contain only the extracted kernel source tree. Queries the Launchpad REST API for the latest published `linux` source package, downloads the `.dsc` + tarballs, extracts the full source tree with `dpkg-source -x` (applying all Ubuntu patches), and commits the -result to the corresponding series branch. +result to the corresponding suite branch. **Schedule**: daily at **04:00 UTC** **Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` @@ -116,7 +116,7 @@ result to the corresponding series branch. | Input | Default | Description | |-------|---------|-------------| -| `series` | `noble` | Ubuntu series to sync — one series per run | +| `suite` | `noble` | Ubuntu suite to sync — one suite per run | | `force` | `false` | Re-sync even if tag already exists | **Idempotent**: checks for the tag before downloading anything. @@ -126,7 +126,7 @@ result to the corresponding series branch. ### `build-kernel.yml` — Build .deb packages -Checks out the series branch (full kernel source tree) and builds `.deb` +Checks out the suite branch (full kernel source tree) and builds `.deb` packages using the Ubuntu `debian/rules` build system on the native arm64 self-hosted runner. @@ -137,7 +137,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `series` | `noble` | Series branch to build from | +| `suite` | `noble` | Suite branch to build from | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | @@ -166,7 +166,7 @@ Go to **Actions** and enable workflows if prompted. | Variable | Default | Description | |----------|---------|-------------| -| `KERNEL_SERIES` | `noble` | Default series for scheduled runs | +| `KERNEL_SUITE` | `noble` | Default suite for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | ### 3. Run the first sync @@ -175,7 +175,7 @@ Go to **Actions** and enable workflows if prompted. # Sync noble sources to the noble branch (creates it if it doesn't exist) gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field series=noble + --field suite=noble ``` --- @@ -217,7 +217,7 @@ The Ubuntu kernel source package is a standard Debian 3.0 (quilt) source package | `linux_X.Y.Z-A.B.debian.tar.xz` | ~5 MB | Ubuntu packaging overlay + patches | `dpkg-source -x` applies all patches and produces the full source tree -that is committed to the series branch. +that is committed to the suite branch. --- @@ -231,25 +231,25 @@ Ubuntu kernel versions follow `X.Y.Z-A.B`: | `A` | `51` | ABI number | | `B` | `52` | Upload number | -Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. +Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. --- -## Supported series +## Supported suites -| Series | Codename | Status | Kernel | -|--------|----------|--------|--------| +| Suite | Codename | Status | Kernel | +|-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | | `questing` | Questing Quokka | 25.04 — add when available | TBD | | `resolute` | Resolute Ringtail | 25.10 — add when available | TBD | -To add a new series, trigger `fetch-source-pkg.yml` with the desired -`series` input — the branch and release tag are created automatically: +To add a new suite, trigger `fetch-source-pkg.yml` with the desired +`suite` input — the branch and release tag are created automatically: ```bash gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field series=questing + --field suite=questing ``` --- diff --git a/scripts/check-version.sh b/scripts/check-version.sh index fbbb386d54fde..6d80f7ff827e0 100755 --- a/scripts/check-version.sh +++ b/scripts/check-version.sh @@ -4,10 +4,10 @@ # check-version.sh - Query the latest published kernel version from Launchpad # # Usage: -# check-version.sh [SERIES] [SOURCE_NAME] +# check-version.sh [SUITE] [SOURCE_NAME] # # Arguments: -# SERIES Ubuntu series (default: noble) +# SUITE Ubuntu suite (default: noble) # SOURCE_NAME Source package name (default: linux) # # Output: @@ -19,7 +19,7 @@ set -euo pipefail -SERIES="${1:-noble}" +SUITE="${1:-noble}" SOURCE_NAME="${2:-linux}" LAUNCHPAD_API="https://api.launchpad.net/1.0" @@ -29,13 +29,13 @@ die() { echo "ERROR: $*" >&2; exit 1; } RESPONSE=$(curl -fsSL \ "${LAUNCHPAD_API}/ubuntu/+archive/primary?ws.op=getPublishedSources\ &source_name=${SOURCE_NAME}\ -&distro_series=/ubuntu/${SERIES}\ +&distro_series=/ubuntu/${SUITE}\ &status=Published\ &order_by_date=true") \ || die "Failed to query Launchpad API" VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') -[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SERIES}'" +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SUITE}'" echo "${VERSION}" diff --git a/scripts/fetch-source-pkg.sh b/scripts/fetch-source-pkg.sh index 2f745a3557065..7151346e08a21 100755 --- a/scripts/fetch-source-pkg.sh +++ b/scripts/fetch-source-pkg.sh @@ -5,10 +5,10 @@ # from Launchpad # # Usage: -# fetch-source-pkg.sh [SERIES] [SOURCE_NAME] [OUTPUT_DIR] +# fetch-source-pkg.sh [SUITE] [SOURCE_NAME] [OUTPUT_DIR] # # Arguments: -# SERIES Ubuntu series (default: noble) +# SUITE Ubuntu suite (default: noble) # SOURCE_NAME Source package name (default: linux) # OUTPUT_DIR Directory to write files into (default: .) # @@ -21,7 +21,7 @@ set -euo pipefail -SERIES="${1:-noble}" +SUITE="${1:-noble}" SOURCE_NAME="${2:-linux}" OUTPUT_DIR="${3:-.}" @@ -38,12 +38,12 @@ hr() { log "$(printf '%0.s─' {1..60})"; } # 1. Query Launchpad for the latest published source # --------------------------------------------------------------------------- hr -log "Querying Launchpad for latest '${SOURCE_NAME}' in Ubuntu ${SERIES}..." +log "Querying Launchpad for latest '${SOURCE_NAME}' in Ubuntu ${SUITE}..." API_URL="${LAUNCHPAD_API}/ubuntu/+archive/primary" API_URL+="?ws.op=getPublishedSources" API_URL+="&source_name=${SOURCE_NAME}" -API_URL+="&distro_series=/ubuntu/${SERIES}" +API_URL+="&distro_series=/ubuntu/${SUITE}" API_URL+="&status=Published" API_URL+="&order_by_date=true" @@ -53,7 +53,7 @@ RESPONSE=$(curl -fsSL "${API_URL}") \ VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') SELF_LINK=$(echo "$RESPONSE" | jq -r '.entries[0].self_link // empty') -[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SERIES}'" +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SUITE}'" [ -n "$SELF_LINK" ] || die "Could not retrieve self_link for '${SOURCE_NAME}' ${VERSION}" # Upstream version: strip Ubuntu revision suffix (e.g. "6.8.0-51.52" → "6.8.0") @@ -103,7 +103,7 @@ hr VERSION_ENV="${OUTPUT_DIR}/version.env" cat > "${VERSION_ENV}" < Date: Mon, 20 Apr 2026 15:06:26 -0700 Subject: [PATCH 005/115] ci: use docker-pkg-build container for kernel build build-kernel.yml: - Checkout qualcomm-linux/docker-pkg-build@main - Build suite-matched docker image via docker_deb_build.py --rebuild - Run fakeroot debian/rules inside ghcr.io/qualcomm-linux/pkg-builder: container (bind-mounted workspace) instead of bare host - Add native build mode as fallback (--build_mode native) - Upload .deb packages to S3 qli-prd-lecore-gh-artifacts in addition to GitHub Actions artifact fetch-source-pkg.yml: - Pass build_mode=docker when dispatching build-kernel.yml --- .github/workflows/build-kernel.yml | 149 +++++++++++++++++-------- .github/workflows/fetch-source-pkg.yml | 3 +- 2 files changed, 103 insertions(+), 49 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 217d94d20ad95..4cb824f83ec14 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -2,19 +2,19 @@ # # build-kernel.yml # -# Builds Ubuntu kernel .deb packages directly from a suite branch +# Builds Ubuntu kernel .deb packages from a suite branch # (e.g. "noble") that was populated by fetch-source-pkg.yml. # -# Output .deb packages are stored in two places: -# 1. GitHub Actions artifact – attached to the workflow run, retained 90 -# days, downloadable from Actions → → Artifacts. -# 2. GitHub Release asset – attached to the - tag/release -# (e.g. noble-6.8.0-51.52) so they are permanently accessible from the -# Releases page of this repository. +# Build modes: +# docker (default) – runs the build inside the suite-matched +# ghcr.io/qualcomm-linux/pkg-builder: container +# via docker-pkg-build; ensures a clean, reproducible +# environment that exactly matches the target distro. +# native – runs dpkg-buildpackage directly on the runner host. # -# Triggers: -# • Automatic – dispatched by fetch-source-pkg.yml on each new sync -# • Manual – workflow_dispatch with configurable inputs +# Output .deb packages are stored in two places: +# 1. S3 bucket – qli-prd-lecore-gh-artifacts (permanent) +# 2. GitHub artifact – attached to the workflow run, retained 90 days name: "Build: Canonical Kernel .deb Packages" @@ -47,6 +47,14 @@ on: - generic - lowlatency - all + build_mode: + description: "Build mode: docker (suite-matched container) or native (host)" + required: false + default: "docker" + type: choice + options: + - docker + - native permissions: contents: write # upload assets to release tag @@ -56,78 +64,111 @@ jobs: # Job: build # -------------------------------------------------------------------------- build: - name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" + name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }}) [${{ inputs.build_mode || 'docker' }}]" runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'noble' }} - ARCH: ${{ inputs.arch || 'arm64' }} - FLAVOR: ${{ inputs.flavor || 'generic' }} + SUITE: ${{ inputs.suite || 'noble' }} + ARCH: ${{ inputs.arch || 'arm64' }} + FLAVOR: ${{ inputs.flavor || 'generic' }} + BUILD_MODE: ${{ inputs.build_mode || 'docker' }} steps: # ----------------------------------------------------------------------- # 1. Checkout the suite branch (contains the full kernel source tree) # ----------------------------------------------------------------------- - - name: Checkout suite branch (${{ inputs.suite }}) + - name: Checkout suite branch (${{ inputs.suite || 'noble' }}) uses: actions/checkout@v4 with: ref: ${{ inputs.suite || 'noble' }} path: kernel-src # ----------------------------------------------------------------------- - # 2. Install base build tools + # 2. Extract org/repo for S3 upload path # ----------------------------------------------------------------------- - - name: Install base build tools + - name: Extract org and repo name run: | - sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends \ - build-essential \ - fakeroot \ - devscripts \ - dpkg-dev \ - debhelper \ - dh-exec \ - rsync + echo "ORG_NAME=${GITHUB_REPOSITORY%%/*}" >> "$GITHUB_ENV" + echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 3. Install kernel-specific build dependencies + # 3. Checkout docker-pkg-build (docker mode only) # ----------------------------------------------------------------------- - - name: Install kernel build dependencies + - name: Checkout docker-pkg-build + if: env.BUILD_MODE == 'docker' + uses: actions/checkout@v4 + with: + repository: qualcomm-linux/docker-pkg-build + ref: main + path: docker-pkg-build + + # ----------------------------------------------------------------------- + # 4. Build the suite-matched pkg-builder docker image (docker mode only) + # ----------------------------------------------------------------------- + - name: Build docker image for suite (${{ inputs.suite || 'noble' }}) + if: env.BUILD_MODE == 'docker' run: | - sudo apt-get build-dep -y kernel-src/ + ./docker-pkg-build/docker_deb_build.py --rebuild -d "${SUITE}" + docker image ls # ----------------------------------------------------------------------- - # 4. Build (native arm64 – no cross-compilation needed) + # 5a. Build kernel packages inside the suite-matched container (docker) + # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - - name: Build kernel packages + - name: Build kernel packages (docker – ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) + if: env.BUILD_MODE == 'docker' run: | JOBS=$(nproc) - echo "Building: suite=${SUITE} flavour=${FLAVOR} arch=${ARCH} jobs=${JOBS}" + if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi - if [ "${FLAVOR}" = "all" ]; then - TARGET="binary" - else - TARGET="binary-${FLAVOR}" - fi + echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" + docker run -i --privileged --rm \ + -v "${GITHUB_WORKSPACE}:${GITHUB_WORKSPACE}" \ + --workdir="${GITHUB_WORKSPACE}" \ + "ghcr.io/qualcomm-linux/pkg-builder:${SUITE}" \ + bash -c " + set -e + sudo apt-get update -qq + sudo apt-get build-dep -y kernel-src/ + cd kernel-src/ + export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' + fakeroot debian/rules ${TARGET} + " + + # ----------------------------------------------------------------------- + # 5b. Build kernel packages natively on the runner (native mode) + # ----------------------------------------------------------------------- + - name: Install build tools (native mode) + if: env.BUILD_MODE == 'native' + run: | + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends \ + build-essential fakeroot devscripts dpkg-dev debhelper dh-exec rsync + sudo apt-get build-dep -y kernel-src/ + + - name: Build kernel packages (native) + if: env.BUILD_MODE == 'native' + run: | + JOBS=$(nproc) + if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi + + echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" - ( - cd kernel-src/ - fakeroot debian/rules "${TARGET}" - ) + (cd kernel-src/ && fakeroot debian/rules "${TARGET}") # ----------------------------------------------------------------------- - # 5. Collect output packages + # 6. Collect output packages # The Ubuntu kernel build drops .deb files one level above the - # source tree (i.e. in the working directory, not inside kernel-src/). + # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). # ----------------------------------------------------------------------- - name: Collect built packages run: | mkdir -p output/ - find . -maxdepth 1 \ + find "${GITHUB_WORKSPACE}" -maxdepth 1 \ \( -name "*.deb" -o -name "*.changes" -o -name "*.buildinfo" \) \ -exec cp {} output/ \; @@ -135,7 +176,17 @@ jobs: ls -lh output/ || echo "(no files found)" # ----------------------------------------------------------------------- - # 6. Upload as workflow artifact (always available for 90 days) + # 7. Upload to S3 (permanent storage) + # ----------------------------------------------------------------------- + - name: Upload kernel .deb packages to S3 + uses: qualcomm-linux/upload-private-artifact-action@aws + with: + s3_bucket: qli-prd-lecore-gh-artifacts + path: ${{ github.workspace }}/output + destination: ${{ env.ORG_NAME }}/pkg/temp/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ + + # ----------------------------------------------------------------------- + # 8. Upload as GitHub Actions artifact (90-day fallback) # ----------------------------------------------------------------------- - name: Upload packages as artifact uses: actions/upload-artifact@v4 @@ -146,7 +197,7 @@ jobs: if-no-files-found: warn # ----------------------------------------------------------------------- - # 7. Attach packages to the release tag (if kernel_version was provided) + # 9. Attach packages to the release tag (if kernel_version was provided) # ----------------------------------------------------------------------- - name: Attach packages to release tag if: inputs.kernel_version != '' @@ -171,11 +222,11 @@ jobs: --clobber echo "Packages attached to ${TAG}" else - echo "Release tag '${TAG}' not found – packages available as artifact only" + echo "Release tag '${TAG}' not found – packages available via S3 and artifact" fi # ----------------------------------------------------------------------- - # 8. Summary + # 10. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() @@ -187,6 +238,8 @@ jobs: echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Build mode | \`${BUILD_MODE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "### Built packages" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 793b9a994f5e1..17db7adcd8b96 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -299,6 +299,7 @@ jobs: --field suite="${SUITE}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ - --field flavor="generic" + --field flavor="generic" \ + --field build_mode="docker" echo "Build workflow dispatched for ${SUITE} ${VERSION}" From 0e148ca7d9926f20d48d5aa9d03f5047f5a4004e Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:13:06 -0700 Subject: [PATCH 006/115] docs: update README with docker-pkg-build pipeline and current workflow details --- README.md | 68 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index bf15fad30ea82..c9731a315e2e2 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ │ │ │ ┌─────────▼──────────┐ │ │ │ Job 2: sync │ │ +│ │ fetch-source-pkg.sh│ │ │ │ download .dsc │ │ │ │ + .orig.tar.gz │ │ │ │ + .debian.tar.xz │ │ @@ -36,27 +37,35 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ │ Job 3: trigger │ │ │ │ gh workflow run │ │ │ │ build-kernel.yml │ │ +│ │ suite=noble │ │ +│ │ build_mode=docker │ │ │ └─────────┬──────────┘ │ │ │ │ │ ┌─────────▼──────────┐ │ │ │ build-kernel.yml │ │ │ │ checkout noble │ │ │ │ branch │ │ -│ │ apt-get build-dep │ │ -│ │ fakeroot │ │ +│ │ checkout │ │ +│ │ docker-pkg-build │ │ +│ │ docker_deb_build.py│ │ +│ │ --rebuild -d noble │ │ +│ │ docker run │ │ +│ │ pkg-builder:noble │ │ │ │ debian/rules │ │ │ │ binary-generic │ │ │ └─────────┬──────────┘ │ │ │ │ -│ ┌──────────────┴──────────────┐ │ -│ ▼ ▼ │ -│ GitHub Actions Artifact GitHub Release Asset │ -│ (90-day retention) noble-6.8.0-51.52 │ -│ Actions → run → Artifacts Releases page → Assets │ -│ (permanent) │ +│ ┌───────────────────┼───────────────────┐ │ +│ ▼ ▼ ▼ │ +│ S3 Bucket GitHub Artifact GitHub Release │ +│ qli-prd-lecore- 90-day retention noble-6.8.0-51.52 │ +│ gh-artifacts Actions → run Releases → Assets │ +│ (permanent) → Artifacts (permanent) │ └─────────────────────────────────────────────────────────────────────┘ ``` +All jobs run on: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted, Ubuntu 24.04 arm64) + --- ## Repository branch layout @@ -110,7 +119,8 @@ with `dpkg-source -x` (applying all Ubuntu patches), and commits the result to the corresponding suite branch. **Schedule**: daily at **04:00 UTC** -**Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` +**Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` +**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` (all three jobs) **Inputs**: @@ -119,19 +129,27 @@ result to the corresponding suite branch. | `suite` | `noble` | Ubuntu suite to sync — one suite per run | | `force` | `false` | Re-sync even if tag already exists | -**Idempotent**: checks for the tag before downloading anything. -**Auto-triggers**: dispatches `build-kernel.yml` on each new sync. +**Jobs**: + +| Job | What it does | +|-----|-------------| +| `check-version` | Queries Launchpad API; checks if tag already exists; sets `should_sync` flag | +| `sync` | Downloads source package via `fetch-source-pkg.sh`; extracts with `dpkg-source -x`; commits to suite branch; creates tag | +| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `build_mode=docker` | + +**Idempotent**: if tag `noble-6.8.0-51.52` already exists, the workflow exits cleanly without downloading anything. --- ### `build-kernel.yml` — Build .deb packages Checks out the suite branch (full kernel source tree) and builds `.deb` -packages using the Ubuntu `debian/rules` build system on the native arm64 -self-hosted runner. +packages inside the suite-matched `ghcr.io/qualcomm-linux/pkg-builder:` +container using `fakeroot debian/rules binary-`. **Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or -manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. +manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. +**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` **Inputs**: @@ -141,17 +159,28 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | - -**Output — two locations**: +| `build_mode` | `docker` | `docker` (suite-matched container) or `native` (host) | + +**Build steps (docker mode)**: +1. Checkout suite branch → `kernel-src/` +2. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` +3. Build docker image: `docker_deb_build.py --rebuild -d ` +4. Run build inside container: + ``` + docker run ghcr.io/qualcomm-linux/pkg-builder: + → apt-get build-dep kernel-src/ + → fakeroot debian/rules binary- + ``` +5. Collect `.deb` files from workspace root + +**Output — three locations**: | Location | How to access | Retention | |----------|---------------|-----------| +| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | | **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | | **GitHub Release asset** | Releases → `noble-6.8.0-X.Y` → Assets | Permanent | -**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` -Native arm64 build — no cross-compilation. - --- ## Setup @@ -172,7 +201,6 @@ Go to **Actions** and enable workflows if prompted. ### 3. Run the first sync ```bash -# Sync noble sources to the noble branch (creates it if it doesn't exist) gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ --field suite=noble From 9a9e99e46a0ba8b3d66f21e6ca3004aa7ef35a6d Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:17:40 -0700 Subject: [PATCH 007/115] ci: switch to ubuntu-24.04-arm GitHub-hosted runner (temporary) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use GitHub-hosted arm64 runner while self-hosted runner group access is being configured for this repository. Changes: - fetch-source-pkg.yml: all three jobs → ubuntu-24.04-arm - build-kernel.yml: job → ubuntu-24.04-arm - build-kernel.yml: add disk cleanup step (GitHub runners have ~14 GB free; kernel build needs ~20 GB) - build-kernel.yml: S3 upload step marked continue-on-error (S3 credentials not available on GitHub-hosted runners) --- .github/workflows/build-kernel.yml | 47 +++++++++++++++++++------- .github/workflows/fetch-source-pkg.yml | 6 ++-- 2 files changed, 38 insertions(+), 15 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 4cb824f83ec14..09264aacbd975 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -65,7 +65,7 @@ jobs: # -------------------------------------------------------------------------- build: name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }}) [${{ inputs.build_mode || 'docker' }}]" - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + runs-on: ubuntu-24.04-arm timeout-minutes: 360 env: @@ -76,7 +76,29 @@ jobs: steps: # ----------------------------------------------------------------------- - # 1. Checkout the suite branch (contains the full kernel source tree) + # 1. Free up disk space + # GitHub-hosted ubuntu-24.04-arm runners ship with ~14 GB free. + # The kernel build needs ~20 GB; removing unused toolchains gives + # enough headroom. + # ----------------------------------------------------------------------- + - name: Free up runner disk space + run: | + echo "Disk before cleanup:" + df -h / + sudo rm -rf \ + /usr/share/dotnet \ + /usr/local/lib/android \ + /opt/ghc \ + /opt/hostedtoolcache/CodeQL \ + /usr/local/share/boost \ + "$AGENT_TOOLSDIRECTORY" \ + 2>/dev/null || true + sudo apt-get clean + echo "Disk after cleanup:" + df -h / + + # ----------------------------------------------------------------------- + # 2. Checkout the suite branch (contains the full kernel source tree) # ----------------------------------------------------------------------- - name: Checkout suite branch (${{ inputs.suite || 'noble' }}) uses: actions/checkout@v4 @@ -85,7 +107,7 @@ jobs: path: kernel-src # ----------------------------------------------------------------------- - # 2. Extract org/repo for S3 upload path + # 3. Extract org/repo for S3 upload path # ----------------------------------------------------------------------- - name: Extract org and repo name run: | @@ -93,7 +115,7 @@ jobs: echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 3. Checkout docker-pkg-build (docker mode only) + # 4. Checkout docker-pkg-build (docker mode only) # ----------------------------------------------------------------------- - name: Checkout docker-pkg-build if: env.BUILD_MODE == 'docker' @@ -104,7 +126,7 @@ jobs: path: docker-pkg-build # ----------------------------------------------------------------------- - # 4. Build the suite-matched pkg-builder docker image (docker mode only) + # 5. Build the suite-matched pkg-builder docker image (docker mode only) # ----------------------------------------------------------------------- - name: Build docker image for suite (${{ inputs.suite || 'noble' }}) if: env.BUILD_MODE == 'docker' @@ -113,7 +135,7 @@ jobs: docker image ls # ----------------------------------------------------------------------- - # 5a. Build kernel packages inside the suite-matched container (docker) + # 6a. Build kernel packages inside the suite-matched container (docker) # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - name: Build kernel packages (docker – ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) @@ -138,7 +160,7 @@ jobs: " # ----------------------------------------------------------------------- - # 5b. Build kernel packages natively on the runner (native mode) + # 6b. Build kernel packages natively on the runner (native mode) # ----------------------------------------------------------------------- - name: Install build tools (native mode) if: env.BUILD_MODE == 'native' @@ -160,7 +182,7 @@ jobs: (cd kernel-src/ && fakeroot debian/rules "${TARGET}") # ----------------------------------------------------------------------- - # 6. Collect output packages + # 7. Collect output packages # The Ubuntu kernel build drops .deb files one level above the # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). # ----------------------------------------------------------------------- @@ -176,9 +198,10 @@ jobs: ls -lh output/ || echo "(no files found)" # ----------------------------------------------------------------------- - # 7. Upload to S3 (permanent storage) + # 8. Upload to S3 (permanent storage, self-hosted runner only) # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 + continue-on-error: true uses: qualcomm-linux/upload-private-artifact-action@aws with: s3_bucket: qli-prd-lecore-gh-artifacts @@ -186,7 +209,7 @@ jobs: destination: ${{ env.ORG_NAME }}/pkg/temp/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ # ----------------------------------------------------------------------- - # 8. Upload as GitHub Actions artifact (90-day fallback) + # 9. Upload as GitHub Actions artifact (90-day fallback) # ----------------------------------------------------------------------- - name: Upload packages as artifact uses: actions/upload-artifact@v4 @@ -197,7 +220,7 @@ jobs: if-no-files-found: warn # ----------------------------------------------------------------------- - # 9. Attach packages to the release tag (if kernel_version was provided) + # 10. Attach packages to the release tag (if kernel_version was provided) # ----------------------------------------------------------------------- - name: Attach packages to release tag if: inputs.kernel_version != '' @@ -226,7 +249,7 @@ jobs: fi # ----------------------------------------------------------------------- - # 10. Summary + # 11. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 17db7adcd8b96..9fcbf134db004 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -51,7 +51,7 @@ jobs: # -------------------------------------------------------------------------- check-version: name: "Check latest version on Launchpad" - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + runs-on: ubuntu-24.04-arm timeout-minutes: 10 outputs: @@ -135,7 +135,7 @@ jobs: # -------------------------------------------------------------------------- sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + runs-on: ubuntu-24.04-arm timeout-minutes: 120 needs: check-version if: needs.check-version.outputs.should_sync == 'true' @@ -283,7 +283,7 @@ jobs: # -------------------------------------------------------------------------- trigger-build: name: "Trigger kernel build" - runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + runs-on: ubuntu-24.04-arm needs: [check-version, sync] if: needs.sync.result == 'success' From 3b2e174ce1bdf2d45871e10a0e1e526af98d3275 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:29:02 -0700 Subject: [PATCH 008/115] ci: add quilt/rsync deps and extraction verification to sync job - Add quilt, rsync, patch to dependencies (required for 3.0 quilt format) - Verify dpkg-source -x produced >5000 files before committing - Verify rsync copied >5000 files to suite-repo before committing - Both checks fail the job with a clear error if extraction is incomplete --- .github/workflows/fetch-source-pkg.yml | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 9fcbf134db004..bfe8ac217a633 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -164,7 +164,7 @@ jobs: run: | sudo apt-get update -qq sudo apt-get install -y --no-install-recommends \ - curl jq devscripts dpkg-dev + curl jq devscripts dpkg-dev quilt rsync patch # ----------------------------------------------------------------------- # 3. Download source package from Launchpad @@ -187,9 +187,21 @@ jobs: dpkg-source -x "${DSC}" kernel-src/ SRC_DIR=$(find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1) + [ -n "${SRC_DIR}" ] || { echo "ERROR: dpkg-source -x produced no directory"; exit 1; } + echo "Extracted to: ${SRC_DIR}" echo "SRC_DIR=${SRC_DIR}" >> "$GITHUB_ENV" + # Verify the extraction produced a real kernel source tree + FILE_COUNT=$(find "${SRC_DIR}" -type f | wc -l) + echo "Extracted ${FILE_COUNT} files" + [ "${FILE_COUNT}" -gt 5000 ] || { + echo "ERROR: Extraction produced only ${FILE_COUNT} files (expected >5000)" + echo "Top-level contents:" + ls -la "${SRC_DIR}/" + exit 1 + } + echo "Top-level contents:" ls "${SRC_DIR}/" @@ -232,6 +244,14 @@ jobs: --exclude='.git' \ "${SRC_DIR}/" suite-repo/ + # Verify rsync copied the files + COPIED=$(find suite-repo/ -type f | wc -l) + echo "Files in suite-repo after rsync: ${COPIED}" + [ "${COPIED}" -gt 5000 ] || { + echo "ERROR: rsync copied only ${COPIED} files" + exit 1 + } + cd suite-repo SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') From a4d61cfd0daece9f69c368357292ce3f9658e247 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:32:26 -0700 Subject: [PATCH 009/115] fix: use exact source_package_name match for Launchpad API queries The Launchpad getPublishedSources source_name= parameter is a prefix match, not an exact match. Querying for 'linux' was returning 'linux-meta-raspi' (most recently published) instead of 'linux'. Fix: filter results with jq select(.source_package_name == $name) to ensure exact package name matching in: - scripts/check-version.sh - scripts/fetch-source-pkg.sh - .github/workflows/fetch-source-pkg.yml (check-version job) --- .github/workflows/fetch-source-pkg.yml | 7 +++++-- scripts/check-version.sh | 7 +++++-- scripts/fetch-source-pkg.sh | 13 +++++++++---- 3 files changed, 19 insertions(+), 8 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index bfe8ac217a633..fb2e37bf98cd1 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -85,9 +85,12 @@ jobs: echo "Querying: ${API}" RESP=$(curl -fsSL "${API}") - VERSION=$(echo "$RESP" | jq -r '.entries[0].source_package_version // empty') + # Filter by exact source_package_name (source_name= is a prefix match) + VERSION=$(echo "$RESP" | jq -r \ + --arg name "${SOURCE}" \ + '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') [ -n "${VERSION}" ] || { - echo "ERROR: No version found for ${SOURCE} in ${SUITE}" >&2 + echo "ERROR: No exact match for '${SOURCE}' in ${SUITE}" >&2 exit 1 } diff --git a/scripts/check-version.sh b/scripts/check-version.sh index 6d80f7ff827e0..b6713d34f6c18 100755 --- a/scripts/check-version.sh +++ b/scripts/check-version.sh @@ -34,8 +34,11 @@ RESPONSE=$(curl -fsSL \ &order_by_date=true") \ || die "Failed to query Launchpad API" -VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') +# Filter by exact source_package_name (source_name= is a prefix match on Launchpad) +VERSION=$(echo "$RESPONSE" | jq -r \ + --arg name "${SOURCE_NAME}" \ + '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') -[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SUITE}'" +[ -n "$VERSION" ] || die "No exact match for '${SOURCE_NAME}' in '${SUITE}'" echo "${VERSION}" diff --git a/scripts/fetch-source-pkg.sh b/scripts/fetch-source-pkg.sh index 7151346e08a21..072095bb249cf 100755 --- a/scripts/fetch-source-pkg.sh +++ b/scripts/fetch-source-pkg.sh @@ -50,10 +50,15 @@ API_URL+="&order_by_date=true" RESPONSE=$(curl -fsSL "${API_URL}") \ || die "Launchpad API request failed: ${API_URL}" -VERSION=$(echo "$RESPONSE" | jq -r '.entries[0].source_package_version // empty') -SELF_LINK=$(echo "$RESPONSE" | jq -r '.entries[0].self_link // empty') - -[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' in '${SUITE}'" +# Filter by exact source_package_name (source_name= is a prefix match on Launchpad) +VERSION=$(echo "$RESPONSE" | jq -r \ + --arg name "${SOURCE_NAME}" \ + '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') +SELF_LINK=$(echo "$RESPONSE" | jq -r \ + --arg name "${SOURCE_NAME}" \ + '[.entries[] | select(.source_package_name == $name)] | .[0].self_link // empty') + +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' (exact) in '${SUITE}'" [ -n "$SELF_LINK" ] || die "Could not retrieve self_link for '${SOURCE_NAME}' ${VERSION}" # Upstream version: strip Ubuntu revision suffix (e.g. "6.8.0-51.52" → "6.8.0") From 8454a04f35352953ecb3935d410230bb997ccab2 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:37:19 -0700 Subject: [PATCH 010/115] ci: improve sync job robustness on GitHub-hosted runners - Add disk cleanup step to sync job (frees ~10 GB before extraction) - Use dpkg-source --no-check to skip GPG verification and redirect verbose 'upstream files modified' output to log file - Add file count verification after extraction and rsync (>5000 files) --- .github/workflows/fetch-source-pkg.yml | 41 +++++++++++++++++++++----- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index fb2e37bf98cd1..4ceb1bca32ff6 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -152,7 +152,26 @@ jobs: steps: # ----------------------------------------------------------------------- - # 1. Checkout main branch to get the helper scripts + # 1. Free up disk space (GitHub-hosted runners have ~14 GB free) + # ----------------------------------------------------------------------- + - name: Free up runner disk space + run: | + echo "Disk before cleanup:" + df -h / + sudo rm -rf \ + /usr/share/dotnet \ + /usr/local/lib/android \ + /opt/ghc \ + /opt/hostedtoolcache/CodeQL \ + /usr/local/share/boost \ + "$AGENT_TOOLSDIRECTORY" \ + 2>/dev/null || true + sudo apt-get clean + echo "Disk after cleanup:" + df -h / + + # ----------------------------------------------------------------------- + # 2. Checkout main branch to get the helper scripts # ----------------------------------------------------------------------- - name: Checkout main (for scripts) uses: actions/checkout@v4 @@ -161,7 +180,7 @@ jobs: path: ci # ----------------------------------------------------------------------- - # 2. Install tools + # 3. Install tools # ----------------------------------------------------------------------- - name: Install dependencies run: | @@ -170,7 +189,7 @@ jobs: curl jq devscripts dpkg-dev quilt rsync patch # ----------------------------------------------------------------------- - # 3. Download source package from Launchpad + # 4. Download source package from Launchpad # ----------------------------------------------------------------------- - name: Fetch source package from Launchpad run: | @@ -181,13 +200,21 @@ jobs: ls -lh source-pkg/ # ----------------------------------------------------------------------- - # 4. Extract source package (applies all Ubuntu patches via quilt) + # 5. Extract source package # ----------------------------------------------------------------------- - name: Extract source package run: | DSC=$(find source-pkg/ -name "*.dsc" | sort -V | tail -1) echo "Extracting: ${DSC}" - dpkg-source -x "${DSC}" kernel-src/ + # --no-check skips GPG signature verification (key not in runner keyring) + # Redirect verbose "upstream files modified" list to log file + dpkg-source --no-check -x "${DSC}" kernel-src/ \ + > /tmp/dpkg-source.log 2>&1 || { + echo "dpkg-source -x failed. Last 30 lines of output:" + tail -30 /tmp/dpkg-source.log + exit 1 + } + echo "Extraction complete" SRC_DIR=$(find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1) [ -n "${SRC_DIR}" ] || { echo "ERROR: dpkg-source -x produced no directory"; exit 1; } @@ -209,7 +236,7 @@ jobs: ls "${SRC_DIR}/" # ----------------------------------------------------------------------- - # 5. Configure git + # 6. Configure git # ----------------------------------------------------------------------- - name: Configure git identity run: | @@ -217,7 +244,7 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 6. Prepare the suite branch + # 7. Prepare the suite branch # ----------------------------------------------------------------------- - name: Clone repository for suite branch update env: From 602506f7aba957c5066011a9bc8c34c78c5fa47a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:40:14 -0700 Subject: [PATCH 011/115] fix: set SRC_DIR correctly when dpkg-source extracts to explicit dir dpkg-source -x extracts the source directly INTO the specified directory, not into a subdirectory. The previous code used 'find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1' which found 'kernel-src/net' (first dir alphabetically) instead of the full tree root. Fix: set SRC_DIR=kernel-src directly since we pass kernel-src/ as the explicit output directory to dpkg-source. --- .github/workflows/fetch-source-pkg.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 4ceb1bca32ff6..5d33b90b84b95 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -216,11 +216,11 @@ jobs: } echo "Extraction complete" - SRC_DIR=$(find kernel-src/ -maxdepth 1 -mindepth 1 -type d | head -1) - [ -n "${SRC_DIR}" ] || { echo "ERROR: dpkg-source -x produced no directory"; exit 1; } - - echo "Extracted to: ${SRC_DIR}" + # When an explicit output directory is given, dpkg-source -x extracts + # the source directly INTO that directory (not into a subdirectory). + SRC_DIR="kernel-src" echo "SRC_DIR=${SRC_DIR}" >> "$GITHUB_ENV" + echo "Extracted to: ${SRC_DIR}" # Verify the extraction produced a real kernel source tree FILE_COUNT=$(find "${SRC_DIR}" -type f | wc -l) From 79d64ec82a7388983b41d84e73def997b6e1910c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:45:20 -0700 Subject: [PATCH 012/115] fix: set SRC_DIR correctly for dpkg-source explicit output dir dpkg-source -x extracts directly INTO the specified directory, not into a subdirectory. Previous code used 'find' which picked up 'kernel-src/net' (first dir alphabetically) instead of the full tree root. Fix: set SRC_DIR=kernel-src directly. Also update README to reflect current state: - Runner: ubuntu-24.04-arm (temporary, pending runner group access) - Source format: Debian 1.0 (.diff.gz not .debian.tar.xz) - S3 upload: self-hosted only, gracefully skipped on GitHub-hosted - Launchpad exact match note documented --- README.md | 79 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 44 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index c9731a315e2e2..805ace21de5b8 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,8 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ ┌─────────▼──────────┐ │ │ │ Job 1: check-version │ │ │ curl Launchpad API │ │ -│ │ → noble 6.8.0-51.52│ │ +│ │ exact match: linux │ │ +│ │ → noble 6.8.0-114 │ │ │ │ tag exists? YES→skip │ │ │ NO ↓ │ │ │ └─────────┬──────────┘ │ @@ -26,8 +27,9 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ │ fetch-source-pkg.sh│ │ │ │ download .dsc │ │ │ │ + .orig.tar.gz │ │ -│ │ + .debian.tar.xz │ │ -│ │ dpkg-source -x │ │ +│ │ + .diff.gz │ │ +│ │ dpkg-source │ │ +│ │ --no-check -x │ │ │ │ → full source tree │ │ │ │ commit to noble │ │ │ │ branch + tag │ │ @@ -58,13 +60,14 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ ┌───────────────────┼───────────────────┐ │ │ ▼ ▼ ▼ │ │ S3 Bucket GitHub Artifact GitHub Release │ -│ qli-prd-lecore- 90-day retention noble-6.8.0-51.52 │ +│ qli-prd-lecore- 90-day retention noble-6.8.0-114.114 │ │ gh-artifacts Actions → run Releases → Assets │ -│ (permanent) → Artifacts (permanent) │ +│ (self-hosted only) → Artifacts (permanent) │ └─────────────────────────────────────────────────────────────────────┘ ``` -All jobs run on: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted, Ubuntu 24.04 arm64) +All jobs run on: `ubuntu-24.04-arm` (GitHub-hosted, Ubuntu 24.04 arm64) +Target runner: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted) — pending runner group access --- @@ -86,7 +89,7 @@ pkg-linux-qcom-canonical ├── noble branch (orphan) │ └── Full Ubuntu Noble 24.04 LTS kernel source tree │ One commit per Canonical upload -│ Tagged noble-6.8.0-51.52, noble-6.8.0-52.53, … +│ Tagged noble-6.8.0-114.114, noble-6.8.0-115.115, … │ └── branch (orphan, added on demand) └── Full kernel source for that suite @@ -114,13 +117,14 @@ and contain only the extracted kernel source tree. ### `fetch-source-pkg.yml` — Sync sources to branch Queries the Launchpad REST API for the latest published `linux` source -package, downloads the `.dsc` + tarballs, extracts the full source tree -with `dpkg-source -x` (applying all Ubuntu patches), and commits the -result to the corresponding suite branch. +package (exact name match — the API does prefix matching), downloads the +source package files, extracts the full source tree with +`dpkg-source --no-check -x`, and commits the result to the corresponding +suite branch. **Schedule**: daily at **04:00 UTC** **Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` -**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` (all three jobs) +**Runner**: `ubuntu-24.04-arm` (all three jobs) **Inputs**: @@ -133,11 +137,11 @@ result to the corresponding suite branch. | Job | What it does | |-----|-------------| -| `check-version` | Queries Launchpad API; checks if tag already exists; sets `should_sync` flag | -| `sync` | Downloads source package via `fetch-source-pkg.sh`; extracts with `dpkg-source -x`; commits to suite branch; creates tag | +| `check-version` | Queries Launchpad API with exact `source_package_name` filter; checks if tag already exists; sets `should_sync` flag | +| `sync` | Frees disk space; downloads source package via `fetch-source-pkg.sh`; extracts with `dpkg-source --no-check -x`; verifies >5000 files; commits to suite branch; creates tag | | `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `build_mode=docker` | -**Idempotent**: if tag `noble-6.8.0-51.52` already exists, the workflow exits cleanly without downloading anything. +**Idempotent**: if tag `noble-6.8.0-114.114` already exists, the workflow exits cleanly without downloading anything. --- @@ -149,7 +153,7 @@ container using `fakeroot debian/rules binary-`. **Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. -**Runner**: `lecore-prd-u2404-arm64-xlrg-od-ephem` +**Runner**: `ubuntu-24.04-arm` **Inputs**: @@ -162,24 +166,25 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | `build_mode` | `docker` | `docker` (suite-matched container) or `native` (host) | **Build steps (docker mode)**: -1. Checkout suite branch → `kernel-src/` -2. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` -3. Build docker image: `docker_deb_build.py --rebuild -d ` -4. Run build inside container: +1. Free up disk space (~10 GB) +2. Checkout suite branch → `kernel-src/` +3. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` +4. Build docker image: `docker_deb_build.py --rebuild -d ` +5. Run build inside container: ``` docker run ghcr.io/qualcomm-linux/pkg-builder: → apt-get build-dep kernel-src/ → fakeroot debian/rules binary- ``` -5. Collect `.deb` files from workspace root +6. Collect `.deb` files from workspace root -**Output — three locations**: +**Output**: -| Location | How to access | Retention | -|----------|---------------|-----------| -| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | -| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | -| **GitHub Release asset** | Releases → `noble-6.8.0-X.Y` → Assets | Permanent | +| Location | How to access | Retention | Notes | +|----------|---------------|-----------|-------| +| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | Self-hosted runner only; skipped gracefully on GitHub-hosted | +| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | Always available | +| **GitHub Release asset** | Releases → `noble-6.8.0-X.Y` → Assets | Permanent | Attached when `kernel_version` is provided | --- @@ -216,7 +221,7 @@ All scripts run on Ubuntu 24.04 arm64. ```bash ./scripts/check-version.sh noble linux -# → 6.8.0-51.52 +# → 6.8.0-114.114 ``` ### Download the source package @@ -236,16 +241,20 @@ All scripts run on Ubuntu 24.04 arm64. ## Source package anatomy -The Ubuntu kernel source package is a standard Debian 3.0 (quilt) source package: +The Ubuntu Noble kernel source package uses **Debian source format 1.0**: | File | Size | Description | |------|------|-------------| | `linux_X.Y.Z-A.B.dsc` | ~10 KB | Source descriptor with SHA256 checksums | -| `linux_X.Y.Z.orig.tar.gz` | ~200 MB | Pristine upstream kernel tarball | -| `linux_X.Y.Z-A.B.debian.tar.xz` | ~5 MB | Ubuntu packaging overlay + patches | +| `linux_X.Y.Z.orig.tar.gz` | ~220 MB | Pristine upstream kernel tarball | +| `linux_X.Y.Z-A.B.diff.gz` | ~8 MB | Ubuntu patch set applied on top of upstream | -`dpkg-source -x` applies all patches and produces the full source tree -that is committed to the suite branch. +`dpkg-source --no-check -x` applies the diff to the orig tarball and +produces the full patched source tree, which is committed to the suite branch. + +> **Note**: The Launchpad API `source_name=` parameter does prefix matching. +> The workflow uses an exact `source_package_name` filter in jq to ensure +> `linux` is fetched and not `linux-meta-raspi` or other `linux-*` packages. --- @@ -256,10 +265,10 @@ Ubuntu kernel versions follow `X.Y.Z-A.B`: | Component | Example | Meaning | |-----------|---------|---------| | `X.Y.Z` | `6.8.0` | Upstream kernel version | -| `A` | `51` | ABI number | -| `B` | `52` | Upload number | +| `A` | `114` | ABI number | +| `B` | `114` | Upload number | -Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-51.52`. +Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. --- From 4cc55c75a25693c24e9807bae041e8ce7327e38f Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:50:49 -0700 Subject: [PATCH 013/115] fix: create debian->debian.master symlink if missing before build The Ubuntu kernel uses debian.master/ as the packaging directory with debian/ as a symlink pointing to it. Git does not always preserve symlinks, so the build would fail with 'No such file: debian/rules'. Add a guard in both docker and native build steps: if [ ! -e debian ] && [ -d debian.master ]; then ln -sf debian.master debian fi Also update README with corrected source package anatomy. --- .github/workflows/build-kernel.yml | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 09264aacbd975..f186512ed084d 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -155,6 +155,12 @@ jobs: sudo apt-get update -qq sudo apt-get build-dep -y kernel-src/ cd kernel-src/ + # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. + # Ensure the symlink exists (git may not preserve it). + if [ ! -e debian ] && [ -d debian.master ]; then + ln -sf debian.master debian + echo 'Created debian -> debian.master symlink' + fi export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' fakeroot debian/rules ${TARGET} " @@ -179,7 +185,15 @@ jobs: echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" - (cd kernel-src/ && fakeroot debian/rules "${TARGET}") + ( + cd kernel-src/ + # Ensure debian/ symlink exists (Ubuntu kernel uses debian.master/) + if [ ! -e debian ] && [ -d debian.master ]; then + ln -sf debian.master debian + echo "Created debian -> debian.master symlink" + fi + fakeroot debian/rules "${TARGET}" + ) # ----------------------------------------------------------------------- # 7. Collect output packages From bf489442331d098b7b4977b734286210a90a59ac Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:52:38 -0700 Subject: [PATCH 014/115] docs: document debian.master/ and debian/ symlink in source tree layout --- README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/README.md b/README.md index 805ace21de5b8..95f817971d85b 100644 --- a/README.md +++ b/README.md @@ -252,6 +252,19 @@ The Ubuntu Noble kernel source package uses **Debian source format 1.0**: `dpkg-source --no-check -x` applies the diff to the orig tarball and produces the full patched source tree, which is committed to the suite branch. +**Extracted source tree layout (noble branch):** + +``` +arch/ drivers/ fs/ kernel/ +debian → debian.master/ (symlink, may need recreating after git checkout) +debian.master/ ← Ubuntu packaging: rules, control, changelog, configs +Makefile net/ scripts/ ... +``` + +The Ubuntu kernel uses `debian.master/` as the actual packaging directory. +`debian/` is a symlink to it. The build workflow automatically recreates +the symlink if git did not preserve it. + > **Note**: The Launchpad API `source_name=` parameter does prefix matching. > The workflow uses an exact `source_package_name` filter in jq to ensure > `linux` is fetched and not `linux-meta-raspi` or other `linux-*` packages. From a5f30389e242a721eb64135277b093c81c5f3ccc Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:54:08 -0700 Subject: [PATCH 015/115] fix: add actions:write permission for workflow dispatch in fetch-source-pkg gh workflow run requires actions:write permission. Without it the trigger-build job fails with HTTP 403 when trying to dispatch build-kernel.yml. --- .github/workflows/fetch-source-pkg.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 5d33b90b84b95..56e76eb77f9b0 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -43,6 +43,7 @@ on: permissions: contents: write # push to suite branch + create tags + actions: write # dispatch build-kernel.yml via gh workflow run jobs: # -------------------------------------------------------------------------- From df3058851cfce02e259fb101a13f41cafa8dd662 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 15:55:28 -0700 Subject: [PATCH 016/115] fix: remove sudo inside docker container and add actions:write permission build-kernel.yml: - Remove sudo from apt-get commands inside docker run bash -c block. The pkg-builder container runs as root; sudo is not installed. fetch-source-pkg.yml: - Add actions:write permission so gh workflow run can dispatch build-kernel.yml from the trigger-build job. --- .github/workflows/build-kernel.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index f186512ed084d..cb7078266541b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -152,8 +152,8 @@ jobs: "ghcr.io/qualcomm-linux/pkg-builder:${SUITE}" \ bash -c " set -e - sudo apt-get update -qq - sudo apt-get build-dep -y kernel-src/ + apt-get update -qq + apt-get build-dep -y kernel-src/ cd kernel-src/ # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. # Ensure the symlink exists (git may not preserve it). From 2ac5363e0067c4e7fb95febadd0c69292486c955 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 16:01:25 -0700 Subject: [PATCH 017/115] fix: handle force re-sync when suite branch content is unchanged When force=true and the extracted source is identical to what is already committed, 'git commit' would fail with 'nothing to commit'. - Skip commit if git diff --cached --quiet (content unchanged) - Use git tag -fa to force-overwrite existing tag - Use git push --force for the tag to overwrite on remote --- .github/workflows/fetch-source-pkg.yml | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 56e76eb77f9b0..8ba76cb9621a5 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -296,21 +296,27 @@ jobs: > /tmp/commit-msg.txt git add --all - git commit -F /tmp/commit-msg.txt - echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" + if git diff --cached --quiet; then + echo "No changes to commit – suite branch already has this content" + else + git commit -F /tmp/commit-msg.txt + echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" + fi - name: Tag the commit run: | cd suite-repo - git tag -a "${TAG}" -m "Ubuntu ${SUITE} kernel ${VERSION}" + # Use -f to overwrite existing tag when force=true + git tag -fa "${TAG}" -m "Ubuntu ${SUITE} kernel ${VERSION}" echo "Tagged: ${TAG}" - name: Push suite branch and tag run: | cd suite-repo git push origin "${SUITE}" - git push origin "${TAG}" + # Use --force to overwrite existing tag when force=true + git push --force origin "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- From 2cbbf05df46583c462a62737a1ead4654ce17789 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 16:07:23 -0700 Subject: [PATCH 018/115] docs: make upstream source URLs suite-generic with noble as example --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 95f817971d85b..2cf884c930151 100644 --- a/README.md +++ b/README.md @@ -103,13 +103,17 @@ and contain only the extracted kernel source tree. ## Upstream source -| Resource | URL | -|----------|-----| -| Launchpad source packages | https://launchpad.net/ubuntu/noble/+source/linux | -| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` | +| Resource | URL pattern | +|----------|-------------| +| Launchpad source packages | `https://launchpad.net/ubuntu//+source/linux` | +| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | | Launchpad REST API | https://api.launchpad.net/1.0/ | | GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | +**Noble (current active suite):** +- Source packages: https://launchpad.net/ubuntu/noble/+source/linux +- Git repository: `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` + --- ## Workflows From 3eeef413430152418f663fedf9d634399af0edc2 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 16:10:39 -0700 Subject: [PATCH 019/115] fix: create debian->debian.master symlink in sync job before commit The Ubuntu kernel source package does not include debian/ as a symlink in the .diff.gz. Add it in the sync job after rsync so the noble branch is self-contained and the build does not need to create it at runtime. This also ensures force re-sync correctly adds the symlink even when the rest of the source content is unchanged. --- .github/workflows/fetch-source-pkg.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 8ba76cb9621a5..59f5aaa898a62 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -283,6 +283,16 @@ jobs: exit 1 } + # Ensure debian/ symlink exists in the suite branch. + # The Ubuntu kernel source package does not include debian/ as a + # symlink in the .diff.gz — it is expected to be created at build + # time. We add it here so the noble branch is self-contained and + # the build can run without extra setup. + if [ ! -e suite-repo/debian ] && [ -d suite-repo/debian.master ]; then + ln -sf debian.master suite-repo/debian + echo "Created debian -> debian.master symlink in suite-repo" + fi + cd suite-repo SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') From 0076781cb476f9da0c2a3274409ef1965b298974 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 20 Apr 2026 16:15:55 -0700 Subject: [PATCH 020/115] fix: enable deb-src in pkg-builder container for apt-get build-dep Ubuntu 24.04 uses DEB822 format (/etc/apt/sources.list.d/ubuntu.sources). The pkg-builder:noble container does not have deb-src enabled by default, causing 'apt-get build-dep' to fail with 'no deb-src URIs in sources.list'. Fix: sed 's/^Types: deb$/Types: deb deb-src/' before apt-get update. --- .github/workflows/build-kernel.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index cb7078266541b..c0d6e6a687c9c 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -152,6 +152,13 @@ jobs: "ghcr.io/qualcomm-linux/pkg-builder:${SUITE}" \ bash -c " set -e + # Enable deb-src so apt-get build-dep can resolve build dependencies. + # Ubuntu 24.04 uses DEB822 format (/etc/apt/sources.list.d/ubuntu.sources). + if [ -f /etc/apt/sources.list.d/ubuntu.sources ]; then + sed -i 's/^Types: deb$/Types: deb deb-src/' /etc/apt/sources.list.d/ubuntu.sources + else + grep '^deb ' /etc/apt/sources.list | sed 's/^deb /deb-src /' >> /etc/apt/sources.list + fi apt-get update -qq apt-get build-dep -y kernel-src/ cd kernel-src/ From 65c445cad5f0c4a50eb57113921cffdbe1cbc321 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 Apr 2026 23:47:44 +0000 Subject: [PATCH 021/115] build(deps): bump actions/upload-artifact from 4 to 7 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/build-kernel.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index c0d6e6a687c9c..5f5838875f7f7 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -233,7 +233,7 @@ jobs: # 9. Upload as GitHub Actions artifact (90-day fallback) # ----------------------------------------------------------------------- - name: Upload packages as artifact - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@v7 with: name: "kernel-debs-${{ inputs.suite }}-${{ inputs.arch }}-${{ inputs.flavor }}" path: output/ From f34f02afc7f0350d8ae8dafdeb1f28ed4cac0efe Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 20 Apr 2026 23:47:47 +0000 Subject: [PATCH 022/115] build(deps): bump actions/checkout from 4 to 6 Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/build-kernel.yml | 4 ++-- .github/workflows/fetch-source-pkg.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 5f5838875f7f7..81c4b36770416 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -101,7 +101,7 @@ jobs: # 2. Checkout the suite branch (contains the full kernel source tree) # ----------------------------------------------------------------------- - name: Checkout suite branch (${{ inputs.suite || 'noble' }}) - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: ref: ${{ inputs.suite || 'noble' }} path: kernel-src @@ -119,7 +119,7 @@ jobs: # ----------------------------------------------------------------------- - name: Checkout docker-pkg-build if: env.BUILD_MODE == 'docker' - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: repository: qualcomm-linux/docker-pkg-build ref: main diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 59f5aaa898a62..efbfc390e2c4c 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -175,7 +175,7 @@ jobs: # 2. Checkout main branch to get the helper scripts # ----------------------------------------------------------------------- - name: Checkout main (for scripts) - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: ref: main path: ci From 0540649375ae5d688d2d6860f3485e9b5e4f028a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 12:45:06 -0700 Subject: [PATCH 023/115] fix: use ./kernel-src/ (local path) for apt-get build-dep apt-get build-dep kernel-src/ tries to find a source package named 'kernel-src/' in the apt repositories. The ./ prefix tells apt-get to read the debian/control file from the local directory instead. --- .github/workflows/build-kernel.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 81c4b36770416..ca243790c07b0 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -160,7 +160,7 @@ jobs: grep '^deb ' /etc/apt/sources.list | sed 's/^deb /deb-src /' >> /etc/apt/sources.list fi apt-get update -qq - apt-get build-dep -y kernel-src/ + apt-get build-dep -y ./kernel-src/ cd kernel-src/ # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. # Ensure the symlink exists (git may not preserve it). @@ -181,7 +181,7 @@ jobs: sudo apt-get update -qq sudo apt-get install -y --no-install-recommends \ build-essential fakeroot devscripts dpkg-dev debhelper dh-exec rsync - sudo apt-get build-dep -y kernel-src/ + sudo apt-get build-dep -y ./kernel-src/ - name: Build kernel packages (native) if: env.BUILD_MODE == 'native' From aa4cc815659d6f60eb6cf20c4a3486ffc630efd1 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 13:01:22 -0700 Subject: [PATCH 024/115] fix: use 'apt-get build-dep linux' instead of local path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The local directory syntax './kernel-src/' is not supported by the apt version in the pkg-builder container. Use the source package name 'linux' directly — deb-src is already enabled so apt can resolve it. --- .github/workflows/build-kernel.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index ca243790c07b0..4ae74567b1e3b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -160,7 +160,11 @@ jobs: grep '^deb ' /etc/apt/sources.list | sed 's/^deb /deb-src /' >> /etc/apt/sources.list fi apt-get update -qq - apt-get build-dep -y ./kernel-src/ + # Install build dependencies for the linux source package. + # Using the source package name is more reliable than pointing + # apt-get at a local directory (requires apt >= 1.1 and a + # debian/control symlink to be in place first). + apt-get build-dep -y linux cd kernel-src/ # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. # Ensure the symlink exists (git may not preserve it). From 157ffdb9f2e797e3068da4c12d649ea4bff73c0d Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 13:33:06 -0700 Subject: [PATCH 025/115] debug+fix: use make -f for rules and add directory diagnostics --- .github/workflows/build-kernel.yml | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 4ae74567b1e3b..0bcb0de08833d 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -166,14 +166,29 @@ jobs: # debian/control symlink to be in place first). apt-get build-dep -y linux cd kernel-src/ + + echo '=== kernel-src top-level ===' + ls -la | head -30 + + echo '=== debian.master/ contents ===' + ls -la debian.master/ 2>/dev/null | head -20 || echo 'debian.master NOT FOUND' + + echo '=== debian symlink before fix ===' + ls -la debian 2>/dev/null || echo 'debian does not exist' + # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. - # Ensure the symlink exists (git may not preserve it). if [ ! -e debian ] && [ -d debian.master ]; then ln -sf debian.master debian echo 'Created debian -> debian.master symlink' fi + + echo '=== debian symlink after fix ===' + ls -la debian 2>/dev/null || echo 'debian STILL does not exist' + ls -la debian/rules 2>/dev/null || echo 'debian/rules NOT FOUND' + export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' - fakeroot debian/rules ${TARGET} + # Use 'make -f' so the rules file does not need to be executable + fakeroot make -f debian/rules ${TARGET} " # ----------------------------------------------------------------------- From 6f6022f9718f997261505300e1364d03995a7e59 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 13:48:00 -0700 Subject: [PATCH 026/115] fix: run debian/reconstruct to generate debian/rules before build The Ubuntu noble kernel source package does not ship debian/rules directly. debian.master/ contains only rules.d/ fragments. The debian/rules file is generated by running debian/reconstruct which assembles the rules from the fragments. --- .github/workflows/build-kernel.yml | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 0bcb0de08833d..d1eb13f55a20b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -186,9 +186,17 @@ jobs: ls -la debian 2>/dev/null || echo 'debian STILL does not exist' ls -la debian/rules 2>/dev/null || echo 'debian/rules NOT FOUND' + # The Ubuntu kernel does not ship debian/rules directly. + # debian.master/ has rules.d/ fragments; debian/rules is generated + # by running the reconstruct script. + echo '=== Running debian/reconstruct ===' + bash debian/reconstruct + + echo '=== debian/rules after reconstruct ===' + ls -la debian/rules 2>/dev/null || echo 'debian/rules STILL not found after reconstruct' + export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' - # Use 'make -f' so the rules file does not need to be executable - fakeroot make -f debian/rules ${TARGET} + fakeroot debian/rules ${TARGET} " # ----------------------------------------------------------------------- From 068e308c090f2afe84b5d7ce5e290d81d7fcdcfb Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 14:41:36 -0700 Subject: [PATCH 027/115] fix: switch sync source from Launchpad source package to Launchpad git MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Problem ─────── The Ubuntu kernel source package (Debian format 1.0) ships only debian.master/ containing rules.d/ Makefile fragments. It does NOT include debian/rules, debian/scripts/, debian/templates/, or any of the other files required to run 'fakeroot debian/rules binary-generic'. These files exist only in the Launchpad git repository: https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble The previous approach (dpkg-source -x on the .dsc/.orig.tar.gz/.diff.gz) produced an incomplete source tree that could not be built. Every attempt to work around this (debian/ symlink, reconstruct script, make -f) failed because the required files were simply not present in the source package. Solution ──────── Replace the source package download+extraction with a shallow git clone from the Launchpad git repository at the tag Ubuntu- (e.g. Ubuntu-6.8.0-114.114). This tag corresponds exactly to the version reported by the Launchpad REST API, so version tracking is unchanged. The git clone produces a complete, buildable source tree including: debian/rules ← main build entry point debian/scripts/ ← build helper scripts debian/templates/ ← package template files debian/cloud-tools/ ← cloud tools debian/config/ ← kernel config fragments debian/control.stub.in ← package control template Changes ─────── .github/workflows/fetch-source-pkg.yml - Remove: apt-get install devscripts dpkg-dev quilt patch - Remove: scripts/fetch-source-pkg.sh call - Remove: dpkg-source --no-check -x extraction step - Remove: debian/ symlink creation - Remove: file count verification (>5000 files check kept) - Add: git clone --depth=1 --branch Ubuntu-${VERSION} from Launchpad git - The Launchpad API is still used to determine the latest version number and map it to the git tag Ubuntu- .github/workflows/build-kernel.yml - Remove: debian/ symlink creation workaround - Remove: debian/reconstruct execution attempt - Remove: all debug ls/echo diagnostic steps - The noble branch now has a complete debian/ directory from git; fakeroot debian/rules binary-generic works without any workarounds scripts/fetch-source-pkg.sh - Rewritten to use git clone instead of downloading source package files - Now produces the same complete, buildable source tree as the CI workflow - Local developers can use this script to get a buildable tree: ./scripts/fetch-source-pkg.sh noble linux ./kernel-src/ ./scripts/build-kernel-deb.sh ./kernel-src/ arm64 generic $(nproc) README.md - Update pipeline diagram: Job 2 now shows git clone instead of dpkg-source - Update fetch-source-pkg.yml description to reflect git clone approach - Add 'Source and build notes' section explaining why git is used - Update scripts table: fetch-source-pkg.sh now clones (not downloads) - Update local usage section with correct workflow --- .github/workflows/build-kernel.yml | 30 +----- .github/workflows/fetch-source-pkg.yml | 134 +++++++------------------ README.md | 66 ++++++------ scripts/fetch-source-pkg.sh | 88 +++++++--------- 4 files changed, 117 insertions(+), 201 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index d1eb13f55a20b..fc5b954de716a 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -167,33 +167,9 @@ jobs: apt-get build-dep -y linux cd kernel-src/ - echo '=== kernel-src top-level ===' - ls -la | head -30 - - echo '=== debian.master/ contents ===' - ls -la debian.master/ 2>/dev/null | head -20 || echo 'debian.master NOT FOUND' - - echo '=== debian symlink before fix ===' - ls -la debian 2>/dev/null || echo 'debian does not exist' - - # The Ubuntu kernel uses debian.master/ with debian/ as a symlink. - if [ ! -e debian ] && [ -d debian.master ]; then - ln -sf debian.master debian - echo 'Created debian -> debian.master symlink' - fi - - echo '=== debian symlink after fix ===' - ls -la debian 2>/dev/null || echo 'debian STILL does not exist' - ls -la debian/rules 2>/dev/null || echo 'debian/rules NOT FOUND' - - # The Ubuntu kernel does not ship debian/rules directly. - # debian.master/ has rules.d/ fragments; debian/rules is generated - # by running the reconstruct script. - echo '=== Running debian/reconstruct ===' - bash debian/reconstruct - - echo '=== debian/rules after reconstruct ===' - ls -la debian/rules 2>/dev/null || echo 'debian/rules STILL not found after reconstruct' + # The noble branch is now sourced from the Launchpad git repository + # which includes the complete debian/ directory with debian/rules. + echo "debian/rules: $(ls -la debian/rules 2>/dev/null || echo 'NOT FOUND')" export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' fakeroot debian/rules ${TARGET} diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index efbfc390e2c4c..57dacc6453e0d 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -2,9 +2,16 @@ # # fetch-source-pkg.yml # -# Fetches the latest Canonical Ubuntu kernel source package from Launchpad, -# extracts it with dpkg-source, and commits the full source tree to a branch -# named after the Ubuntu suite (e.g. "noble", "questing"). +# Clones the Canonical Ubuntu kernel from the Launchpad git repository at +# the tag corresponding to the latest published source package version, and +# commits the full source tree (including the complete debian/ directory) to +# a branch named after the Ubuntu suite (e.g. "noble", "questing"). +# +# Why git instead of the source package? +# The Ubuntu kernel source package (format 1.0) ships only debian.master/ +# with rules.d/ fragments — debian/rules is NOT included. The complete +# debian/ directory (with rules, scripts/, templates/, etc.) lives in the +# Launchpad git repository. Using the git source gives a buildable tree. # # Repository branch layout # ───────────────────────── @@ -13,10 +20,7 @@ # – additional suites added on demand (questing, resolute, …) # # Each commit on a suite branch is tagged - -# (e.g. noble-6.8.0-51.52). -# -# The workflow is idempotent: if the tag already exists it exits cleanly. -# On success it automatically dispatches build-kernel.yml. +# (e.g. noble-6.8.0-114.114). # # Repository variables (Settings → Variables → Actions): # KERNEL_SUITE – default suite to sync when not specified (default: noble) @@ -26,7 +30,6 @@ name: "Sync: Canonical Kernel Sources to Branch" on: schedule: - # Run daily at 04:00 UTC - cron: "0 4 * * *" workflow_dispatch: @@ -42,14 +45,10 @@ on: default: false permissions: - contents: write # push to suite branch + create tags - actions: write # dispatch build-kernel.yml via gh workflow run + contents: write + actions: write jobs: - # -------------------------------------------------------------------------- - # Job 1: check-version - # Queries Launchpad and decides whether a sync is needed. - # -------------------------------------------------------------------------- check-version: name: "Check latest version on Launchpad" runs-on: ubuntu-24.04-arm @@ -67,9 +66,6 @@ jobs: - name: Install dependencies run: sudo apt-get install -y --no-install-recommends jq - # ----------------------------------------------------------------------- - # Query Launchpad REST API - # ----------------------------------------------------------------------- - name: Query Launchpad for latest published source id: query env: @@ -86,7 +82,6 @@ jobs: echo "Querying: ${API}" RESP=$(curl -fsSL "${API}") - # Filter by exact source_package_name (source_name= is a prefix match) VERSION=$(echo "$RESP" | jq -r \ --arg name "${SOURCE}" \ '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') @@ -108,9 +103,6 @@ jobs: echo "Version : ${VERSION}" echo "Tag : ${TAG}" - # ----------------------------------------------------------------------- - # Gate: skip if the tag already exists (unless force=true) - # ----------------------------------------------------------------------- - name: Check whether tag already exists id: gate env: @@ -132,11 +124,6 @@ jobs: echo "Tag '${TAG}' not found – will sync" fi - # -------------------------------------------------------------------------- - # Job 2: sync - # Downloads the source package, extracts it, and commits to the suite - # branch. - # -------------------------------------------------------------------------- sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" runs-on: ubuntu-24.04-arm @@ -153,7 +140,7 @@ jobs: steps: # ----------------------------------------------------------------------- - # 1. Free up disk space (GitHub-hosted runners have ~14 GB free) + # 1. Free up disk space # ----------------------------------------------------------------------- - name: Free up runner disk space run: | @@ -172,72 +159,45 @@ jobs: df -h / # ----------------------------------------------------------------------- - # 2. Checkout main branch to get the helper scripts - # ----------------------------------------------------------------------- - - name: Checkout main (for scripts) - uses: actions/checkout@v6 - with: - ref: main - path: ci - - # ----------------------------------------------------------------------- - # 3. Install tools + # 2. Install tools # ----------------------------------------------------------------------- - name: Install dependencies run: | sudo apt-get update -qq sudo apt-get install -y --no-install-recommends \ - curl jq devscripts dpkg-dev quilt rsync patch + curl jq rsync # ----------------------------------------------------------------------- - # 4. Download source package from Launchpad + # 3. Clone from Launchpad git at the version tag + # The git repository has the complete debian/ directory including + # debian/rules, scripts/, templates/, etc. — unlike the source + # package which only ships debian.master/ with rules.d/ fragments. # ----------------------------------------------------------------------- - - name: Fetch source package from Launchpad + - name: Clone from Launchpad git at version tag run: | - chmod +x ci/scripts/fetch-source-pkg.sh - ci/scripts/fetch-source-pkg.sh "${SUITE}" "${SOURCE}" source-pkg/ + LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" + GIT_TAG="Ubuntu-${VERSION}" - echo "Downloaded files:" - ls -lh source-pkg/ + echo "Cloning ${LAUNCHPAD_URL} at tag ${GIT_TAG} (shallow)..." + git clone --depth=1 --branch "${GIT_TAG}" "${LAUNCHPAD_URL}" kernel-src/ - # ----------------------------------------------------------------------- - # 5. Extract source package - # ----------------------------------------------------------------------- - - name: Extract source package - run: | - DSC=$(find source-pkg/ -name "*.dsc" | sort -V | tail -1) - echo "Extracting: ${DSC}" - # --no-check skips GPG signature verification (key not in runner keyring) - # Redirect verbose "upstream files modified" list to log file - dpkg-source --no-check -x "${DSC}" kernel-src/ \ - > /tmp/dpkg-source.log 2>&1 || { - echo "dpkg-source -x failed. Last 30 lines of output:" - tail -30 /tmp/dpkg-source.log - exit 1 - } - echo "Extraction complete" + echo "SRC_DIR=kernel-src" >> "$GITHUB_ENV" - # When an explicit output directory is given, dpkg-source -x extracts - # the source directly INTO that directory (not into a subdirectory). - SRC_DIR="kernel-src" - echo "SRC_DIR=${SRC_DIR}" >> "$GITHUB_ENV" - echo "Extracted to: ${SRC_DIR}" - - # Verify the extraction produced a real kernel source tree - FILE_COUNT=$(find "${SRC_DIR}" -type f | wc -l) - echo "Extracted ${FILE_COUNT} files" + FILE_COUNT=$(find kernel-src/ -type f | wc -l) + echo "Cloned ${FILE_COUNT} files" [ "${FILE_COUNT}" -gt 5000 ] || { - echo "ERROR: Extraction produced only ${FILE_COUNT} files (expected >5000)" - echo "Top-level contents:" - ls -la "${SRC_DIR}/" + echo "ERROR: Too few files cloned (${FILE_COUNT})" exit 1 } echo "Top-level contents:" - ls "${SRC_DIR}/" + ls kernel-src/ + + echo "debian/ contents:" + ls kernel-src/debian/ | head -20 # ----------------------------------------------------------------------- - # 6. Configure git + # 4. Configure git # ----------------------------------------------------------------------- - name: Configure git identity run: | @@ -245,7 +205,7 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 7. Prepare the suite branch + # 5. Prepare the suite branch # ----------------------------------------------------------------------- - name: Clone repository for suite branch update env: @@ -268,14 +228,13 @@ jobs: git rm -rf --quiet . 2>/dev/null || true fi - - name: Populate suite branch with extracted source + - name: Populate suite branch with cloned source run: | echo "Copying source tree from ${SRC_DIR} → suite-repo/" rsync -a --delete \ --exclude='.git' \ "${SRC_DIR}/" suite-repo/ - # Verify rsync copied the files COPIED=$(find suite-repo/ -type f | wc -l) echo "Files in suite-repo after rsync: ${COPIED}" [ "${COPIED}" -gt 5000 ] || { @@ -283,16 +242,6 @@ jobs: exit 1 } - # Ensure debian/ symlink exists in the suite branch. - # The Ubuntu kernel source package does not include debian/ as a - # symlink in the .diff.gz — it is expected to be created at build - # time. We add it here so the noble branch is self-contained and - # the build can run without extra setup. - if [ ! -e suite-repo/debian ] && [ -d suite-repo/debian.master ]; then - ln -sf debian.master suite-repo/debian - echo "Created debian -> debian.master symlink in suite-repo" - fi - cd suite-repo SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') @@ -301,7 +250,7 @@ jobs: "Ubuntu suite : ${SUITE}" \ "Kernel version : ${VERSION}" \ "Upstream : ${UPSTREAM_VERSION}" \ - "Source : https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION}" \ + "Source : https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE} (tag: Ubuntu-${VERSION})" \ "Synced : ${SYNC_DATE}" \ > /tmp/commit-msg.txt @@ -317,7 +266,6 @@ jobs: - name: Tag the commit run: | cd suite-repo - # Use -f to overwrite existing tag when force=true git tag -fa "${TAG}" -m "Ubuntu ${SUITE} kernel ${VERSION}" echo "Tagged: ${TAG}" @@ -325,12 +273,11 @@ jobs: run: | cd suite-repo git push origin "${SUITE}" - # Use --force to overwrite existing tag when force=true git push --force origin "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- - # 7. Summary + # 6. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() @@ -343,11 +290,8 @@ jobs: echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" + echo "| Source | Launchpad git tag \`Ubuntu-${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - # -------------------------------------------------------------------------- - # Job 3: trigger-build - # -------------------------------------------------------------------------- trigger-build: name: "Trigger kernel build" runs-on: ubuntu-24.04-arm diff --git a/README.md b/README.md index 2cf884c930151..169dfa47cdeae 100644 --- a/README.md +++ b/README.md @@ -24,13 +24,12 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. │ │ │ │ ┌─────────▼──────────┐ │ │ │ Job 2: sync │ │ -│ │ fetch-source-pkg.sh│ │ -│ │ download .dsc │ │ -│ │ + .orig.tar.gz │ │ -│ │ + .diff.gz │ │ -│ │ dpkg-source │ │ -│ │ --no-check -x │ │ -│ │ → full source tree │ │ +│ │ git clone │ │ +│ │ Launchpad git │ │ +│ │ tag Ubuntu-6.8.0- │ │ +│ │ 114.114 (shallow) │ │ +│ │ → complete source │ │ +│ │ incl. debian/rules │ │ │ │ commit to noble │ │ │ │ branch + tag │ │ │ └─────────┬──────────┘ │ @@ -121,9 +120,9 @@ and contain only the extracted kernel source tree. ### `fetch-source-pkg.yml` — Sync sources to branch Queries the Launchpad REST API for the latest published `linux` source -package (exact name match — the API does prefix matching), downloads the -source package files, extracts the full source tree with -`dpkg-source --no-check -x`, and commits the result to the corresponding +package version (exact name match), then clones the Launchpad git +repository at the corresponding tag (`Ubuntu-`) to get the +complete source tree including `debian/rules`, and commits it to the suite branch. **Schedule**: daily at **04:00 UTC** @@ -142,7 +141,7 @@ suite branch. | Job | What it does | |-----|-------------| | `check-version` | Queries Launchpad API with exact `source_package_name` filter; checks if tag already exists; sets `should_sync` flag | -| `sync` | Frees disk space; downloads source package via `fetch-source-pkg.sh`; extracts with `dpkg-source --no-check -x`; verifies >5000 files; commits to suite branch; creates tag | +| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from Launchpad git; verifies >5000 files; commits to suite branch; creates tag | | `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `build_mode=docker` | **Idempotent**: if tag `noble-6.8.0-114.114` already exists, the workflow exits cleanly without downloading anything. @@ -228,12 +227,15 @@ All scripts run on Ubuntu 24.04 arm64. # → 6.8.0-114.114 ``` -### Download the source package +### Clone the kernel source (buildable) ```bash -./scripts/fetch-source-pkg.sh noble linux ./source-pkg/ +# Clones from Launchpad git at the latest Ubuntu- tag +./scripts/fetch-source-pkg.sh noble linux ./kernel-src/ ``` +This produces a complete, buildable source tree with `debian/rules` — the same source the CI workflow uses. + ### Build kernel packages ```bash @@ -243,31 +245,39 @@ All scripts run on Ubuntu 24.04 arm64. --- -## Source package anatomy +## Source and build notes -The Ubuntu Noble kernel source package uses **Debian source format 1.0**: +### Why git instead of the source package? -| File | Size | Description | -|------|------|-------------| -| `linux_X.Y.Z-A.B.dsc` | ~10 KB | Source descriptor with SHA256 checksums | -| `linux_X.Y.Z.orig.tar.gz` | ~220 MB | Pristine upstream kernel tarball | -| `linux_X.Y.Z-A.B.diff.gz` | ~8 MB | Ubuntu patch set applied on top of upstream | +The Ubuntu kernel source package (format 1.0) ships only `debian.master/` +with `rules.d/` fragments — **`debian/rules` is NOT included**. The complete +`debian/` directory (with `rules`, `scripts/`, `templates/`, etc.) lives only +in the Launchpad git repository. -`dpkg-source --no-check -x` applies the diff to the orig tarball and -produces the full patched source tree, which is committed to the suite branch. +The sync workflow therefore clones from: +``` +https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/ +``` +at the tag `Ubuntu-` (e.g. `Ubuntu-6.8.0-114.114`). -**Extracted source tree layout (noble branch):** +### Noble branch source tree layout ``` arch/ drivers/ fs/ kernel/ -debian → debian.master/ (symlink, may need recreating after git checkout) -debian.master/ ← Ubuntu packaging: rules, control, changelog, configs +debian/ ← complete Ubuntu packaging (rules, scripts/, templates/, …) Makefile net/ scripts/ ... ``` -The Ubuntu kernel uses `debian.master/` as the actual packaging directory. -`debian/` is a symlink to it. The build workflow automatically recreates -the symlink if git did not preserve it. +### About the helper scripts + +| Script | Purpose | Used by workflow? | +|--------|---------|-------------------| +| `scripts/check-version.sh` | Query latest version from Launchpad API | No (workflow has inline equivalent) | +| `scripts/fetch-source-pkg.sh` | Clone from Launchpad git at latest version tag (buildable source) | No (workflow has inline equivalent) | +| `scripts/build-kernel-deb.sh` | Build kernel `.deb` packages locally | No (workflow has inline equivalent) | + +`scripts/fetch-source-pkg.sh` clones from the Launchpad git repository (same +as the CI workflow) and produces a complete, buildable source tree locally. > **Note**: The Launchpad API `source_name=` parameter does prefix matching. > The workflow uses an exact `source_package_name` filter in jq to ensure diff --git a/scripts/fetch-source-pkg.sh b/scripts/fetch-source-pkg.sh index 072095bb249cf..2b8f82ea35ed3 100755 --- a/scripts/fetch-source-pkg.sh +++ b/scripts/fetch-source-pkg.sh @@ -1,8 +1,9 @@ #!/usr/bin/env bash # SPDX-License-Identifier: BSD-3-Clause # -# fetch-source-pkg.sh - Download a Canonical Ubuntu kernel source package -# from Launchpad +# fetch-source-pkg.sh - Clone the Canonical Ubuntu kernel source from the +# Launchpad git repository at the version tag matching +# the latest published source package. # # Usage: # fetch-source-pkg.sh [SUITE] [SOURCE_NAME] [OUTPUT_DIR] @@ -10,14 +11,17 @@ # Arguments: # SUITE Ubuntu suite (default: noble) # SOURCE_NAME Source package name (default: linux) -# OUTPUT_DIR Directory to write files into (default: .) +# OUTPUT_DIR Directory to clone into (default: .) # -# The script queries the Launchpad REST API to find the latest published -# source, then downloads all constituent files (.dsc, .orig.tar.gz, -# .debian.tar.xz, etc.) and writes a version.env summary file. +# Why git instead of the source package (.dsc/.orig.tar.gz/.diff.gz)? +# The Ubuntu kernel source package (format 1.0) ships only debian.master/ +# with rules.d/ fragments — debian/rules is NOT included. The complete +# debian/ directory (with rules, scripts/, templates/, etc.) lives only in +# the Launchpad git repository. Cloning from git gives a buildable tree. # -# Environment variables (override defaults): -# LAUNCHPAD_API Base URL for the Launchpad API (default: https://api.launchpad.net/1.0) +# Output: +# A shallow clone of the kernel source at tag Ubuntu- is placed +# in OUTPUT_DIR/. A version.env metadata file is also written. set -euo pipefail @@ -26,6 +30,7 @@ SOURCE_NAME="${2:-linux}" OUTPUT_DIR="${3:-.}" LAUNCHPAD_API="${LAUNCHPAD_API:-https://api.launchpad.net/1.0}" +LAUNCHPAD_GIT="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git" # --------------------------------------------------------------------------- # Helpers @@ -35,7 +40,7 @@ die() { log "ERROR: $*"; exit 1; } hr() { log "$(printf '%0.s─' {1..60})"; } # --------------------------------------------------------------------------- -# 1. Query Launchpad for the latest published source +# 1. Query Launchpad for the latest published version # --------------------------------------------------------------------------- hr log "Querying Launchpad for latest '${SOURCE_NAME}' in Ubuntu ${SUITE}..." @@ -48,75 +53,56 @@ API_URL+="&status=Published" API_URL+="&order_by_date=true" RESPONSE=$(curl -fsSL "${API_URL}") \ - || die "Launchpad API request failed: ${API_URL}" + || die "Launchpad API request failed" -# Filter by exact source_package_name (source_name= is a prefix match on Launchpad) +# Filter by exact source_package_name (source_name= is a prefix match) VERSION=$(echo "$RESPONSE" | jq -r \ --arg name "${SOURCE_NAME}" \ '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') -SELF_LINK=$(echo "$RESPONSE" | jq -r \ - --arg name "${SOURCE_NAME}" \ - '[.entries[] | select(.source_package_name == $name)] | .[0].self_link // empty') -[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' (exact) in '${SUITE}'" -[ -n "$SELF_LINK" ] || die "Could not retrieve self_link for '${SOURCE_NAME}' ${VERSION}" +[ -n "$VERSION" ] || die "No published source found for '${SOURCE_NAME}' (exact) in '${SUITE}'" -# Upstream version: strip Ubuntu revision suffix (e.g. "6.8.0-51.52" → "6.8.0") UPSTREAM_VERSION=$(echo "${VERSION}" | cut -d'-' -f1) +GIT_TAG="Ubuntu-${VERSION}" -log "Found: ${SOURCE_NAME} ${VERSION} (upstream: ${UPSTREAM_VERSION})" -log "Link: ${SELF_LINK}" +log "Found: ${SOURCE_NAME} ${VERSION} (upstream: ${UPSTREAM_VERSION})" +log "Git tag: ${GIT_TAG}" # --------------------------------------------------------------------------- -# 2. Retrieve per-file download URLs +# 2. Clone from Launchpad git at the version tag (shallow) # --------------------------------------------------------------------------- hr -log "Fetching file list..." - -FILE_URLS=$(curl -fsSL "${SELF_LINK}?ws.op=sourceFileUrls" | jq -r '.[]') \ - || die "Failed to retrieve file URLs from ${SELF_LINK}" - -[ -n "$FILE_URLS" ] || die "No files listed for ${SOURCE_NAME} ${VERSION}" - -FILE_COUNT=$(echo "$FILE_URLS" | wc -l) -log "Files to download: ${FILE_COUNT}" +CLONE_URL="${LAUNCHPAD_GIT}/${SUITE}" +log "Cloning ${CLONE_URL} at tag ${GIT_TAG} (shallow)..." -# --------------------------------------------------------------------------- -# 3. Download each file -# --------------------------------------------------------------------------- -hr mkdir -p "${OUTPUT_DIR}" -IDX=0 -while IFS= read -r url; do - IDX=$((IDX + 1)) - FILENAME=$(basename "${url%%\?*}") # strip any query string - DEST="${OUTPUT_DIR}/${FILENAME}" +git clone --depth=1 --branch "${GIT_TAG}" "${CLONE_URL}" "${OUTPUT_DIR}" \ + || die "git clone failed" - log "[${IDX}/${FILE_COUNT}] ${FILENAME}" - curl -fsSL --progress-bar -o "${DEST}" "${url}" \ - || die "Download failed: ${url}" +FILE_COUNT=$(find "${OUTPUT_DIR}" -type f | wc -l) +log "Cloned ${FILE_COUNT} files" - SIZE=$(du -sh "${DEST}" | cut -f1) - log " → ${SIZE} ${DEST}" -done <<< "$FILE_URLS" +[ "${FILE_COUNT}" -gt 5000 ] || \ + die "Too few files cloned (${FILE_COUNT}) — expected >5000" # --------------------------------------------------------------------------- -# 4. Write version metadata +# 3. Write version metadata # --------------------------------------------------------------------------- hr -VERSION_ENV="${OUTPUT_DIR}/version.env" -cat > "${VERSION_ENV}" < "${OUTPUT_DIR}/version.env" < Date: Tue, 21 Apr 2026 14:49:33 -0700 Subject: [PATCH 028/115] ci: add full Launchpad git URL and git tag to sync job summary --- .github/workflows/fetch-source-pkg.yml | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 57dacc6453e0d..d7f8387265704 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -282,15 +282,20 @@ jobs: - name: Print summary if: always() run: | + LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" + GIT_TAG="Ubuntu-${VERSION}" + echo "## Source Sync Summary" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Source | Launchpad git tag \`Ubuntu-${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Cloned from | \`${LAUNCHPAD_URL}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" trigger-build: name: "Trigger kernel build" From f472cb79eb292d6900f959df7825e0d8d4c25b1c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 14:57:01 -0700 Subject: [PATCH 029/115] ci: create GitHub Release with .deb assets + clarify upstream source URLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build-kernel.yml — GitHub Release creation Previously the 'Attach packages to release tag' step only uploaded .deb assets if a GitHub Release already existed for the tag. Since the sync workflow creates only a git tag (not a GitHub Release), no release was ever created and the upload was silently skipped. Fix: step 10 now creates the GitHub Release if it does not exist before uploading the built .deb packages as release assets. Release properties: title: 'Ubuntu kernel (arm64)' tag: '-' (e.g. noble-6.8.0-114.114) assets: linux-image-*.deb, linux-headers-*.deb, linux-modules-*.deb, linux-libc-dev_*.deb latest: false (not marked as latest release) Release notes are written to /tmp/release-notes.md via printf to avoid YAML parsing errors caused by 'Key: value' patterns in multi-line shell strings. After a successful build, .deb packages are permanently accessible at: https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases/tag/ Storage summary after build: GitHub Release assets — permanent, publicly downloadable from GitHub UI GitHub Actions artifact — 90-day retention, downloadable from Actions tab S3 qli-prd-lecore-gh-artifacts — permanent, self-hosted runner only README.md — upstream source table Replace the generic 'Launchpad REST API: https://api.launchpad.net/1.0/' entry with the actual query URL and a 'Used by' column explaining the purpose of each upstream resource: Launchpad REST API → check-version job (version number lookup) Launchpad git repo → sync job (shallow clone at Ubuntu- tag) GitHub Releases → build-kernel job (permanent .deb asset storage) --- .github/workflows/build-kernel.yml | 29 ++++++++++++++++++++--------- README.md | 11 +++++------ 2 files changed, 25 insertions(+), 15 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index fc5b954de716a..12afba40ffc95 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -244,9 +244,9 @@ jobs: if-no-files-found: warn # ----------------------------------------------------------------------- - # 10. Attach packages to the release tag (if kernel_version was provided) + # 10. Create GitHub Release and attach built packages # ----------------------------------------------------------------------- - - name: Attach packages to release tag + - name: Create GitHub Release and attach packages if: inputs.kernel_version != '' env: GH_TOKEN: ${{ github.token }} @@ -259,19 +259,30 @@ jobs: exit 0 fi - if gh release view "${TAG}" \ + # Create the release if it doesn't already exist + if ! gh release view "${TAG}" \ --repo "${{ github.repository }}" \ --json tagName -q .tagName \ >/dev/null 2>&1; then - echo "Attaching packages to release ${TAG}..." - echo "${DEBS}" | xargs gh release upload "${TAG}" \ + echo "Creating GitHub Release ${TAG}..." + printf 'Ubuntu %s kernel %s built for arm64.\n\nFlavour: %s\nBuild mode: %s\nRunner: %s\n' \ + "${SUITE}" "${{ inputs.kernel_version }}" \ + "${FLAVOR}" "${BUILD_MODE}" "${{ runner.name }}" \ + > /tmp/release-notes.md + gh release create "${TAG}" \ --repo "${{ github.repository }}" \ - --clobber - echo "Packages attached to ${TAG}" - else - echo "Release tag '${TAG}' not found – packages available via S3 and artifact" + --title "Ubuntu ${SUITE} kernel ${{ inputs.kernel_version }} (arm64)" \ + --notes-file /tmp/release-notes.md \ + --latest=false fi + echo "Uploading packages to release ${TAG}..." + echo "${DEBS}" | xargs gh release upload "${TAG}" \ + --repo "${{ github.repository }}" \ + --clobber + echo "Packages attached to release ${TAG}" + echo "Release URL: https://github.com/${{ github.repository }}/releases/tag/${TAG}" + # ----------------------------------------------------------------------- # 11. Summary # ----------------------------------------------------------------------- diff --git a/README.md b/README.md index 169dfa47cdeae..3395971ca38b7 100644 --- a/README.md +++ b/README.md @@ -102,12 +102,11 @@ and contain only the extracted kernel source tree. ## Upstream source -| Resource | URL pattern | -|----------|-------------| -| Launchpad source packages | `https://launchpad.net/ubuntu//+source/linux` | -| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | -| Launchpad REST API | https://api.launchpad.net/1.0/ | -| GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | +| Resource | URL pattern | Used by | +|----------|-------------|---------| +| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/` | `check-version` job — queries for the latest published version number | +| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` | +| GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | `build-kernel` job — attaches built `.deb` packages | **Noble (current active suite):** - Source packages: https://launchpad.net/ubuntu/noble/+source/linux From fcb047d7375322a280b61c1adfdca363b6478aab Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 15:00:58 -0700 Subject: [PATCH 030/115] ci: conditional S3 upload + README cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build-kernel.yml: - S3 upload step now conditional on runner.name == 'lecore-prd-u2404-arm64-xlrg-od-ephem' instead of continue-on-error. Cleanly skipped on GitHub-hosted runners where S3 credentials and network access are not available. No silent failures. README.md: - 'Noble (current active suite)' → 'Example (noble suite)' - Fix stray 'A' character at top of file --- .github/workflows/build-kernel.yml | 6 ++++-- README.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 12afba40ffc95..47087b85aac51 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -222,10 +222,12 @@ jobs: ls -lh output/ || echo "(no files found)" # ----------------------------------------------------------------------- - # 8. Upload to S3 (permanent storage, self-hosted runner only) + # 8. Upload to S3 (self-hosted runner only) + # Skipped on GitHub-hosted runners — S3 credentials and network + # access are only available on lecore-prd-u2404-arm64-xlrg-od-ephem. # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 - continue-on-error: true + if: runner.name == 'lecore-prd-u2404-arm64-xlrg-od-ephem' uses: qualcomm-linux/upload-private-artifact-action@aws with: s3_bucket: qli-prd-lecore-gh-artifacts diff --git a/README.md b/README.md index 3395971ca38b7..0b9e3f7c01b22 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ and contain only the extracted kernel source tree. | Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` | | GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | `build-kernel` job — attaches built `.deb` packages | -**Noble (current active suite):** +**Example (noble suite):** - Source packages: https://launchpad.net/ubuntu/noble/+source/linux - Git repository: `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` From 48fdfc5fd1ee46e8b81b41500de2be411825344a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 15:07:25 -0700 Subject: [PATCH 031/115] fix: find and copy .deb files inside container after build The Ubuntu kernel build may drop .deb files at a path not covered by the host-side collect step. Added find+copy inside the docker container to locate all .deb files and copy them to output/ before the container exits. The bind-mounted output/ directory is then visible on the host. --- .github/workflows/build-kernel.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 47087b85aac51..f2a49dbebdb21 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -173,6 +173,15 @@ jobs: export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' fakeroot debian/rules ${TARGET} + + echo '=== Locating built .deb files ===' + find "${GITHUB_WORKSPACE}" -name '*.deb' | sort || true + + echo '=== Copying .deb files to output/ ===' + mkdir -p "${GITHUB_WORKSPACE}/output/" + find "${GITHUB_WORKSPACE}" -name '*.deb' \ + -exec cp {} "${GITHUB_WORKSPACE}/output/" \; + ls -lh "${GITHUB_WORKSPACE}/output/" || echo 'No .deb files found' " # ----------------------------------------------------------------------- From c288d52085f1f8d6289681d2e7e2a39a09aafcb9 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 15:14:09 -0700 Subject: [PATCH 032/115] docs+fix: add Launchpad API explanation to README; find .deb inside container MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md: - Add 'How the sync workflow finds and clones the kernel source' section with the 3-step explanation: REST API query → git tag construction → git clone. Includes the actual JSON response format and the reason the REST API is used (published status, not just tag existence). build-kernel.yml: - After fakeroot debian/rules, find all .deb files anywhere under GITHUB_WORKSPACE inside the container and copy them to output/. The Ubuntu kernel build may drop .deb files at an unexpected path; this ensures they are captured regardless of location. --- README.md | 53 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 0b9e3f7c01b22..dd958089fd987 100644 --- a/README.md +++ b/README.md @@ -246,18 +246,55 @@ This produces a complete, buildable source tree with `debian/rules` — the same ## Source and build notes -### Why git instead of the source package? +### How the sync workflow finds and clones the kernel source -The Ubuntu kernel source package (format 1.0) ships only `debian.master/` -with `rules.d/` fragments — **`debian/rules` is NOT included**. The complete -`debian/` directory (with `rules`, `scripts/`, `templates/`, etc.) lives only -in the Launchpad git repository. +The sync workflow uses two Launchpad services for different purposes: + +**Step 1 — Launchpad REST API: find the latest published version** + +``` +GET https://api.launchpad.net/1.0/ubuntu/+archive/primary + ?ws.op=getPublishedSources + &source_name=linux + &distro_series=/ubuntu/noble + &status=Published + &order_by_date=true + +Response (JSON): +{ + "entries": [ + { + "source_package_name": "linux", + "source_package_version": "6.8.0-114.114", ← we want this + "self_link": "https://api.launchpad.net/..." + }, + ... + ] +} +``` + +The API tells us the exact version string of the latest *officially published* +kernel. A git tag might exist before the package is published to the archive, +so the API is the authoritative source for "what is the current release". + +**Step 2 — Construct the git tag** -The sync workflow therefore clones from: ``` -https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/ +VERSION = "6.8.0-114.114" +GIT_TAG = "Ubuntu-6.8.0-114.114" ``` -at the tag `Ubuntu-` (e.g. `Ubuntu-6.8.0-114.114`). + +**Step 3 — Clone from Launchpad git at that tag** + +```bash +git clone --depth=1 --branch Ubuntu-6.8.0-114.114 \ + https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble +``` + +The git repository has the **complete** `debian/` directory including +`debian/rules`, `debian/scripts/`, `debian/templates/`, etc. — unlike the +source package (`.dsc`/`.orig.tar.gz`/`.diff.gz`) which ships only +`debian.master/` with `rules.d/` fragments and no `debian/rules`. ### Noble branch source tree layout From e22e5478bce6f73208b242b991a9f6bb921900c3 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 21:57:52 -0700 Subject: [PATCH 033/115] fix: use single-quoted bash -c to prevent outer-shell expansion in docker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the 8-minute build with no packages: bash -c "..." (double quotes) caused the outer shell to expand $(...) and ${VAR} BEFORE passing the string to docker. The debug version had '($1 == "DEBIAN")' inside the double-quoted string — the inner double quotes terminated the outer string, causing a bash syntax error on line 41 inside the container. fakeroot debian/rules exited immediately with no packages. Fix: bash -c '...' (single quotes) — nothing is expanded by the outer shell. JOBS, TARGET, and WS are passed as -e env vars and expanded inside the container by the container's own bash. --- .github/workflows/build-kernel.yml | 53 +++++++++++++++++------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index f2a49dbebdb21..cd1cffbcbc353 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -146,43 +146,50 @@ jobs: echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" + # IMPORTANT: use single quotes for bash -c so the outer shell does NOT + # expand $(...) or ${VAR} before passing the string to docker. + # JOBS, TARGET, and WS are passed as -e env vars and expanded inside + # the container by the container's own bash. docker run -i --privileged --rm \ -v "${GITHUB_WORKSPACE}:${GITHUB_WORKSPACE}" \ --workdir="${GITHUB_WORKSPACE}" \ + -e JOBS="${JOBS}" \ + -e TARGET="${TARGET}" \ + -e WS="${GITHUB_WORKSPACE}" \ "ghcr.io/qualcomm-linux/pkg-builder:${SUITE}" \ - bash -c " - set -e - # Enable deb-src so apt-get build-dep can resolve build dependencies. - # Ubuntu 24.04 uses DEB822 format (/etc/apt/sources.list.d/ubuntu.sources). + bash -c ' + set -euo pipefail + + # Enable deb-src (Ubuntu 24.04 uses DEB822 format) if [ -f /etc/apt/sources.list.d/ubuntu.sources ]; then - sed -i 's/^Types: deb$/Types: deb deb-src/' /etc/apt/sources.list.d/ubuntu.sources + sed -i "s/^Types: deb$/Types: deb deb-src/" /etc/apt/sources.list.d/ubuntu.sources else - grep '^deb ' /etc/apt/sources.list | sed 's/^deb /deb-src /' >> /etc/apt/sources.list + grep "^deb " /etc/apt/sources.list | sed "s/^deb /deb-src /" >> /etc/apt/sources.list fi apt-get update -qq - # Install build dependencies for the linux source package. - # Using the source package name is more reliable than pointing - # apt-get at a local directory (requires apt >= 1.1 and a - # debian/control symlink to be in place first). apt-get build-dep -y linux + cd kernel-src/ + echo "debian/debian.env: $(cat debian/debian.env 2>/dev/null || echo NOT FOUND)" + ls -la debian/rules && echo "debian/rules: OK" || { echo "debian/rules: NOT FOUND"; exit 1; } + + # Heartbeat every 2 min — kernel build takes 60-90 min with minimal output + while true; do echo "[$(date -u +%H:%M:%S)] kernel build still running..."; sleep 120; done & + HEARTBEAT=$! - # The noble branch is now sourced from the Launchpad git repository - # which includes the complete debian/ directory with debian/rules. - echo "debian/rules: $(ls -la debian/rules 2>/dev/null || echo 'NOT FOUND')" + echo "=== Starting: fakeroot debian/rules ${TARGET} (jobs=${JOBS}) ===" + export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" + fakeroot debian/rules "${TARGET}" - export DEB_BUILD_OPTIONS='parallel=${JOBS} nocheck' - fakeroot debian/rules ${TARGET} + kill "${HEARTBEAT}" 2>/dev/null || true - echo '=== Locating built .deb files ===' - find "${GITHUB_WORKSPACE}" -name '*.deb' | sort || true + echo "=== Locating built .deb files ===" + find "${WS}" -name "*.deb" | sort || true - echo '=== Copying .deb files to output/ ===' - mkdir -p "${GITHUB_WORKSPACE}/output/" - find "${GITHUB_WORKSPACE}" -name '*.deb' \ - -exec cp {} "${GITHUB_WORKSPACE}/output/" \; - ls -lh "${GITHUB_WORKSPACE}/output/" || echo 'No .deb files found' - " + mkdir -p "${WS}/output/" + find "${WS}" -name "*.deb" -exec cp {} "${WS}/output/" \; + ls -lh "${WS}/output/" || echo "No .deb files found" + ' # ----------------------------------------------------------------------- # 6b. Build kernel packages natively on the runner (native mode) From 0db9e08c4ce623ad332e43eabc5a6462bddc8720 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 21 Apr 2026 22:01:40 -0700 Subject: [PATCH 034/115] ci: remove dead else branch in deb-src sed block pkg-builder: containers are Ubuntu 24.04+ which all use DEB822 format (/etc/apt/sources.list.d/ubuntu.sources). The else branch for traditional sources.list format (Ubuntu 23.10 and earlier) was never executed for noble, questing, or resolute. --- .github/workflows/build-kernel.yml | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index cd1cffbcbc353..f61f3bbd4c01e 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -160,12 +160,9 @@ jobs: bash -c ' set -euo pipefail - # Enable deb-src (Ubuntu 24.04 uses DEB822 format) - if [ -f /etc/apt/sources.list.d/ubuntu.sources ]; then - sed -i "s/^Types: deb$/Types: deb deb-src/" /etc/apt/sources.list.d/ubuntu.sources - else - grep "^deb " /etc/apt/sources.list | sed "s/^deb /deb-src /" >> /etc/apt/sources.list - fi + # Enable deb-src so apt-get build-dep can resolve build dependencies. + # pkg-builder:noble is Ubuntu 24.04 which uses DEB822 format. + sed -i "s/^Types: deb$/Types: deb deb-src/" /etc/apt/sources.list.d/ubuntu.sources apt-get update -qq apt-get build-dep -y linux From f260dd5c996503a950ba3e0d2fc76acae492fef8 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 22 Apr 2026 15:02:34 -0700 Subject: [PATCH 035/115] fix: generate debian/control and certs before kernel build Two issues fixed: 1. CONFIG_RUST_IS_AVAILABLE policy check failure: bindgen-0.65 is not available in pkg-builder container so CONFIG_RUST_IS_AVAILABLE is '-' instead of 'y'. The Ubuntu kernel config policy check (4-checks.mk) requires 'y' for arm64. Fixed by: do_skip_checks=true on the main fakeroot invocation. 2. Missing debian/canonical-certs.pem: The kernel certs/x509_certificate_list target requires debian/canonical-certs.pem to exist before compilation starts. Fixed by: running 'fakeroot make -f debian/rules debian/control' before the main build. Note: 'fakeroot make -f debian/rules' is used instead of 'fakeroot debian/rules' because fakeroot's shell wrapper execs the command via /bin/sh (dash) which resolves the shebang Invoking make explicitly bypasses the shebang lookup. --- .github/workflows/build-kernel.yml | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index f61f3bbd4c01e..27dc0b683d626 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -174,9 +174,26 @@ jobs: while true; do echo "[$(date -u +%H:%M:%S)] kernel build still running..."; sleep 120; done & HEARTBEAT=$! + # Generate debian/canonical-certs.pem and debian/control before + # the kernel build. The kernel certs/x509_certificate_list target + # requires debian/canonical-certs.pem to exist before compilation. + # + # Use "fakeroot make -f debian/rules" rather than + # "fakeroot debian/rules": fakeroot is a shell script that execs + # the command via /bin/sh (dash); dash resolves the shebang + # (#!/usr/bin/make -f) and reports "not found" if make is not yet + # on PATH at exec time. Invoking make explicitly bypasses the + # shebang lookup entirely. + echo "=== Generating debian/control and certificate files ===" + fakeroot make -f debian/rules debian/control + echo "=== Starting: fakeroot debian/rules ${TARGET} (jobs=${JOBS}) ===" export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" - fakeroot debian/rules "${TARGET}" + # do_skip_checks=true skips the config policy check that requires + # CONFIG_RUST_IS_AVAILABLE=y — bindgen-0.65 is not available in + # the container so Rust support is unavailable, causing the check + # to fail. This is the standard approach for non-official builds. + fakeroot debian/rules "${TARGET}" do_skip_checks=true kill "${HEARTBEAT}" 2>/dev/null || true From 3178da7840b5d2f46b10992132f6d7914df6639f Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 22 Apr 2026 15:26:07 -0700 Subject: [PATCH 036/115] ci: remove native build mode, always build in docker container The build-kernel.yml workflow previously supported two build modes: - docker: runs the build inside ghcr.io/qualcomm-linux/pkg-builder: - native: runs dpkg-buildpackage directly on the GitHub-hosted runner The native mode was never used in practice and introduced unnecessary complexity: separate install-tools and build steps, conditional logic on every docker-specific step (if: env.BUILD_MODE == 'docker'), and a build_mode dropdown in the workflow_dispatch inputs. The docker mode is the only supported path because: - It provides a clean, reproducible environment matching the target Ubuntu suite (noble, questing, resolute) exactly - The pkg-builder: container has the correct toolchain versions (gcc, binutils, debhelper) pre-installed for that suite - It isolates the build from the runner's host packages, preventing version skew between the runner's Ubuntu 24.04 and the target suite Changes: - Remove build_mode workflow_dispatch input (docker/native dropdown) - Remove BUILD_MODE job-level env variable - Remove 'if: env.BUILD_MODE == docker' conditionals from steps 4/5/6a - Remove 'Install build tools (native mode)' step - Remove 'Build kernel packages (native)' step - Simplify job display name: drop the [docker] suffix - Remove build_mode field from GitHub Release notes - Remove Build mode row from step summary table --- .github/workflows/build-kernel.yml | 75 ++++++------------------------ 1 file changed, 15 insertions(+), 60 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 27dc0b683d626..97f19c1ad4716 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -5,12 +5,10 @@ # Builds Ubuntu kernel .deb packages from a suite branch # (e.g. "noble") that was populated by fetch-source-pkg.yml. # -# Build modes: -# docker (default) – runs the build inside the suite-matched -# ghcr.io/qualcomm-linux/pkg-builder: container -# via docker-pkg-build; ensures a clean, reproducible -# environment that exactly matches the target distro. -# native – runs dpkg-buildpackage directly on the runner host. +# The build always runs inside the suite-matched +# ghcr.io/qualcomm-linux/pkg-builder: container via +# docker-pkg-build, ensuring a clean, reproducible environment +# that exactly matches the target distro. # # Output .deb packages are stored in two places: # 1. S3 bucket – qli-prd-lecore-gh-artifacts (permanent) @@ -47,14 +45,6 @@ on: - generic - lowlatency - all - build_mode: - description: "Build mode: docker (suite-matched container) or native (host)" - required: false - default: "docker" - type: choice - options: - - docker - - native permissions: contents: write # upload assets to release tag @@ -64,15 +54,14 @@ jobs: # Job: build # -------------------------------------------------------------------------- build: - name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }}) [${{ inputs.build_mode || 'docker' }}]" + name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" runs-on: ubuntu-24.04-arm timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'noble' }} - ARCH: ${{ inputs.arch || 'arm64' }} - FLAVOR: ${{ inputs.flavor || 'generic' }} - BUILD_MODE: ${{ inputs.build_mode || 'docker' }} + SUITE: ${{ inputs.suite || 'noble' }} + ARCH: ${{ inputs.arch || 'arm64' }} + FLAVOR: ${{ inputs.flavor || 'generic' }} steps: # ----------------------------------------------------------------------- @@ -115,10 +104,9 @@ jobs: echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 4. Checkout docker-pkg-build (docker mode only) + # 4. Checkout docker-pkg-build # ----------------------------------------------------------------------- - name: Checkout docker-pkg-build - if: env.BUILD_MODE == 'docker' uses: actions/checkout@v6 with: repository: qualcomm-linux/docker-pkg-build @@ -126,20 +114,18 @@ jobs: path: docker-pkg-build # ----------------------------------------------------------------------- - # 5. Build the suite-matched pkg-builder docker image (docker mode only) + # 5. Build the suite-matched pkg-builder docker image # ----------------------------------------------------------------------- - name: Build docker image for suite (${{ inputs.suite || 'noble' }}) - if: env.BUILD_MODE == 'docker' run: | ./docker-pkg-build/docker_deb_build.py --rebuild -d "${SUITE}" docker image ls # ----------------------------------------------------------------------- - # 6a. Build kernel packages inside the suite-matched container (docker) - # The workspace is bind-mounted so output .deb files land on the host. + # 6. Build kernel packages inside the suite-matched container + # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - - name: Build kernel packages (docker – ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) - if: env.BUILD_MODE == 'docker' + - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) run: | JOBS=$(nproc) if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi @@ -205,36 +191,6 @@ jobs: ls -lh "${WS}/output/" || echo "No .deb files found" ' - # ----------------------------------------------------------------------- - # 6b. Build kernel packages natively on the runner (native mode) - # ----------------------------------------------------------------------- - - name: Install build tools (native mode) - if: env.BUILD_MODE == 'native' - run: | - sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends \ - build-essential fakeroot devscripts dpkg-dev debhelper dh-exec rsync - sudo apt-get build-dep -y ./kernel-src/ - - - name: Build kernel packages (native) - if: env.BUILD_MODE == 'native' - run: | - JOBS=$(nproc) - if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi - - echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" - export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" - - ( - cd kernel-src/ - # Ensure debian/ symlink exists (Ubuntu kernel uses debian.master/) - if [ ! -e debian ] && [ -d debian.master ]; then - ln -sf debian.master debian - echo "Created debian -> debian.master symlink" - fi - fakeroot debian/rules "${TARGET}" - ) - # ----------------------------------------------------------------------- # 7. Collect output packages # The Ubuntu kernel build drops .deb files one level above the @@ -297,9 +253,9 @@ jobs: --json tagName -q .tagName \ >/dev/null 2>&1; then echo "Creating GitHub Release ${TAG}..." - printf 'Ubuntu %s kernel %s built for arm64.\n\nFlavour: %s\nBuild mode: %s\nRunner: %s\n' \ + printf 'Ubuntu %s kernel %s built for arm64.\n\nFlavour: %s\nRunner: %s\n' \ "${SUITE}" "${{ inputs.kernel_version }}" \ - "${FLAVOR}" "${BUILD_MODE}" "${{ runner.name }}" \ + "${FLAVOR}" "${{ runner.name }}" \ > /tmp/release-notes.md gh release create "${TAG}" \ --repo "${{ github.repository }}" \ @@ -328,7 +284,6 @@ jobs: echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Build mode | \`${BUILD_MODE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" From bf20f48f47200b971f9fab204a1233eac4054ac5 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 22 Apr 2026 16:13:03 -0700 Subject: [PATCH 037/115] fix(sync): authenticate git ls-remote and drop stale build_mode field Two fixes to fetch-source-pkg.yml: 1. Gate check: use github.token for git ls-remote tag existence check The previous unauthenticated HTTPS URL silently failed on private repositories (auth error suppressed by >/dev/null 2>&1), causing the gate to always evaluate to 'tag not found' and should_sync=true. The sync job ran daily even when the tag already existed, wasting time cloning the full Launchpad kernel tree unnecessarily. Fix: embed github.token as x-access-token in the URL so git ls-remote can authenticate against the private repository and correctly detect existing tags. 2. trigger-build: remove --field build_mode='docker' The build_mode workflow_dispatch input was removed from build-kernel.yml (docker is now the only build mode). Passing the stale field to 'gh workflow run' would cause an 'Unexpected input' error and prevent the automatic build trigger from firing after a successful sync. --- .github/workflows/fetch-source-pkg.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index d7f8387265704..03addf0116e4a 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -115,7 +115,7 @@ jobs: fi if git ls-remote --exit-code --tags \ - "https://github.com/${{ github.repository }}.git" \ + "https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git" \ "refs/tags/${TAG}" >/dev/null 2>&1; then echo "should_sync=false" >> "$GITHUB_OUTPUT" echo "Tag '${TAG}' already exists – nothing to do" @@ -315,7 +315,6 @@ jobs: --field suite="${SUITE}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ - --field flavor="generic" \ - --field build_mode="docker" + --field flavor="generic" echo "Build workflow dispatched for ${SUITE} ${VERSION}" From bab1f0ecf6ff01c574237d08721b1521de21f2c4 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 22 Apr 2026 16:20:17 -0700 Subject: [PATCH 038/115] fix(sync): increase Launchpad API page size to 300 for noble MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Launchpad getPublishedSources API uses prefix matching for source_name=linux, returning all linux-* packages (linux, linux-meta, linux-restricted-modules, linux-hwe-6.8, linux-oem-6.8, linux-raspi, linux-signed, etc.). The default page size is 75 entries. Noble (24.04 LTS) is a mature suite with many linux-* variants published over time. When more than 75 linux-* packages appear in the date-ordered response, the plain 'linux' package is pushed off the first page and the jq exact-match filter returns empty, causing the 'No exact match for linux in noble' error. Questing (25.04) is a newer suite with far fewer linux-* variants, so linux stays on the first page — explaining why questing worked while noble consistently failed. Fix: add &ws.size=300 to the API URL. The Launchpad API supports a maximum page size of 300, which is sufficient to cover all linux-* variants for any supported suite. --- .github/workflows/fetch-source-pkg.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 03addf0116e4a..406d760d08432 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -78,6 +78,7 @@ jobs: API+="&distro_series=/ubuntu/${SUITE}" API+="&status=Published" API+="&order_by_date=true" + API+="&ws.size=300" echo "Querying: ${API}" RESP=$(curl -fsSL "${API}") From e8b67f34aeb39e69c04906b4ec63c773a93f4c8b Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 22 Apr 2026 16:28:59 -0700 Subject: [PATCH 039/115] docs: update README for docker-only build, sync fixes, and build notes - Redesign pipeline diagram with double-line workflow boxes, inner job boxes, and explicit conditional flow labels - Remove build_mode input from build-kernel.yml inputs table - Update build steps: link to new Build container notes appendix - Remove build_mode=docker from trigger-build job description - Add &ws.size=300 to all Launchpad API URL references - Update prefix-matching note to explain ws.size=300 rationale - Mark noble and questing as active suites - Add Build container notes appendix explaining: - Why fakeroot make -f debian/rules is used (shebang exec failure) - Why do_skip_checks=true is needed (bindgen-0.65 / Rust policy) --- README.md | 218 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 149 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index dd958089fd987..98cecdbc8fd6d 100644 --- a/README.md +++ b/README.md @@ -7,62 +7,80 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. ## End-to-end pipeline ``` -┌─────────────────────────────────────────────────────────────────────┐ -│ DAILY (04:00 UTC) │ -│ │ -│ Launchpad │ -│ api.launchpad.net ──► fetch-source-pkg.yml │ -│ │ │ -│ ┌─────────▼──────────┐ │ -│ │ Job 1: check-version │ -│ │ curl Launchpad API │ │ -│ │ exact match: linux │ │ -│ │ → noble 6.8.0-114 │ │ -│ │ tag exists? YES→skip │ -│ │ NO ↓ │ │ -│ └─────────┬──────────┘ │ -│ │ │ -│ ┌─────────▼──────────┐ │ -│ │ Job 2: sync │ │ -│ │ git clone │ │ -│ │ Launchpad git │ │ -│ │ tag Ubuntu-6.8.0- │ │ -│ │ 114.114 (shallow) │ │ -│ │ → complete source │ │ -│ │ incl. debian/rules │ │ -│ │ commit to noble │ │ -│ │ branch + tag │ │ -│ └─────────┬──────────┘ │ -│ │ │ -│ ┌─────────▼──────────┐ │ -│ │ Job 3: trigger │ │ -│ │ gh workflow run │ │ -│ │ build-kernel.yml │ │ -│ │ suite=noble │ │ -│ │ build_mode=docker │ │ -│ └─────────┬──────────┘ │ -│ │ │ -│ ┌─────────▼──────────┐ │ -│ │ build-kernel.yml │ │ -│ │ checkout noble │ │ -│ │ branch │ │ -│ │ checkout │ │ -│ │ docker-pkg-build │ │ -│ │ docker_deb_build.py│ │ -│ │ --rebuild -d noble │ │ -│ │ docker run │ │ -│ │ pkg-builder:noble │ │ -│ │ debian/rules │ │ -│ │ binary-generic │ │ -│ └─────────┬──────────┘ │ -│ │ │ -│ ┌───────────────────┼───────────────────┐ │ -│ ▼ ▼ ▼ │ -│ S3 Bucket GitHub Artifact GitHub Release │ -│ qli-prd-lecore- 90-day retention noble-6.8.0-114.114 │ -│ gh-artifacts Actions → run Releases → Assets │ -│ (self-hosted only) → Artifacts (permanent) │ -└─────────────────────────────────────────────────────────────────────┘ +SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm +══════════════════════════════════════════════════════════════════════════════ + + Launchpad REST API + (api.launchpad.net) + │ + ▼ +╔════════════════════════════════════════════════════════════════════════════╗ +║ fetch-source-pkg.yml ║ +║ ║ +║ ┌──────────────────────────────────────────────────────────────────────┐ ║ +║ │ Job 1 · check-version │ ║ +║ │ │ ║ +║ │ Query Launchpad API (ws.size=300, exact source_package_name match) │ ║ +║ │ → latest version: noble 6.8.0-114.114 │ ║ +║ │ │ ║ +║ │ git ls-remote (authenticated) → tag noble-6.8.0-114.114 exists? │ ║ +║ │ │ ║ +║ │ YES ──▶ should_sync=false ──▶ workflow exits cleanly │ ║ +║ │ NO ──▶ should_sync=true ──▶ continue ↓ │ ║ +║ └──────────────────────────────────────────────────────────────────────┘ ║ +║ │ should_sync=true ║ +║ ▼ ║ +║ ┌──────────────────────────────────────────────────────────────────────┐ ║ +║ │ Job 2 · sync │ ║ +║ │ │ ║ +║ │ Free disk space (~10 GB) │ ║ +║ │ git clone --depth=1 Launchpad git @ Ubuntu-6.8.0-114.114 │ ║ +║ │ Verify >5000 files cloned │ ║ +║ │ rsync source → noble branch (orphan) │ ║ +║ │ git commit + tag noble-6.8.0-114.114 │ ║ +║ │ git push branch + tag │ ║ +║ └──────────────────────────────────────────────────────────────────────┘ ║ +║ │ sync succeeded ║ +║ ▼ ║ +║ ┌──────────────────────────────────────────────────────────────────────┐ ║ +║ │ Job 3 · trigger-build │ ║ +║ │ │ ║ +║ │ gh workflow run build-kernel.yml │ ║ +║ │ suite=noble kernel_version=6.8.0-114.114 arch=arm64 │ ║ +║ └──────────────────────────────────────────────────────────────────────┘ ║ +╚════════════════════════════════════════════════════════════════════════════╝ + │ + ▼ +╔════════════════════════════════════════════════════════════════════════════╗ +║ build-kernel.yml ║ +║ ║ +║ Checkout noble branch ──▶ kernel-src/ ║ +║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ +║ docker_deb_build.py --rebuild -d noble ║ +║ ║ +║ ┌──────────────────────────────────────────────────────────────────────┐ ║ +║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:noble │ ║ +║ │ │ ║ +║ │ apt-get build-dep linux │ ║ +║ │ fakeroot make -f debian/rules debian/control ← generate certs │ ║ +║ │ fakeroot debian/rules binary-generic do_skip_checks=true │ ║ +║ └──────────────────────────────────────────────────────────────────────┘ ║ +║ ║ +║ Collect .deb files ──▶ output/ ║ +╚════════════════════════════════════════════════════════════════════════════╝ + │ + ├──────────────────────┬───────────────────────┐ + ▼ ▼ ▼ + ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ + │ S3 Bucket │ │ GitHub Artifact │ │ GitHub Release │ + │ │ │ │ │ │ + │ qli-prd- │ │ 90-day retention │ │ noble-6.8.0- │ + │ lecore-gh- │ │ Actions → run │ │ 114.114 │ + │ artifacts │ │ → Artifacts │ │ Releases → │ + │ │ │ │ │ Assets │ + │ self-hosted │ │ always │ │ permanent │ + │ runner only │ │ available │ │ │ + └──────────────┘ └──────────────────┘ └──────────────────┘ ``` All jobs run on: `ubuntu-24.04-arm` (GitHub-hosted, Ubuntu 24.04 arm64) @@ -104,7 +122,7 @@ and contain only the extracted kernel source tree. | Resource | URL pattern | Used by | |----------|-------------|---------| -| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/` | `check-version` job — queries for the latest published version number | +| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/&ws.size=300` | `check-version` job — queries for the latest published version number | | Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` | | GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | `build-kernel` job — attaches built `.deb` packages | @@ -139,9 +157,9 @@ suite branch. | Job | What it does | |-----|-------------| -| `check-version` | Queries Launchpad API with exact `source_package_name` filter; checks if tag already exists; sets `should_sync` flag | +| `check-version` | Queries Launchpad API (`ws.size=300`) with exact `source_package_name` filter; checks tag existence via authenticated `git ls-remote`; sets `should_sync` flag | | `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from Launchpad git; verifies >5000 files; commits to suite branch; creates tag | -| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `build_mode=docker` | +| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=generic` | **Idempotent**: if tag `noble-6.8.0-114.114` already exists, the workflow exits cleanly without downloading anything. @@ -165,19 +183,19 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | -| `build_mode` | `docker` | `docker` (suite-matched container) or `native` (host) | -**Build steps (docker mode)**: +**Build steps**: 1. Free up disk space (~10 GB) 2. Checkout suite branch → `kernel-src/` 3. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` 4. Build docker image: `docker_deb_build.py --rebuild -d ` -5. Run build inside container: +5. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: ``` - docker run ghcr.io/qualcomm-linux/pkg-builder: - → apt-get build-dep kernel-src/ - → fakeroot debian/rules binary- + apt-get build-dep linux + fakeroot make -f debian/rules debian/control + fakeroot debian/rules binary- do_skip_checks=true ``` + See [Build container notes](#build-container-notes) for why these exact invocations are used. 6. Collect `.deb` files from workspace root **Output**: @@ -259,6 +277,7 @@ GET https://api.launchpad.net/1.0/ubuntu/+archive/primary &distro_series=/ubuntu/noble &status=Published &order_by_date=true + &ws.size=300 Response (JSON): { @@ -315,9 +334,11 @@ Makefile net/ scripts/ ... `scripts/fetch-source-pkg.sh` clones from the Launchpad git repository (same as the CI workflow) and produces a complete, buildable source tree locally. -> **Note**: The Launchpad API `source_name=` parameter does prefix matching. -> The workflow uses an exact `source_package_name` filter in jq to ensure -> `linux` is fetched and not `linux-meta-raspi` or other `linux-*` packages. +> **Note**: The Launchpad API `source_name=` parameter does prefix matching, +> returning all `linux-*` packages. The workflow uses `ws.size=300` to ensure +> the full result set is returned, then applies an exact `source_package_name` +> filter in jq to select only `linux` and not `linux-meta`, `linux-hwe-6.8`, +> `linux-raspi`, or other `linux-*` variants. --- @@ -340,7 +361,7 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. | Suite | Codename | Status | Kernel | |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.04 — add when available | TBD | +| `questing` | Questing Quokka | 25.04 — **active** | TBD | | `resolute` | Resolute Ringtail | 25.10 — add when available | TBD | To add a new suite, trigger `fetch-source-pkg.yml` with the desired @@ -354,6 +375,65 @@ gh workflow run fetch-source-pkg.yml \ --- +## Build container notes + +### Pre-generating `debian/control` and certificate files + +Before the main kernel compilation starts, the build runs: + +```bash +fakeroot make -f debian/rules debian/control +``` + +This generates two things the kernel build requires: + +- **`debian/canonical-certs.pem`** — the certificate file referenced by the + kernel's `certs/x509_certificate_list` target. Without it the build fails + immediately with: + ``` + No rule to make target 'debian/canonical-certs.pem', + needed by 'certs/x509_certificate_list' + ``` +- **`debian/control`** — the Debian package control stub used by later build + steps. + +**Why `fakeroot make -f debian/rules` and not `fakeroot debian/rules`?** + +`fakeroot` is a shell script (`/usr/bin/fakeroot`) that execs the given command +via `/bin/sh` (dash). Dash reads the shebang of `debian/rules` +(`#!/usr/bin/make -f`) and tries to resolve the interpreter at exec time. In +the container environment this resolution fails silently, producing: + +``` +/usr/bin/fakeroot: 175: debian/rules: not found (exit 127) +``` + +Invoking `make -f debian/rules` explicitly bypasses the shebang lookup +entirely — `make` is resolved directly from PATH and the Makefile is passed +via `-f`. + +--- + +### Skipping the Rust config policy check (`do_skip_checks=true`) + +The Ubuntu kernel build system runs a config policy check +(`debian/rules.d/4-checks.mk`) that requires `CONFIG_RUST_IS_AVAILABLE=y` +for all supported architectures including arm64. This check fails in the +`pkg-builder` container because `bindgen-0.65` is not available, so +`CONFIG_RUST_IS_AVAILABLE` is set to `-` instead of `y`: + +``` +check-config: CONFIG_RUST_IS_AVAILABLE changed from y to - +make: *** [debian/rules.d/4-checks.mk:15: config-prepare-check-generic] Error 1 +``` + +Passing `do_skip_checks=true` to `fakeroot debian/rules` bypasses this policy +check. This is the standard approach for non-official builds and is equivalent +to how Canonical's own CI handles environments where optional toolchains are +unavailable. + +--- + ## License Scripts and workflows in this repository are licensed under the From 46c59e8ba0d231b580f29e103108cbc19dc7f296 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 23 Apr 2026 14:33:38 -0700 Subject: [PATCH 040/115] refactor: replace debian/control pre-step with debian/rules clean MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously the build used three separate workarounds to prepare the build environment before kernel compilation: 1. fakeroot make -f debian/rules debian/control → generated debian/canonical-certs.pem and debian/control 2. ln -sf ../debian.master/changelog debian/changelog → created the missing changelog symlink that debian/control does not create 3. fakeroot debian/rules binary-generic do_skip_checks=true → bypassed the Rust config policy check The debian/control target is a partial setup path that leaves the build environment in an incomplete state, requiring manual fixups. Replace steps 1 and 2 with the standard Ubuntu kernel build setup entry point: fakeroot make -f debian/rules clean The clean target is the intended setup path used by Canonical's own build infrastructure. It runs debian/control as a dependency (generating debian/canonical-certs.pem and debian/control), creates the debian/changelog → debian.master/changelog symlink required by dh_installchangelogs, and removes any stale build artifacts — all in a single, complete setup step. This reduces the pre-build workarounds from three to one: - do_skip_checks=true remains necessary because bindgen-0.65 is not available in the pkg-builder container, causing the Rust config policy check to fail. The fakeroot make -f invocation is retained (rather than fakeroot debian/rules) to bypass the shebang exec issue in fakeroot's shell wrapper script. Update README Build container notes appendix to document the new single-step setup approach and its rationale. --- .github/workflows/build-kernel.yml | 26 +++++++++++--------- README.md | 38 ++++++++++++++++++++---------- 2 files changed, 40 insertions(+), 24 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 97f19c1ad4716..c7ca5879c153e 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -160,18 +160,22 @@ jobs: while true; do echo "[$(date -u +%H:%M:%S)] kernel build still running..."; sleep 120; done & HEARTBEAT=$! - # Generate debian/canonical-certs.pem and debian/control before - # the kernel build. The kernel certs/x509_certificate_list target - # requires debian/canonical-certs.pem to exist before compilation. + # Run the clean target to set up the full build environment before + # compilation. This is the standard Ubuntu kernel build setup path: + # - runs debian/control as a dependency, generating + # debian/canonical-certs.pem (required by certs/x509_certificate_list) + # and debian/control + # - creates debian/changelog → debian.master/changelog symlink + # (required by dh_installchangelogs at the end of binary-generic) + # - removes any stale build artifacts # - # Use "fakeroot make -f debian/rules" rather than - # "fakeroot debian/rules": fakeroot is a shell script that execs - # the command via /bin/sh (dash); dash resolves the shebang - # (#!/usr/bin/make -f) and reports "not found" if make is not yet - # on PATH at exec time. Invoking make explicitly bypasses the - # shebang lookup entirely. - echo "=== Generating debian/control and certificate files ===" - fakeroot make -f debian/rules debian/control + # Use "fakeroot make -f debian/rules" rather than "fakeroot debian/rules": + # fakeroot is a shell script that execs the command via /bin/sh (dash); + # dash resolves the shebang (#!/usr/bin/make -f) and reports "not found" + # if make is not yet on PATH at exec time. Invoking make explicitly + # bypasses the shebang lookup entirely. + echo "=== Setting up build environment (debian/rules clean) ===" + fakeroot make -f debian/rules clean echo "=== Starting: fakeroot debian/rules ${TARGET} (jobs=${JOBS}) ===" export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" diff --git a/README.md b/README.md index 98cecdbc8fd6d..5b45ed025edca 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:noble │ ║ ║ │ │ ║ ║ │ apt-get build-dep linux │ ║ -║ │ fakeroot make -f debian/rules debian/control ← generate certs │ ║ +║ │ fakeroot make -f debian/rules clean ← setup env │ ║ ║ │ fakeroot debian/rules binary-generic do_skip_checks=true │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ║ ║ @@ -192,7 +192,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow 5. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: ``` apt-get build-dep linux - fakeroot make -f debian/rules debian/control + fakeroot make -f debian/rules clean fakeroot debian/rules binary- do_skip_checks=true ``` See [Build container notes](#build-container-notes) for why these exact invocations are used. @@ -377,25 +377,37 @@ gh workflow run fetch-source-pkg.yml \ ## Build container notes -### Pre-generating `debian/control` and certificate files +### Build environment setup (`debian/rules clean`) Before the main kernel compilation starts, the build runs: ```bash -fakeroot make -f debian/rules debian/control +fakeroot make -f debian/rules clean ``` -This generates two things the kernel build requires: - -- **`debian/canonical-certs.pem`** — the certificate file referenced by the - kernel's `certs/x509_certificate_list` target. Without it the build fails - immediately with: +This is the **standard Ubuntu kernel build setup path** — the same entry point +Canonical's own build infrastructure uses. The `clean` target: + +- Runs `debian/control` as a dependency, which generates: + - **`debian/canonical-certs.pem`** — the X.509 certificate embedded into the + kernel image for module signing. Required by the kernel's + `certs/x509_certificate_list` make target. Without it the build fails + immediately: + ``` + No rule to make target 'debian/canonical-certs.pem', + needed by 'certs/x509_certificate_list' + ``` + - **`debian/control`** — the Debian package control stub +- Creates **`debian/changelog → debian.master/changelog`** symlink. The Ubuntu + kernel source tree does not include `debian/changelog` directly — the + changelog lives in `debian.master/changelog`. The `dh_installchangelogs` + debhelper tool (called at the end of `binary-generic`) requires this symlink + to exist or the build fails after 2+ hours of compilation: ``` - No rule to make target 'debian/canonical-certs.pem', - needed by 'certs/x509_certificate_list' + dh_installchangelogs: error: cannot open file debian/changelog + make: *** [debian/rules.d/2-binary-arch.mk:572: binary-generic] Error 25 ``` -- **`debian/control`** — the Debian package control stub used by later build - steps. +- Removes any stale build artifacts **Why `fakeroot make -f debian/rules` and not `fakeroot debian/rules`?** From faf781c268369678c29973c4401da326c7258df4 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 23 Apr 2026 14:38:50 -0700 Subject: [PATCH 041/115] ci: add debian/changelog safety net after debian/rules clean The clean target is expected to create the debian/changelog symlink as part of its standard build environment setup. However, this behaviour has not been directly verified against the pkg-builder container environment. Add an idempotent guard after the clean step that creates the symlink if it is missing: if [ ! -f debian/changelog ] && [ -f debian.master/changelog ]; then ln -sf ../debian.master/changelog debian/changelog fi This is a no-op when clean already created the symlink (the condition is false). If clean does not create it in this container environment, the guard prevents a 2+ hour wasted build that would otherwise fail at dh_installchangelogs with: dh_installchangelogs: error: cannot open file debian/changelog The guard can be removed once a successful build confirms that clean creates the symlink correctly. --- .github/workflows/build-kernel.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index c7ca5879c153e..d707ca040e31f 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -177,6 +177,16 @@ jobs: echo "=== Setting up build environment (debian/rules clean) ===" fakeroot make -f debian/rules clean + # Safety net: ensure debian/changelog symlink exists. + # The clean target should create it as part of standard setup, + # but if it does not (container environment difference), the + # build would fail at dh_installchangelogs after 2+ hours. + # This guard is a no-op when clean already created the symlink. + if [ ! -f debian/changelog ] && [ -f debian.master/changelog ]; then + ln -sf ../debian.master/changelog debian/changelog + echo "Safety net: created debian/changelog → debian.master/changelog" + fi + echo "=== Starting: fakeroot debian/rules ${TARGET} (jobs=${JOBS}) ===" export DEB_BUILD_OPTIONS="parallel=${JOBS} nocheck" # do_skip_checks=true skips the config policy check that requires From be5a12b69932d66403c017760a20836814b813fd Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 23 Apr 2026 14:55:35 -0700 Subject: [PATCH 042/115] ci: add runner input with three options to build-kernel.yml Add a runner dropdown to the workflow_dispatch inputs so the build can be directed to any of the three available runners without editing the workflow file: - ubuntu-24.04-arm (GitHub-hosted, 2-core, default) - bcollaku-self-hosted (interim personal dev runner) - lecore-prd-u2404-arm64-xlrg-od-ephem (production, pending access) The fetch-source-pkg.yml trigger-build job passes runner=ubuntu-24.04-arm for scheduled/automatic builds so they always use the GitHub-hosted runner by default. The S3 upload step remains gated on runner.name == 'lecore-prd-u2404-arm64-xlrg-od-ephem' and fires only on the production runner. --- .github/workflows/build-kernel.yml | 11 ++++++++++- .github/workflows/fetch-source-pkg.yml | 3 ++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index d707ca040e31f..fa43070da146c 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -45,6 +45,15 @@ on: - generic - lowlatency - all + runner: + description: "Runner to use for the build" + required: false + default: "ubuntu-24.04-arm" + type: choice + options: + - ubuntu-24.04-arm + - bcollaku-self-hosted + - lecore-prd-u2404-arm64-xlrg-od-ephem permissions: contents: write # upload assets to release tag @@ -55,7 +64,7 @@ jobs: # -------------------------------------------------------------------------- build: name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" - runs-on: ubuntu-24.04-arm + runs-on: ${{ inputs.runner || 'ubuntu-24.04-arm' }} timeout-minutes: 360 env: diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 406d760d08432..a14f51a33d7b7 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -316,6 +316,7 @@ jobs: --field suite="${SUITE}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ - --field flavor="generic" + --field flavor="generic" \ + --field runner="ubuntu-24.04-arm" echo "Build workflow dispatched for ${SUITE} ${VERSION}" From c8e4ff5c59cfe224b5f4ed62ce72bae0dd1f5b55 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 23 Apr 2026 15:01:52 -0700 Subject: [PATCH 043/115] ci: add runner input with fromJSON mapping to build-kernel.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a runner dropdown to workflow_dispatch inputs with three options: ubuntu-24.04-arm → runs-on: ubuntu-24.04-arm (GitHub-hosted, default) self-hosted → runs-on: self-hosted (bcollaku dev machine) lecore-production → runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] A fromJSON lookup map resolves the friendly input name to the correct runs-on value at workflow evaluation time, handling both the string form (GitHub-hosted / simple self-hosted) and the array form required by the production runner group. The fetch-source-pkg.yml trigger-build job passes runner=ubuntu-24.04-arm so scheduled/automatic builds always use the GitHub-hosted runner. --- .github/workflows/build-kernel.yml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index fa43070da146c..79fe15bddd24b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -52,8 +52,8 @@ on: type: choice options: - ubuntu-24.04-arm - - bcollaku-self-hosted - - lecore-prd-u2404-arm64-xlrg-od-ephem + - self-hosted + - lecore-production permissions: contents: write # upload assets to release tag @@ -64,7 +64,11 @@ jobs: # -------------------------------------------------------------------------- build: name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" - runs-on: ${{ inputs.runner || 'ubuntu-24.04-arm' }} + # Map friendly runner names to their actual runs-on labels. + # self-hosted → "self-hosted" (bcollaku dev machine) + # lecore-production → ["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"] + # ubuntu-24.04-arm → "ubuntu-24.04-arm" (GitHub-hosted, default) + runs-on: ${{ fromJSON('{"ubuntu-24.04-arm":"ubuntu-24.04-arm","self-hosted":"self-hosted","lecore-production":["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"]}')[inputs.runner || 'ubuntu-24.04-arm'] }} timeout-minutes: 360 env: From d546e351f06d3c77930c3dcb32ec0394c89e74a1 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 23 Apr 2026 15:10:50 -0700 Subject: [PATCH 044/115] ci: add runner input dropdown with three options to build-kernel.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a runner input to workflow_dispatch so the build can be directed to any of three runners without editing the workflow file: ubuntu-24.04-arm → GitHub-hosted 2-core arm64 (default) self-hosted → runs-on: self-hosted (interim dev runner) lecore-production → runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] A fromJSON lookup map resolves the friendly input name to the correct runs-on value, handling both the string form (GitHub-hosted / simple self-hosted) and the array form required by the production runner group. The fetch-source-pkg.yml trigger-build job passes runner=ubuntu-24.04-arm so scheduled/automatic builds always use the GitHub-hosted runner. The S3 upload step remains gated on runner.name == 'lecore-prd-u2404-arm64-xlrg-od-ephem' and fires only on the production runner. Self-hosted runner requirement: Docker socket access is needed even for steps that are conditionally skipped, because GitHub Actions pre-builds Docker-based actions at job startup. Fix: sudo chmod 666 /var/run/docker.sock or add the runner user to the docker group. Update README with runner options table, current default, target production runner, and self-hosted Docker requirements. --- README.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/README.md b/README.md index 5b45ed025edca..74bcd37596158 100644 --- a/README.md +++ b/README.md @@ -183,6 +183,22 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | +| `runner` | `ubuntu-24.04-arm` | Runner to use — see table below | + +**Runner options**: + +| Option | Resolves to | Status | +|--------|-------------|--------| +| `ubuntu-24.04-arm` | GitHub-hosted 2-core arm64 | **default** — used by scheduled builds | +| `self-hosted` | `runs-on: self-hosted` — any registered self-hosted runner | interim dev runner | +| `lecore-production` | `runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem]` | **target** — pending runner group access | + +The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The `lecore-production` runner enables S3 artifact upload (permanent storage) in addition to the GitHub Actions artifact fallback. + +**Self-hosted runner requirements:** +- Ubuntu 24.04 arm64 +- Docker installed; runner user must have access to `/var/run/docker.sock` (add user to `docker` group or `sudo chmod 666 /var/run/docker.sock`) +- ≥ 25 GB free disk space **Build steps**: 1. Free up disk space (~10 GB) From 0caf483ebc72c593acf29e1c52bc0c172414f6bb Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 14:08:28 -0700 Subject: [PATCH 045/115] ci: validate kernel_version tag exists before starting build If kernel_version is provided, verify that the corresponding tag - already exists in the repository before starting the build. This prevents a mislabeled release caused by a user entering a version that was never synced. Without this check, a user entering a non-existent version would trigger a full build that produces correctly-built .deb files but attaches them to a mislabeled release backed by a dangling git tag pointing to the wrong commit. The validation uses authenticated git ls-remote and fails fast (within seconds) with a clear error message pointing to the tags page if the version was never synced. --- .github/workflows/build-kernel.yml | 45 ++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 79fe15bddd24b..177eaf76ae66e 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -117,7 +117,36 @@ jobs: echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 4. Checkout docker-pkg-build + # 4. Validate kernel_version against existing tags (fail fast) + # If kernel_version is provided, verify that the corresponding tag + # - already exists in the repository before + # starting the build. This prevents a mislabeled release caused by + # a user entering a version that was never synced. + # ----------------------------------------------------------------------- + - name: Validate kernel_version tag exists + if: inputs.kernel_version != '' + env: + GH_TOKEN: ${{ github.token }} + run: | + TAG="${SUITE}-${{ inputs.kernel_version }}" + echo "Verifying tag '${TAG}' exists in repository..." + + if git ls-remote --exit-code --tags \ + "https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" \ + "refs/tags/${TAG}" >/dev/null 2>&1; then + echo "✓ Tag '${TAG}' exists — proceeding with build" + else + echo "ERROR: Tag '${TAG}' does not exist in this repository." >&2 + echo "" >&2 + echo "The kernel_version input must match a version that has been" >&2 + echo "synced to the ${SUITE} branch by fetch-source-pkg.yml." >&2 + echo "" >&2 + echo "Existing tags: https://github.com/${{ github.repository }}/tags" >&2 + exit 1 + fi + + # ----------------------------------------------------------------------- + # 5. Checkout docker-pkg-build # ----------------------------------------------------------------------- - name: Checkout docker-pkg-build uses: actions/checkout@v6 @@ -127,7 +156,7 @@ jobs: path: docker-pkg-build # ----------------------------------------------------------------------- - # 5. Build the suite-matched pkg-builder docker image + # 6. Build the suite-matched pkg-builder docker image # ----------------------------------------------------------------------- - name: Build docker image for suite (${{ inputs.suite || 'noble' }}) run: | @@ -135,7 +164,7 @@ jobs: docker image ls # ----------------------------------------------------------------------- - # 6. Build kernel packages inside the suite-matched container + # 7. Build kernel packages inside the suite-matched container # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) @@ -219,7 +248,7 @@ jobs: ' # ----------------------------------------------------------------------- - # 7. Collect output packages + # 8. Collect output packages # The Ubuntu kernel build drops .deb files one level above the # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). # ----------------------------------------------------------------------- @@ -235,7 +264,7 @@ jobs: ls -lh output/ || echo "(no files found)" # ----------------------------------------------------------------------- - # 8. Upload to S3 (self-hosted runner only) + # 9. Upload to S3 (self-hosted runner only) # Skipped on GitHub-hosted runners — S3 credentials and network # access are only available on lecore-prd-u2404-arm64-xlrg-od-ephem. # ----------------------------------------------------------------------- @@ -248,7 +277,7 @@ jobs: destination: ${{ env.ORG_NAME }}/pkg/temp/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ # ----------------------------------------------------------------------- - # 9. Upload as GitHub Actions artifact (90-day fallback) + # 10. Upload as GitHub Actions artifact (90-day fallback) # ----------------------------------------------------------------------- - name: Upload packages as artifact uses: actions/upload-artifact@v7 @@ -259,7 +288,7 @@ jobs: if-no-files-found: warn # ----------------------------------------------------------------------- - # 10. Create GitHub Release and attach built packages + # 11. Create GitHub Release and attach built packages # ----------------------------------------------------------------------- - name: Create GitHub Release and attach packages if: inputs.kernel_version != '' @@ -299,7 +328,7 @@ jobs: echo "Release URL: https://github.com/${{ github.repository }}/releases/tag/${TAG}" # ----------------------------------------------------------------------- - # 11. Summary + # 12. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() From e722b7373975e416af09400d34755cee90dbcacd Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 14:17:33 -0700 Subject: [PATCH 046/115] ci: switch daily default suite from noble to resolute Update the default suite for all scheduled and manual builds from noble (24.04 LTS) to resolute (25.10). Changes: - fetch-source-pkg.yml: suite input default and SUITE env fallback - build-kernel.yml: suite input default, SUITE env fallback, and all step name references - README.md: pipeline diagram, inputs tables, KERNEL_SUITE variable description, supported suites table (resolute marked as daily default), and release asset example Noble remains active and can still be synced/built by specifying suite=noble explicitly. --- .github/workflows/build-kernel.yml | 12 ++++----- .github/workflows/fetch-source-pkg.yml | 4 +-- README.md | 36 +++++++++++++------------- 3 files changed, 26 insertions(+), 26 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 177eaf76ae66e..19d75859db0ed 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -22,7 +22,7 @@ on: suite: description: "Suite branch to build from (e.g. noble, questing, resolute)" required: true - default: "noble" + default: "resolute" type: string kernel_version: description: "Kernel version string (e.g. 6.8.0-51.52) – used for release asset attachment" @@ -72,7 +72,7 @@ jobs: timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'noble' }} + SUITE: ${{ inputs.suite || 'resolute' }} ARCH: ${{ inputs.arch || 'arm64' }} FLAVOR: ${{ inputs.flavor || 'generic' }} @@ -102,10 +102,10 @@ jobs: # ----------------------------------------------------------------------- # 2. Checkout the suite branch (contains the full kernel source tree) # ----------------------------------------------------------------------- - - name: Checkout suite branch (${{ inputs.suite || 'noble' }}) + - name: Checkout suite branch (${{ inputs.suite || 'resolute' }}) uses: actions/checkout@v6 with: - ref: ${{ inputs.suite || 'noble' }} + ref: ${{ inputs.suite || 'resolute' }} path: kernel-src # ----------------------------------------------------------------------- @@ -158,7 +158,7 @@ jobs: # ----------------------------------------------------------------------- # 6. Build the suite-matched pkg-builder docker image # ----------------------------------------------------------------------- - - name: Build docker image for suite (${{ inputs.suite || 'noble' }}) + - name: Build docker image for suite (${{ inputs.suite || 'resolute' }}) run: | ./docker-pkg-build/docker_deb_build.py --rebuild -d "${SUITE}" docker image ls @@ -167,7 +167,7 @@ jobs: # 7. Build kernel packages inside the suite-matched container # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'noble' }}) + - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'resolute' }}) run: | JOBS=$(nproc) if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index a14f51a33d7b7..f66cf0df27442 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -37,7 +37,7 @@ on: suite: description: "Ubuntu suite to sync (e.g. noble, questing, resolute)" required: false - default: "noble" + default: "resolute" type: string force: description: "Force sync even if this version is already committed" @@ -69,7 +69,7 @@ jobs: - name: Query Launchpad for latest published source id: query env: - SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'noble' }} + SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} run: | API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" diff --git a/README.md b/README.md index 74bcd37596158..26e50224107b6 100644 --- a/README.md +++ b/README.md @@ -21,9 +21,9 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 1 · check-version │ ║ ║ │ │ ║ ║ │ Query Launchpad API (ws.size=300, exact source_package_name match) │ ║ -║ │ → latest version: noble 6.8.0-114.114 │ ║ +║ │ → latest version: resolute X.Y.Z-A.B │ ║ ║ │ │ ║ -║ │ git ls-remote (authenticated) → tag noble-6.8.0-114.114 exists? │ ║ +║ │ git ls-remote (authenticated) → tag resolute-X.Y.Z-A.B exists? │ ║ ║ │ │ ║ ║ │ YES ──▶ should_sync=false ──▶ workflow exits cleanly │ ║ ║ │ NO ──▶ should_sync=true ──▶ continue ↓ │ ║ @@ -34,10 +34,10 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 2 · sync │ ║ ║ │ │ ║ ║ │ Free disk space (~10 GB) │ ║ -║ │ git clone --depth=1 Launchpad git @ Ubuntu-6.8.0-114.114 │ ║ +║ │ git clone --depth=1 Launchpad git @ Ubuntu-X.Y.Z-A.B │ ║ ║ │ Verify >5000 files cloned │ ║ -║ │ rsync source → noble branch (orphan) │ ║ -║ │ git commit + tag noble-6.8.0-114.114 │ ║ +║ │ rsync source → resolute branch (orphan) │ ║ +║ │ git commit + tag resolute-X.Y.Z-A.B │ ║ ║ │ git push branch + tag │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ║ │ sync succeeded ║ @@ -46,7 +46,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 3 · trigger-build │ ║ ║ │ │ ║ ║ │ gh workflow run build-kernel.yml │ ║ -║ │ suite=noble kernel_version=6.8.0-114.114 arch=arm64 │ ║ +║ │ suite=resolute kernel_version=X.Y.Z-A.B arch=arm64 │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ │ @@ -54,12 +54,12 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ╔════════════════════════════════════════════════════════════════════════════╗ ║ build-kernel.yml ║ ║ ║ -║ Checkout noble branch ──▶ kernel-src/ ║ +║ Checkout resolute branch ──▶ kernel-src/ ║ ║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ -║ docker_deb_build.py --rebuild -d noble ║ +║ docker_deb_build.py --rebuild -d resolute ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:noble │ ║ +║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:resolute│ ║ ║ │ │ ║ ║ │ apt-get build-dep linux │ ║ ║ │ fakeroot make -f debian/rules clean ← setup env │ ║ @@ -74,8 +74,8 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ S3 Bucket │ │ GitHub Artifact │ │ GitHub Release │ │ │ │ │ │ │ - │ qli-prd- │ │ 90-day retention │ │ noble-6.8.0- │ - │ lecore-gh- │ │ Actions → run │ │ 114.114 │ + │ qli-prd- │ │ 90-day retention │ │ resolute-X.Y.Z- │ + │ lecore-gh- │ │ Actions → run │ │ A.B │ │ artifacts │ │ → Artifacts │ │ Releases → │ │ │ │ │ │ Assets │ │ self-hosted │ │ always │ │ permanent │ @@ -150,7 +150,7 @@ suite branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `noble` | Ubuntu suite to sync — one suite per run | +| `suite` | `resolute` | Ubuntu suite to sync — one suite per run | | `force` | `false` | Re-sync even if tag already exists | **Jobs**: @@ -161,7 +161,7 @@ suite branch. | `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from Launchpad git; verifies >5000 files; commits to suite branch; creates tag | | `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=generic` | -**Idempotent**: if tag `noble-6.8.0-114.114` already exists, the workflow exits cleanly without downloading anything. +**Idempotent**: if the tag for the latest version already exists, the workflow exits cleanly without downloading anything. --- @@ -179,7 +179,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `suite` | `noble` | Suite branch to build from | +| `suite` | `resolute` | Suite branch to build from | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | @@ -220,7 +220,7 @@ The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The ` |----------|---------------|-----------|-------| | **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | Self-hosted runner only; skipped gracefully on GitHub-hosted | | **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | Always available | -| **GitHub Release asset** | Releases → `noble-6.8.0-X.Y` → Assets | Permanent | Attached when `kernel_version` is provided | +| **GitHub Release asset** | Releases → `-X.Y.Z-A.B` → Assets | Permanent | Attached when `kernel_version` is provided | --- @@ -236,7 +236,7 @@ Go to **Actions** and enable workflows if prompted. | Variable | Default | Description | |----------|---------|-------------| -| `KERNEL_SUITE` | `noble` | Default suite for scheduled runs | +| `KERNEL_SUITE` | `resolute` | Default suite for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | ### 3. Run the first sync @@ -378,7 +378,7 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | | `questing` | Questing Quokka | 25.04 — **active** | TBD | -| `resolute` | Resolute Ringtail | 25.10 — add when available | TBD | +| `resolute` | Resolute Ringtail | 25.10 — **active** (daily default) | TBD | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: @@ -386,7 +386,7 @@ To add a new suite, trigger `fetch-source-pkg.yml` with the desired ```bash gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=questing + --field suite=noble ``` --- From 60ac7d9cac0246a098175d892fba5b7849a1c7b8 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 14:36:58 -0700 Subject: [PATCH 047/115] ci: checkout from tag when kernel_version is provided MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When kernel_version is specified, the build now checks out the exact tag - rather than the suite branch HEAD. Previously, the build always checked out the branch HEAD regardless of kernel_version. This meant: - Building with kernel_version=7.0.0-14.14 while the branch HEAD was at 7.0.0-15.15 would produce packages from the wrong source but label the release as 7.0.0-14.14. With this fix: - kernel_version provided → checkout refs/tags/- (exact synced source, correct for any version including older ones) - kernel_version empty → checkout suite branch HEAD (test/dev builds, no release created) The three manual build modes are now semantically clean: Mode A: suite=resolute, kernel_version= → branch HEAD, artifact only (dev/test) Mode B: suite=resolute, kernel_version=7.0.0-15.15 → tag resolute-7.0.0-15.15, creates release Mode C: suite=resolute, kernel_version=7.0.0-14.14 → tag resolute-7.0.0-14.14, rebuilds older release Add Manual build trigger flows section to README with diagrams for all three modes. --- .github/workflows/build-kernel.yml | 11 +++-- README.md | 64 ++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 19d75859db0ed..b2fee8806b915 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -100,12 +100,17 @@ jobs: df -h / # ----------------------------------------------------------------------- - # 2. Checkout the suite branch (contains the full kernel source tree) + # 2. Checkout the kernel source + # - If kernel_version is provided: checkout the exact tag + # - so the build uses the precise synced + # source regardless of what else is on the branch tip. + # - If kernel_version is empty: checkout the suite branch HEAD + # (test/dev builds — no release will be created). # ----------------------------------------------------------------------- - - name: Checkout suite branch (${{ inputs.suite || 'resolute' }}) + - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.suite || 'resolute' }} + ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute', inputs.kernel_version) || inputs.suite || 'resolute' }} path: kernel-src # ----------------------------------------------------------------------- diff --git a/README.md b/README.md index 26e50224107b6..ba61aa58272b0 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,70 @@ Target runner: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted) — pending --- +## Manual build trigger flows + +``` +MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow +══════════════════════════════════════════════════════════════════════════════ + + Three modes depending on kernel_version input: + + ┌─────────────────────────────────────────────────────────────────────────┐ + │ Mode A — Test / dev build (kernel_version left empty) │ + │ │ + │ suite=resolute kernel_version= │ + │ │ │ + │ ▼ │ + │ Checkout resolute branch HEAD │ + │ (includes any commits you pushed on top of the synced source) │ + │ │ │ + │ ▼ │ + │ Build .deb packages │ + │ │ │ + │ ▼ │ + │ GitHub Actions artifact only (90-day) ← no release created │ + └─────────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────────┐ + │ Mode B — Release build for latest synced version │ + │ │ + │ suite=resolute kernel_version=7.0.0-15.15 │ + │ │ │ + │ ▼ │ + │ Validate tag resolute-7.0.0-15.15 exists (fail fast if not) │ + │ │ │ + │ ▼ │ + │ Checkout tag resolute-7.0.0-15.15 ← exact synced source │ + │ │ │ + │ ▼ │ + │ Build .deb packages │ + │ │ │ + │ ▼ │ + │ GitHub Actions artifact (90-day) + GitHub Release resolute-7.0.0-15.15│ + └─────────────────────────────────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────────────────────────────────┐ + │ Mode C — Rebuild / re-release an older synced version │ + │ │ + │ suite=resolute kernel_version=7.0.0-14.14 │ + │ │ │ + │ ▼ │ + │ Validate tag resolute-7.0.0-14.14 exists (fail fast if not) │ + │ │ │ + │ ▼ │ + │ Checkout tag resolute-7.0.0-14.14 ← older synced source (not HEAD) │ + │ │ │ + │ ▼ │ + │ Build .deb packages │ + │ │ │ + │ ▼ │ + │ GitHub Actions artifact (90-day) + GitHub Release resolute-7.0.0-14.14│ + │ (existing release assets are overwritten with --clobber) │ + └─────────────────────────────────────────────────────────────────────────┘ +``` + +--- + ## Repository branch layout ``` From 88f24fb64d749139d73241209a33abf1f0204cfc Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 14:45:45 -0700 Subject: [PATCH 048/115] ci: move tag validation before checkout and clarify checkout comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tag validation step must run before the kernel source checkout. Previously, if kernel_version was invalid, the checkout would fail with a cryptic git 'ref not found' error before the validation step could produce a helpful message. New step order: 1. Free disk space 2. Extract org/repo 3. Validate kernel_version tag (fail fast, clear error) 4. Checkout kernel source (tag or branch HEAD) ... Also add a comment to the checkout step clarifying that actions/checkout is checking out a suite branch (resolute, noble, etc.) of this repository — not the main branch. The suite branches contain the full kernel source tree populated by fetch-source-pkg.yml. --- .github/workflows/build-kernel.yml | 42 ++++++++++++++++-------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index b2fee8806b915..e9f5abfa922db 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -100,21 +100,7 @@ jobs: df -h / # ----------------------------------------------------------------------- - # 2. Checkout the kernel source - # - If kernel_version is provided: checkout the exact tag - # - so the build uses the precise synced - # source regardless of what else is on the branch tip. - # - If kernel_version is empty: checkout the suite branch HEAD - # (test/dev builds — no release will be created). - # ----------------------------------------------------------------------- - - name: Checkout kernel source - uses: actions/checkout@v6 - with: - ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute', inputs.kernel_version) || inputs.suite || 'resolute' }} - path: kernel-src - - # ----------------------------------------------------------------------- - # 3. Extract org/repo for S3 upload path + # 2. Extract org/repo for S3 upload path # ----------------------------------------------------------------------- - name: Extract org and repo name run: | @@ -122,11 +108,9 @@ jobs: echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 4. Validate kernel_version against existing tags (fail fast) - # If kernel_version is provided, verify that the corresponding tag - # - already exists in the repository before - # starting the build. This prevents a mislabeled release caused by - # a user entering a version that was never synced. + # 3. Validate kernel_version against existing tags (fail fast) + # Must run BEFORE checkout so a bad version fails immediately with a + # clear error rather than a cryptic git "ref not found" from checkout. # ----------------------------------------------------------------------- - name: Validate kernel_version tag exists if: inputs.kernel_version != '' @@ -150,6 +134,24 @@ jobs: exit 1 fi + # ----------------------------------------------------------------------- + # 4. Checkout the kernel source + # - If kernel_version is provided: checkout the exact tag + # - so the build uses the precise synced + # source regardless of what else is on the branch tip. + # - If kernel_version is empty: checkout the suite branch HEAD + # (test/dev builds — no release will be created). + # Note: actions/checkout checks out a ref of THIS repository + # (pkg-linux-qcom-canonical). The resolute/noble/questing branches + # of this repo contain the full kernel source tree populated by + # fetch-source-pkg.yml — they are not the main branch. + # ----------------------------------------------------------------------- + - name: Checkout kernel source + uses: actions/checkout@v6 + with: + ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute', inputs.kernel_version) || inputs.suite || 'resolute' }} + path: kernel-src + # ----------------------------------------------------------------------- # 5. Checkout docker-pkg-build # ----------------------------------------------------------------------- From 43b0b409263034fcf57c08a3d960294cee0ef27a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 15:59:22 -0700 Subject: [PATCH 049/115] ci: switch daily default suite from resolute to questing The resolute (25.10) kernel 7.0.0 fails during dtbs_install with a parallel build race condition in Makefile.dtbinst: install: cannot create directory .../device-tree/apm make[4]: *** [.../Makefile.dtbinst:23: .../apm-mustang.dtb] Error 1 Multiple parallel jobs race to create the same device-tree subdirectory. This is a kernel-side bug that requires an upstream fix from Canonical. Switch the daily default back to questing (25.04) which builds successfully. Resolute remains available for manual builds and is documented as having a known build issue. --- .github/workflows/build-kernel.yml | 10 ++--- .github/workflows/fetch-source-pkg.yml | 4 +- README.md | 55 ++++++++++++++------------ 3 files changed, 37 insertions(+), 32 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index e9f5abfa922db..843e91f92e66d 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -22,7 +22,7 @@ on: suite: description: "Suite branch to build from (e.g. noble, questing, resolute)" required: true - default: "resolute" + default: "questing" type: string kernel_version: description: "Kernel version string (e.g. 6.8.0-51.52) – used for release asset attachment" @@ -72,7 +72,7 @@ jobs: timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'resolute' }} + SUITE: ${{ inputs.suite || 'questing' }} ARCH: ${{ inputs.arch || 'arm64' }} FLAVOR: ${{ inputs.flavor || 'generic' }} @@ -149,7 +149,7 @@ jobs: - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute', inputs.kernel_version) || inputs.suite || 'resolute' }} + ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'questing', inputs.kernel_version) || inputs.suite || 'questing' }} path: kernel-src # ----------------------------------------------------------------------- @@ -165,7 +165,7 @@ jobs: # ----------------------------------------------------------------------- # 6. Build the suite-matched pkg-builder docker image # ----------------------------------------------------------------------- - - name: Build docker image for suite (${{ inputs.suite || 'resolute' }}) + - name: Build docker image for suite (${{ inputs.suite || 'questing' }}) run: | ./docker-pkg-build/docker_deb_build.py --rebuild -d "${SUITE}" docker image ls @@ -174,7 +174,7 @@ jobs: # 7. Build kernel packages inside the suite-matched container # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'resolute' }}) + - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'questing' }}) run: | JOBS=$(nproc) if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index f66cf0df27442..0f3854ae4ea79 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -37,7 +37,7 @@ on: suite: description: "Ubuntu suite to sync (e.g. noble, questing, resolute)" required: false - default: "resolute" + default: "questing" type: string force: description: "Force sync even if this version is already committed" @@ -69,7 +69,7 @@ jobs: - name: Query Launchpad for latest published source id: query env: - SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute' }} + SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'questing' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} run: | API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" diff --git a/README.md b/README.md index ba61aa58272b0..77cf99676ec0d 100644 --- a/README.md +++ b/README.md @@ -21,9 +21,9 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 1 · check-version │ ║ ║ │ │ ║ ║ │ Query Launchpad API (ws.size=300, exact source_package_name match) │ ║ -║ │ → latest version: resolute X.Y.Z-A.B │ ║ +║ │ → latest version: questing X.Y.Z-A.B │ ║ ║ │ │ ║ -║ │ git ls-remote (authenticated) → tag resolute-X.Y.Z-A.B exists? │ ║ +║ │ git ls-remote (authenticated) → tag questing-X.Y.Z-A.B exists? │ ║ ║ │ │ ║ ║ │ YES ──▶ should_sync=false ──▶ workflow exits cleanly │ ║ ║ │ NO ──▶ should_sync=true ──▶ continue ↓ │ ║ @@ -36,8 +36,8 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Free disk space (~10 GB) │ ║ ║ │ git clone --depth=1 Launchpad git @ Ubuntu-X.Y.Z-A.B │ ║ ║ │ Verify >5000 files cloned │ ║ -║ │ rsync source → resolute branch (orphan) │ ║ -║ │ git commit + tag resolute-X.Y.Z-A.B │ ║ +║ │ rsync source → questing branch (orphan) │ ║ +║ │ git commit + tag questing-X.Y.Z-A.B │ ║ ║ │ git push branch + tag │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ║ │ sync succeeded ║ @@ -46,7 +46,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 3 · trigger-build │ ║ ║ │ │ ║ ║ │ gh workflow run build-kernel.yml │ ║ -║ │ suite=resolute kernel_version=X.Y.Z-A.B arch=arm64 │ ║ +║ │ suite=questing kernel_version=X.Y.Z-A.B arch=arm64 │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ │ @@ -54,12 +54,12 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ╔════════════════════════════════════════════════════════════════════════════╗ ║ build-kernel.yml ║ ║ ║ -║ Checkout resolute branch ──▶ kernel-src/ ║ +║ Checkout questing branch ──▶ kernel-src/ ║ ║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ -║ docker_deb_build.py --rebuild -d resolute ║ +║ docker_deb_build.py --rebuild -d questing ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:resolute│ ║ +║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:questing│ ║ ║ │ │ ║ ║ │ apt-get build-dep linux │ ║ ║ │ fakeroot make -f debian/rules clean ← setup env │ ║ @@ -74,7 +74,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ S3 Bucket │ │ GitHub Artifact │ │ GitHub Release │ │ │ │ │ │ │ - │ qli-prd- │ │ 90-day retention │ │ resolute-X.Y.Z- │ + │ qli-prd- │ │ 90-day retention │ │ questing-X.Y.Z- │ │ lecore-gh- │ │ Actions → run │ │ A.B │ │ artifacts │ │ → Artifacts │ │ Releases → │ │ │ │ │ │ Assets │ @@ -99,10 +99,10 @@ MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode A — Test / dev build (kernel_version left empty) │ │ │ - │ suite=resolute kernel_version= │ + │ suite=questing kernel_version= │ │ │ │ │ ▼ │ - │ Checkout resolute branch HEAD │ + │ Checkout questing branch HEAD │ │ (includes any commits you pushed on top of the synced source) │ │ │ │ │ ▼ │ @@ -115,37 +115,37 @@ MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode B — Release build for latest synced version │ │ │ - │ suite=resolute kernel_version=7.0.0-15.15 │ + │ suite=questing kernel_version=6.14.0-15.15 │ │ │ │ │ ▼ │ - │ Validate tag resolute-7.0.0-15.15 exists (fail fast if not) │ + │ Validate tag questing-6.14.0-15.15 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag resolute-7.0.0-15.15 ← exact synced source │ + │ Checkout tag questing-6.14.0-15.15 ← exact synced source │ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release resolute-7.0.0-15.15│ + │ GitHub Actions artifact (90-day) + GitHub Release questing-6.14.0-15.15│ └─────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode C — Rebuild / re-release an older synced version │ │ │ - │ suite=resolute kernel_version=7.0.0-14.14 │ + │ suite=questing kernel_version=6.14.0-14.14 │ │ │ │ │ ▼ │ - │ Validate tag resolute-7.0.0-14.14 exists (fail fast if not) │ + │ Validate tag questing-6.14.0-14.14 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag resolute-7.0.0-14.14 ← older synced source (not HEAD) │ + │ Checkout tag questing-6.14.0-14.14 ← older synced source (not HEAD) │ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release resolute-7.0.0-14.14│ + │ GitHub Actions artifact (90-day) + GitHub Release questing-6.14.0-14.14│ │ (existing release assets are overwritten with --clobber) │ └─────────────────────────────────────────────────────────────────────────┘ ``` @@ -214,7 +214,7 @@ suite branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `resolute` | Ubuntu suite to sync — one suite per run | +| `suite` | `questing` | Ubuntu suite to sync — one suite per run | | `force` | `false` | Re-sync even if tag already exists | **Jobs**: @@ -243,7 +243,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `suite` | `resolute` | Suite branch to build from | +| `suite` | `questing` | Suite branch to build from | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | @@ -300,7 +300,7 @@ Go to **Actions** and enable workflows if prompted. | Variable | Default | Description | |----------|---------|-------------| -| `KERNEL_SUITE` | `resolute` | Default suite for scheduled runs | +| `KERNEL_SUITE` | `questing` | Default suite for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | ### 3. Run the first sync @@ -441,8 +441,8 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. | Suite | Codename | Status | Kernel | |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.04 — **active** | TBD | -| `resolute` | Resolute Ringtail | 25.10 — **active** (daily default) | TBD | +| `questing` | Questing Quokka | 25.04 — **active** (daily default) | TBD | +| `resolute` | Resolute Ringtail | 25.10 — **active** (known DTB build issue) | TBD | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: @@ -450,9 +450,14 @@ To add a new suite, trigger `fetch-source-pkg.yml` with the desired ```bash gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=noble + --field suite=resolute ``` +> **Note on resolute builds:** The resolute (25.10) kernel 7.0.0 fails during +> `dtbs_install` with a parallel build race condition +> (`install: cannot create directory .../device-tree/apm`). +> Use `suite=questing` or `suite=noble` for reliable builds until this is resolved. + --- ## Build container notes From 96ce5b78acfe2fdc614010187f9336fd8e8a3e4b Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 16:03:58 -0700 Subject: [PATCH 050/115] docs: update manual trigger flow examples with actual questing versions Replace placeholder 6.14.x version numbers with the actual questing kernel versions currently synced: 6.17.0-24.24 (latest) and 6.17.0-23.23 (older example for Mode C rebuild). --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 77cf99676ec0d..b0782d069e1bc 100644 --- a/README.md +++ b/README.md @@ -115,37 +115,37 @@ MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode B — Release build for latest synced version │ │ │ - │ suite=questing kernel_version=6.14.0-15.15 │ + │ suite=questing kernel_version=6.17.0-24.24 │ │ │ │ │ ▼ │ - │ Validate tag questing-6.14.0-15.15 exists (fail fast if not) │ + │ Validate tag questing-6.17.0-24.24 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag questing-6.14.0-15.15 ← exact synced source │ + │ Checkout tag questing-6.17.0-24.24 ← exact synced source │ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release questing-6.14.0-15.15│ + │ GitHub Actions artifact (90-day) + GitHub Release questing-6.17.0-24.24│ └─────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode C — Rebuild / re-release an older synced version │ │ │ - │ suite=questing kernel_version=6.14.0-14.14 │ + │ suite=questing kernel_version=6.17.0-23.23 │ │ │ │ │ ▼ │ - │ Validate tag questing-6.14.0-14.14 exists (fail fast if not) │ + │ Validate tag questing-6.17.0-23.23 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag questing-6.14.0-14.14 ← older synced source (not HEAD) │ + │ Checkout tag questing-6.17.0-23.23 ← older synced source (not HEAD) │ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release questing-6.14.0-14.14│ + │ GitHub Actions artifact (90-day) + GitHub Release questing-6.17.0-23.23│ │ (existing release assets are overwritten with --clobber) │ └─────────────────────────────────────────────────────────────────────────┘ ``` From c76cfb66cfcb19e0fbc2cde55de29fc8e0edf23c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 16:06:13 -0700 Subject: [PATCH 051/115] docs: update kernel versions in supported suites table questing: 6.17 (confirmed from tag questing-6.17.0-24.24) resolute: 7.0 (confirmed from tag resolute-7.0.0-15.15) Also update manual trigger flow examples to use actual questing 6.17 versions. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b0782d069e1bc..5ecc9a5ab1b32 100644 --- a/README.md +++ b/README.md @@ -441,8 +441,8 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. | Suite | Codename | Status | Kernel | |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.04 — **active** (daily default) | TBD | -| `resolute` | Resolute Ringtail | 25.10 — **active** (known DTB build issue) | TBD | +| `questing` | Questing Quokka | 25.04 — **active** (daily default) | 6.17 | +| `resolute` | Resolute Ringtail | 25.10 — **active** (known DTB build issue) | 7.0 | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: From ddbb06bcadf69cb175198c89ba1339e9ec67d095 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 16:42:39 -0700 Subject: [PATCH 052/115] docs: fix Ubuntu version numbers for questing and resolute suites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Questing Quokka is Ubuntu 25.10 (not 25.04) and Resolute Ringtail is Ubuntu 26.04 LTS (not 25.10). The previous values were off by one release cycle in the Supported suites table and the resolute build note. - questing: 25.04 → 25.10 - resolute: 25.10 → 26.04 LTS --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5ecc9a5ab1b32..fc8678b583a65 100644 --- a/README.md +++ b/README.md @@ -441,8 +441,8 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. | Suite | Codename | Status | Kernel | |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.04 — **active** (daily default) | 6.17 | -| `resolute` | Resolute Ringtail | 25.10 — **active** (known DTB build issue) | 7.0 | +| `questing` | Questing Quokka | 25.10 — **active** (daily default) | 6.17 | +| `resolute` | Resolute Ringtail | 26.04 LTS — **active** (known DTB build issue) | 7.0 | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: @@ -453,7 +453,7 @@ gh workflow run fetch-source-pkg.yml \ --field suite=resolute ``` -> **Note on resolute builds:** The resolute (25.10) kernel 7.0.0 fails during +> **Note on resolute builds:** The resolute (26.04 LTS) kernel 7.0.0 fails during > `dtbs_install` with a parallel build race condition > (`install: cannot create directory .../device-tree/apm`). > Use `suite=questing` or `suite=noble` for reliable builds until this is resolved. From 2ac305eb474abfd64aff07c46baff6e1400103b6 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 16:46:57 -0700 Subject: [PATCH 053/115] docs: correct resolute dtbs_install note to reflect observed behavior Replace 'known DTB build issue' with accurate description of the intermittent dtbs_install failure observed on resolute builds: - Failure is non-deterministic (some runs succeed, others fail) - Error is consistent with a parallel job race condition in scripts/Makefile.dtbinst, not a resolute-specific defect - More likely to surface on kernels with more DTB files (7.0) but not exclusive to resolute - Workaround: re-run on failure; use questing/noble for reliable builds --- README.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index fc8678b583a65..32dddaf0fb34c 100644 --- a/README.md +++ b/README.md @@ -442,7 +442,7 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | | `questing` | Questing Quokka | 25.10 — **active** (daily default) | 6.17 | -| `resolute` | Resolute Ringtail | 26.04 LTS — **active** (known DTB build issue) | 7.0 | +| `resolute` | Resolute Ringtail | 26.04 LTS — **active** (intermittent `dtbs_install` failure observed) | 7.0 | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: @@ -453,10 +453,19 @@ gh workflow run fetch-source-pkg.yml \ --field suite=resolute ``` -> **Note on resolute builds:** The resolute (26.04 LTS) kernel 7.0.0 fails during -> `dtbs_install` with a parallel build race condition -> (`install: cannot create directory .../device-tree/apm`). -> Use `suite=questing` or `suite=noble` for reliable builds until this is resolved. +> **Note on resolute builds:** Intermittent build failures have been observed on +> resolute (26.04 LTS) during `dtbs_install`: +> ``` +> install: cannot create directory .../device-tree/apm +> ``` +> This error is consistent with a parallel job race condition in the kernel's +> `dtbs_install` target (`scripts/Makefile.dtbinst`) — multiple parallel jobs +> racing to create the same vendor subdirectory. The failure is non-deterministic: +> some runs succeed, others fail. It is not exclusive to resolute; it is more +> likely to surface on kernels with a large number of DTB files (such as 7.0) +> because more parallel `install -d` calls increase the probability of a collision. +> If you hit this failure, re-running the build often succeeds. For consistently +> reliable builds, use `suite=questing` or `suite=noble` until this is resolved. --- From 5f14ee8f8edc65d7074d30a334713fa0b5a062c3 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 24 Apr 2026 17:14:26 -0700 Subject: [PATCH 054/115] docs: fix suite version numbers and clean up resolute note MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - questing: correct Ubuntu version 25.04 → 25.10 - resolute: correct Ubuntu version 25.10 → 26.04 LTS - Remove unsubstantiated dtbs_install failure note for resolute; one observed failure out of two runs is insufficient evidence to document as a known issue --- README.md | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/README.md b/README.md index 32dddaf0fb34c..a38da8c5ae37f 100644 --- a/README.md +++ b/README.md @@ -442,7 +442,7 @@ Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | | `questing` | Questing Quokka | 25.10 — **active** (daily default) | 6.17 | -| `resolute` | Resolute Ringtail | 26.04 LTS — **active** (intermittent `dtbs_install` failure observed) | 7.0 | +| `resolute` | Resolute Ringtail | 26.04 LTS — **active** | 7.0 | To add a new suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: @@ -453,20 +453,6 @@ gh workflow run fetch-source-pkg.yml \ --field suite=resolute ``` -> **Note on resolute builds:** Intermittent build failures have been observed on -> resolute (26.04 LTS) during `dtbs_install`: -> ``` -> install: cannot create directory .../device-tree/apm -> ``` -> This error is consistent with a parallel job race condition in the kernel's -> `dtbs_install` target (`scripts/Makefile.dtbinst`) — multiple parallel jobs -> racing to create the same vendor subdirectory. The failure is non-deterministic: -> some runs succeed, others fail. It is not exclusive to resolute; it is more -> likely to surface on kernels with a large number of DTB files (such as 7.0) -> because more parallel `install -d` calls increase the probability of a collision. -> If you hit this failure, re-running the build often succeeds. For consistently -> reliable builds, use `suite=questing` or `suite=noble` until this is resolved. - --- ## Build container notes From 0ba487028aa877ee718c30ba0ec7a12c62bda886 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 10:48:17 -0700 Subject: [PATCH 055/115] feat(sync): add custom_git_url input to fetch-source-pkg workflow Add an optional custom_git_url workflow_dispatch input that allows syncing from a custom Launchpad git repository instead of the official ~ubuntu-kernel repo. When custom_git_url is set: - The Launchpad REST API is bypassed entirely - The latest Ubuntu-* tag is discovered via git ls-remote on the custom repo (sorted with sort -V to handle version ordering) - The resolved URL is passed through to the sync job via a new git_url job output and used for cloning When custom_git_url is empty, all existing behaviour is unchanged. Scheduled daily runs are not affected. Example usage: gh workflow run fetch-source-pkg.yml --field suite=resolute --field custom_git_url=https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute --- .github/workflows/fetch-source-pkg.yml | 108 ++++++++++++++++++------- README.md | 1 + 2 files changed, 79 insertions(+), 30 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 0f3854ae4ea79..d00519fa11f63 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -23,7 +23,7 @@ # (e.g. noble-6.8.0-114.114). # # Repository variables (Settings → Variables → Actions): -# KERNEL_SUITE – default suite to sync when not specified (default: noble) +# KERNEL_SUITE – default suite to sync when not specified (default: questing) # KERNEL_SOURCE – source package name (default: linux) name: "Sync: Canonical Kernel Sources to Branch" @@ -43,6 +43,14 @@ on: description: "Force sync even if this version is already committed" type: boolean default: false + custom_git_url: + description: > + Custom Launchpad git URL to clone from instead of the official + Ubuntu kernel repo. Leave empty to use the default. + Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute + required: false + default: "" + type: string permissions: contents: write @@ -60,36 +68,73 @@ jobs: tag: ${{ steps.query.outputs.tag }} suite: ${{ steps.query.outputs.suite }} source: ${{ steps.query.outputs.source }} + git_url: ${{ steps.query.outputs.git_url }} should_sync: ${{ steps.gate.outputs.should_sync }} steps: - name: Install dependencies run: sudo apt-get install -y --no-install-recommends jq - - name: Query Launchpad for latest published source + - name: Resolve version and git URL id: query env: - SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'questing' }} - SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} + SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'questing' }} + SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} + CUSTOM_GIT_URL: ${{ inputs.custom_git_url || '' }} run: | - API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" - API+="?ws.op=getPublishedSources" - API+="&source_name=${SOURCE}" - API+="&distro_series=/ubuntu/${SUITE}" - API+="&status=Published" - API+="&order_by_date=true" - API+="&ws.size=300" - - echo "Querying: ${API}" - RESP=$(curl -fsSL "${API}") - - VERSION=$(echo "$RESP" | jq -r \ - --arg name "${SOURCE}" \ - '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') - [ -n "${VERSION}" ] || { - echo "ERROR: No exact match for '${SOURCE}' in ${SUITE}" >&2 - exit 1 - } + if [ -n "${CUSTOM_GIT_URL}" ]; then + # ---------------------------------------------------------------- + # Custom repo path: find the latest Ubuntu-* tag via git ls-remote. + # This bypasses the Launchpad REST API entirely — the version is + # derived directly from the tags present in the custom repository. + # ---------------------------------------------------------------- + echo "Custom git URL provided: ${CUSTOM_GIT_URL}" + echo "Querying tags from custom repo..." + + LATEST_TAG=$(git ls-remote --tags "${CUSTOM_GIT_URL}" 'refs/tags/Ubuntu-*' \ + | grep -v '\^{}' \ + | awk '{print $2}' \ + | sed 's|refs/tags/||' \ + | sort -V \ + | tail -1) + + [ -n "${LATEST_TAG}" ] || { + echo "ERROR: No Ubuntu-* tags found in ${CUSTOM_GIT_URL}" >&2 + exit 1 + } + + # Ubuntu-7.0.0-1.1 → 7.0.0-1.1 + VERSION="${LATEST_TAG#Ubuntu-}" + GIT_URL="${CUSTOM_GIT_URL}" + + echo "Latest tag : ${LATEST_TAG}" + else + # ---------------------------------------------------------------- + # Default path: query the Launchpad REST API for the latest + # officially published source package version, then construct + # the standard Launchpad git URL for the suite. + # ---------------------------------------------------------------- + API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" + API+="?ws.op=getPublishedSources" + API+="&source_name=${SOURCE}" + API+="&distro_series=/ubuntu/${SUITE}" + API+="&status=Published" + API+="&order_by_date=true" + API+="&ws.size=300" + + echo "Querying: ${API}" + RESP=$(curl -fsSL "${API}") + + VERSION=$(echo "$RESP" | jq -r \ + --arg name "${SOURCE}" \ + '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') + [ -n "${VERSION}" ] || { + echo "ERROR: No exact match for '${SOURCE}' in ${SUITE}" >&2 + exit 1 + } + + GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" + fi UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) TAG="${SUITE}-${VERSION}" @@ -99,10 +144,12 @@ jobs: echo "tag=${TAG}" >> "$GITHUB_OUTPUT" echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" + echo "git_url=${GIT_URL}" >> "$GITHUB_OUTPUT" echo "Suite : ${SUITE}" echo "Version : ${VERSION}" echo "Tag : ${TAG}" + echo "Git URL : ${GIT_URL}" - name: Check whether tag already exists id: gate @@ -138,6 +185,7 @@ jobs: TAG: ${{ needs.check-version.outputs.tag }} SUITE: ${{ needs.check-version.outputs.suite }} SOURCE: ${{ needs.check-version.outputs.source }} + GIT_URL: ${{ needs.check-version.outputs.git_url }} steps: # ----------------------------------------------------------------------- @@ -169,18 +217,19 @@ jobs: curl jq rsync # ----------------------------------------------------------------------- - # 3. Clone from Launchpad git at the version tag + # 3. Clone from the resolved git URL at the version tag. + # GIT_URL is either the custom repo URL (when custom_git_url input + # was provided) or the standard Launchpad URL for the suite. # The git repository has the complete debian/ directory including # debian/rules, scripts/, templates/, etc. — unlike the source # package which only ships debian.master/ with rules.d/ fragments. # ----------------------------------------------------------------------- - - name: Clone from Launchpad git at version tag + - name: Clone from git repo at version tag run: | - LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" GIT_TAG="Ubuntu-${VERSION}" - echo "Cloning ${LAUNCHPAD_URL} at tag ${GIT_TAG} (shallow)..." - git clone --depth=1 --branch "${GIT_TAG}" "${LAUNCHPAD_URL}" kernel-src/ + echo "Cloning ${GIT_URL} at tag ${GIT_TAG} (shallow)..." + git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ echo "SRC_DIR=kernel-src" >> "$GITHUB_ENV" @@ -251,7 +300,7 @@ jobs: "Ubuntu suite : ${SUITE}" \ "Kernel version : ${VERSION}" \ "Upstream : ${UPSTREAM_VERSION}" \ - "Source : https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE} (tag: Ubuntu-${VERSION})" \ + "Source : ${GIT_URL} (tag: Ubuntu-${VERSION})" \ "Synced : ${SYNC_DATE}" \ > /tmp/commit-msg.txt @@ -283,7 +332,6 @@ jobs: - name: Print summary if: always() run: | - LAUNCHPAD_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" GIT_TAG="Ubuntu-${VERSION}" echo "## Source Sync Summary" >> "$GITHUB_STEP_SUMMARY" @@ -294,7 +342,7 @@ jobs: echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Cloned from | \`${LAUNCHPAD_URL}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Cloned from | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" diff --git a/README.md b/README.md index a38da8c5ae37f..f54ed92cfed56 100644 --- a/README.md +++ b/README.md @@ -216,6 +216,7 @@ suite branch. |-------|---------|-------------| | `suite` | `questing` | Ubuntu suite to sync — one suite per run | | `force` | `false` | Re-sync even if tag already exists | +| `custom_git_url` | *(empty)* | Custom Launchpad git URL to clone from instead of the official Ubuntu kernel repo. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-*` tag is discovered directly from the custom repo via `git ls-remote`. Leave empty for default behaviour. | **Jobs**: From d56bd13f0aee251e821620729cae12b564ad1e23 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 12:50:49 -0700 Subject: [PATCH 056/115] feat: support custom branch names with base suite derivation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes Docker container selection failure when using branch names with custom suffixes (e.g. resolute-qcom) that don't match a valid pkg-builder distro tag. Changes: - fetch-source-pkg.yml: derive BASE_SUITE from suite input by stripping any custom suffix (cut -d'-' -f1). Use BASE_SUITE for Launchpad REST API distro_series and default git URL construction. Add base_suite as a job output. Update suite input description to clarify it is a branch name that can include a custom suffix. - build-kernel.yml: add 'Derive base suite for Docker' step that sets BASE_SUITE env var. Use BASE_SUITE for docker_deb_build.py -d and pkg-builder container image tag. SUITE (full branch name) is still used for checkout ref, artifact name, and release tag. - README.md: update suite input descriptions for both workflows. Example: suite=resolute-qcom → BASE_SUITE=resolute - Branch/tag: resolute-qcom-7.0.0-X.X - Docker: ghcr.io/qualcomm-linux/pkg-builder:resolute - Checkout: resolute-qcom branch or resolute-qcom-7.0.0-X.X tag --- .github/workflows/build-kernel.yml | 28 ++++++++++++++++++++------ .github/workflows/fetch-source-pkg.yml | 21 +++++++++++++++---- README.md | 4 ++-- 3 files changed, 41 insertions(+), 12 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 843e91f92e66d..92cfae839b15b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -20,7 +20,10 @@ on: workflow_dispatch: inputs: suite: - description: "Suite branch to build from (e.g. noble, questing, resolute)" + description: > + Suite branch to build from (e.g. noble, questing, resolute, + resolute-qcom). The base suite (e.g. resolute) is derived + automatically for Docker container selection. required: true default: "questing" type: string @@ -107,6 +110,17 @@ jobs: echo "ORG_NAME=${GITHUB_REPOSITORY%%/*}" >> "$GITHUB_ENV" echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" + # ----------------------------------------------------------------------- + # 2b. Derive base suite for Docker container selection. + # Strips any custom suffix so the correct pkg-builder image is used. + # e.g. resolute-qcom → resolute, noble → noble, questing → questing + # ----------------------------------------------------------------------- + - name: Derive base suite for Docker + run: | + BASE_SUITE=$(echo "${SUITE}" | cut -d'-' -f1) + echo "BASE_SUITE=${BASE_SUITE}" >> "$GITHUB_ENV" + echo "Suite: ${SUITE} → Base suite: ${BASE_SUITE}" + # ----------------------------------------------------------------------- # 3. Validate kernel_version against existing tags (fail fast) # Must run BEFORE checkout so a bad version fails immediately with a @@ -163,11 +177,13 @@ jobs: path: docker-pkg-build # ----------------------------------------------------------------------- - # 6. Build the suite-matched pkg-builder docker image + # 6. Build the base-suite-matched pkg-builder docker image. + # Uses BASE_SUITE (e.g. resolute) not SUITE (e.g. resolute-qcom) + # because pkg-builder images are tagged by Ubuntu distro name only. # ----------------------------------------------------------------------- - name: Build docker image for suite (${{ inputs.suite || 'questing' }}) run: | - ./docker-pkg-build/docker_deb_build.py --rebuild -d "${SUITE}" + ./docker-pkg-build/docker_deb_build.py --rebuild -d "${BASE_SUITE}" docker image ls # ----------------------------------------------------------------------- @@ -179,7 +195,7 @@ jobs: JOBS=$(nproc) if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi - echo "Building: suite=${SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" + echo "Building: suite=${SUITE} base_suite=${BASE_SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" # IMPORTANT: use single quotes for bash -c so the outer shell does NOT # expand $(...) or ${VAR} before passing the string to docker. @@ -191,7 +207,7 @@ jobs: -e JOBS="${JOBS}" \ -e TARGET="${TARGET}" \ -e WS="${GITHUB_WORKSPACE}" \ - "ghcr.io/qualcomm-linux/pkg-builder:${SUITE}" \ + "ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}" \ bash -c ' set -euo pipefail @@ -347,7 +363,7 @@ jobs: echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "### Built packages" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index d00519fa11f63..6f1cb16d5b69a 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -35,7 +35,12 @@ on: workflow_dispatch: inputs: suite: - description: "Ubuntu suite to sync (e.g. noble, questing, resolute)" + description: > + Ubuntu branch name to sync into (e.g. noble, questing, resolute, + resolute-qcom). This becomes the branch and tag prefix in this repo. + When using a custom_git_url with a team-specific suffix (e.g. + resolute-qcom), the base suite (resolute) is derived automatically + for Launchpad API queries and Docker container selection. required: false default: "questing" type: string @@ -67,6 +72,7 @@ jobs: upstream_version: ${{ steps.query.outputs.upstream_version }} tag: ${{ steps.query.outputs.tag }} suite: ${{ steps.query.outputs.suite }} + base_suite: ${{ steps.query.outputs.base_suite }} source: ${{ steps.query.outputs.source }} git_url: ${{ steps.query.outputs.git_url }} should_sync: ${{ steps.gate.outputs.should_sync }} @@ -82,6 +88,13 @@ jobs: SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} CUSTOM_GIT_URL: ${{ inputs.custom_git_url || '' }} run: | + # Derive the base Ubuntu suite by stripping any custom suffix. + # e.g. resolute-qcom → resolute, noble → noble, questing → questing + # Used for Launchpad API queries and default git URL construction. + BASE_SUITE=$(echo "${SUITE}" | cut -d'-' -f1) + echo "Suite : ${SUITE}" + echo "Base suite : ${BASE_SUITE}" + if [ -n "${CUSTOM_GIT_URL}" ]; then # ---------------------------------------------------------------- # Custom repo path: find the latest Ubuntu-* tag via git ls-remote. @@ -117,7 +130,7 @@ jobs: API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" API+="?ws.op=getPublishedSources" API+="&source_name=${SOURCE}" - API+="&distro_series=/ubuntu/${SUITE}" + API+="&distro_series=/ubuntu/${BASE_SUITE}" API+="&status=Published" API+="&order_by_date=true" API+="&ws.size=300" @@ -133,7 +146,7 @@ jobs: exit 1 } - GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${SUITE}" + GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${BASE_SUITE}" fi UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) @@ -143,10 +156,10 @@ jobs: echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" echo "tag=${TAG}" >> "$GITHUB_OUTPUT" echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" + echo "base_suite=${BASE_SUITE}" >> "$GITHUB_OUTPUT" echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" echo "git_url=${GIT_URL}" >> "$GITHUB_OUTPUT" - echo "Suite : ${SUITE}" echo "Version : ${VERSION}" echo "Tag : ${TAG}" echo "Git URL : ${GIT_URL}" diff --git a/README.md b/README.md index f54ed92cfed56..cc510e854eabb 100644 --- a/README.md +++ b/README.md @@ -214,7 +214,7 @@ suite branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `questing` | Ubuntu suite to sync — one suite per run | +| `suite` | `questing` | Ubuntu branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Launchpad API queries and Docker container selection. | | `force` | `false` | Re-sync even if tag already exists | | `custom_git_url` | *(empty)* | Custom Launchpad git URL to clone from instead of the official Ubuntu kernel repo. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-*` tag is discovered directly from the custom repo via `git ls-remote`. Leave empty for default behaviour. | @@ -244,7 +244,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `suite` | `questing` | Suite branch to build from | +| `suite` | `questing` | Suite branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base suite is derived automatically for Docker container selection. | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | From d54a66e564e05df12e62ce4fe942b83096d20be6 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 14:44:40 -0700 Subject: [PATCH 057/115] feat: make resolute-qcom the daily default sync target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change the scheduled daily build to sync from the resolute-qcom git repository instead of the official Ubuntu kernel tree. fetch-source-pkg.yml: - Default suite: questing → resolute-qcom - Default custom_git_url: '' → resolute-qcom Launchpad git URL - SUITE fallback: 'questing' → 'resolute-qcom' - CUSTOM_GIT_URL: add vars.KERNEL_CUSTOM_GIT_URL fallback and hardcode resolute-qcom URL as the final default so scheduled runs use the custom repo without any repository variable setup - Add KERNEL_CUSTOM_GIT_URL to documented repository variables README.md: - Update end-to-end pipeline diagram to show resolute-qcom git repo as the daily source (replaces Launchpad REST API entry) - Update manual build trigger flow examples to use resolute-qcom - Add resolute-qcom branch to repository branch layout - Add new 'Resolute Qcom kernel source' section documenting the custom repo URL, version discovery via git ls-remote, and branch/tag naming conventions - Update 'Upstream source' table to include resolute-qcom repo - Update 'Source and build notes' to document both sync paths (Path A: resolute-qcom via git tags; Path B: official suites via Launchpad REST API) - Separate 'Supported suites' (official Ubuntu codenames only) from new 'Custom branches' section (resolute-qcom and future custom branches) — resolute-qcom is a branch name, not a suite - Update KERNEL_SUITE default to resolute-qcom in setup table - Add KERNEL_CUSTOM_GIT_URL to repository variables table --- .github/workflows/fetch-source-pkg.yml | 18 ++- README.md | 193 +++++++++++++++++-------- 2 files changed, 144 insertions(+), 67 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 6f1cb16d5b69a..17c63996b44d5 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -23,8 +23,10 @@ # (e.g. noble-6.8.0-114.114). # # Repository variables (Settings → Variables → Actions): -# KERNEL_SUITE – default suite to sync when not specified (default: questing) -# KERNEL_SOURCE – source package name (default: linux) +# KERNEL_SUITE – default suite to sync when not specified (default: resolute-qcom) +# KERNEL_SOURCE – source package name (default: linux) +# KERNEL_CUSTOM_GIT_URL – default custom git URL for scheduled runs +# (default: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) name: "Sync: Canonical Kernel Sources to Branch" @@ -42,7 +44,7 @@ on: resolute-qcom), the base suite (resolute) is derived automatically for Launchpad API queries and Docker container selection. required: false - default: "questing" + default: "resolute-qcom" type: string force: description: "Force sync even if this version is already committed" @@ -51,10 +53,12 @@ on: custom_git_url: description: > Custom Launchpad git URL to clone from instead of the official - Ubuntu kernel repo. Leave empty to use the default. + Ubuntu kernel repo. Leave empty to use the configured default + (resolute-qcom repo). Set to 'none' to force the official + Ubuntu kernel repo for this run. Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute required: false - default: "" + default: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" type: string permissions: @@ -84,9 +88,9 @@ jobs: - name: Resolve version and git URL id: query env: - SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'questing' }} + SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute-qcom' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} - CUSTOM_GIT_URL: ${{ inputs.custom_git_url || '' }} + CUSTOM_GIT_URL: ${{ inputs.custom_git_url || vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute' }} run: | # Derive the base Ubuntu suite by stripping any custom suffix. # e.g. resolute-qcom → resolute, noble → noble, questing → questing diff --git a/README.md b/README.md index cc510e854eabb..1ae3807aefa0c 100644 --- a/README.md +++ b/README.md @@ -10,20 +10,20 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ══════════════════════════════════════════════════════════════════════════════ - Launchpad REST API - (api.launchpad.net) + Resolute Qcom git repo + (git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) │ - ▼ + ▼ git ls-remote --tags → latest Ubuntu-* tag → version X.Y.Z-A.B ╔════════════════════════════════════════════════════════════════════════════╗ ║ fetch-source-pkg.yml ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────────┐ ║ ║ │ Job 1 · check-version │ ║ ║ │ │ ║ -║ │ Query Launchpad API (ws.size=300, exact source_package_name match) │ ║ -║ │ → latest version: questing X.Y.Z-A.B │ ║ +║ │ git ls-remote resolute-qcom repo → latest Ubuntu-* tag │ ║ +║ │ → latest version: resolute-qcom X.Y.Z-A.B │ ║ ║ │ │ ║ -║ │ git ls-remote (authenticated) → tag questing-X.Y.Z-A.B exists? │ ║ +║ │ git ls-remote (authenticated) → tag resolute-qcom-X.Y.Z-A.B exists?│ ║ ║ │ │ ║ ║ │ YES ──▶ should_sync=false ──▶ workflow exits cleanly │ ║ ║ │ NO ──▶ should_sync=true ──▶ continue ↓ │ ║ @@ -34,10 +34,10 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 2 · sync │ ║ ║ │ │ ║ ║ │ Free disk space (~10 GB) │ ║ -║ │ git clone --depth=1 Launchpad git @ Ubuntu-X.Y.Z-A.B │ ║ +║ │ git clone --depth=1 resolute-qcom repo @ Ubuntu-X.Y.Z-A.B │ ║ ║ │ Verify >5000 files cloned │ ║ -║ │ rsync source → questing branch (orphan) │ ║ -║ │ git commit + tag questing-X.Y.Z-A.B │ ║ +║ │ rsync source → resolute-qcom branch (orphan) │ ║ +║ │ git commit + tag resolute-qcom-X.Y.Z-A.B │ ║ ║ │ git push branch + tag │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ║ │ sync succeeded ║ @@ -46,7 +46,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 3 · trigger-build │ ║ ║ │ │ ║ ║ │ gh workflow run build-kernel.yml │ ║ -║ │ suite=questing kernel_version=X.Y.Z-A.B arch=arm64 │ ║ +║ │ suite=resolute-qcom kernel_version=X.Y.Z-A.B arch=arm64 │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ │ @@ -54,12 +54,12 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ╔════════════════════════════════════════════════════════════════════════════╗ ║ build-kernel.yml ║ ║ ║ -║ Checkout questing branch ──▶ kernel-src/ ║ -║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ -║ docker_deb_build.py --rebuild -d questing ║ +║ Checkout resolute-qcom branch ──▶ kernel-src/ ║ +║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ +║ docker_deb_build.py --rebuild -d resolute ← base suite derived ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:questing│ ║ +║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:resolute│ ║ ║ │ │ ║ ║ │ apt-get build-dep linux │ ║ ║ │ fakeroot make -f debian/rules clean ← setup env │ ║ @@ -74,8 +74,8 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ S3 Bucket │ │ GitHub Artifact │ │ GitHub Release │ │ │ │ │ │ │ - │ qli-prd- │ │ 90-day retention │ │ questing-X.Y.Z- │ - │ lecore-gh- │ │ Actions → run │ │ A.B │ + │ qli-prd- │ │ 90-day retention │ │ resolute-qcom- │ + │ lecore-gh- │ │ Actions → run │ │ X.Y.Z-A.B │ │ artifacts │ │ → Artifacts │ │ Releases → │ │ │ │ │ │ Assets │ │ self-hosted │ │ always │ │ permanent │ @@ -99,10 +99,10 @@ MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode A — Test / dev build (kernel_version left empty) │ │ │ - │ suite=questing kernel_version= │ + │ suite=resolute-qcom kernel_version= │ │ │ │ │ ▼ │ - │ Checkout questing branch HEAD │ + │ Checkout resolute-qcom branch HEAD │ │ (includes any commits you pushed on top of the synced source) │ │ │ │ │ ▼ │ @@ -115,37 +115,37 @@ MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode B — Release build for latest synced version │ │ │ - │ suite=questing kernel_version=6.17.0-24.24 │ + │ suite=resolute-qcom kernel_version=7.0.0-5.5 │ │ │ │ │ ▼ │ - │ Validate tag questing-6.17.0-24.24 exists (fail fast if not) │ + │ Validate tag resolute-qcom-7.0.0-5.5 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag questing-6.17.0-24.24 ← exact synced source │ + │ Checkout tag resolute-qcom-7.0.0-5.5 ← exact synced source │ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release questing-6.17.0-24.24│ + │ GitHub Actions artifact (90-day) + GitHub Release resolute-qcom-7.0.0-5.5│ └─────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────┐ │ Mode C — Rebuild / re-release an older synced version │ │ │ - │ suite=questing kernel_version=6.17.0-23.23 │ + │ suite=resolute-qcom kernel_version=7.0.0-4.4 │ │ │ │ │ ▼ │ - │ Validate tag questing-6.17.0-23.23 exists (fail fast if not) │ + │ Validate tag resolute-qcom-7.0.0-4.4 exists (fail fast if not) │ │ │ │ │ ▼ │ - │ Checkout tag questing-6.17.0-23.23 ← older synced source (not HEAD) │ + │ Checkout tag resolute-qcom-7.0.0-4.4 ← older synced source (not HEAD)│ │ │ │ │ ▼ │ │ Build .deb packages │ │ │ │ │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release questing-6.17.0-23.23│ + │ GitHub Actions artifact (90-day) + GitHub Release resolute-qcom-7.0.0-4.4│ │ (existing release assets are overwritten with --clobber) │ └─────────────────────────────────────────────────────────────────────────┘ ``` @@ -172,6 +172,11 @@ pkg-linux-qcom-canonical │ One commit per Canonical upload │ Tagged noble-6.8.0-114.114, noble-6.8.0-115.115, … │ +├── resolute-qcom branch (orphan) +│ └── Full resolute-qcom kernel source tree (daily default) +│ One commit per upstream tag +│ Tagged resolute-qcom-7.0.0-X.X, … +│ └── branch (orphan, added on demand) └── Full kernel source for that suite e.g. questing, resolute @@ -182,15 +187,59 @@ and contain only the extracted kernel source tree. --- +## Resolute Qcom kernel source + +The daily scheduled build syncs from a custom resolute kernel repository +maintained separately from the official Ubuntu kernel tree. This repository +is referred to as the **resolute-qcom** source. + +| Resource | URL | +|----------|-----| +| Git repository | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | + +**How version discovery works for resolute-qcom:** + +Unlike the official Ubuntu kernel path (which queries the Launchpad REST API +for the latest published source package), the resolute-qcom path queries the +git repository directly: + +```bash +git ls-remote --tags \ + https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ + 'refs/tags/Ubuntu-*' +``` + +Tags are sorted with `sort -V` (version sort) and the latest `Ubuntu-*` tag +is selected. The version is extracted from the tag name: + +``` +Ubuntu-7.0.0-5.5 → version: 7.0.0-5.5 + → branch tag: resolute-qcom-7.0.0-5.5 +``` + +The Launchpad REST API is **not used** for resolute-qcom — the git tags are +the authoritative source of version information for this repository. + +**Branch and tag naming:** + +| Item | Pattern | Example | +|------|---------|---------| +| Branch in this repo | `resolute-qcom` | `resolute-qcom` | +| Tag in this repo | `resolute-qcom-X.Y.Z-A.B` | `resolute-qcom-7.0.0-5.5` | +| Docker container | `pkg-builder:resolute` | base suite derived automatically | + +--- + ## Upstream source | Resource | URL pattern | Used by | |----------|-------------|---------| -| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/&ws.size=300` | `check-version` job — queries for the latest published version number | -| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` | +| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/&ws.size=300` | `check-version` job — queries for the latest published version number (official suites only; bypassed for resolute-qcom) | +| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` (official suites only) | +| Resolute Qcom git repository | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | `sync` job — daily default; version discovered via `git ls-remote` | | GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | `build-kernel` job — attaches built `.deb` packages | -**Example (noble suite):** +**Example (noble suite — official upstream):** - Source packages: https://launchpad.net/ubuntu/noble/+source/linux - Git repository: `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` @@ -200,11 +249,10 @@ and contain only the extracted kernel source tree. ### `fetch-source-pkg.yml` — Sync sources to branch -Queries the Launchpad REST API for the latest published `linux` source -package version (exact name match), then clones the Launchpad git -repository at the corresponding tag (`Ubuntu-`) to get the -complete source tree including `debian/rules`, and commits it to the -suite branch. +Queries the git repository for the latest `Ubuntu-*` tag (resolute-qcom, daily +default) or the Launchpad REST API (official suites), then clones the source +tree at the corresponding tag to get the complete source including `debian/rules`, +and commits it to the suite branch. **Schedule**: daily at **04:00 UTC** **Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` @@ -214,16 +262,16 @@ suite branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `questing` | Ubuntu branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Launchpad API queries and Docker container selection. | +| `suite` | `resolute-qcom` | Ubuntu branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Docker container selection. | | `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | *(empty)* | Custom Launchpad git URL to clone from instead of the official Ubuntu kernel repo. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-*` tag is discovered directly from the custom repo via `git ls-remote`. Leave empty for default behaviour. | +| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the resolute-qcom repo. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-*` tag is discovered directly from the repo via `git ls-remote`. | **Jobs**: | Job | What it does | |-----|-------------| -| `check-version` | Queries Launchpad API (`ws.size=300`) with exact `source_package_name` filter; checks tag existence via authenticated `git ls-remote`; sets `should_sync` flag | -| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from Launchpad git; verifies >5000 files; commits to suite branch; creates tag | +| `check-version` | For resolute-qcom: queries tags via `git ls-remote` on the custom repo. For official suites: queries Launchpad API (`ws.size=300`). Checks tag existence; sets `should_sync` flag. | +| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from the resolved git URL; verifies >5000 files; commits to suite branch; creates tag | | `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=generic` | **Idempotent**: if the tag for the latest version already exists, the workflow exits cleanly without downloading anything. @@ -233,7 +281,7 @@ suite branch. ### `build-kernel.yml` — Build .deb packages Checks out the suite branch (full kernel source tree) and builds `.deb` -packages inside the suite-matched `ghcr.io/qualcomm-linux/pkg-builder:` +packages inside the base-suite-matched `ghcr.io/qualcomm-linux/pkg-builder:` container using `fakeroot debian/rules binary-`. **Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or @@ -269,15 +317,16 @@ The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The ` 1. Free up disk space (~10 GB) 2. Checkout suite branch → `kernel-src/` 3. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` -4. Build docker image: `docker_deb_build.py --rebuild -d ` -5. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: +4. Derive `BASE_SUITE` from suite (e.g. `resolute-qcom` → `resolute`) +5. Build docker image: `docker_deb_build.py --rebuild -d ` +6. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: ``` apt-get build-dep linux fakeroot make -f debian/rules clean fakeroot debian/rules binary- do_skip_checks=true ``` See [Build container notes](#build-container-notes) for why these exact invocations are used. -6. Collect `.deb` files from workspace root +7. Collect `.deb` files from workspace root **Output**: @@ -301,15 +350,16 @@ Go to **Actions** and enable workflows if prompted. | Variable | Default | Description | |----------|---------|-------------| -| `KERNEL_SUITE` | `questing` | Default suite for scheduled runs | +| `KERNEL_SUITE` | `resolute-qcom` | Default suite for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | +| `KERNEL_CUSTOM_GIT_URL` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Default custom git URL for scheduled runs | ### 3. Run the first sync ```bash gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=noble + --field suite=resolute-qcom ``` --- @@ -347,9 +397,22 @@ This produces a complete, buildable source tree with `debian/rules` — the same ### How the sync workflow finds and clones the kernel source -The sync workflow uses two Launchpad services for different purposes: +The sync workflow supports two paths depending on whether a custom git URL is configured: + +**Path A — Resolute Qcom (daily default): version from git tags** + +```bash +git ls-remote --tags \ + https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ + 'refs/tags/Ubuntu-*' +# → sort -V | tail -1 → Ubuntu-7.0.0-5.5 +# → VERSION=7.0.0-5.5 +``` -**Step 1 — Launchpad REST API: find the latest published version** +The latest `Ubuntu-*` tag in the custom repo is the authoritative version source. +The Launchpad REST API is not used. + +**Path B — Official suites (noble, questing, resolute): version from Launchpad REST API** ``` GET https://api.launchpad.net/1.0/ubuntu/+archive/primary @@ -377,18 +440,10 @@ The API tells us the exact version string of the latest *officially published* kernel. A git tag might exist before the package is published to the archive, so the API is the authoritative source for "what is the current release". -**Step 2 — Construct the git tag** - -``` -VERSION = "6.8.0-114.114" -GIT_TAG = "Ubuntu-6.8.0-114.114" -``` - -**Step 3 — Clone from Launchpad git at that tag** +**Clone step (both paths):** ```bash -git clone --depth=1 --branch Ubuntu-6.8.0-114.114 \ - https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble +git clone --depth=1 --branch Ubuntu- ``` The git repository has the **complete** `debian/` directory including @@ -433,19 +488,21 @@ Ubuntu kernel versions follow `X.Y.Z-A.B`: | `A` | `114` | ABI number | | `B` | `114` | Upload number | -Tags use `-X.Y.Z-A.B`, e.g. `noble-6.8.0-114.114`. +Tags use `-X.Y.Z-A.B`, e.g. `resolute-qcom-7.0.0-5.5`. --- ## Supported suites -| Suite | Codename | Status | Kernel | +Official Ubuntu suites supported by this pipeline: + +| Suite | Codename | Ubuntu | Kernel | |-------|----------|--------|--------| | `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.10 — **active** (daily default) | 6.17 | +| `questing` | Questing Quokka | 25.10 — **active** | 6.17 | | `resolute` | Resolute Ringtail | 26.04 LTS — **active** | 7.0 | -To add a new suite, trigger `fetch-source-pkg.yml` with the desired +To sync an official suite, trigger `fetch-source-pkg.yml` with the desired `suite` input — the branch and release tag are created automatically: ```bash @@ -454,6 +511,22 @@ gh workflow run fetch-source-pkg.yml \ --field suite=resolute ``` +## Custom branches + +In addition to official Ubuntu suites, this repo supports custom branches +that track non-upstream kernel repositories. Custom branches use a +`-` naming convention so the base suite can be derived +automatically for Docker container selection. + +| Branch | Base suite | Source | Daily default | +|--------|-----------|--------|---------------| +| `resolute-qcom` | `resolute` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | ✅ yes | + +Custom branches are **not** Ubuntu suite names — they are branch names in +this repository that happen to be based on a particular Ubuntu suite's kernel. +The `suite` input in both workflows accepts either an official suite name or +a custom branch name. + --- ## Build container notes From a1840ef0f0f85ae3ff5f5fb8857b8eff7cce8843 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 14:51:30 -0700 Subject: [PATCH 058/115] docs: fix build-kernel suite input default and wording in README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - build-kernel.yml inputs table: suite default questing → resolute-qcom - build-kernel.yml inputs table: 'Suite branch' → 'Branch' (resolute-qcom is a branch name, not a Ubuntu suite) - KERNEL_SUITE variable: 'Default suite' → 'Default branch name' - Pipeline diagram: rename source label to 'Ubuntu-qcom Launchpad repository' --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1ae3807aefa0c..26faa89ad47ad 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ══════════════════════════════════════════════════════════════════════════════ - Resolute Qcom git repo + Ubuntu-qcom Launchpad repository (git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) │ ▼ git ls-remote --tags → latest Ubuntu-* tag → version X.Y.Z-A.B @@ -292,7 +292,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `suite` | `questing` | Suite branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base suite is derived automatically for Docker container selection. | +| `suite` | `resolute-qcom` | Branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base suite is derived automatically for Docker container selection. | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | @@ -350,7 +350,7 @@ Go to **Actions** and enable workflows if prompted. | Variable | Default | Description | |----------|---------|-------------| -| `KERNEL_SUITE` | `resolute-qcom` | Default suite for scheduled runs | +| `KERNEL_SUITE` | `resolute-qcom` | Default branch name for scheduled runs | | `KERNEL_SOURCE` | `linux` | Source package name | | `KERNEL_CUSTOM_GIT_URL` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Default custom git URL for scheduled runs | From 57a963fce0e6660b88260942e4c5fe6e952a65b5 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 14:57:53 -0700 Subject: [PATCH 059/115] fix: handle Ubuntu-qcom-* tag format for resolute-qcom repo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Ubuntu-qcom Launchpad repository uses tags of the form Ubuntu-qcom-7.0.0-1003.3, not Ubuntu-7.0.0-1.1. The previous code filtered Ubuntu-* and stripped Ubuntu- prefix, producing the wrong version string (qcom-7.0.0-1003.3 instead of 7.0.0-1003.3) and would clone the wrong tag. fetch-source-pkg.yml: - Add git_tag_prefix input (default: Ubuntu-qcom) that controls both the tag filter pattern and the prefix stripped to extract the version number - Tag filter: refs/tags/Ubuntu-* → refs/tags/${GIT_TAG_PREFIX}-* - Version extraction: ${LATEST_TAG#Ubuntu-} → ${LATEST_TAG#${GIT_TAG_PREFIX}-} - Clone tag: Ubuntu-${VERSION} → ${GIT_TAG_PREFIX}-${VERSION} - Pass git_tag_prefix through as a check-version job output and sync job env var so both jobs use the correct tag format - Official repos (no custom URL) set GIT_TAG_PREFIX=Ubuntu automatically - Update commit message and summary to use ${GIT_TAG_PREFIX}-${VERSION} README.md: - Update pipeline diagram to show Ubuntu-qcom-* tag format - Update Resolute Qcom section: correct tag example to Ubuntu-qcom-7.0.0-1003.3 → 7.0.0-1003.3 - Add upstream git tag row to branch/tag naming table - Add git_tag_prefix input to fetch-source-pkg.yml inputs table - Update versioning scheme example to resolute-qcom-7.0.0-1003.3 - Update Source and build notes Path A to show correct tag format --- .github/workflows/fetch-source-pkg.yml | 95 ++++++++++++++++---------- README.md | 22 +++--- 2 files changed, 70 insertions(+), 47 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 17c63996b44d5..6789d6442f37c 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -23,7 +23,7 @@ # (e.g. noble-6.8.0-114.114). # # Repository variables (Settings → Variables → Actions): -# KERNEL_SUITE – default suite to sync when not specified (default: resolute-qcom) +# KERNEL_SUITE – default branch to sync when not specified (default: resolute-qcom) # KERNEL_SOURCE – source package name (default: linux) # KERNEL_CUSTOM_GIT_URL – default custom git URL for scheduled runs # (default: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) @@ -38,11 +38,11 @@ on: inputs: suite: description: > - Ubuntu branch name to sync into (e.g. noble, questing, resolute, + Branch name to sync into (e.g. noble, questing, resolute, resolute-qcom). This becomes the branch and tag prefix in this repo. - When using a custom_git_url with a team-specific suffix (e.g. - resolute-qcom), the base suite (resolute) is derived automatically - for Launchpad API queries and Docker container selection. + The base suite (e.g. resolute) is derived automatically from the + first component for Launchpad API queries and Docker container + selection. required: false default: "resolute-qcom" type: string @@ -53,13 +53,20 @@ on: custom_git_url: description: > Custom Launchpad git URL to clone from instead of the official - Ubuntu kernel repo. Leave empty to use the configured default - (resolute-qcom repo). Set to 'none' to force the official - Ubuntu kernel repo for this run. + Ubuntu kernel repo. Defaults to the Ubuntu-qcom Launchpad repository. Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute required: false default: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" type: string + git_tag_prefix: + description: > + Tag prefix used to discover and clone versions from a custom git repo. + Only applies when custom_git_url is set. + Default: Ubuntu-qcom (matches Ubuntu-qcom-7.0.0-1003.3 style tags) + Use 'Ubuntu' for repos with standard Ubuntu-X.Y.Z-A.B tags. + required: false + default: "Ubuntu-qcom" + type: string permissions: contents: write @@ -79,6 +86,7 @@ jobs: base_suite: ${{ steps.query.outputs.base_suite }} source: ${{ steps.query.outputs.source }} git_url: ${{ steps.query.outputs.git_url }} + git_tag_prefix: ${{ steps.query.outputs.git_tag_prefix }} should_sync: ${{ steps.gate.outputs.should_sync }} steps: @@ -91,6 +99,7 @@ jobs: SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute-qcom' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} CUSTOM_GIT_URL: ${{ inputs.custom_git_url || vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute' }} + GIT_TAG_PREFIX: ${{ inputs.git_tag_prefix || 'Ubuntu-qcom' }} run: | # Derive the base Ubuntu suite by stripping any custom suffix. # e.g. resolute-qcom → resolute, noble → noble, questing → questing @@ -101,14 +110,20 @@ jobs: if [ -n "${CUSTOM_GIT_URL}" ]; then # ---------------------------------------------------------------- - # Custom repo path: find the latest Ubuntu-* tag via git ls-remote. - # This bypasses the Launchpad REST API entirely — the version is - # derived directly from the tags present in the custom repository. + # Custom repo path: find the latest -* tag via + # git ls-remote. This bypasses the Launchpad REST API entirely — + # the version is derived directly from the tags present in the + # custom repository. + # + # The Ubuntu-qcom Launchpad repository uses tags of the form: + # Ubuntu-qcom-7.0.0-1003.3 + # so GIT_TAG_PREFIX defaults to "Ubuntu-qcom". # ---------------------------------------------------------------- - echo "Custom git URL provided: ${CUSTOM_GIT_URL}" + echo "Custom git URL : ${CUSTOM_GIT_URL}" + echo "Git tag prefix : ${GIT_TAG_PREFIX}" echo "Querying tags from custom repo..." - LATEST_TAG=$(git ls-remote --tags "${CUSTOM_GIT_URL}" 'refs/tags/Ubuntu-*' \ + LATEST_TAG=$(git ls-remote --tags "${CUSTOM_GIT_URL}" "refs/tags/${GIT_TAG_PREFIX}-*" \ | grep -v '\^{}' \ | awk '{print $2}' \ | sed 's|refs/tags/||' \ @@ -116,12 +131,12 @@ jobs: | tail -1) [ -n "${LATEST_TAG}" ] || { - echo "ERROR: No Ubuntu-* tags found in ${CUSTOM_GIT_URL}" >&2 + echo "ERROR: No ${GIT_TAG_PREFIX}-* tags found in ${CUSTOM_GIT_URL}" >&2 exit 1 } - # Ubuntu-7.0.0-1.1 → 7.0.0-1.1 - VERSION="${LATEST_TAG#Ubuntu-}" + # Ubuntu-qcom-7.0.0-1003.3 → 7.0.0-1003.3 + VERSION="${LATEST_TAG#${GIT_TAG_PREFIX}-}" GIT_URL="${CUSTOM_GIT_URL}" echo "Latest tag : ${LATEST_TAG}" @@ -130,7 +145,10 @@ jobs: # Default path: query the Launchpad REST API for the latest # officially published source package version, then construct # the standard Launchpad git URL for the suite. + # Official Ubuntu repos use Ubuntu-X.Y.Z-A.B tag format. # ---------------------------------------------------------------- + GIT_TAG_PREFIX="Ubuntu" + API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" API+="?ws.op=getPublishedSources" API+="&source_name=${SOURCE}" @@ -156,17 +174,19 @@ jobs: UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) TAG="${SUITE}-${VERSION}" - echo "version=${VERSION}" >> "$GITHUB_OUTPUT" - echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" - echo "tag=${TAG}" >> "$GITHUB_OUTPUT" - echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" - echo "base_suite=${BASE_SUITE}" >> "$GITHUB_OUTPUT" - echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" - echo "git_url=${GIT_URL}" >> "$GITHUB_OUTPUT" + echo "version=${VERSION}" >> "$GITHUB_OUTPUT" + echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" + echo "tag=${TAG}" >> "$GITHUB_OUTPUT" + echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" + echo "base_suite=${BASE_SUITE}" >> "$GITHUB_OUTPUT" + echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" + echo "git_url=${GIT_URL}" >> "$GITHUB_OUTPUT" + echo "git_tag_prefix=${GIT_TAG_PREFIX}" >> "$GITHUB_OUTPUT" - echo "Version : ${VERSION}" - echo "Tag : ${TAG}" - echo "Git URL : ${GIT_URL}" + echo "Version : ${VERSION}" + echo "Tag : ${TAG}" + echo "Git URL : ${GIT_URL}" + echo "Tag prefix : ${GIT_TAG_PREFIX}" - name: Check whether tag already exists id: gate @@ -203,6 +223,7 @@ jobs: SUITE: ${{ needs.check-version.outputs.suite }} SOURCE: ${{ needs.check-version.outputs.source }} GIT_URL: ${{ needs.check-version.outputs.git_url }} + GIT_TAG_PREFIX: ${{ needs.check-version.outputs.git_tag_prefix }} steps: # ----------------------------------------------------------------------- @@ -237,13 +258,13 @@ jobs: # 3. Clone from the resolved git URL at the version tag. # GIT_URL is either the custom repo URL (when custom_git_url input # was provided) or the standard Launchpad URL for the suite. - # The git repository has the complete debian/ directory including - # debian/rules, scripts/, templates/, etc. — unlike the source - # package which only ships debian.master/ with rules.d/ fragments. + # GIT_TAG_PREFIX determines the tag format: + # Ubuntu-qcom → Ubuntu-qcom-7.0.0-1003.3 (resolute-qcom repo) + # Ubuntu → Ubuntu-6.8.0-114.114 (official repos) # ----------------------------------------------------------------------- - name: Clone from git repo at version tag run: | - GIT_TAG="Ubuntu-${VERSION}" + GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" echo "Cloning ${GIT_URL} at tag ${GIT_TAG} (shallow)..." git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ @@ -314,17 +335,17 @@ jobs: SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ "${SUITE}: ${VERSION}" \ - "Ubuntu suite : ${SUITE}" \ + "Branch : ${SUITE}" \ "Kernel version : ${VERSION}" \ "Upstream : ${UPSTREAM_VERSION}" \ - "Source : ${GIT_URL} (tag: Ubuntu-${VERSION})" \ + "Source : ${GIT_URL} (tag: ${GIT_TAG_PREFIX}-${VERSION})" \ "Synced : ${SYNC_DATE}" \ > /tmp/commit-msg.txt git add --all if git diff --cached --quiet; then - echo "No changes to commit – suite branch already has this content" + echo "No changes to commit – branch already has this content" else git commit -F /tmp/commit-msg.txt echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" @@ -333,7 +354,7 @@ jobs: - name: Tag the commit run: | cd suite-repo - git tag -fa "${TAG}" -m "Ubuntu ${SUITE} kernel ${VERSION}" + git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" echo "Tagged: ${TAG}" - name: Push suite branch and tag @@ -349,15 +370,15 @@ jobs: - name: Print summary if: always() run: | - GIT_TAG="Ubuntu-${VERSION}" + GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" echo "## Source Sync Summary" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch ref | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Cloned from | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" diff --git a/README.md b/README.md index 26faa89ad47ad..8f20f7954edd8 100644 --- a/README.md +++ b/README.md @@ -13,14 +13,14 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm Ubuntu-qcom Launchpad repository (git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) │ - ▼ git ls-remote --tags → latest Ubuntu-* tag → version X.Y.Z-A.B + ▼ git ls-remote --tags → latest Ubuntu-qcom-* tag → version X.Y.Z-A.B ╔════════════════════════════════════════════════════════════════════════════╗ ║ fetch-source-pkg.yml ║ ║ ║ ║ ┌──────────────────────────────────────────────────────────────────────┐ ║ ║ │ Job 1 · check-version │ ║ ║ │ │ ║ -║ │ git ls-remote resolute-qcom repo → latest Ubuntu-* tag │ ║ +║ │ git ls-remote resolute-qcom repo → latest Ubuntu-qcom-* tag │ ║ ║ │ → latest version: resolute-qcom X.Y.Z-A.B │ ║ ║ │ │ ║ ║ │ git ls-remote (authenticated) → tag resolute-qcom-X.Y.Z-A.B exists?│ ║ @@ -34,7 +34,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ Job 2 · sync │ ║ ║ │ │ ║ ║ │ Free disk space (~10 GB) │ ║ -║ │ git clone --depth=1 resolute-qcom repo @ Ubuntu-X.Y.Z-A.B │ ║ +║ │ git clone --depth=1 resolute-qcom repo @ Ubuntu-qcom-X.Y.Z-A.B │ ║ ║ │ Verify >5000 files cloned │ ║ ║ │ rsync source → resolute-qcom branch (orphan) │ ║ ║ │ git commit + tag resolute-qcom-X.Y.Z-A.B │ ║ @@ -209,12 +209,12 @@ git ls-remote --tags \ 'refs/tags/Ubuntu-*' ``` -Tags are sorted with `sort -V` (version sort) and the latest `Ubuntu-*` tag +Tags are sorted with `sort -V` (version sort) and the latest `Ubuntu-qcom-*` tag is selected. The version is extracted from the tag name: ``` -Ubuntu-7.0.0-5.5 → version: 7.0.0-5.5 - → branch tag: resolute-qcom-7.0.0-5.5 +Ubuntu-qcom-7.0.0-1003.3 → version: 7.0.0-1003.3 + → branch tag: resolute-qcom-7.0.0-1003.3 ``` The Launchpad REST API is **not used** for resolute-qcom — the git tags are @@ -224,8 +224,9 @@ the authoritative source of version information for this repository. | Item | Pattern | Example | |------|---------|---------| +| Upstream git tag | `Ubuntu-qcom-X.Y.Z-A.B` | `Ubuntu-qcom-7.0.0-1003.3` | | Branch in this repo | `resolute-qcom` | `resolute-qcom` | -| Tag in this repo | `resolute-qcom-X.Y.Z-A.B` | `resolute-qcom-7.0.0-5.5` | +| Tag in this repo | `resolute-qcom-X.Y.Z-A.B` | `resolute-qcom-7.0.0-1003.3` | | Docker container | `pkg-builder:resolute` | base suite derived automatically | --- @@ -262,9 +263,10 @@ and commits it to the suite branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `resolute-qcom` | Ubuntu branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Docker container selection. | +| `suite` | `resolute-qcom` | Branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Docker container selection. | | `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the resolute-qcom repo. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-*` tag is discovered directly from the repo via `git ls-remote`. | +| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. When set, the Launchpad REST API is bypassed — the latest tag is discovered directly from the repo via `git ls-remote`. | +| `git_tag_prefix` | `Ubuntu-qcom` | Tag prefix used to filter and strip when discovering versions from a custom repo. Default `Ubuntu-qcom` matches `Ubuntu-qcom-7.0.0-1003.3` style tags. Use `Ubuntu` for repos with standard `Ubuntu-X.Y.Z-A.B` tags. | **Jobs**: @@ -488,7 +490,7 @@ Ubuntu kernel versions follow `X.Y.Z-A.B`: | `A` | `114` | ABI number | | `B` | `114` | Upload number | -Tags use `-X.Y.Z-A.B`, e.g. `resolute-qcom-7.0.0-5.5`. +Tags use `-X.Y.Z-A.B`, e.g. `resolute-qcom-7.0.0-1003.3`. --- From d4b5ac3870f3317eb51c3807b0afc7ab9cffddf6 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:03:51 -0700 Subject: [PATCH 060/115] refactor: make git_tag_prefix an internal variable, not a user input MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The git_tag_prefix was unnecessarily exposed as a workflow_dispatch input box, adding complexity for users who don't need to know about tag prefix internals. fetch-source-pkg.yml: - Remove git_tag_prefix from workflow_dispatch inputs - GIT_TAG_PREFIX is now hardcoded in the env block: custom URL path → Ubuntu-qcom (default) official URL path → Ubuntu (overridden in else branch) - Behavior is identical; users see only 3 inputs: suite, force, custom_git_url README.md: - Remove git_tag_prefix row from fetch-source-pkg.yml inputs table - Update custom_git_url description to mention Ubuntu-qcom-* tags --- .github/workflows/fetch-source-pkg.yml | 14 ++++---------- README.md | 3 +-- 2 files changed, 5 insertions(+), 12 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 6789d6442f37c..11e050bb71d2f 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -58,15 +58,6 @@ on: required: false default: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" type: string - git_tag_prefix: - description: > - Tag prefix used to discover and clone versions from a custom git repo. - Only applies when custom_git_url is set. - Default: Ubuntu-qcom (matches Ubuntu-qcom-7.0.0-1003.3 style tags) - Use 'Ubuntu' for repos with standard Ubuntu-X.Y.Z-A.B tags. - required: false - default: "Ubuntu-qcom" - type: string permissions: contents: write @@ -99,7 +90,10 @@ jobs: SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute-qcom' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} CUSTOM_GIT_URL: ${{ inputs.custom_git_url || vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute' }} - GIT_TAG_PREFIX: ${{ inputs.git_tag_prefix || 'Ubuntu-qcom' }} + # GIT_TAG_PREFIX is derived automatically: + # custom URL → Ubuntu-qcom (Ubuntu-qcom-7.0.0-1003.3 style tags) + # official → Ubuntu (Ubuntu-X.Y.Z-A.B style tags, set in else branch) + GIT_TAG_PREFIX: "Ubuntu-qcom" run: | # Derive the base Ubuntu suite by stripping any custom suffix. # e.g. resolute-qcom → resolute, noble → noble, questing → questing diff --git a/README.md b/README.md index 8f20f7954edd8..a3ced075b0a65 100644 --- a/README.md +++ b/README.md @@ -265,8 +265,7 @@ and commits it to the suite branch. |-------|---------|-------------| | `suite` | `resolute-qcom` | Branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Docker container selection. | | `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. When set, the Launchpad REST API is bypassed — the latest tag is discovered directly from the repo via `git ls-remote`. | -| `git_tag_prefix` | `Ubuntu-qcom` | Tag prefix used to filter and strip when discovering versions from a custom repo. Default `Ubuntu-qcom` matches `Ubuntu-qcom-7.0.0-1003.3` style tags. Use `Ubuntu` for repos with standard `Ubuntu-X.Y.Z-A.B` tags. | +| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-qcom-*` tag is discovered directly from the repo via `git ls-remote`. | **Jobs**: From b790896090b47387418a5b506c8d467a9821de99 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:07:01 -0700 Subject: [PATCH 061/115] fix: remove suite terminology from build-kernel.yml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - suite input default: questing → resolute-qcom - suite input description: 'Suite branch' → 'Branch' - SUITE env fallback: 'questing' → 'resolute-qcom' - File header: 'suite branch' → 'branch', 'suite-matched' → 'base-suite-matched' - Summary table label: 'Suite' → 'Branch' - All 'questing' fallback references updated to 'resolute-qcom' --- .github/workflows/build-kernel.yml | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 92cfae839b15b..cddc2a7f9d07a 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -2,11 +2,11 @@ # # build-kernel.yml # -# Builds Ubuntu kernel .deb packages from a suite branch -# (e.g. "noble") that was populated by fetch-source-pkg.yml. +# Builds Ubuntu kernel .deb packages from a branch +# (e.g. "noble", "resolute-qcom") that was populated by fetch-source-pkg.yml. # -# The build always runs inside the suite-matched -# ghcr.io/qualcomm-linux/pkg-builder: container via +# The build always runs inside the base-suite-matched +# ghcr.io/qualcomm-linux/pkg-builder: container via # docker-pkg-build, ensuring a clean, reproducible environment # that exactly matches the target distro. # @@ -21,11 +21,11 @@ on: inputs: suite: description: > - Suite branch to build from (e.g. noble, questing, resolute, + Branch to build from (e.g. noble, questing, resolute, resolute-qcom). The base suite (e.g. resolute) is derived automatically for Docker container selection. required: true - default: "questing" + default: "resolute-qcom" type: string kernel_version: description: "Kernel version string (e.g. 6.8.0-51.52) – used for release asset attachment" @@ -75,7 +75,7 @@ jobs: timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'questing' }} + SUITE: ${{ inputs.suite || 'resolute-qcom' }} ARCH: ${{ inputs.arch || 'arm64' }} FLAVOR: ${{ inputs.flavor || 'generic' }} @@ -163,7 +163,7 @@ jobs: - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'questing', inputs.kernel_version) || inputs.suite || 'questing' }} + ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute-qcom', inputs.kernel_version) || inputs.suite || 'resolute-qcom' }} path: kernel-src # ----------------------------------------------------------------------- @@ -181,7 +181,7 @@ jobs: # Uses BASE_SUITE (e.g. resolute) not SUITE (e.g. resolute-qcom) # because pkg-builder images are tagged by Ubuntu distro name only. # ----------------------------------------------------------------------- - - name: Build docker image for suite (${{ inputs.suite || 'questing' }}) + - name: Build docker image for suite (${{ inputs.suite || 'resolute-qcom' }}) run: | ./docker-pkg-build/docker_deb_build.py --rebuild -d "${BASE_SUITE}" docker image ls @@ -190,7 +190,7 @@ jobs: # 7. Build kernel packages inside the suite-matched container # The workspace is bind-mounted so output .deb files land on the host. # ----------------------------------------------------------------------- - - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'questing' }}) + - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'resolute-qcom' }}) run: | JOBS=$(nproc) if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi @@ -360,7 +360,7 @@ jobs: echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Suite | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" From 8939cfa7c36d442068c09a74467d64fd608a0eee Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:10:53 -0700 Subject: [PATCH 062/115] docs: remove all suite references for resolute-qcom; use branch terminology MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolute-qcom is a branch name, not a Ubuntu suite. Replace all incorrect suite references throughout: build-kernel.yml: - suite input default: questing → resolute-qcom - suite input description: 'Suite branch' → 'Branch' - SUITE env fallback: 'questing' → 'resolute-qcom' - File header: 'suite branch' → 'branch' - Summary table: 'Suite' → 'Branch' README.md: - Repository branch layout comments: 'suite branch' → 'branch' - build-kernel.yml description: 'Checks out the suite branch' → 'Checks out the branch' - sync job description: 'commits to suite branch' → 'commits to branch' - Build steps: 'Checkout suite branch' → 'Checkout branch' - Build steps: 'Derive BASE_SUITE from suite' → 'from branch name' - Output table: '-X.Y.Z-A.B' → '-X.Y.Z-A.B' - fetch-source-pkg.yml description: 'commits it to the suite branch' → 'commits it to the branch' --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index a3ced075b0a65..bf4c963db6d5a 100644 --- a/README.md +++ b/README.md @@ -159,8 +159,8 @@ pkg-linux-qcom-canonical │ ├── main branch │ ├── .github/workflows/ -│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → suite branch -│ │ └── build-kernel.yml ← build .deb packages from suite branch +│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → branch +│ │ └── build-kernel.yml ← build .deb packages from branch │ ├── scripts/ │ │ ├── check-version.sh ← query latest version from Launchpad │ │ ├── fetch-source-pkg.sh ← download source package files @@ -253,7 +253,7 @@ the authoritative source of version information for this repository. Queries the git repository for the latest `Ubuntu-*` tag (resolute-qcom, daily default) or the Launchpad REST API (official suites), then clones the source tree at the corresponding tag to get the complete source including `debian/rules`, -and commits it to the suite branch. +and commits it to the branch. **Schedule**: daily at **04:00 UTC** **Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` @@ -272,7 +272,7 @@ and commits it to the suite branch. | Job | What it does | |-----|-------------| | `check-version` | For resolute-qcom: queries tags via `git ls-remote` on the custom repo. For official suites: queries Launchpad API (`ws.size=300`). Checks tag existence; sets `should_sync` flag. | -| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from the resolved git URL; verifies >5000 files; commits to suite branch; creates tag | +| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from the resolved git URL; verifies >5000 files; commits to branch; creates tag | | `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=generic` | **Idempotent**: if the tag for the latest version already exists, the workflow exits cleanly without downloading anything. @@ -281,7 +281,7 @@ and commits it to the suite branch. ### `build-kernel.yml` — Build .deb packages -Checks out the suite branch (full kernel source tree) and builds `.deb` +Checks out the branch (full kernel source tree) and builds `.deb` packages inside the base-suite-matched `ghcr.io/qualcomm-linux/pkg-builder:` container using `fakeroot debian/rules binary-`. @@ -316,9 +316,9 @@ The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The ` **Build steps**: 1. Free up disk space (~10 GB) -2. Checkout suite branch → `kernel-src/` +2. Checkout branch → `kernel-src/` 3. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` -4. Derive `BASE_SUITE` from suite (e.g. `resolute-qcom` → `resolute`) +4. Derive `BASE_SUITE` from branch name (e.g. `resolute-qcom` → `resolute`) 5. Build docker image: `docker_deb_build.py --rebuild -d ` 6. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: ``` @@ -335,7 +335,7 @@ The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The ` |----------|---------------|-----------|-------| | **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | Self-hosted runner only; skipped gracefully on GitHub-hosted | | **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | Always available | -| **GitHub Release asset** | Releases → `-X.Y.Z-A.B` → Assets | Permanent | Attached when `kernel_version` is provided | +| **GitHub Release asset** | Releases → `-X.Y.Z-A.B` → Assets | Permanent | Attached when `kernel_version` is provided | --- From b9d64d04902868649fd9b29d96cbdce16cbf7e58 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:14:11 -0700 Subject: [PATCH 063/115] fix: allow syncing official suites by clearing custom_git_url MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When custom_git_url is cleared on a manual run, the workflow was still falling back to the hardcoded carmel-team URL because empty string is falsy in GitHub Actions || expressions. Fix: use github.event_name == 'schedule' to gate the carmel-team URL fallback — it only applies to scheduled runs. Manual runs use exactly what the user entered, so clearing the field correctly triggers the official Launchpad REST API path. Also update custom_git_url description to tell users to clear the field when syncing an official Ubuntu suite. --- .github/workflows/fetch-source-pkg.yml | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 11e050bb71d2f..d6b936965d652 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -52,8 +52,9 @@ on: default: false custom_git_url: description: > - Custom Launchpad git URL to clone from instead of the official - Ubuntu kernel repo. Defaults to the Ubuntu-qcom Launchpad repository. + Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom + Launchpad repository. Clear this field to sync an official Ubuntu + suite (noble, questing, resolute) from the upstream Ubuntu kernel repo. Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute required: false default: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" @@ -89,7 +90,11 @@ jobs: env: SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute-qcom' }} SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} - CUSTOM_GIT_URL: ${{ inputs.custom_git_url || vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute' }} + # For scheduled runs: fall back to vars.KERNEL_CUSTOM_GIT_URL or the + # hardcoded carmel-team URL so the daily build always uses resolute-qcom. + # For manual runs: use exactly what the user entered — empty string means + # "use the official Ubuntu repo" (no custom URL → Launchpad REST API path). + CUSTOM_GIT_URL: ${{ github.event_name == 'schedule' && (vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute') || inputs.custom_git_url }} # GIT_TAG_PREFIX is derived automatically: # custom URL → Ubuntu-qcom (Ubuntu-qcom-7.0.0-1003.3 style tags) # official → Ubuntu (Ubuntu-X.Y.Z-A.B style tags, set in else branch) From 149411f83ae44662630dedf0bab0af5fec0e9743 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:16:44 -0700 Subject: [PATCH 064/115] docs: clarify suite input is a branch name in README tables Both workflow inputs tables now show 'suite (branch name)' and include a note that the parameter is named 'suite' for historical reasons but accepts any branch name (official suite or custom). Also updated custom_git_url description to mention clearing the field for official suite syncs. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bf4c963db6d5a..c1062fa35a9d7 100644 --- a/README.md +++ b/README.md @@ -263,9 +263,9 @@ and commits it to the branch. | Input | Default | Description | |-------|---------|-------------| -| `suite` | `resolute-qcom` | Branch name to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base suite (`resolute`) is derived automatically from the first component for Docker container selection. | +| `suite` *(branch name)* | `resolute-qcom` | Branch to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base Ubuntu suite (`resolute`) is derived automatically from the first component for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | | `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. When set, the Launchpad REST API is bypassed — the latest `Ubuntu-qcom-*` tag is discovered directly from the repo via `git ls-remote`. | +| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. Clear this field to sync an official Ubuntu suite via the Launchpad REST API. When set, the latest `Ubuntu-qcom-*` tag is discovered directly from the repo via `git ls-remote`. | **Jobs**: @@ -293,7 +293,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Input | Default | Description | |-------|---------|-------------| -| `suite` | `resolute-qcom` | Branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base suite is derived automatically for Docker container selection. | +| `suite` *(branch name)* | `resolute-qcom` | Branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base Ubuntu suite is derived automatically for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | | `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | From 9a4f1105b41551bdf412e6c360f802d03c2e6571 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 15:21:44 -0700 Subject: [PATCH 065/115] fix: change custom_git_url default to empty string MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub Actions applies non-empty defaults server-side, so 'clearing' a field with a non-empty default still sends the default value. Changing the default to empty string means leaving the field blank actually results in an empty CUSTOM_GIT_URL, correctly triggering the official Launchpad REST API path. Scheduled runs are unaffected — they use the hardcoded carmel-team URL via the github.event_name == 'schedule' check regardless of the input default. To sync an official suite manually: suite=questing, custom_git_url=(leave empty) To sync resolute-qcom manually: suite=resolute-qcom custom_git_url=https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute --- .github/workflows/fetch-source-pkg.yml | 10 ++++++---- README.md | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index d6b936965d652..0fcf62908677f 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -52,12 +52,14 @@ on: default: false custom_git_url: description: > - Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom - Launchpad repository. Clear this field to sync an official Ubuntu - suite (noble, questing, resolute) from the upstream Ubuntu kernel repo. + Custom Launchpad git URL to clone from. Leave empty to sync an + official Ubuntu suite (noble, questing, resolute) from the upstream + Ubuntu kernel repo via the Launchpad REST API. Fill in the + Ubuntu-qcom Launchpad repository URL for resolute-qcom builds. + Scheduled runs always use the Ubuntu-qcom repo automatically. Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute required: false - default: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" + default: "" type: string permissions: diff --git a/README.md b/README.md index c1062fa35a9d7..1a033af2d19cd 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ and commits it to the branch. |-------|---------|-------------| | `suite` *(branch name)* | `resolute-qcom` | Branch to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base Ubuntu suite (`resolute`) is derived automatically from the first component for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | | `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Custom Launchpad git URL to clone from. Defaults to the Ubuntu-qcom Launchpad repository. Clear this field to sync an official Ubuntu suite via the Launchpad REST API. When set, the latest `Ubuntu-qcom-*` tag is discovered directly from the repo via `git ls-remote`. | +| `custom_git_url` | *(empty)* | Custom Launchpad git URL to clone from. **Leave empty** to sync an official Ubuntu suite (noble, questing, resolute) via the Launchpad REST API. Fill in the Ubuntu-qcom Launchpad repository URL for resolute-qcom builds. Scheduled runs always use the Ubuntu-qcom repo automatically regardless of this field. | **Jobs**: From 1648b2034e2a9b72712ea184a05d00d2a86706a0 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 17:06:02 -0700 Subject: [PATCH 066/115] fix: set qcom as default flavour for resolute-qcom builds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build-kernel.yml: - Default flavour: generic → qcom (qcom is first in dropdown) - FLAVOR env fallback: 'generic' → 'qcom' fetch-source-pkg.yml: - trigger-build: branches with a suffix (e.g. resolute-qcom) → qcom official suite branches (noble, questing, resolute) → generic - Fix corrupted SPDX comment line (was split across two lines) README.md: - Pipeline diagram: binary-generic → binary-qcom - trigger-build job description: flavor=generic → flavor=qcom - build-kernel.yml inputs table: flavor default generic → qcom - Build container notes: binary-generic → binary-qcom - Rust check error example: config-prepare-check-generic → config-prepare-check-qcom --- .github/workflows/build-kernel.yml | 5 ++-- .github/workflows/fetch-source-pkg.yml | 16 ++++++++++-- README.md | 35 ++++++++++++++++---------- 3 files changed, 39 insertions(+), 17 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index cddc2a7f9d07a..e07cfdc31060e 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -42,9 +42,10 @@ on: flavor: description: "Kernel flavour to build" required: false - default: "generic" + default: "qcom" type: choice options: + - qcom - generic - lowlatency - all @@ -77,7 +78,7 @@ jobs: env: SUITE: ${{ inputs.suite || 'resolute-qcom' }} ARCH: ${{ inputs.arch || 'arm64' }} - FLAVOR: ${{ inputs.flavor || 'generic' }} + FLAVOR: ${{ inputs.flavor || 'qcom' }} steps: # ----------------------------------------------------------------------- diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 0fcf62908677f..01972b0abe779 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -398,12 +398,24 @@ jobs: SUITE: ${{ needs.check-version.outputs.suite }} VERSION: ${{ needs.check-version.outputs.version }} run: | + # Derive the kernel flavour from the branch name. + # Custom branches (any branch with a suffix, e.g. resolute-qcom) + # use the 'qcom' flavour. Official Ubuntu suite branches + # (noble, questing, resolute — no suffix) use 'generic'. + # This ensures the daily resolute-qcom build always uses qcom. + if echo "${SUITE}" | grep -q '-'; then + FLAVOR="qcom" + else + FLAVOR="generic" + fi + echo "Branch: ${SUITE} → Flavour: ${FLAVOR}" + gh workflow run build-kernel.yml \ --repo "${{ github.repository }}" \ --field suite="${SUITE}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ - --field flavor="generic" \ + --field flavor="${FLAVOR}" \ --field runner="ubuntu-24.04-arm" - echo "Build workflow dispatched for ${SUITE} ${VERSION}" + echo "Build workflow dispatched for ${SUITE} ${VERSION} (flavor=${FLAVOR})" diff --git a/README.md b/README.md index 1a033af2d19cd..5ed2f94bb1ed9 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ │ ║ ║ │ apt-get build-dep linux │ ║ ║ │ fakeroot make -f debian/rules clean ← setup env │ ║ -║ │ fakeroot debian/rules binary-generic do_skip_checks=true │ ║ +║ │ fakeroot debian/rules binary-qcom do_skip_checks=true │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ Collect .deb files ──▶ output/ ║ @@ -182,7 +182,7 @@ pkg-linux-qcom-canonical e.g. questing, resolute ``` -Suite branches are **orphan branches** — they share no history with `main` +All branches are **orphan branches** — they share no history with `main` and contain only the extracted kernel source tree. --- @@ -273,7 +273,7 @@ and commits it to the branch. |-----|-------------| | `check-version` | For resolute-qcom: queries tags via `git ls-remote` on the custom repo. For official suites: queries Launchpad API (`ws.size=300`). Checks tag existence; sets `should_sync` flag. | | `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from the resolved git URL; verifies >5000 files; commits to branch; creates tag | -| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=generic` | +| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=qcom` (for custom branches) or `flavor=generic` (for official suites) | **Idempotent**: if the tag for the latest version already exists, the workflow exits cleanly without downloading anything. @@ -296,7 +296,7 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | `suite` *(branch name)* | `resolute-qcom` | Branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base Ubuntu suite is derived automatically for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | | `kernel_version` | — | Version string for release asset attachment | | `arch` | `arm64` | Target architecture | -| `flavor` | `generic` | Kernel flavour: `generic`, `lowlatency`, or `all` | +| `flavor` | `qcom` | Kernel flavour: `qcom` (default for resolute-qcom), `generic`, `lowlatency`, or `all` | | `runner` | `ubuntu-24.04-arm` | Runner to use — see table below | **Runner options**: @@ -357,10 +357,19 @@ Go to **Actions** and enable workflows if prompted. ### 3. Run the first sync +**Resolute Qcom (custom branch):** ```bash gh workflow run fetch-source-pkg.yml \ --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=resolute-qcom + --field suite=resolute-qcom \ + --field custom_git_url="https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" +``` + +**Official Ubuntu suite (e.g. noble):** +```bash +gh workflow run fetch-source-pkg.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical \ + --field suite=noble ``` --- @@ -405,12 +414,12 @@ The sync workflow supports two paths depending on whether a custom git URL is co ```bash git ls-remote --tags \ https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ - 'refs/tags/Ubuntu-*' -# → sort -V | tail -1 → Ubuntu-7.0.0-5.5 -# → VERSION=7.0.0-5.5 + 'refs/tags/Ubuntu-qcom-*' +# → sort -V | tail -1 → Ubuntu-qcom-7.0.0-1003.3 +# → VERSION=7.0.0-1003.3 ``` -The latest `Ubuntu-*` tag in the custom repo is the authoritative version source. +The latest `Ubuntu-qcom-*` tag in the custom repo is the authoritative version source. The Launchpad REST API is not used. **Path B — Official suites (noble, questing, resolute): version from Launchpad REST API** @@ -556,11 +565,11 @@ Canonical's own build infrastructure uses. The `clean` target: - Creates **`debian/changelog → debian.master/changelog`** symlink. The Ubuntu kernel source tree does not include `debian/changelog` directly — the changelog lives in `debian.master/changelog`. The `dh_installchangelogs` - debhelper tool (called at the end of `binary-generic`) requires this symlink - to exist or the build fails after 2+ hours of compilation: + debhelper tool (called at the end of `binary-qcom` / `binary-generic`) requires + this symlink to exist or the build fails after 2+ hours of compilation: ``` dh_installchangelogs: error: cannot open file debian/changelog - make: *** [debian/rules.d/2-binary-arch.mk:572: binary-generic] Error 25 + make: *** [debian/rules.d/2-binary-arch.mk:572: binary-qcom] Error 25 ``` - Removes any stale build artifacts @@ -591,7 +600,7 @@ for all supported architectures including arm64. This check fails in the ``` check-config: CONFIG_RUST_IS_AVAILABLE changed from y to - -make: *** [debian/rules.d/4-checks.mk:15: config-prepare-check-generic] Error 1 +make: *** [debian/rules.d/4-checks.mk:15: config-prepare-check-qcom] Error 1 ``` Passing `do_skip_checks=true` to `fakeroot debian/rules` bypasses this policy From 0a7c703c131b3fc4c072ca6b6b89ee1b14a78806 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 17:16:22 -0700 Subject: [PATCH 067/115] ux: shorten workflow input descriptions; revert custom_git_url default to empty MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fetch-source-pkg.yml: - Shorten all input descriptions to single concise lines - custom_git_url default: carmel-team URL → empty string (GitHub Actions cannot clear non-empty defaults from UI; scheduled runs use carmel-team URL automatically via github.event_name == 'schedule' check regardless) build-kernel.yml: - Shorten suite and kernel_version input descriptions - Update kernel_version example to 7.0.0-1003.3 (resolute-qcom format) --- .github/workflows/build-kernel.yml | 7 ++----- .github/workflows/fetch-source-pkg.yml | 17 +++-------------- 2 files changed, 5 insertions(+), 19 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index e07cfdc31060e..135b3d7ab4cdd 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -20,15 +20,12 @@ on: workflow_dispatch: inputs: suite: - description: > - Branch to build from (e.g. noble, questing, resolute, - resolute-qcom). The base suite (e.g. resolute) is derived - automatically for Docker container selection. + description: "Branch to build from (e.g. resolute-qcom, noble, questing, resolute)" required: true default: "resolute-qcom" type: string kernel_version: - description: "Kernel version string (e.g. 6.8.0-51.52) – used for release asset attachment" + description: "Kernel version string (e.g. 7.0.0-1003.3) – used for release asset attachment" required: false type: string arch: diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 01972b0abe779..a311489081356 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -37,27 +37,16 @@ on: workflow_dispatch: inputs: suite: - description: > - Branch name to sync into (e.g. noble, questing, resolute, - resolute-qcom). This becomes the branch and tag prefix in this repo. - The base suite (e.g. resolute) is derived automatically from the - first component for Launchpad API queries and Docker container - selection. + description: "Branch to sync into (e.g. resolute-qcom, noble, questing, resolute)" required: false default: "resolute-qcom" type: string force: - description: "Force sync even if this version is already committed" + description: "Re-sync even if this version is already committed" type: boolean default: false custom_git_url: - description: > - Custom Launchpad git URL to clone from. Leave empty to sync an - official Ubuntu suite (noble, questing, resolute) from the upstream - Ubuntu kernel repo via the Launchpad REST API. Fill in the - Ubuntu-qcom Launchpad repository URL for resolute-qcom builds. - Scheduled runs always use the Ubuntu-qcom repo automatically. - Example: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute + description: "Git URL to clone from. Leave empty to use the official Ubuntu kernel repo." required: false default: "" type: string From ee59b7e7e52a760f97c2d3ac7b73b0a4f54247c4 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 19:09:15 -0700 Subject: [PATCH 068/115] feat: replace rsync snapshot with git fetch + fast-forward mirror MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the shallow-clone + rsync approach with a proper git mirror pattern that preserves the complete Canonical upstream history. Previous approach (shallow clone + rsync): - git clone --depth=1 → no history - rsync files into branch → snapshot commit - git log shows only our sync commits - git diff between versions compares full trees (noisy) - git blame points to our sync commit, not Canonical's author New approach (git fetch + fast-forward): - Clone our repo + add Launchpad as a second remote - git fetch launchpad refs/tags/:refs/tags/ (no --depth — full ancestry preserved; subsequent syncs incremental) - git merge --ff-only (or create branch on first sync) - Our tag (resolute-qcom-7.0.0-1003.3) points to Canonical's commit - No extra snapshot commit added Result: - git log shows complete Canonical history - git diff resolute-qcom-7.0.0-1003.3 resolute-qcom-7.0.0-1004.4 shows exactly what Canonical changed between versions - git blame works against Canonical's actual authors and commits - Subsequent syncs are incremental (only new commits fetched) Edge case handling: - Non-linear upstream history (rebase/rewrite): git reset --hard to maintain clean mirror + push --force-with-lease - File count verified via git ls-tree (no working-tree checkout needed) - rsync dependency removed from Install dependencies step --- .github/workflows/fetch-source-pkg.yml | 252 +++++++++++++++---------- 1 file changed, 152 insertions(+), 100 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index a311489081356..bb6687d8e6355 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -2,10 +2,15 @@ # # fetch-source-pkg.yml # -# Clones the Canonical Ubuntu kernel from the Launchpad git repository at -# the tag corresponding to the latest published source package version, and -# commits the full source tree (including the complete debian/ directory) to -# a branch named after the Ubuntu suite (e.g. "noble", "questing"). +# Mirrors the Canonical Ubuntu kernel from the Launchpad git repository into +# a branch of this repo, preserving the complete upstream git history. +# +# Why git fetch + fast-forward instead of shallow clone + rsync? +# The previous approach (git clone --depth=1 + rsync) created snapshot +# commits with no connection to Canonical's history. The current approach +# fetches the upstream tag with its full ancestry and fast-forwards our +# branch to it, so git log, git diff, and git blame all work against +# Canonical's actual commits. # # Why git instead of the source package? # The Ubuntu kernel source package (format 1.0) ships only debian.master/ @@ -16,11 +21,12 @@ # Repository branch layout # ───────────────────────── # main – CI infrastructure: workflows, scripts, documentation -# noble – Ubuntu Noble (24.04 LTS) kernel source, one commit/upload -# – additional suites added on demand (questing, resolute, …) +# noble – Ubuntu Noble (24.04 LTS) kernel source, full history +# resolute-qcom – Ubuntu-qcom kernel source, full history (daily default) +# – additional branches added on demand # -# Each commit on a suite branch is tagged - -# (e.g. noble-6.8.0-114.114). +# Each branch tip is tagged - +# (e.g. resolute-qcom-7.0.0-1003.3). # # Repository variables (Settings → Variables → Actions): # KERNEL_SUITE – default branch to sync when not specified (default: resolute-qcom) @@ -199,6 +205,25 @@ jobs: echo "Tag '${TAG}' not found – will sync" fi + # ============================================================================ + # sync job + # + # Replaces the previous shallow-clone + rsync approach with a proper git + # mirror pattern: + # + # 1. Clone our GitHub repo + # 2. Add Launchpad as a second remote + # 3. git fetch the upstream tag with its FULL ancestry (no --depth) + # 4. Fast-forward our branch to the upstream tag + # (or create the branch on first sync) + # 5. Place our scoped tag (e.g. resolute-qcom-7.0.0-1003.3) on the + # same Canonical commit — no extra snapshot commit is added + # 6. Push branch + tag + # + # Result: our branch IS the upstream history. git log, git diff, and + # git blame all work against Canonical's actual commits and authors. + # Subsequent syncs are incremental — only new commits are fetched. + # ============================================================================ sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" runs-on: ubuntu-24.04-arm @@ -218,6 +243,8 @@ jobs: steps: # ----------------------------------------------------------------------- # 1. Free up disk space + # The full Canonical history fetch requires more headroom than the + # previous shallow clone. Freeing unused toolchains gives ~10 GB. # ----------------------------------------------------------------------- - name: Free up runner disk space run: | @@ -236,46 +263,15 @@ jobs: df -h / # ----------------------------------------------------------------------- - # 2. Install tools + # 2. Install tools (rsync no longer needed) # ----------------------------------------------------------------------- - name: Install dependencies run: | sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends \ - curl jq rsync + sudo apt-get install -y --no-install-recommends curl jq # ----------------------------------------------------------------------- - # 3. Clone from the resolved git URL at the version tag. - # GIT_URL is either the custom repo URL (when custom_git_url input - # was provided) or the standard Launchpad URL for the suite. - # GIT_TAG_PREFIX determines the tag format: - # Ubuntu-qcom → Ubuntu-qcom-7.0.0-1003.3 (resolute-qcom repo) - # Ubuntu → Ubuntu-6.8.0-114.114 (official repos) - # ----------------------------------------------------------------------- - - name: Clone from git repo at version tag - run: | - GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" - - echo "Cloning ${GIT_URL} at tag ${GIT_TAG} (shallow)..." - git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ - - echo "SRC_DIR=kernel-src" >> "$GITHUB_ENV" - - FILE_COUNT=$(find kernel-src/ -type f | wc -l) - echo "Cloned ${FILE_COUNT} files" - [ "${FILE_COUNT}" -gt 5000 ] || { - echo "ERROR: Too few files cloned (${FILE_COUNT})" - exit 1 - } - - echo "Top-level contents:" - ls kernel-src/ - - echo "debian/ contents:" - ls kernel-src/debian/ | head -20 - - # ----------------------------------------------------------------------- - # 4. Configure git + # 3. Configure git identity # ----------------------------------------------------------------------- - name: Configure git identity run: | @@ -283,79 +279,135 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 5. Prepare the suite branch + # 4. Mirror: fetch upstream history and fast-forward our branch + # + # Strategy: + # • Clone our GitHub repo (contains existing branch history) + # • Add Launchpad as a second remote + # • git fetch the upstream tag with its complete ancestry + # (no --depth — full history is preserved; subsequent syncs + # are incremental, fetching only new commits) + # • Fast-forward our branch to the upstream tag + # If ff is impossible (upstream rebase/rewrite), reset --hard + # to maintain a clean mirror and push with --force-with-lease + # • Place our scoped tag on the same Canonical commit # ----------------------------------------------------------------------- - - name: Clone repository for suite branch update + - name: Mirror upstream history into branch env: GH_TOKEN: ${{ github.token }} run: | + set -euo pipefail + + GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" - git clone "${REPO_URL}" suite-repo - - name: Check out or create suite branch - run: | + # ------------------------------------------------------------------ + # Clone our GitHub repository + # ------------------------------------------------------------------ + echo "=== Cloning our repository ===" + git clone "${REPO_URL}" suite-repo cd suite-repo - if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then - echo "Branch '${SUITE}' exists – checking out" - git checkout "${SUITE}" - git rm -rf --quiet . 2>/dev/null || true - else - echo "Branch '${SUITE}' does not exist – creating orphan branch" - git checkout --orphan "${SUITE}" - git rm -rf --quiet . 2>/dev/null || true - fi - - - name: Populate suite branch with cloned source - run: | - echo "Copying source tree from ${SRC_DIR} → suite-repo/" - rsync -a --delete \ - --exclude='.git' \ - "${SRC_DIR}/" suite-repo/ - - COPIED=$(find suite-repo/ -type f | wc -l) - echo "Files in suite-repo after rsync: ${COPIED}" - [ "${COPIED}" -gt 5000 ] || { - echo "ERROR: rsync copied only ${COPIED} files" + # ------------------------------------------------------------------ + # Add Launchpad as a second remote + # ------------------------------------------------------------------ + echo "=== Adding Launchpad remote ===" + git remote add launchpad "${GIT_URL}" + echo "Remotes:" + git remote -v + + # ------------------------------------------------------------------ + # Fetch the upstream tag with its complete ancestry. + # + # No --depth: the full Canonical history is preserved in our branch. + # The first sync fetches the complete history; subsequent syncs are + # incremental — only commits reachable from the new tag that are not + # already present locally are transferred. + # + # The refspec refs/tags/:refs/tags/ fetches the tag + # object itself in addition to the commit it points to, so the tag + # is available locally for checkout and verification. + # ------------------------------------------------------------------ + echo "=== Fetching ${GIT_TAG} from ${GIT_URL} ===" + git fetch launchpad "refs/tags/${GIT_TAG}:refs/tags/${GIT_TAG}" + + CANONICAL_COMMIT=$(git rev-parse "${GIT_TAG}") + echo "Upstream commit : ${CANONICAL_COMMIT}" + echo "Upstream tag : ${GIT_TAG}" + + # ------------------------------------------------------------------ + # Verify the source tree at the fetched tag + # (git ls-tree is accurate and requires no working-tree checkout) + # ------------------------------------------------------------------ + FILE_COUNT=$(git ls-tree -r --name-only "${GIT_TAG}" | wc -l) + echo "Files at ${GIT_TAG}: ${FILE_COUNT}" + [ "${FILE_COUNT}" -gt 5000 ] || { + echo "ERROR: Too few files at tag (${FILE_COUNT}) — expected >5000" exit 1 } - cd suite-repo + echo "Top-level contents:" + git ls-tree --name-only "${GIT_TAG}" - SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ - "${SUITE}: ${VERSION}" \ - "Branch : ${SUITE}" \ - "Kernel version : ${VERSION}" \ - "Upstream : ${UPSTREAM_VERSION}" \ - "Source : ${GIT_URL} (tag: ${GIT_TAG_PREFIX}-${VERSION})" \ - "Synced : ${SYNC_DATE}" \ - > /tmp/commit-msg.txt + echo "debian/ contents:" + git ls-tree --name-only "${GIT_TAG}:debian" 2>/dev/null | head -20 \ + || echo "(debian/ not found at root — may be in a subdirectory)" - git add --all + # ------------------------------------------------------------------ + # Create or update our branch + # ------------------------------------------------------------------ + FORCE_PUSH=false - if git diff --cached --quiet; then - echo "No changes to commit – branch already has this content" + if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then + echo "=== Branch '${SUITE}' exists — updating ===" + git checkout "${SUITE}" + + # Fast-forward to the new upstream tag. + # Ubuntu/carmel-team kernel history is linear so --ff-only + # succeeds as long as the new tag descends from the branch tip. + if git merge --ff-only "${GIT_TAG}"; then + echo "✓ Fast-forwarded '${SUITE}' → ${GIT_TAG}" + else + # Non-linear history (upstream rebase or rewrite). + # Reset to maintain a clean mirror of the upstream. + echo "WARNING: Cannot fast-forward — upstream history is non-linear" + echo "Resetting '${SUITE}' to ${GIT_TAG} (mirror reset)" + git reset --hard "${GIT_TAG}" + FORCE_PUSH=true + fi else - git commit -F /tmp/commit-msg.txt - echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" + echo "=== Branch '${SUITE}' does not exist — creating ===" + git checkout -b "${SUITE}" "${GIT_TAG}" fi - - name: Tag the commit - run: | - cd suite-repo - git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" - echo "Tagged: ${TAG}" + echo "Branch tip: $(git rev-parse HEAD)" - - name: Push suite branch and tag - run: | - cd suite-repo - git push origin "${SUITE}" + # ------------------------------------------------------------------ + # Place our scoped tag on the Canonical commit. + # e.g. resolute-qcom-7.0.0-1003.3 → same commit as Ubuntu-qcom-7.0.0-1003.3 + # This preserves the full history chain — no extra snapshot commit. + # ------------------------------------------------------------------ + echo "=== Tagging ===" + git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" + echo "Tagged: ${TAG} → $(git rev-parse HEAD)" + + # ------------------------------------------------------------------ + # Push branch and tag to GitHub. + # --force-with-lease for the branch: safe force-push that fails if + # the remote was updated by another process since our clone. + # --force for the tag: tags are always overwritten on re-sync. + # ------------------------------------------------------------------ + echo "=== Pushing ===" + if [ "${FORCE_PUSH}" = "true" ]; then + git push --force-with-lease origin "${SUITE}" + else + git push origin "${SUITE}" + fi git push --force origin "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- - # 6. Summary + # 5. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() @@ -366,13 +418,13 @@ jobs: echo "" >> "$GITHUB_STEP_SUMMARY" echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch ref | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Cloned from | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Branch ref | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" + echo "| Upstream | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Upstream tag| \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" trigger-build: name: "Trigger kernel build" From 84d20cc570cc5ca12555f8bcb318c1ef46b97fb9 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 19:13:55 -0700 Subject: [PATCH 069/115] fix: add retry logic and depth limit to git fetch in sync job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The full Ubuntu kernel history fetch (no --depth) was hitting Launchpad HTTP 503 errors and would be too large for CI runners. Changes to the fetch step: - --depth=200: fetches the last 200 commits — enough for meaningful diffs between recent versions without overloading Launchpad's servers - --update-shallow: on subsequent syncs, extends the shallow boundary to include new commits while reusing already-fetched objects - Retry loop: up to 3 attempts with 30-second back-off to handle transient Launchpad 503 / RPC errors 200 commits covers several months of Ubuntu kernel development, which is more than sufficient for git diff and git log between versions. --- .github/workflows/fetch-source-pkg.yml | 36 +++++++++++++++++++------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index bb6687d8e6355..c31995fbc0c28 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -317,19 +317,37 @@ jobs: git remote -v # ------------------------------------------------------------------ - # Fetch the upstream tag with its complete ancestry. + # Fetch the upstream tag with its ancestry. # - # No --depth: the full Canonical history is preserved in our branch. - # The first sync fetches the complete history; subsequent syncs are - # incremental — only commits reachable from the new tag that are not - # already present locally are transferred. + # --depth=200: fetches the last 200 commits of history, which covers + # several months of kernel development — enough for meaningful diffs + # between versions without overloading Launchpad's git servers. + # Subsequent syncs use --update-shallow to extend the shallow boundary + # to include new commits while reusing already-fetched objects. # - # The refspec refs/tags/:refs/tags/ fetches the tag - # object itself in addition to the commit it points to, so the tag - # is available locally for checkout and verification. + # Retry up to 3 times with a 30-second back-off to handle transient + # Launchpad 503 / RPC errors. # ------------------------------------------------------------------ echo "=== Fetching ${GIT_TAG} from ${GIT_URL} ===" - git fetch launchpad "refs/tags/${GIT_TAG}:refs/tags/${GIT_TAG}" + FETCH_DEPTH=200 + MAX_RETRIES=3 + RETRY_DELAY=30 + + for attempt in $(seq 1 ${MAX_RETRIES}); do + echo "Fetch attempt ${attempt}/${MAX_RETRIES} (depth=${FETCH_DEPTH})..." + if git fetch --depth="${FETCH_DEPTH}" --update-shallow \ + launchpad "refs/tags/${GIT_TAG}:refs/tags/${GIT_TAG}"; then + echo "✓ Fetch succeeded on attempt ${attempt}" + break + fi + if [ "${attempt}" -lt "${MAX_RETRIES}" ]; then + echo "Fetch failed — retrying in ${RETRY_DELAY}s..." + sleep "${RETRY_DELAY}" + else + echo "ERROR: Fetch failed after ${MAX_RETRIES} attempts" >&2 + exit 1 + fi + done CANONICAL_COMMIT=$(git rev-parse "${GIT_TAG}") echo "Upstream commit : ${CANONICAL_COMMIT}" From 9b7b3c8c08645d98f4983cdc2da8a452b5da70db Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 19:31:58 -0700 Subject: [PATCH 070/115] feat: replace git fetch with full git clone for Launchpad mirror MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit git fetch --update-shallow into an existing repo uses a different protocol path than git clone. Launchpad's servers handle git clone reliably (same as local usage) but return 503 on git fetch operations. New sync approach: git clone --branch kernel-src/ (no --depth) git remote add github git checkout -b git push --force github This is exactly what a developer does locally. Complete upstream history is preserved — git log, git diff, and git blame work against all upstream commits and authors. --- .github/workflows/fetch-source-pkg.yml | 178 +++++++------------------ 1 file changed, 50 insertions(+), 128 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index c31995fbc0c28..37cfb8ab9d8b4 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -3,14 +3,12 @@ # fetch-source-pkg.yml # # Mirrors the Canonical Ubuntu kernel from the Launchpad git repository into -# a branch of this repo, preserving the complete upstream git history. +# a branch of this repo, preserving the upstream git history. # -# Why git fetch + fast-forward instead of shallow clone + rsync? -# The previous approach (git clone --depth=1 + rsync) created snapshot -# commits with no connection to Canonical's history. The current approach -# fetches the upstream tag with its full ancestry and fast-forwards our -# branch to it, so git log, git diff, and git blame all work against -# Canonical's actual commits. +# Approach: full git clone + push +# Full git clone from Launchpad (same as local usage — reliably served). +# The complete upstream history is preserved. git log, git diff, and +# git blame work against all upstream commits. # # Why git instead of the source package? # The Ubuntu kernel source package (format 1.0) ships only debian.master/ @@ -21,8 +19,8 @@ # Repository branch layout # ───────────────────────── # main – CI infrastructure: workflows, scripts, documentation -# noble – Ubuntu Noble (24.04 LTS) kernel source, full history -# resolute-qcom – Ubuntu-qcom kernel source, full history (daily default) +# noble – Ubuntu Noble (24.04 LTS) kernel source, one commit/upload +# resolute-qcom – Ubuntu-qcom kernel source, one commit/upload (daily default) # – additional branches added on demand # # Each branch tip is tagged - @@ -208,21 +206,18 @@ jobs: # ============================================================================ # sync job # - # Replaces the previous shallow-clone + rsync approach with a proper git - # mirror pattern: + # Approach: full git clone from Launchpad + push to GitHub # - # 1. Clone our GitHub repo - # 2. Add Launchpad as a second remote - # 3. git fetch the upstream tag with its FULL ancestry (no --depth) - # 4. Fast-forward our branch to the upstream tag - # (or create the branch on first sync) - # 5. Place our scoped tag (e.g. resolute-qcom-7.0.0-1003.3) on the - # same Canonical commit — no extra snapshot commit is added - # 6. Push branch + tag + # 1. Full git clone --branch from Launchpad (no --depth) + # Same operation as local usage — reliably served by Launchpad. + # git fetch --update-shallow causes 503 errors; git clone does not. + # 2. Add our GitHub repo as a remote + # 3. Create our branch at the cloned HEAD + # 4. Add our scoped tag on the same upstream commit + # 5. Push branch + tag to GitHub (--force for mirror semantics) # - # Result: our branch IS the upstream history. git log, git diff, and - # git blame all work against Canonical's actual commits and authors. - # Subsequent syncs are incremental — only new commits are fetched. + # Result: our branch IS the upstream history. Complete git log, git diff, + # and git blame work against all upstream commits and authors. # ============================================================================ sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" @@ -243,8 +238,6 @@ jobs: steps: # ----------------------------------------------------------------------- # 1. Free up disk space - # The full Canonical history fetch requires more headroom than the - # previous shallow clone. Freeing unused toolchains gives ~10 GB. # ----------------------------------------------------------------------- - name: Free up runner disk space run: | @@ -263,7 +256,7 @@ jobs: df -h / # ----------------------------------------------------------------------- - # 2. Install tools (rsync no longer needed) + # 2. Install tools # ----------------------------------------------------------------------- - name: Install dependencies run: | @@ -279,20 +272,20 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 4. Mirror: fetch upstream history and fast-forward our branch + # 4. Clone from Launchpad and push to GitHub + # + # git clone is reliably served by Launchpad (same as local usage). + # git fetch --update-shallow into an existing repo uses a different + # protocol path that Launchpad's servers handle poorly (503 errors). # # Strategy: - # • Clone our GitHub repo (contains existing branch history) - # • Add Launchpad as a second remote - # • git fetch the upstream tag with its complete ancestry - # (no --depth — full history is preserved; subsequent syncs - # are incremental, fetching only new commits) - # • Fast-forward our branch to the upstream tag - # If ff is impossible (upstream rebase/rewrite), reset --hard - # to maintain a clean mirror and push with --force-with-lease - # • Place our scoped tag on the same Canonical commit + # • Full git clone --branch from Launchpad (no --depth) + # • Add our GitHub repo as a remote + # • Create our branch at the cloned HEAD + # • Add our scoped tag on the same commit + # • Push branch + tag to GitHub # ----------------------------------------------------------------------- - - name: Mirror upstream history into branch + - name: Clone from Launchpad and push to GitHub env: GH_TOKEN: ${{ github.token }} run: | @@ -302,126 +295,55 @@ jobs: REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" # ------------------------------------------------------------------ - # Clone our GitHub repository - # ------------------------------------------------------------------ - echo "=== Cloning our repository ===" - git clone "${REPO_URL}" suite-repo - cd suite-repo - - # ------------------------------------------------------------------ - # Add Launchpad as a second remote + # Full clone from Launchpad at the version tag. + # git clone is the same operation as local usage — reliably served. + # No --depth: complete upstream history is preserved. # ------------------------------------------------------------------ - echo "=== Adding Launchpad remote ===" - git remote add launchpad "${GIT_URL}" - echo "Remotes:" - git remote -v + echo "=== Cloning from Launchpad (full history) ===" + git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ + cd kernel-src - # ------------------------------------------------------------------ - # Fetch the upstream tag with its ancestry. - # - # --depth=200: fetches the last 200 commits of history, which covers - # several months of kernel development — enough for meaningful diffs - # between versions without overloading Launchpad's git servers. - # Subsequent syncs use --update-shallow to extend the shallow boundary - # to include new commits while reusing already-fetched objects. - # - # Retry up to 3 times with a 30-second back-off to handle transient - # Launchpad 503 / RPC errors. - # ------------------------------------------------------------------ - echo "=== Fetching ${GIT_TAG} from ${GIT_URL} ===" - FETCH_DEPTH=200 - MAX_RETRIES=3 - RETRY_DELAY=30 - - for attempt in $(seq 1 ${MAX_RETRIES}); do - echo "Fetch attempt ${attempt}/${MAX_RETRIES} (depth=${FETCH_DEPTH})..." - if git fetch --depth="${FETCH_DEPTH}" --update-shallow \ - launchpad "refs/tags/${GIT_TAG}:refs/tags/${GIT_TAG}"; then - echo "✓ Fetch succeeded on attempt ${attempt}" - break - fi - if [ "${attempt}" -lt "${MAX_RETRIES}" ]; then - echo "Fetch failed — retrying in ${RETRY_DELAY}s..." - sleep "${RETRY_DELAY}" - else - echo "ERROR: Fetch failed after ${MAX_RETRIES} attempts" >&2 - exit 1 - fi - done - - CANONICAL_COMMIT=$(git rev-parse "${GIT_TAG}") + CANONICAL_COMMIT=$(git rev-parse HEAD) echo "Upstream commit : ${CANONICAL_COMMIT}" echo "Upstream tag : ${GIT_TAG}" # ------------------------------------------------------------------ - # Verify the source tree at the fetched tag - # (git ls-tree is accurate and requires no working-tree checkout) + # Verify the source tree # ------------------------------------------------------------------ - FILE_COUNT=$(git ls-tree -r --name-only "${GIT_TAG}" | wc -l) - echo "Files at ${GIT_TAG}: ${FILE_COUNT}" + FILE_COUNT=$(find . -not -path './.git*' -type f | wc -l) + echo "Files: ${FILE_COUNT}" [ "${FILE_COUNT}" -gt 5000 ] || { - echo "ERROR: Too few files at tag (${FILE_COUNT}) — expected >5000" + echo "ERROR: Too few files (${FILE_COUNT}) — expected >5000" exit 1 } echo "Top-level contents:" - git ls-tree --name-only "${GIT_TAG}" + ls echo "debian/ contents:" - git ls-tree --name-only "${GIT_TAG}:debian" 2>/dev/null | head -20 \ + ls debian/ 2>/dev/null | head -20 \ || echo "(debian/ not found at root — may be in a subdirectory)" # ------------------------------------------------------------------ - # Create or update our branch + # Add our GitHub repo as a remote # ------------------------------------------------------------------ - FORCE_PUSH=false - - if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then - echo "=== Branch '${SUITE}' exists — updating ===" - git checkout "${SUITE}" - - # Fast-forward to the new upstream tag. - # Ubuntu/carmel-team kernel history is linear so --ff-only - # succeeds as long as the new tag descends from the branch tip. - if git merge --ff-only "${GIT_TAG}"; then - echo "✓ Fast-forwarded '${SUITE}' → ${GIT_TAG}" - else - # Non-linear history (upstream rebase or rewrite). - # Reset to maintain a clean mirror of the upstream. - echo "WARNING: Cannot fast-forward — upstream history is non-linear" - echo "Resetting '${SUITE}' to ${GIT_TAG} (mirror reset)" - git reset --hard "${GIT_TAG}" - FORCE_PUSH=true - fi - else - echo "=== Branch '${SUITE}' does not exist — creating ===" - git checkout -b "${SUITE}" "${GIT_TAG}" - fi - - echo "Branch tip: $(git rev-parse HEAD)" + git remote add github "${REPO_URL}" # ------------------------------------------------------------------ - # Place our scoped tag on the Canonical commit. - # e.g. resolute-qcom-7.0.0-1003.3 → same commit as Ubuntu-qcom-7.0.0-1003.3 - # This preserves the full history chain — no extra snapshot commit. + # Create our branch at the current HEAD (the Launchpad tag commit) + # and add our scoped tag on the same commit. # ------------------------------------------------------------------ - echo "=== Tagging ===" + git checkout -b "${SUITE}" git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" echo "Tagged: ${TAG} → $(git rev-parse HEAD)" # ------------------------------------------------------------------ # Push branch and tag to GitHub. - # --force-with-lease for the branch: safe force-push that fails if - # the remote was updated by another process since our clone. - # --force for the tag: tags are always overwritten on re-sync. + # --force: safe for a mirror — we always move to a newer upstream tag. # ------------------------------------------------------------------ echo "=== Pushing ===" - if [ "${FORCE_PUSH}" = "true" ]; then - git push --force-with-lease origin "${SUITE}" - else - git push origin "${SUITE}" - fi - git push --force origin "${TAG}" + git push --force github "${SUITE}" + git push --force github "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- From 773260f4b3eda50c0950d5cb91e9509c671dacd5 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 19:39:20 -0700 Subject: [PATCH 071/115] feat: full git clone from Launchpad with retry and buffer tuning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use git clone --branch (no --depth) for complete upstream history. Add HTTP buffer tuning and retry logic to handle large repo transfers: http.postBuffer=500MB — prevents curl 18 / early EOF on large clones http.lowSpeedLimit=0 — disables low-speed timeout http.lowSpeedTime=max — prevents timeout on slow connections retry 3× / 30s delay — handles transient connection drops git clone is the same operation as local usage and is reliably served by Launchpad. Complete upstream history preserved. --- .github/workflows/fetch-source-pkg.yml | 28 +++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 37cfb8ab9d8b4..9fa271fed36eb 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -296,11 +296,33 @@ jobs: # ------------------------------------------------------------------ # Full clone from Launchpad at the version tag. - # git clone is the same operation as local usage — reliably served. - # No --depth: complete upstream history is preserved. + # Increase HTTP buffer to handle large repos without mid-transfer + # connection drops (curl 18 / early EOF errors). + # Retry up to 3× with 30s back-off for transient failures. # ------------------------------------------------------------------ + git config --global http.postBuffer 524288000 + git config --global http.lowSpeedLimit 0 + git config --global http.lowSpeedTime 999999 + echo "=== Cloning from Launchpad (full history) ===" - git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ + MAX_RETRIES=3 + RETRY_DELAY=30 + + for attempt in $(seq 1 ${MAX_RETRIES}); do + echo "Clone attempt ${attempt}/${MAX_RETRIES}..." + if git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then + echo "✓ Clone succeeded on attempt ${attempt}" + break + fi + rm -rf kernel-src/ + if [ "${attempt}" -lt "${MAX_RETRIES}" ]; then + echo "Clone failed — retrying in ${RETRY_DELAY}s..." + sleep "${RETRY_DELAY}" + else + echo "ERROR: Clone failed after ${MAX_RETRIES} attempts" >&2 + exit 1 + fi + done cd kernel-src CANONICAL_COMMIT=$(git rev-parse HEAD) From 46a3be367c2c3a5b15c7e17ac7269da86e6eaff1 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 19:59:04 -0700 Subject: [PATCH 072/115] =?UTF-8?q?fix:=20add=20--depth=3D200=20to=20clone?= =?UTF-8?q?=20=E2=80=94=20full=20history=20exceeds=20GitHub=20push=20limit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The full Ubuntu kernel history (~83k files, several GB) clones from Launchpad successfully but fails on git push to GitHub with HTTP 500: 'send-pack: unexpected disconnect while reading sideband packet' GitHub has a ~2GB push size limit per operation. The full kernel history exceeds this. --depth=200 covers months of kernel history for meaningful git diff between versions while staying within limits. --- .github/workflows/fetch-source-pkg.yml | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 9fa271fed36eb..d7b808fd20687 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -295,22 +295,24 @@ jobs: REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" # ------------------------------------------------------------------ - # Full clone from Launchpad at the version tag. - # Increase HTTP buffer to handle large repos without mid-transfer - # connection drops (curl 18 / early EOF errors). + # Clone from Launchpad at the version tag with --depth=200. + # Full history (no --depth) exceeds GitHub's push size limit and + # results in HTTP 500 on git push. --depth=200 covers months of + # kernel history for meaningful diffs while staying within limits. + # Increase HTTP buffer to prevent curl 18 / early EOF on clone. # Retry up to 3× with 30s back-off for transient failures. # ------------------------------------------------------------------ git config --global http.postBuffer 524288000 git config --global http.lowSpeedLimit 0 git config --global http.lowSpeedTime 999999 - echo "=== Cloning from Launchpad (full history) ===" + echo "=== Cloning from Launchpad (depth=200) ===" MAX_RETRIES=3 RETRY_DELAY=30 for attempt in $(seq 1 ${MAX_RETRIES}); do echo "Clone attempt ${attempt}/${MAX_RETRIES}..." - if git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then + if git clone --depth=200 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then echo "✓ Clone succeeded on attempt ${attempt}" break fi From f6f77f769e796940dce661954603cf406d6478b0 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 20:12:35 -0700 Subject: [PATCH 073/115] feat: full git clone + push for complete Canonical history mirror MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revert to full git clone (no --depth) with HTTP buffer tuning. The noble repo failed with HTTP 500 (too large), but the carmel-team resolute repo is a newer/smaller fork and should push successfully. git clone --branch (no --depth, full history) git remote add github git checkout -b git push --force github HTTP tuning to prevent curl 18 / early EOF: http.postBuffer=500MB http.lowSpeedLimit=0 http.lowSpeedTime=999999 Retry 3× with 30s back-off for transient failures. --- .github/workflows/fetch-source-pkg.yml | 43 +++++++++++++------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index d7b808fd20687..9af162f084c10 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -5,10 +5,10 @@ # Mirrors the Canonical Ubuntu kernel from the Launchpad git repository into # a branch of this repo, preserving the upstream git history. # -# Approach: full git clone + push -# Full git clone from Launchpad (same as local usage — reliably served). -# The complete upstream history is preserved. git log, git diff, and -# git blame work against all upstream commits. +# Approach: full git clone + push (complete mirror) +# Full git clone from Launchpad (same as local usage) then push to GitHub. +# The complete upstream history is preserved — git log, git diff, and +# git blame work against all of Canonical's actual commits and authors. # # Why git instead of the source package? # The Ubuntu kernel source package (format 1.0) ships only debian.master/ @@ -206,18 +206,18 @@ jobs: # ============================================================================ # sync job # - # Approach: full git clone from Launchpad + push to GitHub + # Approach: full git clone from Launchpad + push to GitHub (complete mirror) # # 1. Full git clone --branch from Launchpad (no --depth) # Same operation as local usage — reliably served by Launchpad. - # git fetch --update-shallow causes 503 errors; git clone does not. + # HTTP buffer tuned to prevent curl 18 / early EOF on large clones. # 2. Add our GitHub repo as a remote - # 3. Create our branch at the cloned HEAD - # 4. Add our scoped tag on the same upstream commit + # 3. Create our branch at the cloned HEAD (Canonical's commit) + # 4. Add our scoped tag on the same commit # 5. Push branch + tag to GitHub (--force for mirror semantics) # - # Result: our branch IS the upstream history. Complete git log, git diff, - # and git blame work against all upstream commits and authors. + # Result: our branch IS the complete upstream history. git log shows all + # of Canonical's commits. git diff and git blame work against upstream. # ============================================================================ sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" @@ -293,26 +293,27 @@ jobs: GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" + MAX_RETRIES=3 + RETRY_DELAY=30 # ------------------------------------------------------------------ - # Clone from Launchpad at the version tag with --depth=200. - # Full history (no --depth) exceeds GitHub's push size limit and - # results in HTTP 500 on git push. --depth=200 covers months of - # kernel history for meaningful diffs while staying within limits. - # Increase HTTP buffer to prevent curl 18 / early EOF on clone. - # Retry up to 3× with 30s back-off for transient failures. + # Tune git HTTP for large repo transfers. + # Prevents curl 18 (early EOF) on large clones. # ------------------------------------------------------------------ git config --global http.postBuffer 524288000 git config --global http.lowSpeedLimit 0 git config --global http.lowSpeedTime 999999 - echo "=== Cloning from Launchpad (depth=200) ===" - MAX_RETRIES=3 - RETRY_DELAY=30 + # ------------------------------------------------------------------ + # Full clone from Launchpad at the version tag (no --depth). + # Same operation as local usage — reliably served by Launchpad. + # Retry up to 3× with 30s back-off for transient failures. + # ------------------------------------------------------------------ + echo "=== Cloning from Launchpad (full history) ===" for attempt in $(seq 1 ${MAX_RETRIES}); do echo "Clone attempt ${attempt}/${MAX_RETRIES}..." - if git clone --depth=200 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then + if git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then echo "✓ Clone succeeded on attempt ${attempt}" break fi @@ -354,7 +355,7 @@ jobs: git remote add github "${REPO_URL}" # ------------------------------------------------------------------ - # Create our branch at the current HEAD (the Launchpad tag commit) + # Create our branch at the current HEAD (Canonical's commit) # and add our scoped tag on the same commit. # ------------------------------------------------------------------ git checkout -b "${SUITE}" From c55ec76b170de0749799f14f89e79ceb479d3908 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 23:44:20 -0700 Subject: [PATCH 074/115] =?UTF-8?q?fix:=20switch=20to=20--depth=3D1=20?= =?UTF-8?q?=E2=80=94=20full=20clone=20exceeds=20GitHub=202GB=20push=20limi?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full git clone of the kernel repo pushes successfully from Launchpad but fails on git push to GitHub (HTTP 500 / send-pack disconnect) because the full history exceeds GitHub's 2GB per-push limit. --depth=1 is the only depth that works end-to-end: full clone → HTTP 500 on push (too large) --depth=N>1 → push rejected (cannot resolve shallow boundary parents) --depth=1 → works (single root commit, no parent to resolve) Key property preserved: the branch tip has the SAME commit hash as Canonical's branch. PRs raised on top will apply cleanly on Canonical's branch via git cherry-pick — the parent commit is identical. This is fundamentally different from the old orphan approach which used rsync + git commit (creating a new commit with a different hash that would NOT apply on Canonical's branch). --- .github/workflows/fetch-source-pkg.yml | 50 ++++++++++++++------------ 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 9af162f084c10..16408b7b269de 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -5,10 +5,12 @@ # Mirrors the Canonical Ubuntu kernel from the Launchpad git repository into # a branch of this repo, preserving the upstream git history. # -# Approach: full git clone + push (complete mirror) -# Full git clone from Launchpad (same as local usage) then push to GitHub. -# The complete upstream history is preserved — git log, git diff, and -# git blame work against all of Canonical's actual commits and authors. +# Approach: git clone --depth=1 + push (Canonical commit hash preserved) +# Clones the exact Canonical commit at the version tag (--depth=1) and +# pushes it directly to GitHub — no rsync, no extra commit. The branch +# tip has the same hash as Canonical's commit, so PRs raised on top will +# apply cleanly on Canonical's branch via git cherry-pick. +# Full history push exceeds GitHub's 2GB push limit for large repos. # # Why git instead of the source package? # The Ubuntu kernel source package (format 1.0) ships only debian.master/ @@ -206,18 +208,24 @@ jobs: # ============================================================================ # sync job # - # Approach: full git clone from Launchpad + push to GitHub (complete mirror) + # Approach: git clone --depth=1 + push (Canonical commit hash preserved) # - # 1. Full git clone --branch from Launchpad (no --depth) - # Same operation as local usage — reliably served by Launchpad. - # HTTP buffer tuned to prevent curl 18 / early EOF on large clones. + # 1. git clone --depth=1 --branch from Launchpad + # Fetches exactly the Canonical commit at the version tag. + # --depth=1 is the only depth that works end-to-end: + # full clone → GitHub push fails (HTTP 500, exceeds 2GB limit) + # --depth=N>1 → GitHub push fails (cannot resolve shallow parents) + # --depth=1 → works (single root commit, no parent to resolve) # 2. Add our GitHub repo as a remote - # 3. Create our branch at the cloned HEAD (Canonical's commit) + # 3. Create our branch at the cloned HEAD (Canonical's actual commit) # 4. Add our scoped tag on the same commit # 5. Push branch + tag to GitHub (--force for mirror semantics) # - # Result: our branch IS the complete upstream history. git log shows all - # of Canonical's commits. git diff and git blame work against upstream. + # Key property: the branch tip has the SAME commit hash as Canonical's + # branch. PRs raised on top will apply cleanly on Canonical's branch + # via git cherry-pick — the parent commit is identical. + # git diff compares the two source trees and shows exactly + # what Canonical changed between releases. # ============================================================================ sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" @@ -297,23 +305,19 @@ jobs: RETRY_DELAY=30 # ------------------------------------------------------------------ - # Tune git HTTP for large repo transfers. - # Prevents curl 18 (early EOF) on large clones. - # ------------------------------------------------------------------ - git config --global http.postBuffer 524288000 - git config --global http.lowSpeedLimit 0 - git config --global http.lowSpeedTime 999999 - - # ------------------------------------------------------------------ - # Full clone from Launchpad at the version tag (no --depth). - # Same operation as local usage — reliably served by Launchpad. + # Clone from Launchpad at the version tag with --depth=1. + # Fetches exactly the Canonical commit — no extra history. + # --depth=1 is the only value that works end-to-end with GitHub: + # full clone → HTTP 500 on push (exceeds GitHub's 2GB limit) + # --depth=N>1 → push rejected (cannot resolve shallow parents) + # --depth=1 → works (single root commit, no parent to resolve) # Retry up to 3× with 30s back-off for transient failures. # ------------------------------------------------------------------ - echo "=== Cloning from Launchpad (full history) ===" + echo "=== Cloning from Launchpad (depth=1) ===" for attempt in $(seq 1 ${MAX_RETRIES}); do echo "Clone attempt ${attempt}/${MAX_RETRIES}..." - if git clone --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then + if git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then echo "✓ Clone succeeded on attempt ${attempt}" break fi From e97b97c0d2cbfde3913cc71b2ccc70a7db332f1a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 27 Apr 2026 23:51:57 -0700 Subject: [PATCH 075/115] fix: add --no-thin to git push to fix 'did not receive expected object' When pushing a shallow clone (--depth=1), git creates a thin pack that uses objects already on GitHub as delta bases. If GitHub does not have the expected base objects (e.g. after a force-push replaced the branch), the remote index-pack fails with: 'remote: fatal: did not receive expected object ' --no-thin sends all objects explicitly in the pack, without relying on the remote having any base objects. This fixes the push for both new branches and existing branches that were previously force-pushed. --- .github/workflows/fetch-source-pkg.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 16408b7b269de..4f5f1ca21d3d2 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -369,10 +369,12 @@ jobs: # ------------------------------------------------------------------ # Push branch and tag to GitHub. # --force: safe for a mirror — we always move to a newer upstream tag. + # --no-thin: send all objects explicitly; thin packs fail when GitHub + # does not have the expected base objects for delta resolution. # ------------------------------------------------------------------ echo "=== Pushing ===" - git push --force github "${SUITE}" - git push --force github "${TAG}" + git push --force --no-thin github "${SUITE}" + git push --force --no-thin github "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- From cbd7c97d6d265a77de945b29d9afdc3221741d31 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 28 Apr 2026 08:33:16 -0700 Subject: [PATCH 076/115] revert: restore original shallow clone + rsync sync approach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revert to the proven orphan/rsync approach that was in place before 'feat: replace rsync snapshot with git fetch + fast-forward mirror'. The git mirror approach was attempted in multiple forms but all failed due to GitHub infrastructure constraints: - Full clone push → HTTP 500 (exceeds GitHub's 2GB push limit) - --depth=N>1 push → 'did not receive expected object' (shallow boundary) - --depth=1 push → same 'did not receive expected object' error - --no-thin flag → did not resolve the issue The original approach (git clone --depth=1 + rsync + git commit) is reliable and has been working since the repo was created. It is the correct approach for GitHub-hosted CI with these infrastructure limits. Note: the sync job retains all other improvements made during this session (resolute-qcom default, qcom flavor derivation, custom_git_url support, Ubuntu-qcom tag prefix handling, etc.). --- .github/workflows/fetch-source-pkg.yml | 222 ++++++++++++------------- 1 file changed, 102 insertions(+), 120 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 4f5f1ca21d3d2..44c58cdb1158e 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -2,15 +2,14 @@ # # fetch-source-pkg.yml # -# Mirrors the Canonical Ubuntu kernel from the Launchpad git repository into -# a branch of this repo, preserving the upstream git history. +# Syncs the Canonical Ubuntu kernel from the Launchpad git repository into +# a branch of this repo, building an incremental commit history of syncs. # -# Approach: git clone --depth=1 + push (Canonical commit hash preserved) -# Clones the exact Canonical commit at the version tag (--depth=1) and -# pushes it directly to GitHub — no rsync, no extra commit. The branch -# tip has the same hash as Canonical's commit, so PRs raised on top will -# apply cleanly on Canonical's branch via git cherry-pick. -# Full history push exceeds GitHub's 2GB push limit for large repos. +# Approach: shallow clone + rsync + incremental commit +# git clone --depth=1 fetches exactly one commit from Launchpad — reliably +# served without triggering rate limits or 503 errors. Each sync adds a +# commit to our branch showing exactly what changed between kernel versions. +# git diff between any two of our tags shows the precise delta. # # Why git instead of the source package? # The Ubuntu kernel source package (format 1.0) ships only debian.master/ @@ -205,28 +204,6 @@ jobs: echo "Tag '${TAG}' not found – will sync" fi - # ============================================================================ - # sync job - # - # Approach: git clone --depth=1 + push (Canonical commit hash preserved) - # - # 1. git clone --depth=1 --branch from Launchpad - # Fetches exactly the Canonical commit at the version tag. - # --depth=1 is the only depth that works end-to-end: - # full clone → GitHub push fails (HTTP 500, exceeds 2GB limit) - # --depth=N>1 → GitHub push fails (cannot resolve shallow parents) - # --depth=1 → works (single root commit, no parent to resolve) - # 2. Add our GitHub repo as a remote - # 3. Create our branch at the cloned HEAD (Canonical's actual commit) - # 4. Add our scoped tag on the same commit - # 5. Push branch + tag to GitHub (--force for mirror semantics) - # - # Key property: the branch tip has the SAME commit hash as Canonical's - # branch. PRs raised on top will apply cleanly on Canonical's branch - # via git cherry-pick — the parent commit is identical. - # git diff compares the two source trees and shows exactly - # what Canonical changed between releases. - # ============================================================================ sync: name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" runs-on: ubuntu-24.04-arm @@ -269,10 +246,41 @@ jobs: - name: Install dependencies run: | sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends curl jq + sudo apt-get install -y --no-install-recommends \ + curl jq rsync # ----------------------------------------------------------------------- - # 3. Configure git identity + # 3. Clone from the resolved git URL at the version tag. + # GIT_URL is either the custom repo URL (when custom_git_url input + # was provided) or the standard Launchpad URL for the suite. + # GIT_TAG_PREFIX determines the tag format: + # Ubuntu-qcom → Ubuntu-qcom-7.0.0-1003.3 (resolute-qcom repo) + # Ubuntu → Ubuntu-6.8.0-114.114 (official repos) + # ----------------------------------------------------------------------- + - name: Clone from git repo at version tag + run: | + GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" + + echo "Cloning ${GIT_URL} at tag ${GIT_TAG} (shallow)..." + git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ + + echo "SRC_DIR=kernel-src" >> "$GITHUB_ENV" + + FILE_COUNT=$(find kernel-src/ -type f | wc -l) + echo "Cloned ${FILE_COUNT} files" + [ "${FILE_COUNT}" -gt 5000 ] || { + echo "ERROR: Too few files cloned (${FILE_COUNT})" + exit 1 + } + + echo "Top-level contents:" + ls kernel-src/ + + echo "debian/ contents:" + ls kernel-src/debian/ | head -20 + + # ----------------------------------------------------------------------- + # 4. Configure git # ----------------------------------------------------------------------- - name: Configure git identity run: | @@ -280,105 +288,79 @@ jobs: git config --global user.name "github-actions[bot]" # ----------------------------------------------------------------------- - # 4. Clone from Launchpad and push to GitHub - # - # git clone is reliably served by Launchpad (same as local usage). - # git fetch --update-shallow into an existing repo uses a different - # protocol path that Launchpad's servers handle poorly (503 errors). - # - # Strategy: - # • Full git clone --branch from Launchpad (no --depth) - # • Add our GitHub repo as a remote - # • Create our branch at the cloned HEAD - # • Add our scoped tag on the same commit - # • Push branch + tag to GitHub + # 5. Prepare the suite branch # ----------------------------------------------------------------------- - - name: Clone from Launchpad and push to GitHub + - name: Clone repository for suite branch update env: GH_TOKEN: ${{ github.token }} run: | - set -euo pipefail - - GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" - MAX_RETRIES=3 - RETRY_DELAY=30 - - # ------------------------------------------------------------------ - # Clone from Launchpad at the version tag with --depth=1. - # Fetches exactly the Canonical commit — no extra history. - # --depth=1 is the only value that works end-to-end with GitHub: - # full clone → HTTP 500 on push (exceeds GitHub's 2GB limit) - # --depth=N>1 → push rejected (cannot resolve shallow parents) - # --depth=1 → works (single root commit, no parent to resolve) - # Retry up to 3× with 30s back-off for transient failures. - # ------------------------------------------------------------------ - echo "=== Cloning from Launchpad (depth=1) ===" - - for attempt in $(seq 1 ${MAX_RETRIES}); do - echo "Clone attempt ${attempt}/${MAX_RETRIES}..." - if git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/; then - echo "✓ Clone succeeded on attempt ${attempt}" - break - fi - rm -rf kernel-src/ - if [ "${attempt}" -lt "${MAX_RETRIES}" ]; then - echo "Clone failed — retrying in ${RETRY_DELAY}s..." - sleep "${RETRY_DELAY}" - else - echo "ERROR: Clone failed after ${MAX_RETRIES} attempts" >&2 - exit 1 - fi - done - cd kernel-src - - CANONICAL_COMMIT=$(git rev-parse HEAD) - echo "Upstream commit : ${CANONICAL_COMMIT}" - echo "Upstream tag : ${GIT_TAG}" - - # ------------------------------------------------------------------ - # Verify the source tree - # ------------------------------------------------------------------ - FILE_COUNT=$(find . -not -path './.git*' -type f | wc -l) - echo "Files: ${FILE_COUNT}" - [ "${FILE_COUNT}" -gt 5000 ] || { - echo "ERROR: Too few files (${FILE_COUNT}) — expected >5000" + git clone "${REPO_URL}" suite-repo + + - name: Check out or create suite branch + run: | + cd suite-repo + + if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then + echo "Branch '${SUITE}' exists – checking out" + git checkout "${SUITE}" + git rm -rf --quiet . 2>/dev/null || true + else + echo "Branch '${SUITE}' does not exist – creating orphan branch" + git checkout --orphan "${SUITE}" + git rm -rf --quiet . 2>/dev/null || true + fi + + - name: Populate suite branch with cloned source + run: | + echo "Copying source tree from ${SRC_DIR} → suite-repo/" + rsync -a --delete \ + --exclude='.git' \ + "${SRC_DIR}/" suite-repo/ + + COPIED=$(find suite-repo/ -type f | wc -l) + echo "Files in suite-repo after rsync: ${COPIED}" + [ "${COPIED}" -gt 5000 ] || { + echo "ERROR: rsync copied only ${COPIED} files" exit 1 } - echo "Top-level contents:" - ls + cd suite-repo - echo "debian/ contents:" - ls debian/ 2>/dev/null | head -20 \ - || echo "(debian/ not found at root — may be in a subdirectory)" - - # ------------------------------------------------------------------ - # Add our GitHub repo as a remote - # ------------------------------------------------------------------ - git remote add github "${REPO_URL}" - - # ------------------------------------------------------------------ - # Create our branch at the current HEAD (Canonical's commit) - # and add our scoped tag on the same commit. - # ------------------------------------------------------------------ - git checkout -b "${SUITE}" + SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') + printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ + "${SUITE}: ${VERSION}" \ + "Branch : ${SUITE}" \ + "Kernel version : ${VERSION}" \ + "Upstream : ${UPSTREAM_VERSION}" \ + "Source : ${GIT_URL} (tag: ${GIT_TAG_PREFIX}-${VERSION})" \ + "Synced : ${SYNC_DATE}" \ + > /tmp/commit-msg.txt + + git add --all + + if git diff --cached --quiet; then + echo "No changes to commit – branch already has this content" + else + git commit -F /tmp/commit-msg.txt + echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" + fi + + - name: Tag the commit + run: | + cd suite-repo git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" - echo "Tagged: ${TAG} → $(git rev-parse HEAD)" - - # ------------------------------------------------------------------ - # Push branch and tag to GitHub. - # --force: safe for a mirror — we always move to a newer upstream tag. - # --no-thin: send all objects explicitly; thin packs fail when GitHub - # does not have the expected base objects for delta resolution. - # ------------------------------------------------------------------ - echo "=== Pushing ===" - git push --force --no-thin github "${SUITE}" - git push --force --no-thin github "${TAG}" + echo "Tagged: ${TAG}" + + - name: Push suite branch and tag + run: | + cd suite-repo + git push origin "${SUITE}" + git push --force origin "${TAG}" echo "Pushed branch '${SUITE}' and tag '${TAG}'" # ----------------------------------------------------------------------- - # 5. Summary + # 6. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() @@ -393,8 +375,8 @@ jobs: echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Branch ref | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Upstream | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Upstream tag| \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Cloned from | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" + echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" trigger-build: From cd2a62dadb98dd969f050f64b3e1bf2ec75c6833 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 28 Apr 2026 11:12:23 -0700 Subject: [PATCH 077/115] fix: production runner S3/gh issues + default daily builds to lecore-production MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build-kernel.yml: - S3 upload condition: runner.name == '...' → inputs.runner == 'lecore-production' (runner.name checks the display name which may not match the label; inputs.runner is reliable since it's the explicit workflow input) - Add 'Install GitHub CLI (if not present)' step before the release step (gh is not pre-installed on self-hosted runners; installs from official GitHub CLI apt repo if missing, no-op if already present) - Renumber steps 11→12 (release), 12→13 (summary) fetch-source-pkg.yml: - trigger-build: runner=ubuntu-24.04-arm → runner=lecore-production (all daily scheduled builds now run on the production runner) --- .github/workflows/build-kernel.yml | 32 +++++++++++++++++++++----- .github/workflows/fetch-source-pkg.yml | 4 ++-- 2 files changed, 28 insertions(+), 8 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 135b3d7ab4cdd..11ff6e86496fc 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -285,12 +285,13 @@ jobs: ls -lh output/ || echo "(no files found)" # ----------------------------------------------------------------------- - # 9. Upload to S3 (self-hosted runner only) - # Skipped on GitHub-hosted runners — S3 credentials and network - # access are only available on lecore-prd-u2404-arm64-xlrg-od-ephem. + # 9. Upload to S3 (lecore-production runner only) + # Skipped on GitHub-hosted and generic self-hosted runners. + # Condition checks the runner input, not runner.name, because the + # runner's registered display name may differ from its label. # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 - if: runner.name == 'lecore-prd-u2404-arm64-xlrg-od-ephem' + if: inputs.runner == 'lecore-production' uses: qualcomm-linux/upload-private-artifact-action@aws with: s3_bucket: qli-prd-lecore-gh-artifacts @@ -309,7 +310,26 @@ jobs: if-no-files-found: warn # ----------------------------------------------------------------------- - # 11. Create GitHub Release and attach built packages + # 11. Install GitHub CLI if not present (required on self-hosted runners) + # ----------------------------------------------------------------------- + - name: Install GitHub CLI (if not present) + if: inputs.kernel_version != '' + run: | + if ! command -v gh &>/dev/null; then + echo "gh not found — installing..." + curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \ + | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg + echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" \ + | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null + sudo apt-get update -qq + sudo apt-get install -y gh + echo "✓ gh installed: $(gh --version | head -1)" + else + echo "✓ gh already available: $(gh --version | head -1)" + fi + + # ----------------------------------------------------------------------- + # 12. Create GitHub Release and attach built packages # ----------------------------------------------------------------------- - name: Create GitHub Release and attach packages if: inputs.kernel_version != '' @@ -349,7 +369,7 @@ jobs: echo "Release URL: https://github.com/${{ github.repository }}/releases/tag/${TAG}" # ----------------------------------------------------------------------- - # 12. Summary + # 13. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 44c58cdb1158e..de6bc6c626034 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -410,6 +410,6 @@ jobs: --field kernel_version="${VERSION}" \ --field arch="arm64" \ --field flavor="${FLAVOR}" \ - --field runner="ubuntu-24.04-arm" + --field runner="lecore-production" - echo "Build workflow dispatched for ${SUITE} ${VERSION} (flavor=${FLAVOR})" + echo "Build workflow dispatched for ${SUITE} ${VERSION} (flavor=${FLAVOR}, runner=lecore-production)" From 7b99835a37a30a8af2c0acaf0551fdfc2a067cc5 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 28 Apr 2026 14:42:53 -0700 Subject: [PATCH 078/115] docs: update README to reflect lecore-production as active build runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Pipeline diagram header: ubuntu-24.04-arm → lecore-production - trigger-build box: add runner=lecore-production field - Footer line: split into sync runner (ubuntu-24.04-arm) and build runner (lecore-production) - build-kernel.yml Runner section: 'pending runner group access' → 'active — used by scheduled builds' - build-kernel.yml Runner description: ubuntu-24.04-arm → lecore-production (scheduled builds) - Dispatch sentence: 'always dispatches with runner=ubuntu-24.04-arm' → 'runner=lecore-production' - S3 output table: 'Self-hosted runner only' → 'lecore-production runner only' --- README.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 5ed2f94bb1ed9..f3665a6cbcb1b 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. ## End-to-end pipeline ``` -SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm +SCHEDULE: daily 04:00 UTC · RUNNER: lecore-production ══════════════════════════════════════════════════════════════════════════════ Ubuntu-qcom Launchpad repository @@ -47,6 +47,7 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm ║ │ │ ║ ║ │ gh workflow run build-kernel.yml │ ║ ║ │ suite=resolute-qcom kernel_version=X.Y.Z-A.B arch=arm64 │ ║ +║ │ runner=lecore-production │ ║ ║ └──────────────────────────────────────────────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ │ @@ -83,8 +84,8 @@ SCHEDULE: daily 04:00 UTC · RUNNER: ubuntu-24.04-arm └──────────────┘ └──────────────────┘ └──────────────────┘ ``` -All jobs run on: `ubuntu-24.04-arm` (GitHub-hosted, Ubuntu 24.04 arm64) -Target runner: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted) — pending runner group access +Sync jobs run on: `ubuntu-24.04-arm` (GitHub-hosted, Ubuntu 24.04 arm64) +Build jobs run on: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted, lecore-production) --- @@ -287,7 +288,7 @@ container using `fakeroot debian/rules binary-`. **Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. -**Runner**: `ubuntu-24.04-arm` +**Runner**: `lecore-production` (scheduled builds); selectable for manual builds **Inputs**: @@ -303,11 +304,11 @@ manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow | Option | Resolves to | Status | |--------|-------------|--------| -| `ubuntu-24.04-arm` | GitHub-hosted 2-core arm64 | **default** — used by scheduled builds | +| `ubuntu-24.04-arm` | GitHub-hosted 2-core arm64 | manual builds / fallback | | `self-hosted` | `runs-on: self-hosted` — any registered self-hosted runner | interim dev runner | -| `lecore-production` | `runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem]` | **target** — pending runner group access | +| `lecore-production` | `runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem]` | **active** — used by scheduled builds | -The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The `lecore-production` runner enables S3 artifact upload (permanent storage) in addition to the GitHub Actions artifact fallback. +The scheduled daily sync always dispatches with `runner=lecore-production`. The `lecore-production` runner enables S3 artifact upload (permanent storage) in addition to the GitHub Actions artifact fallback. **Self-hosted runner requirements:** - Ubuntu 24.04 arm64 @@ -333,7 +334,7 @@ The scheduled daily sync always dispatches with `runner=ubuntu-24.04-arm`. The ` | Location | How to access | Retention | Notes | |----------|---------------|-----------|-------| -| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | Self-hosted runner only; skipped gracefully on GitHub-hosted | +| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | `lecore-production` runner only; skipped on other runners | | **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | Always available | | **GitHub Release asset** | Releases → `-X.Y.Z-A.B` → Assets | Permanent | Attached when `kernel_version` is provided | From ee49fc2a49c26f24129feb8c16f9c1879a6852a1 Mon Sep 17 00:00:00 2001 From: GitHub Service Bot Date: Thu, 21 May 2026 21:51:19 +0000 Subject: [PATCH 079/115] chore: sync workflows from qcom-build-utils Signed-off-by: GitHub Service Bot --- .github/workflows/build-debian-package.yml | 40 ++++++++++++++++ .github/workflows/promote-prebuilt.yml | 50 ++++++++++++++++++++ .github/workflows/promote-upstream.yml | 33 +++++++++++++ .github/workflows/release.yml | 55 ++++++++++++++++++++++ 4 files changed, 178 insertions(+) create mode 100644 .github/workflows/build-debian-package.yml create mode 100644 .github/workflows/promote-prebuilt.yml create mode 100644 .github/workflows/promote-upstream.yml create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/build-debian-package.yml b/.github/workflows/build-debian-package.yml new file mode 100644 index 0000000000000..17ec9a6a1fb37 --- /dev/null +++ b/.github/workflows/build-debian-package.yml @@ -0,0 +1,40 @@ +name: Build Debian Package +description: | + Builds the debian package represented by this repo at the ref pointed by the debain-ref argument. + +on: + workflow_dispatch: + inputs: + + debian-ref: + description: The debian ref to build. For example branch "debian/qcom-next" or tag "debian/1.0.0-1" + type: string + required: true + default: debian/qcom-next + + suite: + description: The distribution codename or Debian suite to build for. Ex noble, questing, resolute, trixie, sid, unstable + type: choice + default: unstable + options: + - noble + - questing + - resolute + - unstable + - forky + - trixie + - bookworm + - sid + +permissions: + contents: read + packages: read + +jobs: + build: + uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-build-pkg-reusable-workflow.yml@main + with: + qcom-build-utils-ref: main + debian-ref: ${{ inputs.debian-ref }} + suite: ${{ inputs.suite }} + debusine-parent-workspace: ${{ vars.DEBUSINE_PARENT_WORKSPACE }} diff --git a/.github/workflows/promote-prebuilt.yml b/.github/workflows/promote-prebuilt.yml new file mode 100644 index 0000000000000..1a388f66a5931 --- /dev/null +++ b/.github/workflows/promote-prebuilt.yml @@ -0,0 +1,50 @@ +name: Promote New Prebuilt Binary Version + +on: + workflow_dispatch: + inputs: + + debian-branch: + description: The debian branch to apply the promotion to. For example "debian/qcom-next" + type: string + required: false + default: debian/qcom-next + + new-tag: + description: | + The new Artifactory TAG to promote to (e.g. 251030.2). + This is the TAG field in upstream.conf. + type: string + required: true + + new-package-name: + description: | + The new tarball filename if it has changed (e.g. qcom-adreno_1.838.2_armv8-2a.tar.gz). + Leave empty to keep the current PACKAGE_NAME in upstream.conf unchanged. + type: string + required: false + default: "" + + new-debian-version: + description: | + The new debian changelog version (e.g. 1.838.2~251030 or 1.838.2-1). + If omitted, the version is automatically derived from the PACKAGE_NAME + (e.g. qcom-adreno_1.838.2_armv8-2a.tar.gz -> 1.838.2-1). + type: string + required: false + default: "" + +permissions: + contents: write + pull-requests: write + +jobs: + promote: + uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-prebuilt-reusable-workflow.yml@dev/pkg-bin + with: + qcom-build-utils-ref: dev/pkg-bin + debian-branch: ${{inputs.debian-branch}} + new-tag: ${{inputs.new-tag}} + new-package-name: ${{inputs.new-package-name}} + new-debian-version: ${{inputs.new-debian-version}} + secrets: inherit diff --git a/.github/workflows/promote-upstream.yml b/.github/workflows/promote-upstream.yml new file mode 100644 index 0000000000000..4fa36550b4383 --- /dev/null +++ b/.github/workflows/promote-upstream.yml @@ -0,0 +1,33 @@ +name: Promote New Upstream Version + +on: + workflow_dispatch: + inputs: + + debian-branch: + description: The debian branch to apply the promotion to. For example branch "debian/qcom-next" + type: string + required: false + default: debian/qcom-next + + upstream-tag: + description: The upstream tag to promote this package repo to. Eg, v1.1.0 or 1.2.0, depending on the versioning style + type: string + required: true + +permissions: + contents: write + packages: read + pull-requests: write + +jobs: + promote: + + uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-upstream-reusable-workflow.yml@main + with: + qcom-build-utils-ref: main + debian-branch: ${{inputs.debian-branch}} + upstream-tag: ${{inputs.upstream-tag}} + upstream-repo: ${{vars.UPSTREAM_REPO_GITHUB_NAME}} + secrets: + PAT: ${{secrets.DEB_PKG_BOT_CI_TOKEN}} # If the source repo is private, this secret is necessary to be provided in the repo \ No newline at end of file diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000000000..61b0b047691b7 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,55 @@ +name: Release Version +description: | + Release a new package version. + This workflow is manually triggered via workflow_dispatch. + +on: + workflow_dispatch: + inputs: + + suite: + description: The distribution codename or Debian suite to release for. Ex noble, questing, resolute, trixie, bookworm, sid, unstable + type: choice + default: noble + options: + - noble + - questing + - resolute + - unstable + - forky + - trixie + - bookworm + - sid + + debian-branch: + description: The debian branch to use to execute the release from. For example branch "debian/qcom-next" + type: string + required: false + default: debian/qcom-next + + test-run: + description: | + Debian suites: if true, stop after the Debusine build and installability test. + Ubuntu codenames: preserve the previous release flow, including upload to the test S3 location. + type: boolean + default: true + +permissions: + contents: write + packages: read + +jobs: + release: + + uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-release-reusable-workflow.yml@main + with: + qcom-build-utils-ref: main + suite: ${{ github.event.inputs.suite }} + debian-branch: ${{ github.event.inputs.debian-branch }} + test-run: ${{ github.event.inputs.test-run == 'true' && true || false }} + debusine-parent-workspace: ${{ vars.DEBUSINE_PARENT_WORKSPACE }} + secrets: + PAT: ${{ secrets.DEB_PKG_BOT_CI_TOKEN }} + DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} + DEBUSINE_RELEASE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} From 297e9f065fe2a52ff8498c230b6b8e8060cf97ab Mon Sep 17 00:00:00 2001 From: GitHub Service Bot Date: Sat, 23 May 2026 00:34:13 +0000 Subject: [PATCH 080/115] chore: sync workflows from qcom-build-utils Signed-off-by: GitHub Service Bot --- .github/workflows/build-debian-package.yml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/.github/workflows/build-debian-package.yml b/.github/workflows/build-debian-package.yml index 17ec9a6a1fb37..7ad7432d26186 100644 --- a/.github/workflows/build-debian-package.yml +++ b/.github/workflows/build-debian-package.yml @@ -26,6 +26,11 @@ on: - bookworm - sid + force-docker-build: + description: Force local pkg-builder instead of Debusine for Debian-family suites + type: boolean + default: false + permissions: contents: read packages: read @@ -37,4 +42,8 @@ jobs: qcom-build-utils-ref: main debian-ref: ${{ inputs.debian-ref }} suite: ${{ inputs.suite }} + force-docker-build: ${{ inputs.force-docker-build }} debusine-parent-workspace: ${{ vars.DEBUSINE_PARENT_WORKSPACE }} + secrets: + DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} + DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} From 3cc5e31e391abbf65a07875dc160e93ff19cf991 Mon Sep 17 00:00:00 2001 From: qcom-service-bot Date: Thu, 28 May 2026 11:37:19 -0700 Subject: [PATCH 081/115] chore: sync workflows from qcom-build-utils (#6) Signed-off-by: GitHub Service Bot --- .github/workflows/promote-prebuilt.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/promote-prebuilt.yml b/.github/workflows/promote-prebuilt.yml index 1a388f66a5931..0334e816c978b 100644 --- a/.github/workflows/promote-prebuilt.yml +++ b/.github/workflows/promote-prebuilt.yml @@ -37,12 +37,13 @@ on: permissions: contents: write pull-requests: write + packages: read jobs: promote: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-prebuilt-reusable-workflow.yml@dev/pkg-bin + uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-prebuilt-reusable-workflow.yml@main with: - qcom-build-utils-ref: dev/pkg-bin + qcom-build-utils-ref: main debian-branch: ${{inputs.debian-branch}} new-tag: ${{inputs.new-tag}} new-package-name: ${{inputs.new-package-name}} From bde4d7c8c2db4ac0eed7bb71d0c24c0a7f025004 Mon Sep 17 00:00:00 2001 From: GitHub Service Bot Date: Tue, 9 Jun 2026 19:10:58 +0000 Subject: [PATCH 082/115] chore: sync workflows from qcom-build-utils Signed-off-by: GitHub Service Bot --- .../workflows/{build-debian-package.yml => pkg-build.yml} | 6 +++--- .../{promote-prebuilt.yml => pkg-promote-prebuilt.yml} | 6 +++--- .../workflows/{promote-upstream.yml => pkg-promote.yml} | 8 ++++---- .github/workflows/{release.yml => pkg-release.yml} | 6 +++--- 4 files changed, 13 insertions(+), 13 deletions(-) rename .github/workflows/{build-debian-package.yml => pkg-build.yml} (87%) rename .github/workflows/{promote-prebuilt.yml => pkg-promote-prebuilt.yml} (88%) rename .github/workflows/{promote-upstream.yml => pkg-promote.yml} (81%) rename .github/workflows/{release.yml => pkg-release.yml} (89%) diff --git a/.github/workflows/build-debian-package.yml b/.github/workflows/pkg-build.yml similarity index 87% rename from .github/workflows/build-debian-package.yml rename to .github/workflows/pkg-build.yml index 7ad7432d26186..958b08725b6da 100644 --- a/.github/workflows/build-debian-package.yml +++ b/.github/workflows/pkg-build.yml @@ -7,10 +7,10 @@ on: inputs: debian-ref: - description: The debian ref to build. For example branch "debian/qcom-next" or tag "debian/1.0.0-1" + description: The debian ref to build. For example branch "qcom/debian/latest" or tag "debian/1.0.0-1" type: string required: true - default: debian/qcom-next + default: qcom/debian/latest suite: description: The distribution codename or Debian suite to build for. Ex noble, questing, resolute, trixie, sid, unstable @@ -37,7 +37,7 @@ permissions: jobs: build: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-build-pkg-reusable-workflow.yml@main + uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-build-reusable-workflow.yml@main with: qcom-build-utils-ref: main debian-ref: ${{ inputs.debian-ref }} diff --git a/.github/workflows/promote-prebuilt.yml b/.github/workflows/pkg-promote-prebuilt.yml similarity index 88% rename from .github/workflows/promote-prebuilt.yml rename to .github/workflows/pkg-promote-prebuilt.yml index 0334e816c978b..e1b2bb9ef5057 100644 --- a/.github/workflows/promote-prebuilt.yml +++ b/.github/workflows/pkg-promote-prebuilt.yml @@ -5,10 +5,10 @@ on: inputs: debian-branch: - description: The debian branch to apply the promotion to. For example "debian/qcom-next" + description: The debian branch to apply the promotion to. For example "qcom/debian/latest" type: string required: false - default: debian/qcom-next + default: qcom/debian/latest new-tag: description: | @@ -41,7 +41,7 @@ permissions: jobs: promote: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-prebuilt-reusable-workflow.yml@main + uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-promote-prebuilt-reusable-workflow.yml@main with: qcom-build-utils-ref: main debian-branch: ${{inputs.debian-branch}} diff --git a/.github/workflows/promote-upstream.yml b/.github/workflows/pkg-promote.yml similarity index 81% rename from .github/workflows/promote-upstream.yml rename to .github/workflows/pkg-promote.yml index 4fa36550b4383..564114fbdcaa4 100644 --- a/.github/workflows/promote-upstream.yml +++ b/.github/workflows/pkg-promote.yml @@ -5,10 +5,10 @@ on: inputs: debian-branch: - description: The debian branch to apply the promotion to. For example branch "debian/qcom-next" + description: The debian branch to apply the promotion to. For example branch "qcom/debian/latest" type: string required: false - default: debian/qcom-next + default: qcom/debian/latest upstream-tag: description: The upstream tag to promote this package repo to. Eg, v1.1.0 or 1.2.0, depending on the versioning style @@ -23,11 +23,11 @@ permissions: jobs: promote: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-promote-upstream-reusable-workflow.yml@main + uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-promote-reusable-workflow.yml@main with: qcom-build-utils-ref: main debian-branch: ${{inputs.debian-branch}} upstream-tag: ${{inputs.upstream-tag}} upstream-repo: ${{vars.UPSTREAM_REPO_GITHUB_NAME}} secrets: - PAT: ${{secrets.DEB_PKG_BOT_CI_TOKEN}} # If the source repo is private, this secret is necessary to be provided in the repo \ No newline at end of file + PAT: ${{secrets.DEB_PKG_BOT_CI_TOKEN}} # If the source repo is private, this secret is necessary to be provided in the repo diff --git a/.github/workflows/release.yml b/.github/workflows/pkg-release.yml similarity index 89% rename from .github/workflows/release.yml rename to .github/workflows/pkg-release.yml index 61b0b047691b7..542be328eb08c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/pkg-release.yml @@ -22,10 +22,10 @@ on: - sid debian-branch: - description: The debian branch to use to execute the release from. For example branch "debian/qcom-next" + description: The debian branch to use to execute the release from. For example branch "qcom/debian/latest" type: string required: false - default: debian/qcom-next + default: qcom/debian/latest test-run: description: | @@ -41,7 +41,7 @@ permissions: jobs: release: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/qcom-release-reusable-workflow.yml@main + uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-release-reusable-workflow.yml@main with: qcom-build-utils-ref: main suite: ${{ github.event.inputs.suite }} From 859167284122b8f749c1745a9155a835d1f5718b Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 19 Jun 2026 15:10:39 -0700 Subject: [PATCH 083/115] ci: remove vestigial qcom-build-utils pkg-* workflows PRs #4-#8 ("chore: sync workflows from qcom-build-utils") auto-synced four reusable-workflow wrappers into this repo: pkg-build.yml, pkg-promote.yml, pkg-promote-prebuilt.yml, and pkg-release.yml. They call qualcomm-linux/qcom-build-utils reusable workflows that require a qcom/debian/latest gbp/quilt packaging branch and an upstream.conf, neither of which exists here. This repo builds through its own engine (fetch-source-pkg.yml -> build-kernel.yml), so these wrappers are inapplicable and have only ever failed (e.g. "a branch or tag with the name 'qcom/debian/latest' could not be found"). No other file references them; removing them drops four dormant, non-functional workflows from the Actions tab. The build pipeline is unaffected. Signed-off-by: Bjordis Collaku --- .github/workflows/pkg-build.yml | 49 ------------------- .github/workflows/pkg-promote-prebuilt.yml | 51 -------------------- .github/workflows/pkg-promote.yml | 33 ------------- .github/workflows/pkg-release.yml | 55 ---------------------- 4 files changed, 188 deletions(-) delete mode 100644 .github/workflows/pkg-build.yml delete mode 100644 .github/workflows/pkg-promote-prebuilt.yml delete mode 100644 .github/workflows/pkg-promote.yml delete mode 100644 .github/workflows/pkg-release.yml diff --git a/.github/workflows/pkg-build.yml b/.github/workflows/pkg-build.yml deleted file mode 100644 index 958b08725b6da..0000000000000 --- a/.github/workflows/pkg-build.yml +++ /dev/null @@ -1,49 +0,0 @@ -name: Build Debian Package -description: | - Builds the debian package represented by this repo at the ref pointed by the debain-ref argument. - -on: - workflow_dispatch: - inputs: - - debian-ref: - description: The debian ref to build. For example branch "qcom/debian/latest" or tag "debian/1.0.0-1" - type: string - required: true - default: qcom/debian/latest - - suite: - description: The distribution codename or Debian suite to build for. Ex noble, questing, resolute, trixie, sid, unstable - type: choice - default: unstable - options: - - noble - - questing - - resolute - - unstable - - forky - - trixie - - bookworm - - sid - - force-docker-build: - description: Force local pkg-builder instead of Debusine for Debian-family suites - type: boolean - default: false - -permissions: - contents: read - packages: read - -jobs: - build: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-build-reusable-workflow.yml@main - with: - qcom-build-utils-ref: main - debian-ref: ${{ inputs.debian-ref }} - suite: ${{ inputs.suite }} - force-docker-build: ${{ inputs.force-docker-build }} - debusine-parent-workspace: ${{ vars.DEBUSINE_PARENT_WORKSPACE }} - secrets: - DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} diff --git a/.github/workflows/pkg-promote-prebuilt.yml b/.github/workflows/pkg-promote-prebuilt.yml deleted file mode 100644 index e1b2bb9ef5057..0000000000000 --- a/.github/workflows/pkg-promote-prebuilt.yml +++ /dev/null @@ -1,51 +0,0 @@ -name: Promote New Prebuilt Binary Version - -on: - workflow_dispatch: - inputs: - - debian-branch: - description: The debian branch to apply the promotion to. For example "qcom/debian/latest" - type: string - required: false - default: qcom/debian/latest - - new-tag: - description: | - The new Artifactory TAG to promote to (e.g. 251030.2). - This is the TAG field in upstream.conf. - type: string - required: true - - new-package-name: - description: | - The new tarball filename if it has changed (e.g. qcom-adreno_1.838.2_armv8-2a.tar.gz). - Leave empty to keep the current PACKAGE_NAME in upstream.conf unchanged. - type: string - required: false - default: "" - - new-debian-version: - description: | - The new debian changelog version (e.g. 1.838.2~251030 or 1.838.2-1). - If omitted, the version is automatically derived from the PACKAGE_NAME - (e.g. qcom-adreno_1.838.2_armv8-2a.tar.gz -> 1.838.2-1). - type: string - required: false - default: "" - -permissions: - contents: write - pull-requests: write - packages: read - -jobs: - promote: - uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-promote-prebuilt-reusable-workflow.yml@main - with: - qcom-build-utils-ref: main - debian-branch: ${{inputs.debian-branch}} - new-tag: ${{inputs.new-tag}} - new-package-name: ${{inputs.new-package-name}} - new-debian-version: ${{inputs.new-debian-version}} - secrets: inherit diff --git a/.github/workflows/pkg-promote.yml b/.github/workflows/pkg-promote.yml deleted file mode 100644 index 564114fbdcaa4..0000000000000 --- a/.github/workflows/pkg-promote.yml +++ /dev/null @@ -1,33 +0,0 @@ -name: Promote New Upstream Version - -on: - workflow_dispatch: - inputs: - - debian-branch: - description: The debian branch to apply the promotion to. For example branch "qcom/debian/latest" - type: string - required: false - default: qcom/debian/latest - - upstream-tag: - description: The upstream tag to promote this package repo to. Eg, v1.1.0 or 1.2.0, depending on the versioning style - type: string - required: true - -permissions: - contents: write - packages: read - pull-requests: write - -jobs: - promote: - - uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-promote-reusable-workflow.yml@main - with: - qcom-build-utils-ref: main - debian-branch: ${{inputs.debian-branch}} - upstream-tag: ${{inputs.upstream-tag}} - upstream-repo: ${{vars.UPSTREAM_REPO_GITHUB_NAME}} - secrets: - PAT: ${{secrets.DEB_PKG_BOT_CI_TOKEN}} # If the source repo is private, this secret is necessary to be provided in the repo diff --git a/.github/workflows/pkg-release.yml b/.github/workflows/pkg-release.yml deleted file mode 100644 index 542be328eb08c..0000000000000 --- a/.github/workflows/pkg-release.yml +++ /dev/null @@ -1,55 +0,0 @@ -name: Release Version -description: | - Release a new package version. - This workflow is manually triggered via workflow_dispatch. - -on: - workflow_dispatch: - inputs: - - suite: - description: The distribution codename or Debian suite to release for. Ex noble, questing, resolute, trixie, bookworm, sid, unstable - type: choice - default: noble - options: - - noble - - questing - - resolute - - unstable - - forky - - trixie - - bookworm - - sid - - debian-branch: - description: The debian branch to use to execute the release from. For example branch "qcom/debian/latest" - type: string - required: false - default: qcom/debian/latest - - test-run: - description: | - Debian suites: if true, stop after the Debusine build and installability test. - Ubuntu codenames: preserve the previous release flow, including upload to the test S3 location. - type: boolean - default: true - -permissions: - contents: write - packages: read - -jobs: - release: - - uses: qualcomm-linux/qcom-build-utils/.github/workflows/pkg-release-reusable-workflow.yml@main - with: - qcom-build-utils-ref: main - suite: ${{ github.event.inputs.suite }} - debian-branch: ${{ github.event.inputs.debian-branch }} - test-run: ${{ github.event.inputs.test-run == 'true' && true || false }} - debusine-parent-workspace: ${{ vars.DEBUSINE_PARENT_WORKSPACE }} - secrets: - PAT: ${{ secrets.DEB_PKG_BOT_CI_TOKEN }} - DEBUSINE_USER: ${{ secrets.DEBUSINE_USER }} - DEBUSINE_TOKEN: ${{ secrets.DEBUSINE_TOKEN }} - DEBUSINE_RELEASE_TOKEN: ${{ secrets.DEBUSINE_RELEASE_TOKEN }} From e4a89bc271707498ef4b9d9339550fabbaed2cd2 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 23 Jun 2026 14:49:39 -0700 Subject: [PATCH 084/115] Update fetch-source-pkg.yml to remove chron job Signed-off-by: Bjordis Collaku --- .github/workflows/fetch-source-pkg.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index de6bc6c626034..bc4374cf662e3 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -36,9 +36,6 @@ name: "Sync: Canonical Kernel Sources to Branch" on: - schedule: - - cron: "0 4 * * *" - workflow_dispatch: inputs: suite: From 3f85ef82d42f88a816d0434cd92fcd60f579007e Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 23 Jun 2026 15:40:30 -0700 Subject: [PATCH 085/115] Reference aws-v4 tag for upload artifacts Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 11ff6e86496fc..8575b109fb72c 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -292,7 +292,7 @@ jobs: # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 if: inputs.runner == 'lecore-production' - uses: qualcomm-linux/upload-private-artifact-action@aws + uses: qualcomm-linux/upload-private-artifact-action@aws-v4 with: s3_bucket: qli-prd-lecore-gh-artifacts path: ${{ github.workspace }}/output From ec20c58e8a5de41bf652d544b05965b635b05636 Mon Sep 17 00:00:00 2001 From: GitHub Service Bot Date: Wed, 24 Jun 2026 21:52:54 +0000 Subject: [PATCH 086/115] chore: remove qcom-preflight-checks workflow Signed-off-by: GitHub Service Bot --- .github/workflows/qcom-preflight-checks.yml | 24 --------------------- 1 file changed, 24 deletions(-) delete mode 100644 .github/workflows/qcom-preflight-checks.yml diff --git a/.github/workflows/qcom-preflight-checks.yml b/.github/workflows/qcom-preflight-checks.yml deleted file mode 100644 index 1b068996ed5ab..0000000000000 --- a/.github/workflows/qcom-preflight-checks.yml +++ /dev/null @@ -1,24 +0,0 @@ -name: QC Preflight Checks - -on: - pull_request: - push: - branches: [main] - workflow_dispatch: - -jobs: - preflight: - name: Run QC Preflight Checks - uses: qualcomm/qcom-reusable-workflows/.github/workflows/reusable-qcom-preflight-checks-orchestrator.yml@v2 - with: - enable-semgrep-scan: true - enable-dependency-review: true - enable-repolinter-check: true - enable-copyright-license-check: true - enable-commit-email-check: true - enable-commit-msg-check: false - enable-armor-checkers: false - - permissions: - contents: read - security-events: write From 6ec2e6bdcc7b80e8addca1fdb8978daf4f944acd Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 24 Jun 2026 09:58:20 -0700 Subject: [PATCH 087/115] ci: preserve Canonical kernel history with verbatim-tag mirror Replace the legacy shallow-clone + rsync + squash sync (one flattened commit per upload, no upstream history) with a history-preserving mirror, and align it with the operating model the team settled on. Mirror model - The resolute-qcom branch faithfully mirrors the upstream Canonical tree with full ancestry. It is a movable "latest" pointer; the upstream Canonical tag is mirrored VERBATIM (Ubuntu-qcom-X.Y.Z-A.B) as the immutable per-upload record -- no re-tagging into a - namespace. Sync (scripts/sync-mirror.sh + fetch-source-pkg.yml) - Pure fetch + repoint: fetch the upstream tag (delta only), advance the branch, atomic lease-pinned push of branch + the verbatim tag. Never merges/rebases, so it cannot conflict or be blocked by developer patches on the branch. - Manual workflow_dispatch only (no cron, no inputs): this repo always mirrors the carmel-team resolute repository into resolute-qcom, so the upstream URL, branch, and tag prefix are fixed constants. Bootstrap (scripts/seed-history.sh + bootstrap-history.yml) - One-time, automated, single-shot full clone (Launchpad's shallow/deepen path is broken; it can serve a full clone), then a sliced <2 GB push of the verbatim upstream tags. Pushes to a -seed branch (never the live branch) for a human to review and promote. The clone is cached so a retry after a failed publish skips re-downloading the history. Build (build-kernel.yml) - Output goes ONLY to S3 (lecore-production runner, the default): the GitHub Actions artifact and GitHub Release steps are removed; token scope reduced to contents: read. - Version builds check out the verbatim tag Ubuntu-qcom-. Validated end-to-end in CI on the seeded 1.43M-commit resolute-qcom branch (build green, .deb to S3). actionlint + shellcheck clean. Signed-off-by: Bjordis Collaku --- .github/workflows/bootstrap-history.yml | 181 ++++++++++ .github/workflows/build-kernel.yml | 161 +++------ .github/workflows/fetch-source-pkg.yml | 454 ++++++------------------ scripts/seed-history.sh | 187 ++++++++++ scripts/sync-mirror.sh | 242 +++++++++++++ 5 files changed, 762 insertions(+), 463 deletions(-) create mode 100644 .github/workflows/bootstrap-history.yml create mode 100755 scripts/seed-history.sh create mode 100755 scripts/sync-mirror.sh diff --git a/.github/workflows/bootstrap-history.yml b/.github/workflows/bootstrap-history.yml new file mode 100644 index 0000000000000..57c36f0a08d65 --- /dev/null +++ b/.github/workflows/bootstrap-history.yml @@ -0,0 +1,181 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# bootstrap-history.yml +# +# One-time, automated bootstrap of a suite branch with the COMPLETE Canonical +# kernel history from Launchpad. Run this ONCE per suite (via "Run workflow"); +# thereafter fetch-source-pkg.yml keeps it current incrementally. +# +# This exists because the steady-state sync is incremental and therefore needs a +# real-history base to exist first, but the branch starts with no usable history +# (the legacy pipeline stored flattened snapshots with no upstream ancestry). +# +# How it works (single shot) +# ────────────────────────── +# scripts/seed-history.sh does a FULL `git clone` of the upstream history +# (Launchpad's shallow/deepen path is broken, but it can serve a full clone), +# then pushes the seed in <2 GB slices (GitHub caps a single push at 2 GB) with +# the upstream upload tags mirrored verbatim. +# +# * The seed is pushed to a -seed branch (e.g. resolute-qcom-seed), +# NEVER the live branch -- a human reviews and promotes it. +# * The cloned repo is cached (actions/cache); if the publish fails, a manual +# re-dispatch restores the cache and skips re-downloading the history. + +name: "Bootstrap: Seed Canonical Kernel History" + +on: + workflow_dispatch: + inputs: + suite: + description: "Suite to seed (e.g. resolute-qcom, noble, questing, resolute). Pushed to -seed." + required: false + default: "resolute-qcom" + type: string + custom_git_url: + description: "Git URL to seed from. Leave empty to use the official Ubuntu kernel repo for the base suite." + required: false + default: "" + type: string + +permissions: + contents: write # push the seeded branch and tags + +jobs: + bootstrap: + name: "Seed ${{ inputs.suite }}-seed" + runs-on: ubuntu-24.04-arm + timeout-minutes: 350 + + steps: + # ----------------------------------------------------------------------- + # 1. Free disk space -- a full kernel history is multi-GB. + # ----------------------------------------------------------------------- + - name: Free up runner disk space + run: | + sudo rm -rf \ + /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/share/boost \ + "$AGENT_TOOLSDIRECTORY" 2>/dev/null || true + sudo apt-get clean + df -h / + + # ----------------------------------------------------------------------- + # 2. Check out this repo just for the scripts/ helpers. + # ----------------------------------------------------------------------- + - name: Checkout CI scripts + uses: actions/checkout@v6 + + # ----------------------------------------------------------------------- + # 3. Resolve the seed configuration (URL, tag prefix, seed branch). + # ----------------------------------------------------------------------- + - name: Resolve configuration + id: cfg + env: + SUITE: ${{ inputs.suite || 'resolute-qcom' }} + CUSTOM_GIT_URL: ${{ inputs.custom_git_url }} + run: | + BASE_SUITE="$(echo "${SUITE}" | cut -d'-' -f1)" + + if [ -n "${CUSTOM_GIT_URL}" ]; then + UPSTREAM_URL="${CUSTOM_GIT_URL}" + UPSTREAM_PREFIX="Ubuntu-qcom" + elif echo "${SUITE}" | grep -q '-'; then + # A custom (suffixed) suite with no URL given defaults to the + # Qualcomm-Ubuntu carmel-team repository for that base suite. + UPSTREAM_URL="https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/${BASE_SUITE}" + UPSTREAM_PREFIX="Ubuntu-qcom" + else + UPSTREAM_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${BASE_SUITE}" + UPSTREAM_PREFIX="Ubuntu" + fi + + { + echo "suite=${SUITE}" + echo "upstream_url=${UPSTREAM_URL}" + echo "upstream_prefix=${UPSTREAM_PREFIX}" + echo "seed_branch=${SUITE}-seed" + echo "cache_path=${RUNNER_TEMP}/history-seed" + } >> "$GITHUB_OUTPUT" + + # Seed pushed to -seed (NOT the live branch); tags mirrored verbatim. + echo "Suite ${SUITE} -> seed branch ${SUITE}-seed from ${UPSTREAM_URL} (tags ${UPSTREAM_PREFIX}-* verbatim)" + + # ----------------------------------------------------------------------- + # 4. Restore a previously-cloned seed (so a retry after a failed publish + # skips re-downloading the multi-GB history). + # ----------------------------------------------------------------------- + - name: Restore cached clone + uses: actions/cache/restore@v4 + with: + path: ${{ steps.cfg.outputs.cache_path }} + key: history-seed-${{ steps.cfg.outputs.suite }}-${{ github.run_id }} + restore-keys: | + history-seed-${{ steps.cfg.outputs.suite }}- + + # ----------------------------------------------------------------------- + # 5. Full-clone the history and publish the seed. Capture the exit code so + # the clone can be cached before the job is failed on a publish error. + # ----------------------------------------------------------------------- + - name: Seed history + id: seed + env: + MIRROR_URL: https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git + UPSTREAM_URL: ${{ steps.cfg.outputs.upstream_url }} + BRANCH: ${{ steps.cfg.outputs.seed_branch }} + UPSTREAM_PREFIX: ${{ steps.cfg.outputs.upstream_prefix }} + WORKDIR: ${{ steps.cfg.outputs.cache_path }} + run: | + git config --global user.email "github-actions[bot]@users.noreply.github.com" + git config --global user.name "github-actions[bot]" + + set +e + bash scripts/seed-history.sh + rc=$? + set -e + + echo "rc=${rc}" >> "$GITHUB_OUTPUT" + if [ "${rc}" -eq 0 ]; then + echo "status=complete" >> "$GITHUB_OUTPUT" + else + echo "status=failed" >> "$GITHUB_OUTPUT" + fi + + # ----------------------------------------------------------------------- + # 6. On failure, cache the cloned history so a manual re-dispatch can reuse + # it instead of re-downloading from the slow Launchpad server. + # ----------------------------------------------------------------------- + - name: Cache clone for retry + if: steps.seed.outputs.status == 'failed' + uses: actions/cache/save@v4 + with: + path: ${{ steps.cfg.outputs.cache_path }} + key: history-seed-${{ steps.cfg.outputs.suite }}-${{ github.run_id }} + + # ----------------------------------------------------------------------- + # 7. Surface a publish/clone failure as a red job, AFTER the clone is cached. + # ----------------------------------------------------------------------- + - name: Fail on seed error + if: steps.seed.outputs.status == 'failed' + run: | + echo "::error::seed-history.sh failed (rc=${{ steps.seed.outputs.rc }})." \ + "The clone is cached; investigate and re-dispatch to retry the publish." + exit 1 + + # ----------------------------------------------------------------------- + # 8. Summary. + # ----------------------------------------------------------------------- + - name: Print summary + if: always() + run: | + { + echo "## Bootstrap Summary" + echo "" + echo "| Field | Value |" + echo "|-------|-------|" + echo "| Seed branch | \`${{ steps.cfg.outputs.seed_branch }}\` |" + echo "| Status | \`${{ steps.seed.outputs.status || 'failed' }}\` |" + if [ "${{ steps.seed.outputs.status }}" = "complete" ]; then + echo "| Result | Seed pushed -- review and promote \`${{ steps.cfg.outputs.seed_branch }}\` to the live branch |" + fi + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 8575b109fb72c..1788113799d84 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -10,9 +10,9 @@ # docker-pkg-build, ensuring a clean, reproducible environment # that exactly matches the target distro. # -# Output .deb packages are stored in two places: -# 1. S3 bucket – qli-prd-lecore-gh-artifacts (permanent) -# 2. GitHub artifact – attached to the workflow run, retained 90 days +# Output .deb packages are uploaded ONLY to the S3 bucket +# (qli-prd-lecore-gh-artifacts), on the lecore-production runner. No GitHub +# Actions artifacts and no GitHub Releases are produced. name: "Build: Canonical Kernel .deb Packages" @@ -20,12 +20,12 @@ on: workflow_dispatch: inputs: suite: - description: "Branch to build from (e.g. resolute-qcom, noble, questing, resolute)" + description: "Branch to build from (e.g. resolute-qcom)" required: true default: "resolute-qcom" type: string kernel_version: - description: "Kernel version string (e.g. 7.0.0-1003.3) – used for release asset attachment" + description: "Version (e.g. 7.0.0-1006.8) - builds the exact Ubuntu-qcom- tag. Empty = branch HEAD." required: false type: string arch: @@ -47,29 +47,29 @@ on: - lowlatency - all runner: - description: "Runner to use for the build" + description: "Runner to use for the build (lecore-production is the only one that uploads to S3)" required: false - default: "ubuntu-24.04-arm" + default: "lecore-production" type: choice options: + - lecore-production - ubuntu-24.04-arm - self-hosted - - lecore-production permissions: - contents: write # upload assets to release tag + contents: read # checkout + tag ls-remote only; output goes to S3 jobs: # -------------------------------------------------------------------------- # Job: build # -------------------------------------------------------------------------- build: - name: "Build ${{ inputs.suite }} ${{ inputs.flavor }} (${{ inputs.arch }})" + name: "Build" # Map friendly runner names to their actual runs-on labels. - # self-hosted → "self-hosted" (bcollaku dev machine) - # lecore-production → ["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"] - # ubuntu-24.04-arm → "ubuntu-24.04-arm" (GitHub-hosted, default) - runs-on: ${{ fromJSON('{"ubuntu-24.04-arm":"ubuntu-24.04-arm","self-hosted":"self-hosted","lecore-production":["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"]}')[inputs.runner || 'ubuntu-24.04-arm'] }} + # lecore-production → ["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"] (default; uploads to S3) + # ubuntu-24.04-arm → "ubuntu-24.04-arm" (GitHub-hosted) + # self-hosted → "self-hosted" + runs-on: ${{ fromJSON('{"ubuntu-24.04-arm":"ubuntu-24.04-arm","self-hosted":"self-hosted","lecore-production":["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"]}')[inputs.runner || 'lecore-production'] }} timeout-minutes: 360 env: @@ -109,14 +109,21 @@ jobs: echo "REPO_NAME=${GITHUB_REPOSITORY#*/}" >> "$GITHUB_ENV" # ----------------------------------------------------------------------- - # 2b. Derive base suite for Docker container selection. - # Strips any custom suffix so the correct pkg-builder image is used. - # e.g. resolute-qcom → resolute, noble → noble, questing → questing + # 2b. Derive base suite (for Docker image) and upstream tag prefix. + # BASE_SUITE strips any custom suffix so the correct pkg-builder image + # is used (resolute-qcom → resolute). UPSTREAM_PREFIX is the verbatim + # Canonical tag prefix the source branch was mirrored with + # (custom/qcom suites → Ubuntu-qcom; official suites → Ubuntu). # ----------------------------------------------------------------------- - - name: Derive base suite for Docker + - name: Derive base suite and tag prefix run: | BASE_SUITE=$(echo "${SUITE}" | cut -d'-' -f1) echo "BASE_SUITE=${BASE_SUITE}" >> "$GITHUB_ENV" + if echo "${SUITE}" | grep -q '-'; then + echo "UPSTREAM_PREFIX=Ubuntu-qcom" >> "$GITHUB_ENV" + else + echo "UPSTREAM_PREFIX=Ubuntu" >> "$GITHUB_ENV" + fi echo "Suite: ${SUITE} → Base suite: ${BASE_SUITE}" # ----------------------------------------------------------------------- @@ -129,7 +136,7 @@ jobs: env: GH_TOKEN: ${{ github.token }} run: | - TAG="${SUITE}-${{ inputs.kernel_version }}" + TAG="${UPSTREAM_PREFIX}-${{ inputs.kernel_version }}" echo "Verifying tag '${TAG}' exists in repository..." if git ls-remote --exit-code --tags \ @@ -148,20 +155,20 @@ jobs: # ----------------------------------------------------------------------- # 4. Checkout the kernel source - # - If kernel_version is provided: checkout the exact tag - # - so the build uses the precise synced - # source regardless of what else is on the branch tip. + # - If kernel_version is provided: checkout the exact verbatim Canonical + # tag - (e.g. Ubuntu-qcom-7.0.0-1006.8) + # so the build uses the precise synced source. # - If kernel_version is empty: checkout the suite branch HEAD - # (test/dev builds — no release will be created). + # (test/dev builds). # Note: actions/checkout checks out a ref of THIS repository - # (pkg-linux-qcom-canonical). The resolute/noble/questing branches - # of this repo contain the full kernel source tree populated by + # (pkg-linux-qcom-canonical). The resolute-qcom/noble/... branches of + # this repo contain the full kernel source tree mirrored by # fetch-source-pkg.yml — they are not the main branch. # ----------------------------------------------------------------------- - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', inputs.suite || 'resolute-qcom', inputs.kernel_version) || inputs.suite || 'resolute-qcom' }} + ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', env.UPSTREAM_PREFIX, inputs.kernel_version) || inputs.suite || 'resolute-qcom' }} path: kernel-src # ----------------------------------------------------------------------- @@ -299,92 +306,24 @@ jobs: destination: ${{ env.ORG_NAME }}/pkg/temp/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ # ----------------------------------------------------------------------- - # 10. Upload as GitHub Actions artifact (90-day fallback) - # ----------------------------------------------------------------------- - - name: Upload packages as artifact - uses: actions/upload-artifact@v7 - with: - name: "kernel-debs-${{ inputs.suite }}-${{ inputs.arch }}-${{ inputs.flavor }}" - path: output/ - retention-days: 90 - if-no-files-found: warn - - # ----------------------------------------------------------------------- - # 11. Install GitHub CLI if not present (required on self-hosted runners) - # ----------------------------------------------------------------------- - - name: Install GitHub CLI (if not present) - if: inputs.kernel_version != '' - run: | - if ! command -v gh &>/dev/null; then - echo "gh not found — installing..." - curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \ - | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg - echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" \ - | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null - sudo apt-get update -qq - sudo apt-get install -y gh - echo "✓ gh installed: $(gh --version | head -1)" - else - echo "✓ gh already available: $(gh --version | head -1)" - fi - - # ----------------------------------------------------------------------- - # 12. Create GitHub Release and attach built packages - # ----------------------------------------------------------------------- - - name: Create GitHub Release and attach packages - if: inputs.kernel_version != '' - env: - GH_TOKEN: ${{ github.token }} - run: | - TAG="${SUITE}-${{ inputs.kernel_version }}" - DEBS=$(find output/ -name "*.deb" | sort) - - if [ -z "${DEBS}" ]; then - echo "No .deb files to attach" - exit 0 - fi - - # Create the release if it doesn't already exist - if ! gh release view "${TAG}" \ - --repo "${{ github.repository }}" \ - --json tagName -q .tagName \ - >/dev/null 2>&1; then - echo "Creating GitHub Release ${TAG}..." - printf 'Ubuntu %s kernel %s built for arm64.\n\nFlavour: %s\nRunner: %s\n' \ - "${SUITE}" "${{ inputs.kernel_version }}" \ - "${FLAVOR}" "${{ runner.name }}" \ - > /tmp/release-notes.md - gh release create "${TAG}" \ - --repo "${{ github.repository }}" \ - --title "Ubuntu ${SUITE} kernel ${{ inputs.kernel_version }} (arm64)" \ - --notes-file /tmp/release-notes.md \ - --latest=false - fi - - echo "Uploading packages to release ${TAG}..." - echo "${DEBS}" | xargs gh release upload "${TAG}" \ - --repo "${{ github.repository }}" \ - --clobber - echo "Packages attached to release ${TAG}" - echo "Release URL: https://github.com/${{ github.repository }}/releases/tag/${TAG}" - - # ----------------------------------------------------------------------- - # 13. Summary + # 10. Summary # ----------------------------------------------------------------------- - name: Print summary if: always() run: | - echo "## Kernel Build Summary" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" - echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Architecture | \`${ARCH}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Flavour | \`${FLAVOR}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Runner | \`${{ runner.name }}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "### Built packages" >> "$GITHUB_STEP_SUMMARY" - echo '```' >> "$GITHUB_STEP_SUMMARY" - ls -lh output/*.deb 2>/dev/null || echo "(none)" - echo '```' >> "$GITHUB_STEP_SUMMARY" + { + echo "## Kernel Build Summary" + echo "" + echo "| Field | Value |" + echo "|-------|-------|" + echo "| Branch | \`${SUITE}\` |" + echo "| Architecture | \`${ARCH}\` |" + echo "| Flavour | \`${FLAVOR}\` |" + echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" + echo "| Runner | \`${{ runner.name }}\` |" + echo "" + echo "### Built packages" + echo '```' + ls -lh output/*.deb 2>/dev/null || echo "(none)" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index bc4374cf662e3..dec5f110884a6 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -2,411 +2,161 @@ # # fetch-source-pkg.yml # -# Syncs the Canonical Ubuntu kernel from the Launchpad git repository into -# a branch of this repo, building an incremental commit history of syncs. +# Mirrors the latest Qualcomm-Ubuntu kernel upload from the carmel-team Launchpad +# repository into the resolute-qcom branch (preserving full upstream history), +# then triggers a kernel build. # -# Approach: shallow clone + rsync + incremental commit -# git clone --depth=1 fetches exactly one commit from Launchpad — reliably -# served without triggering rate limits or 503 errors. Each sync adds a -# commit to our branch showing exactly what changed between kernel versions. -# git diff between any two of our tags shows the precise delta. +# This repo mirrors exactly ONE source into ONE branch, so the workflow takes no +# inputs -- everything is fixed: +# upstream : git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute +# branch : resolute-qcom +# tags : Ubuntu-qcom-X.Y.Z-A.B (mirrored verbatim from upstream) # -# Why git instead of the source package? -# The Ubuntu kernel source package (format 1.0) ships only debian.master/ -# with rules.d/ fragments — debian/rules is NOT included. The complete -# debian/ directory (with rules, scripts/, templates/, etc.) lives in the -# Launchpad git repository. Using the git source gives a buildable tree. +# History-preserving "mirror-repoint" model +# * resolute-qcom is a movable "latest Canonical" pointer. +# * The upstream Ubuntu-qcom-X.Y.Z-A.B tag is mirrored verbatim as the immutable +# per-upload record. +# * A sync is pure fetch + repoint (scripts/sync-mirror.sh): it never merges or +# rebases, so it cannot conflict. # -# Repository branch layout -# ───────────────────────── -# main – CI infrastructure: workflows, scripts, documentation -# noble – Ubuntu Noble (24.04 LTS) kernel source, one commit/upload -# resolute-qcom – Ubuntu-qcom kernel source, one commit/upload (daily default) -# – additional branches added on demand -# -# Each branch tip is tagged - -# (e.g. resolute-qcom-7.0.0-1003.3). -# -# Repository variables (Settings → Variables → Actions): -# KERNEL_SUITE – default branch to sync when not specified (default: resolute-qcom) -# KERNEL_SOURCE – source package name (default: linux) -# KERNEL_CUSTOM_GIT_URL – default custom git URL for scheduled runs -# (default: https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) +# Prerequisite: resolute-qcom must first be seeded with full history by the +# "Bootstrap: Seed Canonical Kernel History" workflow. This job is incremental +# and refuses to run against an un-seeded branch. name: "Sync: Canonical Kernel Sources to Branch" +run-name: "Mirror latest Ubuntu-qcom upload into resolute-qcom" on: - workflow_dispatch: - inputs: - suite: - description: "Branch to sync into (e.g. resolute-qcom, noble, questing, resolute)" - required: false - default: "resolute-qcom" - type: string - force: - description: "Re-sync even if this version is already committed" - type: boolean - default: false - custom_git_url: - description: "Git URL to clone from. Leave empty to use the official Ubuntu kernel repo." - required: false - default: "" - type: string + workflow_dispatch: # manual only; always mirrors the latest carmel-team upload permissions: contents: write actions: write +# Only one sync at a time -- a second run's lease-pinned push would be rejected. +concurrency: + group: mirror-sync-resolute-qcom + cancel-in-progress: false + +env: + BRANCH: resolute-qcom + UPSTREAM_URL: "https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" + UPSTREAM_PREFIX: Ubuntu-qcom + jobs: + # ========================================================================== + # Job 1 - cheap "is there a new upload?" gate (no clone). + # ========================================================================== check-version: - name: "Check latest version on Launchpad" + name: "Check for new uploads" runs-on: ubuntu-24.04-arm timeout-minutes: 10 - outputs: - version: ${{ steps.query.outputs.version }} - upstream_version: ${{ steps.query.outputs.upstream_version }} - tag: ${{ steps.query.outputs.tag }} - suite: ${{ steps.query.outputs.suite }} - base_suite: ${{ steps.query.outputs.base_suite }} - source: ${{ steps.query.outputs.source }} - git_url: ${{ steps.query.outputs.git_url }} - git_tag_prefix: ${{ steps.query.outputs.git_tag_prefix }} - should_sync: ${{ steps.gate.outputs.should_sync }} - + should_sync: ${{ steps.gate.outputs.should_sync }} steps: - - name: Install dependencies - run: sudo apt-get install -y --no-install-recommends jq - - - name: Resolve version and git URL - id: query - env: - SUITE: ${{ inputs.suite || vars.KERNEL_SUITE || 'resolute-qcom' }} - SOURCE: ${{ vars.KERNEL_SOURCE || 'linux' }} - # For scheduled runs: fall back to vars.KERNEL_CUSTOM_GIT_URL or the - # hardcoded carmel-team URL so the daily build always uses resolute-qcom. - # For manual runs: use exactly what the user entered — empty string means - # "use the official Ubuntu repo" (no custom URL → Launchpad REST API path). - CUSTOM_GIT_URL: ${{ github.event_name == 'schedule' && (vars.KERNEL_CUSTOM_GIT_URL || 'https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute') || inputs.custom_git_url }} - # GIT_TAG_PREFIX is derived automatically: - # custom URL → Ubuntu-qcom (Ubuntu-qcom-7.0.0-1003.3 style tags) - # official → Ubuntu (Ubuntu-X.Y.Z-A.B style tags, set in else branch) - GIT_TAG_PREFIX: "Ubuntu-qcom" - run: | - # Derive the base Ubuntu suite by stripping any custom suffix. - # e.g. resolute-qcom → resolute, noble → noble, questing → questing - # Used for Launchpad API queries and default git URL construction. - BASE_SUITE=$(echo "${SUITE}" | cut -d'-' -f1) - echo "Suite : ${SUITE}" - echo "Base suite : ${BASE_SUITE}" - - if [ -n "${CUSTOM_GIT_URL}" ]; then - # ---------------------------------------------------------------- - # Custom repo path: find the latest -* tag via - # git ls-remote. This bypasses the Launchpad REST API entirely — - # the version is derived directly from the tags present in the - # custom repository. - # - # The Ubuntu-qcom Launchpad repository uses tags of the form: - # Ubuntu-qcom-7.0.0-1003.3 - # so GIT_TAG_PREFIX defaults to "Ubuntu-qcom". - # ---------------------------------------------------------------- - echo "Custom git URL : ${CUSTOM_GIT_URL}" - echo "Git tag prefix : ${GIT_TAG_PREFIX}" - echo "Querying tags from custom repo..." - - LATEST_TAG=$(git ls-remote --tags "${CUSTOM_GIT_URL}" "refs/tags/${GIT_TAG_PREFIX}-*" \ - | grep -v '\^{}' \ - | awk '{print $2}' \ - | sed 's|refs/tags/||' \ - | sort -V \ - | tail -1) - - [ -n "${LATEST_TAG}" ] || { - echo "ERROR: No ${GIT_TAG_PREFIX}-* tags found in ${CUSTOM_GIT_URL}" >&2 - exit 1 - } - - # Ubuntu-qcom-7.0.0-1003.3 → 7.0.0-1003.3 - VERSION="${LATEST_TAG#${GIT_TAG_PREFIX}-}" - GIT_URL="${CUSTOM_GIT_URL}" - - echo "Latest tag : ${LATEST_TAG}" - else - # ---------------------------------------------------------------- - # Default path: query the Launchpad REST API for the latest - # officially published source package version, then construct - # the standard Launchpad git URL for the suite. - # Official Ubuntu repos use Ubuntu-X.Y.Z-A.B tag format. - # ---------------------------------------------------------------- - GIT_TAG_PREFIX="Ubuntu" - - API="https://api.launchpad.net/1.0/ubuntu/+archive/primary" - API+="?ws.op=getPublishedSources" - API+="&source_name=${SOURCE}" - API+="&distro_series=/ubuntu/${BASE_SUITE}" - API+="&status=Published" - API+="&order_by_date=true" - API+="&ws.size=300" - - echo "Querying: ${API}" - RESP=$(curl -fsSL "${API}") - - VERSION=$(echo "$RESP" | jq -r \ - --arg name "${SOURCE}" \ - '[.entries[] | select(.source_package_name == $name)] | .[0].source_package_version // empty') - [ -n "${VERSION}" ] || { - echo "ERROR: No exact match for '${SOURCE}' in ${SUITE}" >&2 - exit 1 - } - - GIT_URL="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/${BASE_SUITE}" - fi - - UPSTREAM=$(echo "${VERSION}" | cut -d'-' -f1) - TAG="${SUITE}-${VERSION}" - - echo "version=${VERSION}" >> "$GITHUB_OUTPUT" - echo "upstream_version=${UPSTREAM}" >> "$GITHUB_OUTPUT" - echo "tag=${TAG}" >> "$GITHUB_OUTPUT" - echo "suite=${SUITE}" >> "$GITHUB_OUTPUT" - echo "base_suite=${BASE_SUITE}" >> "$GITHUB_OUTPUT" - echo "source=${SOURCE}" >> "$GITHUB_OUTPUT" - echo "git_url=${GIT_URL}" >> "$GITHUB_OUTPUT" - echo "git_tag_prefix=${GIT_TAG_PREFIX}" >> "$GITHUB_OUTPUT" - - echo "Version : ${VERSION}" - echo "Tag : ${TAG}" - echo "Git URL : ${GIT_URL}" - echo "Tag prefix : ${GIT_TAG_PREFIX}" - - - name: Check whether tag already exists + - name: Gate on un-mirrored uploads id: gate env: - TAG: ${{ steps.query.outputs.tag }} + MIRROR_URL: https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git run: | - if [ "${{ inputs.force }}" = "true" ]; then - echo "should_sync=true" >> "$GITHUB_OUTPUT" - echo "Force flag set – will re-sync" - exit 0 - fi - - if git ls-remote --exit-code --tags \ - "https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git" \ - "refs/tags/${TAG}" >/dev/null 2>&1; then - echo "should_sync=false" >> "$GITHUB_OUTPUT" - echo "Tag '${TAG}' already exists – nothing to do" - else - echo "should_sync=true" >> "$GITHUB_OUTPUT" - echo "Tag '${TAG}' not found – will sync" - fi - + # Compare upstream upload tags against our mirrored tags via two cheap + # ls-remote calls (no clone). should_sync is true iff any upstream + # upload has no matching tag here. Capture ls-remote first so a transport + # failure is not mistaken for "upstream has no tags". + up_raw="$(git ls-remote --tags "${UPSTREAM_URL}" "refs/tags/${UPSTREAM_PREFIX}-*")" \ + || { echo "::error::git ls-remote failed for upstream (network/auth?)"; exit 1; } + mapfile -t UP < <( + printf '%s\n' "${up_raw}" \ + | grep -v '\^{}' | sed -E "s#.*refs/tags/${UPSTREAM_PREFIX}-##" | sort -V + ) + [ "${#UP[@]}" -gt 0 ] || { echo "No ${UPSTREAM_PREFIX}-* tags upstream"; exit 1; } + LATEST="${UP[-1]}" + + OURS="$( + git ls-remote --tags "${MIRROR_URL}" "refs/tags/${UPSTREAM_PREFIX}-*" \ + | grep -v '\^{}' | sed -E "s#.*refs/tags/${UPSTREAM_PREFIX}-##" || true + )" + + # -F: match the version as a fixed string (dots are not wildcards). + SHOULD_SYNC=false + for ver in "${UP[@]}"; do + if ! grep -qxF "${ver}" <<<"${OURS}"; then SHOULD_SYNC=true; break; fi + done + + echo "should_sync=${SHOULD_SYNC}" >> "$GITHUB_OUTPUT" + echo "Latest upstream: ${LATEST}; new uploads to mirror: ${SHOULD_SYNC}" + + # ========================================================================== + # Job 2 - mirror every new upload (history-preserving), via sync-mirror.sh. + # ========================================================================== sync: - name: "Sync ${{ needs.check-version.outputs.suite }} sources to branch" + name: "Mirror" runs-on: ubuntu-24.04-arm - timeout-minutes: 120 + timeout-minutes: 60 needs: check-version if: needs.check-version.outputs.should_sync == 'true' - env: - VERSION: ${{ needs.check-version.outputs.version }} - UPSTREAM_VERSION: ${{ needs.check-version.outputs.upstream_version }} - TAG: ${{ needs.check-version.outputs.tag }} - SUITE: ${{ needs.check-version.outputs.suite }} - SOURCE: ${{ needs.check-version.outputs.source }} - GIT_URL: ${{ needs.check-version.outputs.git_url }} - GIT_TAG_PREFIX: ${{ needs.check-version.outputs.git_tag_prefix }} + outputs: + synced_version: ${{ steps.sync.outputs.synced_version }} + synced_count: ${{ steps.sync.outputs.synced_count }} steps: - # ----------------------------------------------------------------------- - # 1. Free up disk space - # ----------------------------------------------------------------------- - name: Free up runner disk space run: | - echo "Disk before cleanup:" - df -h / sudo rm -rf \ - /usr/share/dotnet \ - /usr/local/lib/android \ - /opt/ghc \ - /opt/hostedtoolcache/CodeQL \ - /usr/local/share/boost \ - "$AGENT_TOOLSDIRECTORY" \ - 2>/dev/null || true + /usr/share/dotnet /usr/local/lib/android /opt/ghc \ + /opt/hostedtoolcache/CodeQL /usr/local/share/boost \ + "$AGENT_TOOLSDIRECTORY" 2>/dev/null || true sudo apt-get clean - echo "Disk after cleanup:" - df -h / - - # ----------------------------------------------------------------------- - # 2. Install tools - # ----------------------------------------------------------------------- - - name: Install dependencies - run: | - sudo apt-get update -qq - sudo apt-get install -y --no-install-recommends \ - curl jq rsync - - # ----------------------------------------------------------------------- - # 3. Clone from the resolved git URL at the version tag. - # GIT_URL is either the custom repo URL (when custom_git_url input - # was provided) or the standard Launchpad URL for the suite. - # GIT_TAG_PREFIX determines the tag format: - # Ubuntu-qcom → Ubuntu-qcom-7.0.0-1003.3 (resolute-qcom repo) - # Ubuntu → Ubuntu-6.8.0-114.114 (official repos) - # ----------------------------------------------------------------------- - - name: Clone from git repo at version tag - run: | - GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" - - echo "Cloning ${GIT_URL} at tag ${GIT_TAG} (shallow)..." - git clone --depth=1 --branch "${GIT_TAG}" "${GIT_URL}" kernel-src/ - - echo "SRC_DIR=kernel-src" >> "$GITHUB_ENV" - - FILE_COUNT=$(find kernel-src/ -type f | wc -l) - echo "Cloned ${FILE_COUNT} files" - [ "${FILE_COUNT}" -gt 5000 ] || { - echo "ERROR: Too few files cloned (${FILE_COUNT})" - exit 1 - } - echo "Top-level contents:" - ls kernel-src/ + - name: Checkout CI scripts + uses: actions/checkout@v6 - echo "debian/ contents:" - ls kernel-src/debian/ | head -20 - - # ----------------------------------------------------------------------- - # 4. Configure git - # ----------------------------------------------------------------------- - - name: Configure git identity + # BRANCH, UPSTREAM_URL and UPSTREAM_PREFIX come from the workflow-level env + # above; sync-mirror.sh reads them directly. + - name: Mirror new uploads + id: sync + env: + MIRROR_URL: https://x-access-token:${{ github.token }}@github.com/${{ github.repository }}.git + WORKDIR: ${{ runner.temp }}/mirror-sync run: | git config --global user.email "github-actions[bot]@users.noreply.github.com" git config --global user.name "github-actions[bot]" + bash scripts/sync-mirror.sh - # ----------------------------------------------------------------------- - # 5. Prepare the suite branch - # ----------------------------------------------------------------------- - - name: Clone repository for suite branch update - env: - GH_TOKEN: ${{ github.token }} - run: | - REPO_URL="https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" - git clone "${REPO_URL}" suite-repo - - - name: Check out or create suite branch - run: | - cd suite-repo - - if git ls-remote --exit-code origin "refs/heads/${SUITE}" >/dev/null 2>&1; then - echo "Branch '${SUITE}' exists – checking out" - git checkout "${SUITE}" - git rm -rf --quiet . 2>/dev/null || true - else - echo "Branch '${SUITE}' does not exist – creating orphan branch" - git checkout --orphan "${SUITE}" - git rm -rf --quiet . 2>/dev/null || true - fi - - - name: Populate suite branch with cloned source - run: | - echo "Copying source tree from ${SRC_DIR} → suite-repo/" - rsync -a --delete \ - --exclude='.git' \ - "${SRC_DIR}/" suite-repo/ - - COPIED=$(find suite-repo/ -type f | wc -l) - echo "Files in suite-repo after rsync: ${COPIED}" - [ "${COPIED}" -gt 5000 ] || { - echo "ERROR: rsync copied only ${COPIED} files" - exit 1 - } - - cd suite-repo - - SYNC_DATE=$(date -u '+%Y-%m-%dT%H:%M:%SZ') - printf '%s\n\n%s\n%s\n%s\n%s\n%s\n' \ - "${SUITE}: ${VERSION}" \ - "Branch : ${SUITE}" \ - "Kernel version : ${VERSION}" \ - "Upstream : ${UPSTREAM_VERSION}" \ - "Source : ${GIT_URL} (tag: ${GIT_TAG_PREFIX}-${VERSION})" \ - "Synced : ${SYNC_DATE}" \ - > /tmp/commit-msg.txt - - git add --all - - if git diff --cached --quiet; then - echo "No changes to commit – branch already has this content" - else - git commit -F /tmp/commit-msg.txt - echo "Committed ${SUITE} branch at $(git rev-parse HEAD)" - fi - - - name: Tag the commit - run: | - cd suite-repo - git tag -fa "${TAG}" -m "${SUITE} kernel ${VERSION}" - echo "Tagged: ${TAG}" - - - name: Push suite branch and tag - run: | - cd suite-repo - git push origin "${SUITE}" - git push --force origin "${TAG}" - echo "Pushed branch '${SUITE}' and tag '${TAG}'" - - # ----------------------------------------------------------------------- - # 6. Summary - # ----------------------------------------------------------------------- - name: Print summary if: always() run: | - GIT_TAG="${GIT_TAG_PREFIX}-${VERSION}" - - echo "## Source Sync Summary" >> "$GITHUB_STEP_SUMMARY" - echo "" >> "$GITHUB_STEP_SUMMARY" - echo "| Field | Value |" >> "$GITHUB_STEP_SUMMARY" - echo "|-------|-------|" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch | \`${SUITE}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Version | \`${VERSION}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Branch ref | [\`${SUITE}\`](https://github.com/${{ github.repository }}/tree/${SUITE}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Tag | [\`${TAG}\`](https://github.com/${{ github.repository }}/releases/tag/${TAG}) |" >> "$GITHUB_STEP_SUMMARY" - echo "| Cloned from | \`${GIT_URL}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Git tag | \`${GIT_TAG}\` |" >> "$GITHUB_STEP_SUMMARY" - echo "| Launchpad | https://launchpad.net/ubuntu/${SUITE}/+source/${SOURCE}/${VERSION} |" >> "$GITHUB_STEP_SUMMARY" - + { + echo "## Sync Summary" + echo "" + echo "| Field | Value |" + echo "|-------|-------|" + echo "| Branch | \`${BRANCH}\` |" + echo "| Uploads mirrored | ${{ steps.sync.outputs.synced_count || 0 }} |" + echo "| Latest version | \`${{ steps.sync.outputs.synced_version }}\` |" + } >> "$GITHUB_STEP_SUMMARY" + + # ========================================================================== + # Job 3 - trigger a kernel build for the newest mirrored upload. + # ========================================================================== trigger-build: name: "Trigger kernel build" runs-on: ubuntu-24.04-arm needs: [check-version, sync] - if: needs.sync.result == 'success' + if: needs.sync.result == 'success' && needs.sync.outputs.synced_count != '0' steps: - name: Dispatch build-kernel workflow env: GH_TOKEN: ${{ github.token }} - SUITE: ${{ needs.check-version.outputs.suite }} - VERSION: ${{ needs.check-version.outputs.version }} + VERSION: ${{ needs.sync.outputs.synced_version }} run: | - # Derive the kernel flavour from the branch name. - # Custom branches (any branch with a suffix, e.g. resolute-qcom) - # use the 'qcom' flavour. Official Ubuntu suite branches - # (noble, questing, resolute — no suffix) use 'generic'. - # This ensures the daily resolute-qcom build always uses qcom. - if echo "${SUITE}" | grep -q '-'; then - FLAVOR="qcom" - else - FLAVOR="generic" - fi - echo "Branch: ${SUITE} → Flavour: ${FLAVOR}" - + echo "Building ${BRANCH} ${VERSION} (flavour qcom)" gh workflow run build-kernel.yml \ --repo "${{ github.repository }}" \ - --field suite="${SUITE}" \ + --field suite="${BRANCH}" \ --field kernel_version="${VERSION}" \ --field arch="arm64" \ - --field flavor="${FLAVOR}" \ + --field flavor="qcom" \ --field runner="lecore-production" - - echo "Build workflow dispatched for ${SUITE} ${VERSION} (flavor=${FLAVOR}, runner=lecore-production)" diff --git a/scripts/seed-history.sh b/scripts/seed-history.sh new file mode 100755 index 0000000000000..b83c1237a85d6 --- /dev/null +++ b/scripts/seed-history.sh @@ -0,0 +1,187 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# +# seed-history.sh - One-time automated bootstrap: seed a branch of this repo with +# the FULL Canonical kernel history from Launchpad, pushing the +# upstream upload tags (e.g. Ubuntu-qcom-X.Y.Z-A.B) verbatim. +# +# Why a bootstrap is needed +# ───────────────────────── +# The steady-state sync (sync-mirror.sh) is incremental: it only works once the +# branch already holds real upstream history, so each new upload is a small +# delta. The branch starts empty, so the FULL history must be transferred from +# Launchpad exactly once. This script does that. +# +# How the history is transferred (both points proven empirically) +# ─────────────────────────────────────────────────────────────── +# * A FULL `git clone` is used, NOT shallow + `git fetch --deepen`: Launchpad's +# shallow-deepen path is broken (it stalls and throws "error processing +# shallow info"). Launchpad CAN serve a full clone, but spends many minutes +# server-side computing the pack (the client sees ~0 bytes meanwhile), so the +# git low-speed abort is relaxed to tolerate that quiet phase. +# * GitHub caps a single push at 2 GB, so the seed is pushed in <2 GB slices. +# +# Single shot (no resume loop): the full clone either completes within the job's +# time budget or fails. On failure, re-dispatch -- the caller caches the cloned +# repo, so a retry skips re-downloading it and just re-attempts the publish. +# +# Usage: +# seed-history.sh +# +# Required environment: +# MIRROR_URL Authenticated push URL of THIS repo +# UPSTREAM_URL Canonical/Launchpad git URL to seed from +# BRANCH Seed branch to create (e.g. resolute-qcom-seed) +# UPSTREAM_PREFIX Upstream tag prefix, mirrored verbatim (e.g. Ubuntu-qcom) +# WORKDIR Scratch dir (cached across runs by the caller) +# +# Optional environment: +# PUSH_SLICE_COMMITS Commits per push slice, to stay under GitHub's 2 GB +# per-push limit (default: 20000) +# +# Exit codes: +# 0 Seed complete and published +# 1 Hard error + +set -euo pipefail + +log() { printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2; } +die() { log "ERROR: $*"; exit 1; } +hr() { log "────────────────────────────────────────────────────────────"; } +redact() { sed -E 's#(https?://)[^@/]*@#\1***@#g' <<<"$1"; } + +: "${MIRROR_URL:?MIRROR_URL is required}" +: "${UPSTREAM_URL:?UPSTREAM_URL is required}" +: "${BRANCH:?BRANCH is required}" +: "${UPSTREAM_PREFIX:?UPSTREAM_PREFIX is required}" +: "${WORKDIR:?WORKDIR is required (cached across runs by the caller)}" + +PUSH_SLICE_COMMITS="${PUSH_SLICE_COMMITS:-20000}" +SEED="${WORKDIR}/seed.git" + +# Abort only a TRULY dead transfer. Launchpad spends many minutes server-side +# computing the pack for a full kernel history (the client sees ~0 bytes during +# "Counting/Compressing objects"); too short a low-speed window kills that +# legitimate compute phase. Tolerate a long quiet period, abort only if nothing +# moves for the full window. +export GIT_HTTP_LOW_SPEED_LIMIT="${GIT_HTTP_LOW_SPEED_LIMIT:-1000}" +export GIT_HTTP_LOW_SPEED_TIME="${GIT_HTTP_LOW_SPEED_TIME:-2400}" # 40 min + +hr +log "Canonical kernel history bootstrap" +log " Seed branch : ${BRANCH}" +log " Upstream : $(redact "${UPSTREAM_URL}")" +log " Tags : ${UPSTREAM_PREFIX}-* (verbatim)" +log " Workdir : ${WORKDIR}" +hr + +mkdir -p "${WORKDIR}" + +# --------------------------------------------------------------------------- +# 1. Full-clone the upstream history (fresh), or reuse a cached clone (retry). +# A full clone is used rather than shallow + `git fetch --deepen`: Launchpad's +# shallow-deepen path is broken, but it can serve a full clone. Retried a few +# times to ride out transient stalls. +# --------------------------------------------------------------------------- +if [ ! -d "${SEED}" ]; then + log "No cached clone found -- full-cloning upstream history (this is slow)..." + for attempt in 1 2 3; do + if git clone --bare "${UPSTREAM_URL}" "${SEED}"; then break; fi + log "Full clone attempt ${attempt} failed; cleaning up and backing off..." + rm -rf "${SEED}" + sleep "$((attempt * 30))" + done + [ -d "${SEED}" ] || die "Full clone of ${UPSTREAM_URL} failed after retries." +else + log "Reusing cached clone at ${SEED}." +fi + +cd "${SEED}" +git remote get-url upstream >/dev/null 2>&1 || git remote add upstream "${UPSTREAM_URL}" + +# A full clone is never shallow; assert it defensively, since GitHub rejects a +# shallow push outright. +[ ! -f "${SEED}/shallow" ] \ + || die "Clone is unexpectedly shallow -- cannot publish a shallow history." +log "Full history cloned: $(git rev-list --all --count) commits." + +# --------------------------------------------------------------------------- +# 3. Fetch EVERY upload tag (full; the shared base is already local, so these +# are cheap deltas) so each upload -- including any that upstream rebased onto +# a divergent line -- is preserved under its own immutable tag. +# --------------------------------------------------------------------------- +log "Fetching all ${UPSTREAM_PREFIX}-* upload tags for full preservation..." +# On a resume, these tags may already be local; a re-pointed upstream tag then +# yields a non-zero "would clobber" instead of a silent overwrite (G10). Surface +# it clearly rather than letting `set -e` report an opaque failure. +if ! git fetch --no-tags upstream \ + "refs/tags/${UPSTREAM_PREFIX}-*:refs/tags/${UPSTREAM_PREFIX}-*" 2>fetch.err; then + if grep -q 'would clobber existing tag' fetch.err; then + die "An upstream ${UPSTREAM_PREFIX}-* tag was re-pointed since a previous \ +attempt (immutability violation upstream). Refusing to clobber a preserved tag." + fi + cat fetch.err >&2 + die "Failed to fetch upstream upload tags." +fi + +mapfile -t VERSIONS < <( + git tag --list "${UPSTREAM_PREFIX}-*" \ + | sed -E "s#^${UPSTREAM_PREFIX}-##" | sort -V +) +[ "${#VERSIONS[@]}" -gt 0 ] || die "No upload tags present after fetch." +LATEST_VERSION="${VERSIONS[-1]}" +log "Uploads to preserve: ${VERSIONS[*]}" + +# The upstream ${UPSTREAM_PREFIX}-* tags fetched above ARE the preservation tags, +# mirrored verbatim (Canonical's tag names) -- no re-tagging. Point the seed +# branch at the latest upload. +git update-ref "refs/heads/${BRANCH}" \ + "$(git rev-parse "${UPSTREAM_PREFIX}-${LATEST_VERSION}^{commit}")" + +# --------------------------------------------------------------------------- +# 4. Sliced push: GitHub caps a single push at 2 GB, so push the latest branch's +# history in commit-count checkpoints (each slice carries only the objects +# between checkpoints), then the branch ref and every preservation tag. +# --------------------------------------------------------------------------- +# `git clone` set origin to the UPSTREAM (Launchpad) URL; repoint it at our +# mirror so the seed is pushed to GitHub, not back to Launchpad. +if git remote get-url origin >/dev/null 2>&1; then + git remote set-url origin "${MIRROR_URL}" +else + git remote add origin "${MIRROR_URL}" +fi + +# The scratch ref is a DISPOSABLE chunking aid, never history. It is force-pushed +# and cleared on every exit so a leftover from a failed prior attempt can never +# wedge a resume into a non-fast-forward (which would otherwise make the bootstrap +# non-convergent without manual intervention). +SCRATCH="refs/heads/_seed-progress" +cleanup_scratch() { git push origin ":${SCRATCH}" >/dev/null 2>&1 || true; } +trap cleanup_scratch EXIT +cleanup_scratch # clear any leftover before we start + +log "Slicing history into <2 GB pushes (every ${PUSH_SLICE_COMMITS} commits)..." +mapfile -t CHECKPOINTS < <( + git rev-list --first-parent --reverse "refs/heads/${BRANCH}" \ + | awk -v n="${PUSH_SLICE_COMMITS}" 'NR % n == 0' +) +for cp in "${CHECKPOINTS[@]}"; do + log " push checkpoint ${cp:0:12} -> ${SCRATCH}" + git push --force origin "${cp}:${SCRATCH}" +done + +log "Pushing branch ${BRANCH} and ${#VERSIONS[@]} preservation tag(s)..." +git push origin "refs/heads/${BRANCH}" +git push origin "refs/tags/${UPSTREAM_PREFIX}-*" +# scratch ref is removed by the EXIT trap + +hr +log "Bootstrap complete." +log " Branch ${BRANCH} seeded with $(git rev-list --count "${BRANCH}") commits." +log " Preserved uploads: ${VERSIONS[*]}" +log " Latest: ${LATEST_VERSION}" + +if [ -n "${GITHUB_OUTPUT:-}" ]; then + echo "latest_version=${LATEST_VERSION}" >> "${GITHUB_OUTPUT}" + echo "preserved_count=${#VERSIONS[@]}" >> "${GITHUB_OUTPUT}" +fi diff --git a/scripts/sync-mirror.sh b/scripts/sync-mirror.sh new file mode 100755 index 0000000000000..d7ebfae932be2 --- /dev/null +++ b/scripts/sync-mirror.sh @@ -0,0 +1,242 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: BSD-3-Clause +# +# sync-mirror.sh - Incrementally mirror new Canonical kernel upload tags into a +# branch of this repository while PRESERVING full upstream +# history. +# +# This is the steady-state "mirror-repoint" sync. It replaces the legacy +# shallow-clone + rsync + squash approach, which discarded all upstream history +# (one flattened snapshot commit per upload). Here, the branch becomes a faithful +# mirror of the upstream kernel tree: each upload is fetched with its real +# ancestry and frozen under an immutable per-upload tag. +# +# Mental model +# ──────────── +# * BRANCH (e.g. resolute-qcom) -- a MOVABLE "latest Canonical" +# pointer; disposable by design. +# * TAG (e.g. Ubuntu-qcom-7.0.0-1006.8) -- the upstream Canonical tag, +# mirrored VERBATIM. IMMUTABLE +# per-upload record; the sole +# anchor that preserves history. +# +# A sync is PURE FETCH + REPOINT -- it never merges or rebases, so it can never +# conflict and can never be blocked by developer patches layered on the branch. +# Developer patches are deliberately out of scope: the branch is force-advanced +# past them (they live on developers' own branches and are re-applied manually). +# +# Operational guardrails (each empirically validated by the history-preservation +# stress test; numbers refer to that test's findings): +# G1 Pin the lease: --force-with-lease=:, never bare +# --force-with-lease (a pre-push fetch silently defeats the bare form) and +# never blind --force. +# G2 Push the tag and the branch ATOMICALLY (--atomic) so they land together +# or not at all -- prevents tag-lands/branch-rejected split-brain. +# G3 Fail fast: abort the whole run on any rejected push. Never let the next +# idempotent run mask an incomplete one. +# G4 Tag every missing upload, ascending, and create the tag BEFORE moving the +# branch. Latest-only silently drops unique commits from rebased uploads. +# G5 Never use --depth on the incremental fetch -- the closure must be complete +# and non-shallow. +# G8 Operate on a BARE clone (no worktree) so the branch ref can be updated +# without the "branch used by worktree" failure. +# G10 Do not assume `git fetch` always exits 0 (a re-pointed upstream tag yields +# a non-zero "would clobber existing tag" -- handled explicitly). +# +# Usage: +# sync-mirror.sh +# +# Required environment: +# MIRROR_URL Authenticated push URL of THIS repo +# (e.g. https://x-access-token:TOKEN@github.com/org/repo.git) +# UPSTREAM_URL Canonical/Launchpad git URL to mirror from +# BRANCH Branch to advance (e.g. resolute-qcom) +# UPSTREAM_PREFIX Upstream tag prefix, mirrored verbatim (e.g. Ubuntu-qcom) +# +# Optional environment: +# WORKDIR Scratch directory for the bare mirror (default: mktemp) +# MIN_HISTORY_COMMITS Bootstrap sentinel: refuse to sync if BRANCH has fewer +# than this many commits, i.e. it has not been seeded with +# real history yet (default: 1000). Run bootstrap first. +# GITHUB_OUTPUT If set, the latest synced version is written as +# `synced_version=` and `synced_count=` for the caller. +# +# Exit codes: +# 0 Up to date or one-or-more uploads synced successfully +# 1 Hard error (bootstrap required, push rejected, etc.) + +set -euo pipefail + +# --------------------------------------------------------------------------- +# Logging helpers +# --------------------------------------------------------------------------- +log() { printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2; } +die() { log "ERROR: $*"; exit 1; } +hr() { log "────────────────────────────────────────────────────────────"; } + +# --------------------------------------------------------------------------- +# Validate inputs +# --------------------------------------------------------------------------- +: "${MIRROR_URL:?MIRROR_URL is required}" +: "${UPSTREAM_URL:?UPSTREAM_URL is required}" +: "${BRANCH:?BRANCH is required}" +: "${UPSTREAM_PREFIX:?UPSTREAM_PREFIX is required}" + +MIN_HISTORY_COMMITS="${MIN_HISTORY_COMMITS:-1000}" +WORKDIR="${WORKDIR:-$(mktemp -d)}" +MIRROR="${WORKDIR}/mirror.git" + +# Redact credentials from any URL before printing it. +redact() { sed -E 's#(https?://)[^@/]*@#\1***@#g' <<<"$1"; } + +hr +log "Canonical kernel mirror sync (history-preserving)" +log " Branch : ${BRANCH}" +log " Upstream : $(redact "${UPSTREAM_URL}")" +log " Tags (verbatim) : ${UPSTREAM_PREFIX}-*" +hr + +# --------------------------------------------------------------------------- +# 1. Clone OUR mirror as a BARE repo (G8: no worktree -> clean ref updates). +# --filter=blob:none keeps the clone small: we need the commit/tree graph for +# fetch negotiation and ref moves, not the file blobs (the upstream fetch +# brings the new blobs, and the push only sends objects the mirror lacks). +# --single-branch limits it to BRANCH so unrelated suite branches are not +# pulled. +# --------------------------------------------------------------------------- +log "Cloning mirror (bare, blobless, single-branch ${BRANCH})..." +rm -rf "${MIRROR}" # self-heal if an operator reuses a fixed WORKDIR +git clone --bare --filter=blob:none --single-branch --branch "${BRANCH}" \ + "${MIRROR_URL}" "${MIRROR}" \ + || die "Mirror does not yet contain branch '${BRANCH}'. Run bootstrap-history first." +cd "${MIRROR}" + +# --------------------------------------------------------------------------- +# 2. Bootstrap sentinel: the incremental model only works if BRANCH already +# carries real upstream history (so the upstream fetch transfers a small +# delta, not the whole tree). A freshly created or legacy-squashed branch has +# a handful of commits -- refuse, and point the operator at the bootstrap. +# --------------------------------------------------------------------------- +HISTORY_COUNT="$(git rev-list --count "${BRANCH}")" +log "Branch '${BRANCH}' currently has ${HISTORY_COUNT} commits." +if [ "${HISTORY_COUNT}" -lt "${MIN_HISTORY_COMMITS}" ]; then + die "Branch '${BRANCH}' has only ${HISTORY_COUNT} commits (< ${MIN_HISTORY_COMMITS}); \ +it has not been seeded with full Canonical history. Run the 'Bootstrap' workflow first." +fi + +# A bare mirror must never be shallow (G5): a shallow base breaks negotiation and +# GitHub rejects shallow pushes outright. +if [ -f shallow ]; then + die "Mirror clone is shallow -- refusing to sync. The seed must be fully \ +unshallowed before incremental syncs can run." +fi + +# --------------------------------------------------------------------------- +# 3. Discover which upstream uploads we have not mirrored yet. +# Missing = upstream ${UPSTREAM_PREFIX}- tags not yet present in our +# mirror (we mirror Canonical's tag names verbatim). Sorted ascending so a +# rebased middle upload is preserved before the branch advances past it (G4). +# --------------------------------------------------------------------------- +git remote add upstream "${UPSTREAM_URL}" + +# Capture ls-remote first so a transport failure (network/auth/5xx) is not +# silently flattened to an empty list and misreported as "no tags upstream". +upstream_raw="$(git ls-remote --tags upstream "refs/tags/${UPSTREAM_PREFIX}-*")" \ + || die "git ls-remote failed for upstream (network/auth?)." +mapfile -t UPSTREAM_VERSIONS < <( + printf '%s\n' "${upstream_raw}" \ + | grep -v '\^{}' \ + | sed -E "s#.*refs/tags/${UPSTREAM_PREFIX}-##" \ + | sort -V +) +[ "${#UPSTREAM_VERSIONS[@]}" -gt 0 ] \ + || die "No ${UPSTREAM_PREFIX}-* tags found upstream." + +MISSING=() +for ver in "${UPSTREAM_VERSIONS[@]}"; do + if ! git rev-parse -q --verify "refs/tags/${UPSTREAM_PREFIX}-${ver}" >/dev/null; then + MISSING+=("${ver}") + fi +done + +if [ "${#MISSING[@]}" -eq 0 ]; then + log "Already up to date -- no new uploads to mirror." + [ -n "${GITHUB_OUTPUT:-}" ] && { + echo "synced_version=" >> "${GITHUB_OUTPUT}" + echo "synced_count=0" >> "${GITHUB_OUTPUT}" + } + exit 0 +fi + +log "Uploads to mirror (ascending): ${MISSING[*]}" + +# --------------------------------------------------------------------------- +# 4. Mirror each missing upload in order: fetch the upstream tag (delta) -> +# advance branch -> atomic, lease-pinned push of branch + the upstream tag. +# Halt on the first failure (G3). +# --------------------------------------------------------------------------- +SYNCED=0 +LAST_VERSION="" +for ver in "${MISSING[@]}"; do + hr + log "Mirroring upload ${ver}" + + upstream_tag="${UPSTREAM_PREFIX}-${ver}" + + # Lease baseline (G1): the branch value we are advancing FROM. Empty if the + # branch somehow vanished between clone and now (treated as a create). + old_sha="$(git rev-parse -q --verify "refs/heads/${BRANCH}" || true)" + + # Fetch ONLY this upload's tag, full depth (G5), into the SAME ref name so the + # upstream annotated tag object becomes our preservation tag verbatim -- we + # mirror Canonical's tag names exactly. Because the mirror already holds the + # shared base, negotiation transfers just the new objects. A re-pointed + # upstream tag returns non-zero "would clobber" (G10) -- surface it clearly + # rather than letting `set -e` report a generic failure. + if ! git fetch --no-tags upstream \ + "refs/tags/${upstream_tag}:refs/tags/${upstream_tag}" 2>fetch.err; then + if grep -q 'would clobber existing tag' fetch.err; then + die "Upstream tag '${upstream_tag}' was re-pointed (immutability violation \ +upstream). Refusing to move our preservation tag. Manual review required." + fi + cat fetch.err >&2 + die "Failed to fetch upstream tag '${upstream_tag}'." + fi + + new_sha="$(git rev-parse "${upstream_tag}^{commit}")" + + # Advance the movable "latest" pointer (bare repo: update-ref == branch -f, + # with no worktree guard, G8). The upstream tag fetched above is already the + # immutable per-upload record (G4); no re-tagging needed. + git update-ref "refs/heads/${BRANCH}" "${new_sha}" + + # Push the branch and the tag ATOMICALLY (G2) with a PINNED lease (G1). + # An empty old_sha means "create" -- assert the remote ref is absent. + # Array (not a bare string) so a BRANCH with unexpected characters cannot + # word-split or glob the push arguments. + if [ -n "${old_sha}" ]; then + lease=("--force-with-lease=refs/heads/${BRANCH}:${old_sha}") + else + lease=("--force-with-lease=refs/heads/${BRANCH}:") + fi + + log "Pushing branch + tag atomically (lease pinned to ${old_sha:-})..." + if ! git push --atomic "${lease[@]}" origin \ + "refs/heads/${BRANCH}" \ + "refs/tags/${upstream_tag}"; then + die "Atomic push rejected for ${ver} (stale lease or protected ref). \ +Halting so the next run does not mask a partial sync (G3)." + fi + + log "Mirrored ${ver}: branch ${BRANCH} -> ${new_sha:0:12}, tag ${upstream_tag}" + SYNCED=$((SYNCED + 1)) + LAST_VERSION="${ver}" +done + +hr +log "Sync complete: ${SYNCED} upload(s) mirrored; branch now at ${LAST_VERSION}." + +if [ -n "${GITHUB_OUTPUT:-}" ]; then + echo "synced_version=${LAST_VERSION}" >> "${GITHUB_OUTPUT}" + echo "synced_count=${SYNCED}" >> "${GITHUB_OUTPUT}" +fi From 6dcfd7082fbf60839e26d921f3811879ee3b3ee9 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 24 Jun 2026 09:58:20 -0700 Subject: [PATCH 088/115] docs: split README into an audience-scoped doc set The 351-line README mixed a developer/consumer landing page with the full CI operator runbook. Split it by audience: - README.md - slim public landing page: branches, the two-sided integration branch (Qualcomm pushes / Canonical pulls), latest-tag badge + fallback, and a documentation index. - docs/INTEGRATION.md - the integration contract for Qualcomm developers (push patches to resolute-qcom-devel) and Canonical engineers (extract the delta to upstream); versioning; what the mirrored tree contains. - docs/PIPELINE.md - maintainer runbook plus the guardrail rationale: bootstrap, sync, build, the mirror-repoint model, and the Launchpad/GitHub limits. Also fix CONTRIBUTING.md, which was an unfilled template routing all contributors to "develop on main" - wrong for kernel patches; scope it to the CI driver and point kernel patches at resolute-qcom-devel. The current tag and baseline are not hardcoded; they resolve from git. Signed-off-by: Bjordis Collaku --- CONTRIBUTING.md | 32 ++- README.md | 643 +++----------------------------------------- docs/INTEGRATION.md | 40 +++ docs/PIPELINE.md | 298 ++++++++++++++++++++ 4 files changed, 402 insertions(+), 611 deletions(-) create mode 100644 docs/INTEGRATION.md create mode 100644 docs/PIPELINE.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 455b5f8f94b78..8082beccd7284 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,20 +1,34 @@ -# Contributing to +# Contributing to pkg-linux-qcom-canonical Hi there! -We’re thrilled that you’d like to contribute to this project. +We're thrilled that you'd like to contribute to this project. Your help is essential for keeping this project great and for making it better. +## Two kinds of contribution + +This repository hosts both a CI orchestrator and a mirror of the Canonical kernel, and +the two have **different** contribution flows: + +- **Kernel patches** (changes to the kernel source) do **not** go through this + document. They belong on the `resolute-qcom-devel` integration branch - never on + the `resolute-qcom` mirror. See **[docs/INTEGRATION.md](docs/INTEGRATION.md)**. +- **CI changes** (the GitHub Actions workflows, the `scripts/` helpers, and + the docs - everything on `main`) follow the pull-request flow below. + +The rest of this guide is about **CI changes on `main`**. + ## Branching Strategy -In general, contributors should develop on branches based off of `main` and pull requests should be made against `main`. +For CI changes, develop on branches based off of `main`, and open pull +requests against `main`. ## Submitting a pull request 1. Please read our [code of conduct](CODE-OF-CONDUCT.md) and [license](LICENSE.txt). -1. [Fork](https://github.com/qualcomm//fork) and clone the repository. +1. [Fork](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/fork) and clone the repository. ```bash - git clone https://github.com//.git + git clone https://github.com//pkg-linux-qcom-canonical.git ``` 1. Create a new branch based on `main`: @@ -26,14 +40,14 @@ In general, contributors should develop on branches based off of `main` and pull 1. Create an upstream `remote` to make it easier to keep your branches up-to-date: ```bash - git remote add upstream https://github.com/qualcomm/.git + git remote add upstream https://github.com/qualcomm-linux/pkg-linux-qcom-canonical.git ``` 1. Make your changes, add tests, and make sure the tests still pass. 1. Commit your changes using the [DCO](https://developercertificate.org/). You can attest to the DCO by commiting with the **-s** or **--signoff** options or manually adding the "Signed-off-by": ```bash - git commit -s -m "Really useful commit message"` + git commit -s -m "Really useful commit message" ``` 1. After committing your changes on the topic branch, sync it with the upstream branch: @@ -50,7 +64,7 @@ In general, contributors should develop on branches based off of `main` and pull The `-u` is shorthand for `--set-upstream`. This will set up the tracking reference so subsequent runs of `git push` or `git pull` can omit the remote and branch. -1. [Submit a pull request](https://github.com/qualcomm//pulls) from your branch to `main`. +1. [Submit a pull request](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/pulls) from your branch to `main`. 1. Pat yourself on the back and wait for your pull request to be reviewed. ## Security Analysis of Pull Requests @@ -68,7 +82,7 @@ By submitting a PR, you agree to participate in this process and help us keep th Here are a few things you can do that will increase the likelihood of your pull request to be accepted: -- Follow the existing style where possible. **INSERT LINK TO STYLE, e.g. PEP8 for python** +- Follow the existing style. Shell scripts should pass [shellcheck](https://www.shellcheck.net/); GitHub Actions workflows should pass [actionlint](https://github.com/rhysd/actionlint). - Write tests. - Keep your change as focused as possible. If you want to make multiple independent changes, please consider submitting them as separate pull requests. diff --git a/README.md b/README.md index f3665a6cbcb1b..d00677a43b026 100644 --- a/README.md +++ b/README.md @@ -1,621 +1,60 @@ # pkg-linux-qcom-canonical -Mirror and CI build pipeline for Canonical Ubuntu kernel source packages. +Mirrors the Qualcomm-Ubuntu (Canonical) kernel on `resolute-qcom` and builds it +into `.deb` packages. Qualcomm developers add downstream patches on the +`resolute-qcom-devel` integration branch. ---- +> [!TIP] +> Latest upload: see the **[tags page](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/tags)**. -## End-to-end pipeline +## At a glance -``` -SCHEDULE: daily 04:00 UTC · RUNNER: lecore-production -══════════════════════════════════════════════════════════════════════════════ - - Ubuntu-qcom Launchpad repository - (git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) - │ - ▼ git ls-remote --tags → latest Ubuntu-qcom-* tag → version X.Y.Z-A.B -╔════════════════════════════════════════════════════════════════════════════╗ -║ fetch-source-pkg.yml ║ -║ ║ -║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ Job 1 · check-version │ ║ -║ │ │ ║ -║ │ git ls-remote resolute-qcom repo → latest Ubuntu-qcom-* tag │ ║ -║ │ → latest version: resolute-qcom X.Y.Z-A.B │ ║ -║ │ │ ║ -║ │ git ls-remote (authenticated) → tag resolute-qcom-X.Y.Z-A.B exists?│ ║ -║ │ │ ║ -║ │ YES ──▶ should_sync=false ──▶ workflow exits cleanly │ ║ -║ │ NO ──▶ should_sync=true ──▶ continue ↓ │ ║ -║ └──────────────────────────────────────────────────────────────────────┘ ║ -║ │ should_sync=true ║ -║ ▼ ║ -║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ Job 2 · sync │ ║ -║ │ │ ║ -║ │ Free disk space (~10 GB) │ ║ -║ │ git clone --depth=1 resolute-qcom repo @ Ubuntu-qcom-X.Y.Z-A.B │ ║ -║ │ Verify >5000 files cloned │ ║ -║ │ rsync source → resolute-qcom branch (orphan) │ ║ -║ │ git commit + tag resolute-qcom-X.Y.Z-A.B │ ║ -║ │ git push branch + tag │ ║ -║ └──────────────────────────────────────────────────────────────────────┘ ║ -║ │ sync succeeded ║ -║ ▼ ║ -║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ Job 3 · trigger-build │ ║ -║ │ │ ║ -║ │ gh workflow run build-kernel.yml │ ║ -║ │ suite=resolute-qcom kernel_version=X.Y.Z-A.B arch=arm64 │ ║ -║ │ runner=lecore-production │ ║ -║ └──────────────────────────────────────────────────────────────────────┘ ║ -╚════════════════════════════════════════════════════════════════════════════╝ - │ - ▼ -╔════════════════════════════════════════════════════════════════════════════╗ -║ build-kernel.yml ║ -║ ║ -║ Checkout resolute-qcom branch ──▶ kernel-src/ ║ -║ Checkout docker-pkg-build ──▶ docker-pkg-build/ ║ -║ docker_deb_build.py --rebuild -d resolute ← base suite derived ║ -║ ║ -║ ┌──────────────────────────────────────────────────────────────────────┐ ║ -║ │ docker run --privileged ghcr.io/qualcomm-linux/pkg-builder:resolute│ ║ -║ │ │ ║ -║ │ apt-get build-dep linux │ ║ -║ │ fakeroot make -f debian/rules clean ← setup env │ ║ -║ │ fakeroot debian/rules binary-qcom do_skip_checks=true │ ║ -║ └──────────────────────────────────────────────────────────────────────┘ ║ -║ ║ -║ Collect .deb files ──▶ output/ ║ -╚════════════════════════════════════════════════════════════════════════════╝ - │ - ├──────────────────────┬───────────────────────┐ - ▼ ▼ ▼ - ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ - │ S3 Bucket │ │ GitHub Artifact │ │ GitHub Release │ - │ │ │ │ │ │ - │ qli-prd- │ │ 90-day retention │ │ resolute-qcom- │ - │ lecore-gh- │ │ Actions → run │ │ X.Y.Z-A.B │ - │ artifacts │ │ → Artifacts │ │ Releases → │ - │ │ │ │ │ Assets │ - │ self-hosted │ │ always │ │ permanent │ - │ runner only │ │ available │ │ │ - └──────────────┘ └──────────────────┘ └──────────────────┘ -``` - -Sync jobs run on: `ubuntu-24.04-arm` (GitHub-hosted, Ubuntu 24.04 arm64) -Build jobs run on: `lecore-prd-u2404-arm64-xlrg-od-ephem` (self-hosted, lecore-production) - ---- - -## Manual build trigger flows - -``` -MANUAL: Actions → Build: Canonical Kernel .deb Packages → Run workflow -══════════════════════════════════════════════════════════════════════════════ - - Three modes depending on kernel_version input: - - ┌─────────────────────────────────────────────────────────────────────────┐ - │ Mode A — Test / dev build (kernel_version left empty) │ - │ │ - │ suite=resolute-qcom kernel_version= │ - │ │ │ - │ ▼ │ - │ Checkout resolute-qcom branch HEAD │ - │ (includes any commits you pushed on top of the synced source) │ - │ │ │ - │ ▼ │ - │ Build .deb packages │ - │ │ │ - │ ▼ │ - │ GitHub Actions artifact only (90-day) ← no release created │ - └─────────────────────────────────────────────────────────────────────────┘ - - ┌─────────────────────────────────────────────────────────────────────────┐ - │ Mode B — Release build for latest synced version │ - │ │ - │ suite=resolute-qcom kernel_version=7.0.0-5.5 │ - │ │ │ - │ ▼ │ - │ Validate tag resolute-qcom-7.0.0-5.5 exists (fail fast if not) │ - │ │ │ - │ ▼ │ - │ Checkout tag resolute-qcom-7.0.0-5.5 ← exact synced source │ - │ │ │ - │ ▼ │ - │ Build .deb packages │ - │ │ │ - │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release resolute-qcom-7.0.0-5.5│ - └─────────────────────────────────────────────────────────────────────────┘ - - ┌─────────────────────────────────────────────────────────────────────────┐ - │ Mode C — Rebuild / re-release an older synced version │ - │ │ - │ suite=resolute-qcom kernel_version=7.0.0-4.4 │ - │ │ │ - │ ▼ │ - │ Validate tag resolute-qcom-7.0.0-4.4 exists (fail fast if not) │ - │ │ │ - │ ▼ │ - │ Checkout tag resolute-qcom-7.0.0-4.4 ← older synced source (not HEAD)│ - │ │ │ - │ ▼ │ - │ Build .deb packages │ - │ │ │ - │ ▼ │ - │ GitHub Actions artifact (90-day) + GitHub Release resolute-qcom-7.0.0-4.4│ - │ (existing release assets are overwritten with --clobber) │ - └─────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## Repository branch layout - -``` -pkg-linux-qcom-canonical -│ -├── main branch -│ ├── .github/workflows/ -│ │ ├── fetch-source-pkg.yml ← sync Launchpad sources → branch -│ │ └── build-kernel.yml ← build .deb packages from branch -│ ├── scripts/ -│ │ ├── check-version.sh ← query latest version from Launchpad -│ │ ├── fetch-source-pkg.sh ← download source package files -│ │ └── build-kernel-deb.sh ← build kernel .deb packages locally -│ └── README.md -│ -├── noble branch (orphan) -│ └── Full Ubuntu Noble 24.04 LTS kernel source tree -│ One commit per Canonical upload -│ Tagged noble-6.8.0-114.114, noble-6.8.0-115.115, … -│ -├── resolute-qcom branch (orphan) -│ └── Full resolute-qcom kernel source tree (daily default) -│ One commit per upstream tag -│ Tagged resolute-qcom-7.0.0-X.X, … -│ -└── branch (orphan, added on demand) - └── Full kernel source for that suite - e.g. questing, resolute -``` - -All branches are **orphan branches** — they share no history with `main` -and contain only the extracted kernel source tree. - ---- - -## Resolute Qcom kernel source - -The daily scheduled build syncs from a custom resolute kernel repository -maintained separately from the official Ubuntu kernel tree. This repository -is referred to as the **resolute-qcom** source. - -| Resource | URL | -|----------|-----| -| Git repository | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | - -**How version discovery works for resolute-qcom:** - -Unlike the official Ubuntu kernel path (which queries the Launchpad REST API -for the latest published source package), the resolute-qcom path queries the -git repository directly: - -```bash -git ls-remote --tags \ - https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ - 'refs/tags/Ubuntu-*' -``` - -Tags are sorted with `sort -V` (version sort) and the latest `Ubuntu-qcom-*` tag -is selected. The version is extracted from the tag name: - -``` -Ubuntu-qcom-7.0.0-1003.3 → version: 7.0.0-1003.3 - → branch tag: resolute-qcom-7.0.0-1003.3 -``` - -The Launchpad REST API is **not used** for resolute-qcom — the git tags are -the authoritative source of version information for this repository. - -**Branch and tag naming:** - -| Item | Pattern | Example | -|------|---------|---------| -| Upstream git tag | `Ubuntu-qcom-X.Y.Z-A.B` | `Ubuntu-qcom-7.0.0-1003.3` | -| Branch in this repo | `resolute-qcom` | `resolute-qcom` | -| Tag in this repo | `resolute-qcom-X.Y.Z-A.B` | `resolute-qcom-7.0.0-1003.3` | -| Docker container | `pkg-builder:resolute` | base suite derived automatically | - ---- - -## Upstream source - -| Resource | URL pattern | Used by | -|----------|-------------|---------| -| Launchpad REST API | `https://api.launchpad.net/1.0/ubuntu/+archive/primary?ws.op=getPublishedSources&source_name=linux&distro_series=/ubuntu/&ws.size=300` | `check-version` job — queries for the latest published version number (official suites only; bypassed for resolute-qcom) | -| Launchpad git repository | `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/` | `sync` job — clones the complete source tree at tag `Ubuntu-` (official suites only) | -| Resolute Qcom git repository | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | `sync` job — daily default; version discovered via `git ls-remote` | -| GitHub Releases | https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/releases | `build-kernel` job — attaches built `.deb` packages | - -**Example (noble suite — official upstream):** -- Source packages: https://launchpad.net/ubuntu/noble/+source/linux -- Git repository: `https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble` - ---- - -## Workflows - -### `fetch-source-pkg.yml` — Sync sources to branch - -Queries the git repository for the latest `Ubuntu-*` tag (resolute-qcom, daily -default) or the Launchpad REST API (official suites), then clones the source -tree at the corresponding tag to get the complete source including `debian/rules`, -and commits it to the branch. - -**Schedule**: daily at **04:00 UTC** -**Manual trigger**: `Actions → Sync: Canonical Kernel Sources to Branch → Run workflow` -**Runner**: `ubuntu-24.04-arm` (all three jobs) - -**Inputs**: - -| Input | Default | Description | -|-------|---------|-------------| -| `suite` *(branch name)* | `resolute-qcom` | Branch to sync into (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). Becomes the branch and tag prefix in this repo. The base Ubuntu suite (`resolute`) is derived automatically from the first component for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | -| `force` | `false` | Re-sync even if tag already exists | -| `custom_git_url` | *(empty)* | Custom Launchpad git URL to clone from. **Leave empty** to sync an official Ubuntu suite (noble, questing, resolute) via the Launchpad REST API. Fill in the Ubuntu-qcom Launchpad repository URL for resolute-qcom builds. Scheduled runs always use the Ubuntu-qcom repo automatically regardless of this field. | - -**Jobs**: - -| Job | What it does | -|-----|-------------| -| `check-version` | For resolute-qcom: queries tags via `git ls-remote` on the custom repo. For official suites: queries Launchpad API (`ws.size=300`). Checks tag existence; sets `should_sync` flag. | -| `sync` | Frees disk space; `git clone --depth=1 --branch Ubuntu-` from the resolved git URL; verifies >5000 files; commits to branch; creates tag | -| `trigger-build` | Dispatches `build-kernel.yml` with `suite`, `kernel_version`, `arch=arm64`, `flavor=qcom` (for custom branches) or `flavor=generic` (for official suites) | - -**Idempotent**: if the tag for the latest version already exists, the workflow exits cleanly without downloading anything. - ---- - -### `build-kernel.yml` — Build .deb packages - -Checks out the branch (full kernel source tree) and builds `.deb` -packages inside the base-suite-matched `ghcr.io/qualcomm-linux/pkg-builder:` -container using `fakeroot debian/rules binary-`. - -**Trigger**: dispatched automatically by `fetch-source-pkg.yml`, or -manually via `Actions → Build: Canonical Kernel .deb Packages → Run workflow`. -**Runner**: `lecore-production` (scheduled builds); selectable for manual builds - -**Inputs**: - -| Input | Default | Description | -|-------|---------|-------------| -| `suite` *(branch name)* | `resolute-qcom` | Branch to build from (e.g. `noble`, `questing`, `resolute`, `resolute-qcom`). The base Ubuntu suite is derived automatically for Docker container selection. Note: the parameter is named `suite` for historical reasons but accepts any branch name. | -| `kernel_version` | — | Version string for release asset attachment | -| `arch` | `arm64` | Target architecture | -| `flavor` | `qcom` | Kernel flavour: `qcom` (default for resolute-qcom), `generic`, `lowlatency`, or `all` | -| `runner` | `ubuntu-24.04-arm` | Runner to use — see table below | - -**Runner options**: - -| Option | Resolves to | Status | -|--------|-------------|--------| -| `ubuntu-24.04-arm` | GitHub-hosted 2-core arm64 | manual builds / fallback | -| `self-hosted` | `runs-on: self-hosted` — any registered self-hosted runner | interim dev runner | -| `lecore-production` | `runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem]` | **active** — used by scheduled builds | - -The scheduled daily sync always dispatches with `runner=lecore-production`. The `lecore-production` runner enables S3 artifact upload (permanent storage) in addition to the GitHub Actions artifact fallback. - -**Self-hosted runner requirements:** -- Ubuntu 24.04 arm64 -- Docker installed; runner user must have access to `/var/run/docker.sock` (add user to `docker` group or `sudo chmod 666 /var/run/docker.sock`) -- ≥ 25 GB free disk space - -**Build steps**: -1. Free up disk space (~10 GB) -2. Checkout branch → `kernel-src/` -3. Checkout `qualcomm-linux/docker-pkg-build@main` → `docker-pkg-build/` -4. Derive `BASE_SUITE` from branch name (e.g. `resolute-qcom` → `resolute`) -5. Build docker image: `docker_deb_build.py --rebuild -d ` -6. Run build inside `ghcr.io/qualcomm-linux/pkg-builder:` container: - ``` - apt-get build-dep linux - fakeroot make -f debian/rules clean - fakeroot debian/rules binary- do_skip_checks=true - ``` - See [Build container notes](#build-container-notes) for why these exact invocations are used. -7. Collect `.deb` files from workspace root - -**Output**: - -| Location | How to access | Retention | Notes | -|----------|---------------|-----------|-------| -| **S3** | `s3://qli-prd-lecore-gh-artifacts//pkg/temp///` | Permanent | `lecore-production` runner only; skipped on other runners | -| **GitHub Actions artifact** | Actions → workflow run → *Artifacts* | 90 days | Always available | -| **GitHub Release asset** | Releases → `-X.Y.Z-A.B` → Assets | Permanent | Attached when `kernel_version` is provided | - ---- - -## Setup - -### 1. Enable workflows - -Go to **Actions** and enable workflows if prompted. - -### 2. Configure repository variables *(optional)* - -**Settings → Secrets and variables → Actions → Variables**: - -| Variable | Default | Description | -|----------|---------|-------------| -| `KERNEL_SUITE` | `resolute-qcom` | Default branch name for scheduled runs | -| `KERNEL_SOURCE` | `linux` | Source package name | -| `KERNEL_CUSTOM_GIT_URL` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | Default custom git URL for scheduled runs | - -### 3. Run the first sync - -**Resolute Qcom (custom branch):** -```bash -gh workflow run fetch-source-pkg.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=resolute-qcom \ - --field custom_git_url="https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute" -``` - -**Official Ubuntu suite (e.g. noble):** -```bash -gh workflow run fetch-source-pkg.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=noble -``` - ---- - -## Local usage - -All scripts run on Ubuntu 24.04 arm64. - -### Check the latest version - -```bash -./scripts/check-version.sh noble linux -# → 6.8.0-114.114 -``` - -### Clone the kernel source (buildable) - -```bash -# Clones from Launchpad git at the latest Ubuntu- tag -./scripts/fetch-source-pkg.sh noble linux ./kernel-src/ -``` - -This produces a complete, buildable source tree with `debian/rules` — the same source the CI workflow uses. - -### Build kernel packages - -```bash -# arm64 generic (native build on arm64 host) -./scripts/build-kernel-deb.sh ./kernel-src/ arm64 generic $(nproc) -``` - ---- - -## Source and build notes +| | | +|---|---| +| **Upstream** | [carmel-team Qualcomm-Ubuntu kernel](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) | +| **`resolute-qcom`** | Mirror of that kernel - sync-only, do not commit here | +| **`resolute-qcom-devel`** | Integration branch - Qualcomm's downstream patches (via PR) | +| **Output** | Kernel `.deb` packages, uploaded to S3 | -### How the sync workflow finds and clones the kernel source +> [!NOTE] +> `main` is the **CI orchestrator** - the workflows, scripts, and docs that drive the sync and build. -The sync workflow supports two paths depending on whether a custom git URL is configured: +## Add a downstream patch -**Path A — Resolute Qcom (daily default): version from git tags** +> [!IMPORTANT] +> Patches reach `resolute-qcom-devel` through a **feature branch + pull request**. Branch off the integration branch, push your branch, then open a PR back into it. ```bash -git ls-remote --tags \ - https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ - 'refs/tags/Ubuntu-qcom-*' -# → sort -V | tail -1 → Ubuntu-qcom-7.0.0-1003.3 -# → VERSION=7.0.0-1003.3 +git clone https://github.com/qualcomm-linux/pkg-linux-qcom-canonical.git +cd pkg-linux-qcom-canonical +git checkout -b my-feature origin/resolute-qcom-devel # branch off the integration branch +# add your patches, then commit DCO-signed: +git commit -s +git push origin my-feature ``` -The latest `Ubuntu-qcom-*` tag in the custom repo is the authoritative version source. -The Launchpad REST API is not used. - -**Path B — Official suites (noble, questing, resolute): version from Launchpad REST API** - -``` -GET https://api.launchpad.net/1.0/ubuntu/+archive/primary - ?ws.op=getPublishedSources - &source_name=linux - &distro_series=/ubuntu/noble - &status=Published - &order_by_date=true - &ws.size=300 - -Response (JSON): -{ - "entries": [ - { - "source_package_name": "linux", - "source_package_version": "6.8.0-114.114", ← we want this - "self_link": "https://api.launchpad.net/..." - }, - ... - ] -} -``` - -The API tells us the exact version string of the latest *officially published* -kernel. A git tag might exist before the package is published to the archive, -so the API is the authoritative source for "what is the current release". - -**Clone step (both paths):** - -```bash -git clone --depth=1 --branch Ubuntu- -``` - -The git repository has the **complete** `debian/` directory including -`debian/rules`, `debian/scripts/`, `debian/templates/`, etc. — unlike the -source package (`.dsc`/`.orig.tar.gz`/`.diff.gz`) which ships only -`debian.master/` with `rules.d/` fragments and no `debian/rules`. - -### Noble branch source tree layout - -``` -arch/ drivers/ fs/ kernel/ -debian/ ← complete Ubuntu packaging (rules, scripts/, templates/, …) -Makefile net/ scripts/ ... -``` - -### About the helper scripts - -| Script | Purpose | Used by workflow? | -|--------|---------|-------------------| -| `scripts/check-version.sh` | Query latest version from Launchpad API | No (workflow has inline equivalent) | -| `scripts/fetch-source-pkg.sh` | Clone from Launchpad git at latest version tag (buildable source) | No (workflow has inline equivalent) | -| `scripts/build-kernel-deb.sh` | Build kernel `.deb` packages locally | No (workflow has inline equivalent) | - -`scripts/fetch-source-pkg.sh` clones from the Launchpad git repository (same -as the CI workflow) and produces a complete, buildable source tree locally. - -> **Note**: The Launchpad API `source_name=` parameter does prefix matching, -> returning all `linux-*` packages. The workflow uses `ws.size=300` to ensure -> the full result set is returned, then applies an exact `source_package_name` -> filter in jq to select only `linux` and not `linux-meta`, `linux-hwe-6.8`, -> `linux-raspi`, or other `linux-*` variants. +Then open a PR from `my-feature` **into `resolute-qcom-devel`**. See +**[docs/INTEGRATION.md](docs/INTEGRATION.md)** for the full workflow. ---- +## Golden rules -## Versioning scheme +> [!WARNING] +> - :no_entry: **Never commit to `resolute-qcom`.** The sync is its only writer and force-advances it on every upload. +> - :no_entry: **Never push directly to `resolute-qcom-devel`.** Patches land through a feature branch + pull request. +> - :white_check_mark: DCO `Signed-off-by` is required on every commit (`git commit -s`). -Ubuntu kernel versions follow `X.Y.Z-A.B`: +## Documentation -| Component | Example | Meaning | -|-----------|---------|---------| -| `X.Y.Z` | `6.8.0` | Upstream kernel version | -| `A` | `114` | ABI number | -| `B` | `114` | Upload number | - -Tags use `-X.Y.Z-A.B`, e.g. `resolute-qcom-7.0.0-1003.3`. - ---- - -## Supported suites - -Official Ubuntu suites supported by this pipeline: - -| Suite | Codename | Ubuntu | Kernel | -|-------|----------|--------|--------| -| `noble` | Noble Numbat | 24.04 LTS — **active** | 6.8 | -| `questing` | Questing Quokka | 25.10 — **active** | 6.17 | -| `resolute` | Resolute Ringtail | 26.04 LTS — **active** | 7.0 | - -To sync an official suite, trigger `fetch-source-pkg.yml` with the desired -`suite` input — the branch and release tag are created automatically: - -```bash -gh workflow run fetch-source-pkg.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical \ - --field suite=resolute -``` - -## Custom branches - -In addition to official Ubuntu suites, this repo supports custom branches -that track non-upstream kernel repositories. Custom branches use a -`-` naming convention so the base suite can be derived -automatically for Docker container selection. - -| Branch | Base suite | Source | Daily default | -|--------|-----------|--------|---------------| -| `resolute-qcom` | `resolute` | `https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute` | ✅ yes | - -Custom branches are **not** Ubuntu suite names — they are branch names in -this repository that happen to be based on a particular Ubuntu suite's kernel. -The `suite` input in both workflows accepts either an official suite name or -a custom branch name. - ---- - -## Build container notes - -### Build environment setup (`debian/rules clean`) - -Before the main kernel compilation starts, the build runs: - -```bash -fakeroot make -f debian/rules clean -``` - -This is the **standard Ubuntu kernel build setup path** — the same entry point -Canonical's own build infrastructure uses. The `clean` target: - -- Runs `debian/control` as a dependency, which generates: - - **`debian/canonical-certs.pem`** — the X.509 certificate embedded into the - kernel image for module signing. Required by the kernel's - `certs/x509_certificate_list` make target. Without it the build fails - immediately: - ``` - No rule to make target 'debian/canonical-certs.pem', - needed by 'certs/x509_certificate_list' - ``` - - **`debian/control`** — the Debian package control stub -- Creates **`debian/changelog → debian.master/changelog`** symlink. The Ubuntu - kernel source tree does not include `debian/changelog` directly — the - changelog lives in `debian.master/changelog`. The `dh_installchangelogs` - debhelper tool (called at the end of `binary-qcom` / `binary-generic`) requires - this symlink to exist or the build fails after 2+ hours of compilation: - ``` - dh_installchangelogs: error: cannot open file debian/changelog - make: *** [debian/rules.d/2-binary-arch.mk:572: binary-qcom] Error 25 - ``` -- Removes any stale build artifacts - -**Why `fakeroot make -f debian/rules` and not `fakeroot debian/rules`?** - -`fakeroot` is a shell script (`/usr/bin/fakeroot`) that execs the given command -via `/bin/sh` (dash). Dash reads the shebang of `debian/rules` -(`#!/usr/bin/make -f`) and tries to resolve the interpreter at exec time. In -the container environment this resolution fails silently, producing: - -``` -/usr/bin/fakeroot: 175: debian/rules: not found (exit 127) -``` - -Invoking `make -f debian/rules` explicitly bypasses the shebang lookup -entirely — `make` is resolved directly from PATH and the Makefile is passed -via `-f`. - ---- - -### Skipping the Rust config policy check (`do_skip_checks=true`) - -The Ubuntu kernel build system runs a config policy check -(`debian/rules.d/4-checks.mk`) that requires `CONFIG_RUST_IS_AVAILABLE=y` -for all supported architectures including arm64. This check fails in the -`pkg-builder` container because `bindgen-0.65` is not available, so -`CONFIG_RUST_IS_AVAILABLE` is set to `-` instead of `y`: - -``` -check-config: CONFIG_RUST_IS_AVAILABLE changed from y to - -make: *** [debian/rules.d/4-checks.mk:15: config-prepare-check-qcom] Error 1 -``` - -Passing `do_skip_checks=true` to `fakeroot debian/rules` bypasses this policy -check. This is the standard approach for non-official builds and is equivalent -to how Canonical's own CI handles environments where optional toolchains are -unavailable. - ---- +| Doc | For | +|-----|-----| +| **[docs/INTEGRATION.md](docs/INTEGRATION.md)** | Qualcomm developers - working on `resolute-qcom-devel` | +| **[docs/PIPELINE.md](docs/PIPELINE.md)** | Maintainers - sync, build, and mirror operations | +| **[CONTRIBUTING.md](CONTRIBUTING.md)** | Contributing to the CI orchestrator on `main` | +| **[SECURITY.md](SECURITY.md)** | Reporting security issues | ## License -Scripts and workflows in this repository are licensed under the -[BSD 3-Clause License](LICENSE.txt). - -The kernel source code fetched from Launchpad is subject to the -[GNU General Public License v2](https://www.kernel.org/doc/html/latest/process/license-rules.html) -and the individual licences of its components. +| Scope | License | +|-------|---------| +| CI scripts and workflows | BSD 3-Clause - see **[LICENSE.txt](LICENSE.txt)** | +| Mirrored kernel source | GPL-2.0 and the individual licences of its components | diff --git a/docs/INTEGRATION.md b/docs/INTEGRATION.md new file mode 100644 index 0000000000000..8c3fe87b887e7 --- /dev/null +++ b/docs/INTEGRATION.md @@ -0,0 +1,40 @@ +# Developing on `resolute-qcom-devel` + +Qualcomm's downstream kernel patches live on the `resolute-qcom-devel` branch, on +top of the `resolute-qcom` mirror of the Canonical kernel. This guide is for +developers adding those patches. + +For how the mirror and build pipeline work, see [PIPELINE.md](PIPELINE.md). + +## The branches + +| Branch | Role | +|--------|------| +| `resolute-qcom` | The mirror of the upstream Canonical kernel. Only the automated sync writes to it - **do not commit here**. | +| `resolute-qcom-devel` | The integration branch: Qualcomm's downstream patches on top of `resolute-qcom`. **Push your patches here.** | + +The `-devel` suffix follows Canonical's own `devel` naming convention. + +## Contributing patches + +Downstream kernel patches land on `resolute-qcom-devel` through a **pull request** - +work on a feature branch and open a PR into `resolute-qcom-devel`. Never commit to +`resolute-qcom`: the sync is its only writer and force-advances it on every new +upload. + +```bash +git clone https://github.com/qualcomm-linux/pkg-linux-qcom-canonical.git +cd pkg-linux-qcom-canonical +git checkout -b my-feature origin/resolute-qcom-devel # feature branch off the integration branch +# add your patches, commit DCO-signed (git commit -s), then: +git push origin my-feature +``` + +Then open a pull request from your feature branch **into `resolute-qcom-devel`**. +Commits must carry a DCO `Signed-off-by` line (`git commit -s`); see +[CONTRIBUTING.md](../CONTRIBUTING.md) for the sign-off policy. + +## Building + +To build `resolute-qcom-devel` (or any mirrored upload) into `.deb` packages, see +[PIPELINE.md](PIPELINE.md#manual-build-triggers). diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md new file mode 100644 index 0000000000000..23270e530e3a0 --- /dev/null +++ b/docs/PIPELINE.md @@ -0,0 +1,298 @@ +# Pipeline operations (maintainers) + +How the mirror and build pipeline work, and the reasoning behind the parts that +are easy to break. + +The pipeline has two halves: a **one-time bootstrap** that seeds the full history, +and the **incremental sync + build** that runs per upload thereafter. + +## Repository branch layout + +``` +pkg-linux-qcom-canonical +│ +├── main branch ← CI orchestrator: workflows, scripts, docs +│ ├── .github/workflows/ +│ │ ├── fetch-source-pkg.yml ← manual incremental mirror sync +│ │ ├── bootstrap-history.yml ← one-time full-history seed +│ │ └── build-kernel.yml ← build .deb packages +│ ├── scripts/ +│ │ ├── sync-mirror.sh ← incremental mirror-repoint (CI) +│ │ └── seed-history.sh ← one-time full-history bootstrap (CI) +│ └── README.md +│ +├── resolute-qcom branch ← full-history mirror of the carmel-team tree (SYNC-ONLY) +│ └── ~1.4M commits; immutable tag per upload: Ubuntu-qcom-X.Y.Z-A.B (verbatim) +│ +├── resolute-qcom-devel branch ← developer integration branch (see INTEGRATION.md) +│ +└── resolute-qcom-seed branch ← transient bootstrap staging (only during a seed) +``` + +`resolute-qcom` shares **no history with `main`** (the CI orchestrator); it +carries the **complete upstream commit history** of the kernel it mirrors. + +## Architecture and flow + +### One-time bootstrap (run once, by hand) + +```mermaid +flowchart LR + LP[("Launchpad
~carmel-team / resolute
Ubuntu-qcom-* tags")] + A1["bootstrap-history.yml
manual, once"] + A2["seed-history.sh
full clone ~1.43M commits
relaxed slow-transfer abort"] + A3["sliced push < 2 GB
verbatim Ubuntu-qcom-* tags"] + A4["resolute-qcom-seed
(staging branch)"] + A5{{"human review
and promote"}} + A6[("resolute-qcom
the live mirror")] + A1 --> A2 + LP -->|"clone full history (slow)"| A2 + A2 --> A3 --> A4 --> A5 -->|"promote"| A6 +``` + +Launchpad cannot serve shallow `--deepen`, so the seed is a single-shot **full** +clone; GitHub caps a push at 2 GB, so it is pushed in `< 2 GB` slices. The seed +lands on `resolute-qcom-seed` (never the live branch) for a human to promote. + +### Incremental sync + build (per new upstream upload, manual dispatch) + +```mermaid +flowchart TB + LP[("Launchpad
newest Ubuntu-qcom-* tag")] + G{"check-version gate
upstream tags vs ours
anything new?"} + NO(["no-op, exit green"]) + S1["sync-mirror.sh
fetch delta tag (no --depth)"] + S2["repoint resolute-qcom
create verbatim tag"] + S3["atomic lease-pinned push
branch + tag"] + RQ[("resolute-qcom
advanced to new upload")] + T["trigger-build"] + B1["build-kernel.yml
checkout Ubuntu-qcom-<ver> tag"] + B2["build .deb
pkg-builder:resolute"] + S3DB[("S3 only
qli-prd-lecore-gh-artifacts")] + LP --> G + G -->|"no"| NO + G -->|"yes"| S1 --> S2 --> S3 --> RQ --> T --> B1 --> B2 --> S3DB +``` + +The gate is a cheap two-call `ls-remote` (no clone). Because the branch already +holds full history, the per-upload fetch transfers only the delta, the push is +small, and the branch fast-forwards onto the new verbatim tag. + +Sync jobs run on `ubuntu-24.04-arm` (GitHub-hosted). The build runs on the +`lecore-production` self-hosted runner (`lecore-prd-u2404-arm64-xlrg-od-ephem`). + +## The mirror-repoint model and its guardrails + +`resolute-qcom` is a **faithful mirror of the upstream Canonical kernel tree with +full commit history preserved**. Every upload is fetched with its real ancestry +and frozen under an immutable per-upload tag. + +| Ref | Role | +|-----|------| +| **branch** `resolute-qcom` | A movable "latest Canonical" pointer. Disposable by design - force-advanced to each new upload. | +| **tag** `Ubuntu-qcom-X.Y.Z-A.B` | The upstream Canonical tag, mirrored **verbatim**. An **immutable** per-upload record; the sole anchor that preserves that upload's history (`git diff` between any two tags is a clean kernel delta). | + +A sync is **pure fetch + repoint** ([`scripts/sync-mirror.sh`](../scripts/sync-mirror.sh)): +it only fast-forwards/repoints the mirror (no merge, no rebase), so it can never +conflict. + +**Guardrails - do not "optimize" these away.** They are what keeps the ~1.43M-commit +history intact; several look redundant until the day they save the branch: + +- **Never `--depth` / never shallow.** The incremental fetch and the seed must + produce a complete, non-shallow closure. A shallow base breaks fetch negotiation, + and GitHub rejects a shallow push outright. +- **Pinned lease, atomic push.** The branch + verbatim tag are pushed together with + `--atomic` and `--force-with-lease=:` (never a bare + `--force-with-lease`, never a blind `--force`). This prevents a tag-lands / + branch-rejected split-brain and a concurrent run clobbering the branch. +- **Tag before moving the branch, ascending.** Every missing upload is tagged in + version order before the branch advances past it, so a rebased middle upload is + never silently dropped. +- **Never move a preservation tag.** A re-pointed upstream tag returns a non-zero + "would clobber existing tag"; the sync surfaces it and refuses, rather than + overwriting a frozen record. +- **Bare clone, fail-fast.** The mirror is operated as a bare repo (clean ref + updates); any rejected push aborts the whole run so the next idempotent run can + never mask a partial sync. +- **A freshly created branch is refused.** `sync-mirror.sh` aborts if the branch + has fewer than `MIN_HISTORY_COMMITS` (1000) - i.e. it has not been seeded with + real history yet. Run the bootstrap first. + +### Launchpad and GitHub limits (hard-won) + +- Launchpad's shallow `git fetch --deepen` path is **broken** (it stalls and throws + `error processing shallow info`). It *can* serve a full clone, but spends several + minutes computing the pack server-side, so the seed relaxes the git slow-transfer + abort to ride out that quiet phase. +- GitHub caps a single push at **2 GB**, so the seed is pushed in `< 2 GB` slices. + +## Upstream source and version discovery + +The mirror tracks the carmel-team Qualcomm-Ubuntu kernel on Launchpad: +[https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute). + +Version discovery is **purely git-tag based - there is no Launchpad REST API call**. +The upstream `Ubuntu-qcom-*` tag is the authoritative version source; the +`check-version` gate compares it against our mirrored tags. To check the newest +upstream upload directly, independent of the mirror: + +```bash +git ls-remote --tags \ + https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ + 'refs/tags/Ubuntu-qcom-*' | sort -V | tail -1 +# -> Ubuntu-qcom-7.0.0-1006.8 (example output) +``` + +## Bootstrapping + +`resolute-qcom` must be **seeded with full history once** before the sync can run +incrementally. This is automated: + +```bash +gh workflow run bootstrap-history.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical + # defaults: suite=resolute-qcom (seeds into the resolute-qcom-seed branch) +``` + +How it works ([`scripts/seed-history.sh`](../scripts/seed-history.sh)): + +- **Single-shot full clone** of the full history (see + [Launchpad and GitHub limits](#launchpad-and-github-limits-hard-won) for why + shallow `--deepen` cannot be used). +- **Sliced push** in `< 2 GB` slices (the GitHub push cap, same section); the + upstream `Ubuntu-qcom-*` tags are pushed **verbatim**. +- **Pushes to `resolute-qcom-seed`, never the live branch.** A human reviews and + promotes it (the production `resolute-qcom` branch is too valuable to clobber + automatically). The cloned repo is cached, so a retry after a failed publish + skips re-downloading the multi-GB history. + +Once seeded and promoted, +[`fetch-source-pkg.yml`](../.github/workflows/fetch-source-pkg.yml) keeps +`resolute-qcom` current; it **refuses to run against an un-seeded branch**. + +## Running a sync + +The sync takes **no inputs** - it always mirrors the latest carmel-team upload +into `resolute-qcom`: + +```bash +gh workflow run fetch-source-pkg.yml \ + --repo qualcomm-linux/pkg-linux-qcom-canonical +# or: Actions → "Sync: Canonical Kernel Sources to Branch" → Run workflow +``` + +If the newest `Ubuntu-qcom-*` tag is already mirrored, the run exits cleanly +(idempotent). Otherwise it mirrors every un-mirrored upload in version order, then +triggers a build of the newest one. + +## Manual build triggers + +Trigger manually: **Actions → Build: Canonical Kernel .deb Packages → Run +workflow**. It builds the `resolute-qcom-devel` integration branch by default; set +`suite` to `resolute-qcom` to build the mirror instead. The `kernel_version` input +controls what is checked out: + +| `kernel_version` | Source checked out | Use | +|------------------|--------------------|-----| +| *empty* | the selected branch HEAD (default `resolute-qcom-devel`) | build the current branch tip | +| `X.Y.Z-A.B` | the tag `Ubuntu-qcom-X.Y.Z-A.B` (validated first; fails fast if absent) | build an exact mirrored upload | + +The only output is the `.deb` upload to **S3** (`lecore-production` runner). No +GitHub Actions artifacts and no GitHub Releases are produced. + +## Workflows reference + +### `fetch-source-pkg.yml` - Sync + +Mirrors new upstream uploads into `resolute-qcom`, history-preserving, via +[`scripts/sync-mirror.sh`](../scripts/sync-mirror.sh). + +- **Trigger**: manual `workflow_dispatch` only (no cron). **No inputs** - the + upstream URL, branch, and tag prefix are fixed constants in the workflow. +- **Runner**: `ubuntu-24.04-arm` (all jobs). + +| Job | What it does | +|-----|-------------| +| `check-version` | Two `ls-remote` calls (no clone): newest upstream `Ubuntu-qcom-*` tag vs our tags. Sets `should_sync`. | +| `sync` | Bare-clones our mirror; for each un-mirrored upload (ascending): fetches the upstream tag (delta only), advances `resolute-qcom`, and atomically pushes the branch + the verbatim tag (lease-pinned, fail-fast). Never merges/rebases. | +| `trigger-build` | Dispatches `build-kernel.yml` for the newest synced upload of the mirror (`suite=resolute-qcom`, `arch=arm64`, `flavor=qcom`, `runner=lecore-production`). | + +Idempotent: if the newest upload is already mirrored, the run exits without cloning +anything. + +### `build-kernel.yml` - Build + +Checks out the kernel source (a verbatim tag, or the branch HEAD) and builds `.deb` +packages inside the base-suite-matched +`ghcr.io/qualcomm-linux/pkg-builder:` container. + +- **Trigger**: dispatched by the sync, or manually. +- **Output**: **S3 only** (`lecore-production` runner). No artifacts, no releases. + +| Input | Default | Description | +|-------|---------|-------------| +| `suite` | `resolute-qcom-devel` | Branch to build from; base suite (`resolute`) derived for container selection. | +| `kernel_version` | *(empty)* | Builds the exact `Ubuntu-qcom-` tag (validated first). Empty = branch HEAD. | +| `arch` | `arm64` | Target architecture. | +| `flavor` | `qcom` | Kernel flavour. **`qcom` is the only flavour this tree builds** (`generic`/`lowlatency` do not exist for the qcom derivative). `all` builds `qcom` + `qcom-rt`. | +| `runner` | `ubuntu-24.04-arm` | `ubuntu-24.04-arm` (hosted, no S3), `self-hosted`, or `lecore-production` (the only runner that uploads to S3). | + +**Build steps**: +1. Free disk space. +2. Checkout the tag/branch → `kernel-src/`. +3. Checkout `qualcomm-linux/docker-pkg-build@main`. +4. Derive `BASE_SUITE` (`resolute-qcom` → `resolute`). +5. Build the image: `docker_deb_build.py --rebuild -d `. +6. In-container: `apt-get build-dep linux`; `fakeroot make -f debian/rules clean`; + `fakeroot debian/rules binary- do_skip_checks=true` + (see [Build container notes](#build-container-notes)). +7. Collect `.deb` files and upload to S3. + +**Self-hosted runner requirements** (`lecore-production`): Ubuntu 24.04 arm64, +Docker (runner user in the `docker` group), ≥ 25 GB free disk. + +## The CI scripts + +| Script | Purpose | Used by | +|--------|---------|---------| +| `scripts/sync-mirror.sh` | Incremental history-preserving mirror-repoint of new uploads | `fetch-source-pkg.yml` | +| `scripts/seed-history.sh` | One-time full-history bootstrap of `resolute-qcom-seed` | `bootstrap-history.yml` | + +`scripts/check-version.sh`, `scripts/fetch-source-pkg.sh`, and +`scripts/build-kernel-deb.sh` are **legacy** standalone local helpers that predate +the mirror model; they are **not** part of the CI path. + +## Build container notes + +### Build environment setup (`debian/rules clean`) + +Before compilation the build runs `fakeroot make -f debian/rules clean`, the +standard Ubuntu kernel build setup path. The `clean` target: + +- Runs `debian/control` as a dependency, generating **`debian/canonical-certs.pem`** + (the X.509 cert the kernel's `certs/x509_certificate_list` target requires - + without it the build fails immediately) and **`debian/control`**. +- Creates **`debian/changelog`** (the active derivative's changelog, + `debian.qcom/changelog`). `dh_installchangelogs`, run at the end of + `binary-qcom`, fails without it - after 2+ hours of compilation. +- Removes stale build artifacts. + +**Why `fakeroot make -f debian/rules` and not `fakeroot debian/rules`?** +`fakeroot` execs the command via `/bin/sh` (dash), which resolves the +`#!/usr/bin/make -f` shebang at exec time; in the container that resolution fails +silently with `debian/rules: not found (exit 127)`. Passing `make -f debian/rules` +explicitly bypasses the shebang lookup. + +### Skipping the config policy check (`do_skip_checks=true`) + +The Ubuntu kernel build runs a config-policy check (`annotations --check`) that +expects `CONFIG_RUST_IS_AVAILABLE=y`. In the `pkg-builder` container `bindgen` is +unavailable, so `olddefconfig` drops that symbol and the policy diff fails. Passing +`do_skip_checks=true` bypasses the check - the standard approach for non-official +builds where optional toolchains are absent. + +## License + +See the [main README](../README.md#license): BSD 3-Clause for the CI scripts and +workflows, GPL-2 for the mirrored kernel source. From fc601e67b7abf56d4db2a74f51d341a151908073 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Wed, 1 Jul 2026 16:22:06 -0700 Subject: [PATCH 089/115] ci: default manual kernel builds to resolute-qcom-devel Manual dispatches of build-kernel.yml now build the integration branch by default, since that is what developers test. Sync-triggered builds are unchanged: fetch-source-pkg.yml passes suite=resolute-qcom explicitly, so automatic builds still target the mirror. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 1788113799d84..ba27900ac3f7b 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -20,9 +20,9 @@ on: workflow_dispatch: inputs: suite: - description: "Branch to build from (e.g. resolute-qcom)" + description: "Branch to build from (e.g. resolute-qcom-devel, resolute-qcom)" required: true - default: "resolute-qcom" + default: "resolute-qcom-devel" type: string kernel_version: description: "Version (e.g. 7.0.0-1006.8) - builds the exact Ubuntu-qcom- tag. Empty = branch HEAD." From 791b7909e1dbb3aa7ef82e87c15c073bc5b89b9c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 16:53:00 -0700 Subject: [PATCH 090/115] ci(build-kernel): always build binary-indep to produce linux-qcom-headers When flavor is not 'all', the build target was set to binary-${FLAVOR} (e.g. binary-qcom), which skips the binary-indep target entirely. The arch-independent common headers package (linux-qcom-headers-PKGVER-ABINUM, arch: all) is only produced by binary-indep, so it was never built and never uploaded to S3. The flavour-specific linux-headers-PKGVER-ABINUM-FLAVOUR package has a hard Depends on linux-qcom-headers-PKGVER-ABINUM (declared in debian.qcom/control.d/flavour-control.stub), so without the indep package in S3 the rootfs build fails to install linux-headers at all. Fix: append binary-indep to TARGET whenever flavor != 'all', so the common headers package is always built and collected alongside the flavour-specific debs. Remove quotes around ${TARGET} in the fakeroot invocation so make receives the two targets as separate arguments rather than a single string. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index ba27900ac3f7b..ff9435a65ca5d 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -198,7 +198,7 @@ jobs: - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'resolute-qcom' }}) run: | JOBS=$(nproc) - if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR}"; fi + if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR} binary-indep"; fi echo "Building: suite=${SUITE} base_suite=${BASE_SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" @@ -263,7 +263,7 @@ jobs: # CONFIG_RUST_IS_AVAILABLE=y — bindgen-0.65 is not available in # the container so Rust support is unavailable, causing the check # to fail. This is the standard approach for non-official builds. - fakeroot debian/rules "${TARGET}" do_skip_checks=true + fakeroot debian/rules ${TARGET} do_skip_checks=true kill "${HEARTBEAT}" 2>/dev/null || true From a69e95e20cd562dbc655638ef94dfdcf58c325fc Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 15:09:55 -0700 Subject: [PATCH 091/115] ci: add devel_prs input, workflow_call trigger, and premerge-pr build check build-kernel.yml: - Add workflow_call trigger so premerge-pr.yml can call it as a reusable workflow without duplicating build logic. - Add devel_prs input (space-separated PR numbers against resolute-qcom-devel). Each PR is fetched and merged with --no-ff --no-commit before the build. Conflict or invalid input aborts immediately with a clear error. Modeled on qcom-next-pr in pkg-linux-qcom/build-kernel-deb.yml. - Add ref input to workflow_call so callers can pin an exact SHA (used by premerge-pr.yml to build the PR head, not the branch tip). - Remove dead if: inputs.runner == 'lecore-production' guard on S3 upload (runner is now always lecore-production). premerge-pr.yml: - New workflow triggered on pull_request_target targeting resolute-qcom-devel (opened, synchronize, reopened). - Uses pull_request_target (not pull_request) because resolute-qcom-devel is a kernel source tree with no .github/; pull_request_target resolves the workflow from the default branch. - Calls build-kernel.yml with suite=resolute-qcom-devel and ref=PR head SHA so each PR builds its exact commit in isolation. - Concurrency group per PR number cancels stale runs on new pushes. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 73 +++++++++++++++++++++++++++++- .github/workflows/premerge-pr.yml | 40 ++++++++++++++++ 2 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/premerge-pr.yml diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index ff9435a65ca5d..ef1a7da6f84e1 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -55,6 +55,34 @@ on: - lecore-production - ubuntu-24.04-arm - self-hosted + devel_prs: + description: "Space-separated PR numbers against resolute-qcom-devel to merge before building (e.g. 42 43)" + required: false + default: "" + type: string + + workflow_call: + inputs: + suite: + description: "Branch to build from" + required: false + default: "resolute-qcom-devel" + type: string + kernel_version: + description: "Version tag to build. Empty = branch HEAD." + required: false + default: "" + type: string + devel_prs: + description: "Space-separated PR numbers against resolute-qcom-devel to merge before building" + required: false + default: "" + type: string + ref: + description: "Exact git ref to checkout. Overrides suite/kernel_version when set (used by premerge-pr.yml to pin the PR head SHA)." + required: false + default: "" + type: string permissions: contents: read # checkout + tag ls-remote only; output goes to S3 @@ -168,9 +196,48 @@ jobs: - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.kernel_version != '' && format('{0}-{1}', env.UPSTREAM_PREFIX, inputs.kernel_version) || inputs.suite || 'resolute-qcom' }} + ref: ${{ inputs.ref != '' && inputs.ref || (inputs.kernel_version != '' && format('{0}-{1}', env.UPSTREAM_PREFIX, inputs.kernel_version) || inputs.suite || 'resolute-qcom') }} path: kernel-src + # ----------------------------------------------------------------------- + # 4b. Merge additional PRs against resolute-qcom-devel into the + # checked-out source tree (engineering builds only). + # Each PR number is validated as numeric before use. Conflicts abort + # the build immediately with a clear error. + # ----------------------------------------------------------------------- + - name: Merge devel PRs into source tree + if: inputs.devel_prs != '' + env: + DEVEL_PRS: ${{ inputs.devel_prs }} + GH_TOKEN: ${{ github.token }} + run: | + cd kernel-src/ + git config user.email "github-actions[bot]@users.noreply.github.com" + git config user.name "github-actions[bot]" + git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${{ github.repository }}.git" + + echo "PRs to merge: ${DEVEL_PRS}" + for pr in ${DEVEL_PRS}; do + [[ "${pr}" =~ ^[0-9]+$ ]] || { + echo "ERROR: invalid PR number '${pr}' -- must be numeric" >&2 + exit 1 + } + echo "::group::Merging resolute-qcom-devel PR #${pr}" + git fetch --no-tags origin "refs/pull/${pr}/head:pr-${pr}" + if ! git merge --no-ff --no-commit "pr-${pr}"; then + echo "ERROR: merge conflict while merging PR #${pr}. Aborting." >&2 + git merge --abort || true + exit 1 + fi + if git diff --cached --quiet; then + echo "PR #${pr} already present in tree -- nothing to merge." + else + git commit -m "Merged resolute-qcom-devel PR #${pr}" + echo "PR #${pr} merged successfully." + fi + echo "::endgroup::" + done + # ----------------------------------------------------------------------- # 5. Checkout docker-pkg-build # ----------------------------------------------------------------------- @@ -298,7 +365,6 @@ jobs: # runner's registered display name may differ from its label. # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 - if: inputs.runner == 'lecore-production' uses: qualcomm-linux/upload-private-artifact-action@aws-v4 with: s3_bucket: qli-prd-lecore-gh-artifacts @@ -321,6 +387,9 @@ jobs: echo "| Flavour | \`${FLAVOR}\` |" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" echo "| Runner | \`${{ runner.name }}\` |" + if [ -n "${{ inputs.devel_prs }}" ]; then + echo "| Devel PRs | \`${{ inputs.devel_prs }}\` |" + fi echo "" echo "### Built packages" echo '```' diff --git a/.github/workflows/premerge-pr.yml b/.github/workflows/premerge-pr.yml new file mode 100644 index 0000000000000..0cca085391633 --- /dev/null +++ b/.github/workflows/premerge-pr.yml @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: BSD-3-Clause +# +# premerge-pr.yml +# +# Triggered on every PR opened or updated against resolute-qcom-devel. +# Runs a full kernel build and reports pass/fail as a status check on the PR. +# +# Uses pull_request_target so the workflow is resolved from the default +# branch (main) rather than from the base branch of the PR +# (resolute-qcom-devel, which is a kernel source tree with no .github/). +# +# The build checks out the exact PR head SHA so concurrent PRs are +# fully isolated. A new push to the same PR cancels the in-flight run. + +name: "Pre-merge PR build" + +on: + pull_request_target: + branches: + - resolute-qcom-devel + types: + - opened + - synchronize + - reopened + +concurrency: + group: premerge-pr-${{ github.event.pull_request.number }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: read + +jobs: + build: + name: "Build check" + uses: ./.github/workflows/build-kernel.yml + with: + suite: resolute-qcom-devel + ref: ${{ github.event.pull_request.head.sha }} From 075bef79c013ba4134d4eb27d5a4b8e786585ad5 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 15:10:21 -0700 Subject: [PATCH 092/115] ci: remove redundant arch, flavor, runner inputs from build-kernel.yml arch, flavor, and runner are constants for this repo and never vary: arch = arm64 (arm64-only kernel) flavor = qcom (only the qcom flavour is built) runner = lecore-production (only runner with S3 access) Remove them from workflow_dispatch inputs and hardcode directly in the job. The fromJSON runner map is replaced with the lecore label directly on runs-on. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 33 +++--------------------------- 1 file changed, 3 insertions(+), 30 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index ef1a7da6f84e1..6e99d459a29bf 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -28,33 +28,6 @@ on: description: "Version (e.g. 7.0.0-1006.8) - builds the exact Ubuntu-qcom- tag. Empty = branch HEAD." required: false type: string - arch: - description: "Target Debian architecture" - required: false - default: "arm64" - type: choice - options: - - arm64 - - amd64 - flavor: - description: "Kernel flavour to build" - required: false - default: "qcom" - type: choice - options: - - qcom - - generic - - lowlatency - - all - runner: - description: "Runner to use for the build (lecore-production is the only one that uploads to S3)" - required: false - default: "lecore-production" - type: choice - options: - - lecore-production - - ubuntu-24.04-arm - - self-hosted devel_prs: description: "Space-separated PR numbers against resolute-qcom-devel to merge before building (e.g. 42 43)" required: false @@ -97,13 +70,13 @@ jobs: # lecore-production → ["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"] (default; uploads to S3) # ubuntu-24.04-arm → "ubuntu-24.04-arm" (GitHub-hosted) # self-hosted → "self-hosted" - runs-on: ${{ fromJSON('{"ubuntu-24.04-arm":"ubuntu-24.04-arm","self-hosted":"self-hosted","lecore-production":["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"]}')[inputs.runner || 'lecore-production'] }} + runs-on: ["self-hosted", "lecore-prd-u2404-arm64-xlrg-od-ephem"] timeout-minutes: 360 env: SUITE: ${{ inputs.suite || 'resolute-qcom' }} - ARCH: ${{ inputs.arch || 'arm64' }} - FLAVOR: ${{ inputs.flavor || 'qcom' }} + ARCH: arm64 + FLAVOR: qcom steps: # ----------------------------------------------------------------------- From 333713e52a93dcfa6a106c47efb4d1cb68a24739 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 15:32:59 -0700 Subject: [PATCH 093/115] ci: add skip_s3 input to build-kernel.yml and use it in premerge-pr.yml Premerge builds should never upload to S3. The runner input removal made the S3 upload unconditional since the job always runs on lecore-production. Add a workflow_call-only skip_s3 boolean input to gate the upload step, and set it in premerge-pr.yml. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 10 +++++++--- .github/workflows/premerge-pr.yml | 1 + 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 6e99d459a29bf..13ed84e4c8c5d 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -56,6 +56,11 @@ on: required: false default: "" type: string + skip_s3: + description: "Skip uploading built .deb packages to S3 (used by premerge-pr.yml for build-only checks)." + required: false + default: false + type: boolean permissions: contents: read # checkout + tag ls-remote only; output goes to S3 @@ -333,11 +338,10 @@ jobs: # ----------------------------------------------------------------------- # 9. Upload to S3 (lecore-production runner only) - # Skipped on GitHub-hosted and generic self-hosted runners. - # Condition checks the runner input, not runner.name, because the - # runner's registered display name may differ from its label. + # Skipped when skip_s3 is set (premerge-pr.yml build-only checks). # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 + if: inputs.skip_s3 != true uses: qualcomm-linux/upload-private-artifact-action@aws-v4 with: s3_bucket: qli-prd-lecore-gh-artifacts diff --git a/.github/workflows/premerge-pr.yml b/.github/workflows/premerge-pr.yml index 0cca085391633..58022717ec011 100644 --- a/.github/workflows/premerge-pr.yml +++ b/.github/workflows/premerge-pr.yml @@ -38,3 +38,4 @@ jobs: with: suite: resolute-qcom-devel ref: ${{ github.event.pull_request.head.sha }} + skip_s3: true From 485fa4bcbc9eba2c9e2f1a6d390ce161fbab4c53 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 15:33:13 -0700 Subject: [PATCH 094/115] ci: remove stale runner comment and fix SUITE default mismatch The runner comment referenced ubuntu-24.04-arm and self-hosted options that were dropped when the runner input was removed. Also align the workflow_dispatch SUITE env default with the actual default input value (resolute-qcom-devel). Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 13ed84e4c8c5d..eef965b261bed 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -71,15 +71,12 @@ jobs: # -------------------------------------------------------------------------- build: name: "Build" - # Map friendly runner names to their actual runs-on labels. - # lecore-production → ["self-hosted","lecore-prd-u2404-arm64-xlrg-od-ephem"] (default; uploads to S3) - # ubuntu-24.04-arm → "ubuntu-24.04-arm" (GitHub-hosted) - # self-hosted → "self-hosted" + # Runner is always lecore-production for this repo. runs-on: ["self-hosted", "lecore-prd-u2404-arm64-xlrg-od-ephem"] timeout-minutes: 360 env: - SUITE: ${{ inputs.suite || 'resolute-qcom' }} + SUITE: ${{ inputs.suite || 'resolute-qcom-devel' }} ARCH: arm64 FLAVOR: qcom From d4a55f85ca7733075dabc457fa239adae1783c0c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 16:12:23 -0700 Subject: [PATCH 095/115] ci: remove stale arch/flavor/runner fields from fetch-source-pkg dispatch build-kernel.yml no longer accepts arch, flavor, or runner as workflow_dispatch inputs (removed in a060b472). Passing them via gh workflow run causes GitHub to return HTTP 422 Unexpected inputs, silently breaking the trigger-build job on every Canonical sync run. Drop the three stale --field lines. suite and kernel_version are the only inputs needed; arch, flavor, and runner are now hardcoded inside build-kernel.yml. Signed-off-by: Bjordis Collaku --- .github/workflows/fetch-source-pkg.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index dec5f110884a6..442dbac66755c 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -156,7 +156,4 @@ jobs: gh workflow run build-kernel.yml \ --repo "${{ github.repository }}" \ --field suite="${BRANCH}" \ - --field kernel_version="${VERSION}" \ - --field arch="arm64" \ - --field flavor="qcom" \ - --field runner="lecore-production" + --field kernel_version="${VERSION}" From 8ecf4010885afdd1cb780baf7f788d0f3f19301c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 6 Jul 2026 16:18:22 -0700 Subject: [PATCH 096/115] docs: update PIPELINE.md for pr-build-pipeline changes - remove arch, flavor, runner from build-kernel.yml inputs table (removed as inputs in a060b472, now hardcoded) - add devel_prs, ref, skip_s3 inputs to the table with workflow_dispatch vs workflow_call scope noted - fix trigger-build job description to drop the stale arch/flavor/runner fields (matches the fetch-source-pkg.yml fix in 1744ced9) - update build steps list: note ref override on checkout, add devel_prs merge step, renumber, note skip_s3 on S3 upload step - add premerge-pr.yml section to workflows reference Signed-off-by: Bjordis Collaku --- docs/PIPELINE.md | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 23270e530e3a0..7699ba5121280 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -15,7 +15,8 @@ pkg-linux-qcom-canonical │ ├── .github/workflows/ │ │ ├── fetch-source-pkg.yml ← manual incremental mirror sync │ │ ├── bootstrap-history.yml ← one-time full-history seed -│ │ └── build-kernel.yml ← build .deb packages +│ │ ├── build-kernel.yml ← build .deb packages +│ │ └── premerge-pr.yml ← PR build check for resolute-qcom-devel │ ├── scripts/ │ │ ├── sync-mirror.sh ← incremental mirror-repoint (CI) │ │ └── seed-history.sh ← one-time full-history bootstrap (CI) @@ -190,7 +191,7 @@ triggers a build of the newest one. Trigger manually: **Actions → Build: Canonical Kernel .deb Packages → Run workflow**. It builds the `resolute-qcom-devel` integration branch by default; set -`suite` to `resolute-qcom` to build the mirror instead. The `kernel_version` input +the **Branch to build from** field to `resolute-qcom` to build from the mirror branch instead. The `kernel_version` input controls what is checked out: | `kernel_version` | Source checked out | Use | @@ -216,7 +217,7 @@ Mirrors new upstream uploads into `resolute-qcom`, history-preserving, via |-----|-------------| | `check-version` | Two `ls-remote` calls (no clone): newest upstream `Ubuntu-qcom-*` tag vs our tags. Sets `should_sync`. | | `sync` | Bare-clones our mirror; for each un-mirrored upload (ascending): fetches the upstream tag (delta only), advances `resolute-qcom`, and atomically pushes the branch + the verbatim tag (lease-pinned, fail-fast). Never merges/rebases. | -| `trigger-build` | Dispatches `build-kernel.yml` for the newest synced upload of the mirror (`suite=resolute-qcom`, `arch=arm64`, `flavor=qcom`, `runner=lecore-production`). | +| `trigger-build` | Dispatches `build-kernel.yml` for the newest synced upload of the mirror (`suite=resolute-qcom`, `kernel_version=`). | Idempotent: if the newest upload is already mirrored, the run exits without cloning anything. @@ -230,28 +231,39 @@ packages inside the base-suite-matched - **Trigger**: dispatched by the sync, or manually. - **Output**: **S3 only** (`lecore-production` runner). No artifacts, no releases. -| Input | Default | Description | -|-------|---------|-------------| -| `suite` | `resolute-qcom-devel` | Branch to build from; base suite (`resolute`) derived for container selection. | -| `kernel_version` | *(empty)* | Builds the exact `Ubuntu-qcom-` tag (validated first). Empty = branch HEAD. | -| `arch` | `arm64` | Target architecture. | -| `flavor` | `qcom` | Kernel flavour. **`qcom` is the only flavour this tree builds** (`generic`/`lowlatency` do not exist for the qcom derivative). `all` builds `qcom` + `qcom-rt`. | -| `runner` | `ubuntu-24.04-arm` | `ubuntu-24.04-arm` (hosted, no S3), `self-hosted`, or `lecore-production` (the only runner that uploads to S3). | +| Input | Default | Trigger | Description | +|-------|---------|---------|-------------| +| `suite` | `resolute-qcom-devel` | dispatch + call | Branch to build from; base suite (`resolute`) derived for container selection. | +| `kernel_version` | *(empty)* | dispatch + call | Builds the exact `Ubuntu-qcom-` tag (validated first). Empty = branch HEAD. | +| `devel_prs` | *(empty)* | dispatch + call | Space-separated PR numbers against `resolute-qcom-devel` to merge before building (e.g. `42 43`). Conflict aborts with a clear error. | +| `ref` | *(empty)* | call only | Exact git ref to checkout, overrides `suite`/`kernel_version` when set. Used by `premerge-pr.yml` to pin the PR head SHA. | +| `skip_s3` | `false` | call only | Skip the S3 upload step. Used by `premerge-pr.yml` so premerge builds are build-only and never upload. | **Build steps**: 1. Free disk space. -2. Checkout the tag/branch → `kernel-src/`. -3. Checkout `qualcomm-linux/docker-pkg-build@main`. -4. Derive `BASE_SUITE` (`resolute-qcom` → `resolute`). -5. Build the image: `docker_deb_build.py --rebuild -d `. -6. In-container: `apt-get build-dep linux`; `fakeroot make -f debian/rules clean`; +2. Checkout the tag/branch (or `ref` if set) → `kernel-src/`. +3. If `devel_prs` is set: merge each PR against `resolute-qcom-devel` into `kernel-src/` (conflict aborts). +4. Checkout `qualcomm-linux/docker-pkg-build@main`. +5. Derive `BASE_SUITE` (`resolute-qcom` → `resolute`). +6. Build the image: `docker_deb_build.py --rebuild -d `. +7. In-container: `apt-get build-dep linux`; `fakeroot make -f debian/rules clean`; `fakeroot debian/rules binary- do_skip_checks=true` (see [Build container notes](#build-container-notes)). -7. Collect `.deb` files and upload to S3. +8. Collect `.deb` files and upload to S3 (skipped when `skip_s3=true`). **Self-hosted runner requirements** (`lecore-production`): Ubuntu 24.04 arm64, Docker (runner user in the `docker` group), ≥ 25 GB free disk. +### `premerge-pr.yml` - PR build check + +Runs a build-only kernel build as a status check on every PR against `resolute-qcom-devel`. + +- **Trigger**: `pull_request_target` on `resolute-qcom-devel` (opened, synchronize, reopened). +- **Why `pull_request_target` and not `pull_request`**: `resolute-qcom-devel` is a kernel source tree with no `.github/` directory. `pull_request` resolves the workflow from the base branch of the PR -- since `resolute-qcom-devel` has no workflows, it would never fire. `pull_request_target` resolves workflows from the default branch (`main`) where they live, so the trigger works correctly. +- **Runner**: `lecore-production` (hardcoded, same as all builds). +- **Concurrency**: one build per PR number; a new push cancels the in-progress build for that PR. +- **What it does**: calls `build-kernel.yml` via `workflow_call` with `suite=resolute-qcom-devel`, `ref=` (pins the exact commit under test), and `skip_s3=true` (build-only, no S3 upload). + ## The CI scripts | Script | Purpose | Used by | From d907a8823f8513cb3b52992539ed7f5b5ccbaf8c Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 7 Jul 2026 09:38:17 -0700 Subject: [PATCH 097/115] ci: fix premerge PR check design and harden devel_prs Address review of the pre-merge build feature: premerge-pr.yml (moved to .github/devel-workflows/ as a source, installed on resolute-qcom-devel): - Was pull_request_target reading the workflow from main and building the raw PR head SHA on the S3-credentialed self-hosted runner - a pwn-request, and it would never have fired: pull_request(_target) resolves the workflow from the PR base branch (resolute-qcom-devel), not the default branch. - Now pull_request (read-only token, no secrets) building the PR merge ref via build-kernel.yml@main, gated to same-repo (non-fork) PRs. Installed on the integration branch so the trigger actually fires. Static job name yields the stable 'Build check / Build' required-check name. build-kernel.yml: - devel_prs now validates each PR is open and targets the suite branch, and deepens the shallow checkout until a merge base exists before merging (the old code failed on any PR not based on the exact branch tip and misreported it as a conflict). - Fix shell-injection of inputs.devel_prs in the summary step (route via env). - Align the empty-suite checkout fallback with the resolute-qcom-devel default. docs/PIPELINE.md: correct the pull_request_target rationale, document the install-on-devel step and the residual credentialed-runner security note. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 54 +++++++++++++++--- .github/workflows/premerge-pr.yml | 41 -------------- docs/PIPELINE.md | 90 ++++++++++++++++++++++++++---- 3 files changed, 124 insertions(+), 61 deletions(-) delete mode 100644 .github/workflows/premerge-pr.yml diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index eef965b261bed..617ff69bc52c9 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -52,7 +52,7 @@ on: default: "" type: string ref: - description: "Exact git ref to checkout. Overrides suite/kernel_version when set (used by premerge-pr.yml to pin the PR head SHA)." + description: "Exact git ref to checkout. Overrides suite/kernel_version when set (used by premerge-pr.yml to build the PR merge ref)." required: false default: "" type: string @@ -171,7 +171,7 @@ jobs: - name: Checkout kernel source uses: actions/checkout@v6 with: - ref: ${{ inputs.ref != '' && inputs.ref || (inputs.kernel_version != '' && format('{0}-{1}', env.UPSTREAM_PREFIX, inputs.kernel_version) || inputs.suite || 'resolute-qcom') }} + ref: ${{ inputs.ref != '' && inputs.ref || (inputs.kernel_version != '' && format('{0}-{1}', env.UPSTREAM_PREFIX, inputs.kernel_version) || inputs.suite || 'resolute-qcom-devel') }} path: kernel-src # ----------------------------------------------------------------------- @@ -197,17 +197,53 @@ jobs: echo "ERROR: invalid PR number '${pr}' -- must be numeric" >&2 exit 1 } - echo "::group::Merging resolute-qcom-devel PR #${pr}" - git fetch --no-tags origin "refs/pull/${pr}/head:pr-${pr}" - if ! git merge --no-ff --no-commit "pr-${pr}"; then + + # Validate: the PR must exist, be open, and target the suite branch. + # A closed or wrong-base PR number would otherwise be merged silently. + info="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${pr}" \ + --jq '[.state, .base.ref] | @tsv')" || { + echo "ERROR: PR #${pr} not found in ${GITHUB_REPOSITORY}." >&2 + exit 1 + } + state="$(printf '%s' "${info}" | cut -f1)" + base="$(printf '%s' "${info}" | cut -f2)" + [ "${state}" = "open" ] || { + echo "ERROR: PR #${pr} is '${state}' -- only open PRs can be merged." >&2 + exit 1 + } + [ "${base}" = "${SUITE}" ] || { + echo "ERROR: PR #${pr} targets '${base}', not '${SUITE}'." >&2 + exit 1 + } + + echo "::group::Merging ${SUITE} PR #${pr}" + # The build checkout is shallow (depth 1), so the merge base with the + # PR is not present. Fetch the PR head and deepen BOTH histories until + # a merge base exists - PRs fork from a recent tip, so this stays a + # small fetch and never pulls the full ~1.43M-commit history. + git fetch --no-tags --depth=1 origin "refs/pull/${pr}/head:refs/pr/${pr}" + step=100; total=0; max=6400 + until git merge-base HEAD "refs/pr/${pr}" >/dev/null 2>&1; do + if [ "${total}" -ge "${max}" ]; then + echo "ERROR: no common ancestor with PR #${pr} within ${max} commits." >&2 + echo "Rebase the PR onto ${SUITE} and retry." >&2 + exit 1 + fi + echo " deepening history by ${step} commits to find the merge base..." + git fetch --deepen="${step}" origin "refs/heads/${SUITE}" "refs/pull/${pr}/head" + total=$((total + step)); step=$((step * 2)) + done + + if ! git merge --no-ff --no-commit "refs/pr/${pr}"; then echo "ERROR: merge conflict while merging PR #${pr}. Aborting." >&2 git merge --abort || true exit 1 fi if git diff --cached --quiet; then echo "PR #${pr} already present in tree -- nothing to merge." + git merge --abort >/dev/null 2>&1 || true else - git commit -m "Merged resolute-qcom-devel PR #${pr}" + git commit -m "Merged ${SUITE} PR #${pr}" echo "PR #${pr} merged successfully." fi echo "::endgroup::" @@ -350,6 +386,8 @@ jobs: # ----------------------------------------------------------------------- - name: Print summary if: always() + env: + DEVEL_PRS: ${{ inputs.devel_prs }} run: | { echo "## Kernel Build Summary" @@ -361,8 +399,8 @@ jobs: echo "| Flavour | \`${FLAVOR}\` |" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" echo "| Runner | \`${{ runner.name }}\` |" - if [ -n "${{ inputs.devel_prs }}" ]; then - echo "| Devel PRs | \`${{ inputs.devel_prs }}\` |" + if [ -n "${DEVEL_PRS}" ]; then + echo "| Devel PRs | \`${DEVEL_PRS}\` |" fi echo "" echo "### Built packages" diff --git a/.github/workflows/premerge-pr.yml b/.github/workflows/premerge-pr.yml deleted file mode 100644 index 58022717ec011..0000000000000 --- a/.github/workflows/premerge-pr.yml +++ /dev/null @@ -1,41 +0,0 @@ -# SPDX-License-Identifier: BSD-3-Clause -# -# premerge-pr.yml -# -# Triggered on every PR opened or updated against resolute-qcom-devel. -# Runs a full kernel build and reports pass/fail as a status check on the PR. -# -# Uses pull_request_target so the workflow is resolved from the default -# branch (main) rather than from the base branch of the PR -# (resolute-qcom-devel, which is a kernel source tree with no .github/). -# -# The build checks out the exact PR head SHA so concurrent PRs are -# fully isolated. A new push to the same PR cancels the in-flight run. - -name: "Pre-merge PR build" - -on: - pull_request_target: - branches: - - resolute-qcom-devel - types: - - opened - - synchronize - - reopened - -concurrency: - group: premerge-pr-${{ github.event.pull_request.number }} - cancel-in-progress: true - -permissions: - contents: read - pull-requests: read - -jobs: - build: - name: "Build check" - uses: ./.github/workflows/build-kernel.yml - with: - suite: resolute-qcom-devel - ref: ${{ github.event.pull_request.head.sha }} - skip_s3: true diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 7699ba5121280..f422f1199b4e8 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -15,8 +15,7 @@ pkg-linux-qcom-canonical │ ├── .github/workflows/ │ │ ├── fetch-source-pkg.yml ← manual incremental mirror sync │ │ ├── bootstrap-history.yml ← one-time full-history seed -│ │ ├── build-kernel.yml ← build .deb packages -│ │ └── premerge-pr.yml ← PR build check for resolute-qcom-devel +│ │ └── build-kernel.yml ← build .deb packages (+ reusable workflow_call) │ ├── scripts/ │ │ ├── sync-mirror.sh ← incremental mirror-repoint (CI) │ │ └── seed-history.sh ← one-time full-history bootstrap (CI) @@ -26,6 +25,7 @@ pkg-linux-qcom-canonical │ └── ~1.4M commits; immutable tag per upload: Ubuntu-qcom-X.Y.Z-A.B (verbatim) │ ├── resolute-qcom-devel branch ← developer integration branch (see INTEGRATION.md) +│ └── .github/workflows/premerge-pr.yml ← pre-merge PR build check (lives here, not on main) │ └── resolute-qcom-seed branch ← transient bootstrap staging (only during a seed) ``` @@ -236,7 +236,7 @@ packages inside the base-suite-matched | `suite` | `resolute-qcom-devel` | dispatch + call | Branch to build from; base suite (`resolute`) derived for container selection. | | `kernel_version` | *(empty)* | dispatch + call | Builds the exact `Ubuntu-qcom-` tag (validated first). Empty = branch HEAD. | | `devel_prs` | *(empty)* | dispatch + call | Space-separated PR numbers against `resolute-qcom-devel` to merge before building (e.g. `42 43`). Conflict aborts with a clear error. | -| `ref` | *(empty)* | call only | Exact git ref to checkout, overrides `suite`/`kernel_version` when set. Used by `premerge-pr.yml` to pin the PR head SHA. | +| `ref` | *(empty)* | call only | Exact git ref to checkout, overrides `suite`/`kernel_version` when set. Used by `premerge-pr.yml` to build the PR merge ref. | | `skip_s3` | `false` | call only | Skip the S3 upload step. Used by `premerge-pr.yml` so premerge builds are build-only and never upload. | **Build steps**: @@ -254,15 +254,81 @@ packages inside the base-suite-matched **Self-hosted runner requirements** (`lecore-production`): Ubuntu 24.04 arm64, Docker (runner user in the `docker` group), ≥ 25 GB free disk. -### `premerge-pr.yml` - PR build check - -Runs a build-only kernel build as a status check on every PR against `resolute-qcom-devel`. - -- **Trigger**: `pull_request_target` on `resolute-qcom-devel` (opened, synchronize, reopened). -- **Why `pull_request_target` and not `pull_request`**: `resolute-qcom-devel` is a kernel source tree with no `.github/` directory. `pull_request` resolves the workflow from the base branch of the PR -- since `resolute-qcom-devel` has no workflows, it would never fire. `pull_request_target` resolves workflows from the default branch (`main`) where they live, so the trigger works correctly. -- **Runner**: `lecore-production` (hardcoded, same as all builds). -- **Concurrency**: one build per PR number; a new push cancels the in-progress build for that PR. -- **What it does**: calls `build-kernel.yml` via `workflow_call` with `suite=resolute-qcom-devel`, `ref=` (pins the exact commit under test), and `skip_s3=true` (build-only, no S3 upload). +### Pre-merge validation + +The `premerge-pr.yml` check gives every PR into `resolute-qcom-devel` a build-only +kernel build as a status check. It builds the PR's **merge ref** (the PR as it would look merged into the +current branch HEAD) and uploads nothing (`skip_s3=true`). + +- **Trigger**: `pull_request` on `resolute-qcom-devel` (opened, synchronize, + reopened), run via `build-kernel.yml`'s `workflow_call` interface. +- **Where it lives**: GitHub resolves `pull_request` workflows from the PR's + **base branch** - here `resolute-qcom-devel`, which is a kernel tree carrying + none of `main`'s workflows. (Base equals the default branch only in the common + case; here they differ, so a workflow on `main` alone would never fire.) So the + workflow lives **on the integration branch itself**, at + `resolute-qcom-devel:.github/workflows/premerge-pr.yml` - a thin caller that + invokes `build-kernel.yml@main`, so no build logic lives on the kernel branch. + It is added and maintained by its own PR into `resolute-qcom-devel` (it is + **not** carried on `main`); the reference copy is below. +- **Security**: it uses `pull_request`, **not** `pull_request_target`, so the run + has a read-only token and no secrets, and it builds the merge ref rather than a + raw attacker-controlled head. Contributor PRs come from **forks**, so the build + runs untrusted PR code (`debian/rules`) on the S3-credentialed + `lecore-production` runner. The repo **must** enable **"Require approval for all + fork pull requests"** (Settings > Actions > Fork pull request workflows) so a + maintainer reviews and approves each run before it executes - without it, this + is a pwn-request. Stronger still, and recommended before going public: run + pre-merge builds on a non-credentialed runner, or scope S3 to short-lived OIDC + so validation builds carry no write capability at all. +- **Concurrency**: one build per PR; a new push cancels the in-progress build. + +Before the check is added: `main` must already carry the `workflow_call` interface +(added here), and the repo must have **"Require approval for all fork pull +requests"** enabled (see Security above) so fork PRs cannot run code on the runner +unreviewed. To make the check blocking, add **`Build check / Build`** as a required +status check on the `resolute-qcom-devel` ruleset. + +
+Reference: resolute-qcom-devel:.github/workflows/premerge-pr.yml + +Copy this onto `resolute-qcom-devel` (via a PR into that branch) to add or restore +the check. + +```yaml +# SPDX-License-Identifier: BSD-3-Clause +# +# premerge-pr.yml - pre-merge validation build for resolute-qcom-devel. +# Lives on resolute-qcom-devel (GitHub resolves pull_request workflows from the PR +# base branch). Thin caller of the reusable build-kernel.yml on main. Contributor +# PRs come from forks, so the repo must enable "Require approval for all fork pull +# requests". Rationale + security note: docs/PIPELINE.md#pre-merge-validation. + +name: "Pre-merge PR build" + +on: + pull_request: + branches: + - resolute-qcom-devel + types: [opened, synchronize, reopened] + +permissions: + contents: read + +concurrency: + group: premerge-pr-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + build: + name: "Build check" + uses: qualcomm-linux/pkg-linux-qcom-canonical/.github/workflows/build-kernel.yml@main + with: + suite: resolute-qcom-devel + ref: refs/pull/${{ github.event.pull_request.number }}/merge + skip_s3: true +``` +
## The CI scripts From 8ccb117c2ff019e640d7991ccf2890e7ffa23848 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 10 Jul 2026 17:16:51 -0700 Subject: [PATCH 098/115] ci: add daily 15:00 PST kernel build Add a schedule trigger to build-kernel.yml so resolute-qcom-devel HEAD is built and uploaded to S3 daily at 15:00 PST (23:00 UTC). A scheduled run passes no inputs, so the existing defaults apply: SUITE and the checkout ref fall back to resolute-qcom-devel, the runner is the hardcoded lecore-production host, and the S3 upload runs because skip_s3 is unset. No other changes are needed. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 617ff69bc52c9..4209df50a3b4f 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -17,6 +17,13 @@ name: "Build: Canonical Kernel .deb Packages" on: + # Daily build at 15:00 PST (23:00 UTC; GitHub cron runs in UTC). A scheduled run + # passes no inputs, so it builds resolute-qcom-devel HEAD (the SUITE default) on + # the lecore-production runner and uploads the .deb packages to S3 (skip_s3 is + # unset, so the upload step runs). + schedule: + - cron: "0 23 * * *" + workflow_dispatch: inputs: suite: From d949d2d2442c5945e72c27b7571a8060b6163516 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 13 Jul 2026 17:34:42 -0700 Subject: [PATCH 099/115] ci(sync-mirror): detect mirrored tags from the remote, not local clone The --single-branch mirror clone only holds tags reachable from the branch tip, so a rebased or divergent upload that was already mirrored is invisible locally. The missing-upload check used that local view, re-listed such uploads, and the atomic push then failed with "tag already exists", halting the sync. Judge "already mirrored" against the mirror's actual tags via git ls-remote --tags origin, matching the check-version gate in fetch-source-pkg.yml. Also guarantee the branch ends at the newest upload after the loop: a backfilled older upload or a prior partial run could otherwise leave the mirror HEAD behind the latest tag. Signed-off-by: Bjordis Collaku --- scripts/sync-mirror.sh | 41 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/scripts/sync-mirror.sh b/scripts/sync-mirror.sh index d7ebfae932be2..d9d7a2c352f39 100755 --- a/scripts/sync-mirror.sh +++ b/scripts/sync-mirror.sh @@ -152,11 +152,24 @@ mapfile -t UPSTREAM_VERSIONS < <( [ "${#UPSTREAM_VERSIONS[@]}" -gt 0 ] \ || die "No ${UPSTREAM_PREFIX}-* tags found upstream." +# "Missing" is judged against the tags ACTUALLY ON THE MIRROR (ls-remote origin), +# NOT the local clone: the clone is --single-branch, so it only holds tags +# reachable from the branch tip. A rebased/divergent upload we already mirrored is +# invisible locally, and re-listing it here would make the atomic push below fail +# with "tag already exists". This matches how the check-version gate decides what +# is new (fetch-source-pkg.yml). +mirror_raw="$(git ls-remote --tags origin "refs/tags/${UPSTREAM_PREFIX}-*")" \ + || die "git ls-remote failed for the mirror (origin)." +mirrored_versions="$( + printf '%s\n' "${mirror_raw}" \ + | grep -v '\^{}' \ + | sed -E "s#.*refs/tags/${UPSTREAM_PREFIX}-##" || true +)" + MISSING=() for ver in "${UPSTREAM_VERSIONS[@]}"; do - if ! git rev-parse -q --verify "refs/tags/${UPSTREAM_PREFIX}-${ver}" >/dev/null; then - MISSING+=("${ver}") - fi + # -F: the version is a fixed string (dots are literal, not globs). + grep -qxF "${ver}" <<<"${mirrored_versions}" || MISSING+=("${ver}") done if [ "${#MISSING[@]}" -eq 0 ]; then @@ -233,6 +246,28 @@ Halting so the next run does not mask a partial sync (G3)." LAST_VERSION="${ver}" done +# --------------------------------------------------------------------------- +# 5. Guarantee the branch ends at the NEWEST upload. The loop advances the branch +# as it mirrors, but a late-arriving OLDER upload (backfill) or a prior partial +# run can leave the branch behind the newest tag. Force it forward so the mirror +# HEAD is always the latest upload (a no-op when the loop already ended there). +# --------------------------------------------------------------------------- +newest="${UPSTREAM_VERSIONS[-1]}" +newest_tag="${UPSTREAM_PREFIX}-${newest}" +git rev-parse -q --verify "${newest_tag}^{commit}" >/dev/null 2>&1 \ + || git fetch --no-tags origin "refs/tags/${newest_tag}:refs/tags/${newest_tag}" 2>/dev/null \ + || git fetch --no-tags upstream "refs/tags/${newest_tag}:refs/tags/${newest_tag}" \ + || die "Could not obtain newest tag ${newest_tag} to position the branch." +newest_sha="$(git rev-parse "${newest_tag}^{commit}")" +current_sha="$(git rev-parse -q --verify "refs/heads/${BRANCH}" || true)" +if [ "${newest_sha}" != "${current_sha}" ]; then + log "Advancing branch ${BRANCH} to newest upload ${newest} (${current_sha:0:12} -> ${newest_sha:0:12})..." + git update-ref "refs/heads/${BRANCH}" "${newest_sha}" + git push --force-with-lease="refs/heads/${BRANCH}:${current_sha}" origin "refs/heads/${BRANCH}" \ + || die "Failed to advance branch to newest upload ${newest} (stale lease or protected ref). Halting (G3)." +fi +LAST_VERSION="${newest}" + hr log "Sync complete: ${SYNCED} upload(s) mirrored; branch now at ${LAST_VERSION}." From 1e158926d7362337f55a56d0d7f7a6ee86d7e280 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 10 Jul 2026 17:37:14 -0700 Subject: [PATCH 100/115] docs: clarify and streamline PIPELINE.md Tighten and clarify the pipeline doc: trim deep operational detail to what a reader needs and simplify the wording throughout. Docs only; no workflow or script changes. Signed-off-by: Bjordis Collaku --- docs/PIPELINE.md | 372 ++++------------------------------------------- 1 file changed, 31 insertions(+), 341 deletions(-) diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index f422f1199b4e8..064ceba4484ad 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -1,10 +1,7 @@ # Pipeline operations (maintainers) -How the mirror and build pipeline work, and the reasoning behind the parts that -are easy to break. - -The pipeline has two halves: a **one-time bootstrap** that seeds the full history, -and the **incremental sync + build** that runs per upload thereafter. +How the mirror and build pipeline work. To contribute patches, see +[INTEGRATION.md](INTEGRATION.md). ## Repository branch layout @@ -14,15 +11,13 @@ pkg-linux-qcom-canonical ├── main branch ← CI orchestrator: workflows, scripts, docs │ ├── .github/workflows/ │ │ ├── fetch-source-pkg.yml ← manual incremental mirror sync -│ │ ├── bootstrap-history.yml ← one-time full-history seed +│ │ ├── bootstrap-history.yml ← one-time history seed │ │ └── build-kernel.yml ← build .deb packages (+ reusable workflow_call) -│ ├── scripts/ -│ │ ├── sync-mirror.sh ← incremental mirror-repoint (CI) -│ │ └── seed-history.sh ← one-time full-history bootstrap (CI) +│ ├── scripts/ ← sync-mirror.sh, seed-history.sh (self-documenting) │ └── README.md │ -├── resolute-qcom branch ← full-history mirror of the carmel-team tree (SYNC-ONLY) -│ └── ~1.4M commits; immutable tag per upload: Ubuntu-qcom-X.Y.Z-A.B (verbatim) +├── resolute-qcom branch ← upstream Ubuntu kernel mirror (SYNC-ONLY) +│ └── immutable tag per upload: Ubuntu-qcom-X.Y.Z-A.B │ ├── resolute-qcom-devel branch ← developer integration branch (see INTEGRATION.md) │ └── .github/workflows/premerge-pr.yml ← pre-merge PR build check (lives here, not on main) @@ -30,347 +25,42 @@ pkg-linux-qcom-canonical └── resolute-qcom-seed branch ← transient bootstrap staging (only during a seed) ``` -`resolute-qcom` shares **no history with `main`** (the CI orchestrator); it -carries the **complete upstream commit history** of the kernel it mirrors. - -## Architecture and flow - -### One-time bootstrap (run once, by hand) - -```mermaid -flowchart LR - LP[("Launchpad
~carmel-team / resolute
Ubuntu-qcom-* tags")] - A1["bootstrap-history.yml
manual, once"] - A2["seed-history.sh
full clone ~1.43M commits
relaxed slow-transfer abort"] - A3["sliced push < 2 GB
verbatim Ubuntu-qcom-* tags"] - A4["resolute-qcom-seed
(staging branch)"] - A5{{"human review
and promote"}} - A6[("resolute-qcom
the live mirror")] - A1 --> A2 - LP -->|"clone full history (slow)"| A2 - A2 --> A3 --> A4 --> A5 -->|"promote"| A6 -``` - -Launchpad cannot serve shallow `--deepen`, so the seed is a single-shot **full** -clone; GitHub caps a push at 2 GB, so it is pushed in `< 2 GB` slices. The seed -lands on `resolute-qcom-seed` (never the live branch) for a human to promote. - -### Incremental sync + build (per new upstream upload, manual dispatch) - -```mermaid -flowchart TB - LP[("Launchpad
newest Ubuntu-qcom-* tag")] - G{"check-version gate
upstream tags vs ours
anything new?"} - NO(["no-op, exit green"]) - S1["sync-mirror.sh
fetch delta tag (no --depth)"] - S2["repoint resolute-qcom
create verbatim tag"] - S3["atomic lease-pinned push
branch + tag"] - RQ[("resolute-qcom
advanced to new upload")] - T["trigger-build"] - B1["build-kernel.yml
checkout Ubuntu-qcom-<ver> tag"] - B2["build .deb
pkg-builder:resolute"] - S3DB[("S3 only
qli-prd-lecore-gh-artifacts")] - LP --> G - G -->|"no"| NO - G -->|"yes"| S1 --> S2 --> S3 --> RQ --> T --> B1 --> B2 --> S3DB -``` - -The gate is a cheap two-call `ls-remote` (no clone). Because the branch already -holds full history, the per-upload fetch transfers only the delta, the push is -small, and the branch fast-forwards onto the new verbatim tag. - -Sync jobs run on `ubuntu-24.04-arm` (GitHub-hosted). The build runs on the -`lecore-production` self-hosted runner (`lecore-prd-u2404-arm64-xlrg-od-ephem`). - -## The mirror-repoint model and its guardrails - -`resolute-qcom` is a **faithful mirror of the upstream Canonical kernel tree with -full commit history preserved**. Every upload is fetched with its real ancestry -and frozen under an immutable per-upload tag. - -| Ref | Role | -|-----|------| -| **branch** `resolute-qcom` | A movable "latest Canonical" pointer. Disposable by design - force-advanced to each new upload. | -| **tag** `Ubuntu-qcom-X.Y.Z-A.B` | The upstream Canonical tag, mirrored **verbatim**. An **immutable** per-upload record; the sole anchor that preserves that upload's history (`git diff` between any two tags is a clean kernel delta). | - -A sync is **pure fetch + repoint** ([`scripts/sync-mirror.sh`](../scripts/sync-mirror.sh)): -it only fast-forwards/repoints the mirror (no merge, no rebase), so it can never -conflict. - -**Guardrails - do not "optimize" these away.** They are what keeps the ~1.43M-commit -history intact; several look redundant until the day they save the branch: - -- **Never `--depth` / never shallow.** The incremental fetch and the seed must - produce a complete, non-shallow closure. A shallow base breaks fetch negotiation, - and GitHub rejects a shallow push outright. -- **Pinned lease, atomic push.** The branch + verbatim tag are pushed together with - `--atomic` and `--force-with-lease=:` (never a bare - `--force-with-lease`, never a blind `--force`). This prevents a tag-lands / - branch-rejected split-brain and a concurrent run clobbering the branch. -- **Tag before moving the branch, ascending.** Every missing upload is tagged in - version order before the branch advances past it, so a rebased middle upload is - never silently dropped. -- **Never move a preservation tag.** A re-pointed upstream tag returns a non-zero - "would clobber existing tag"; the sync surfaces it and refuses, rather than - overwriting a frozen record. -- **Bare clone, fail-fast.** The mirror is operated as a bare repo (clean ref - updates); any rejected push aborts the whole run so the next idempotent run can - never mask a partial sync. -- **A freshly created branch is refused.** `sync-mirror.sh` aborts if the branch - has fewer than `MIN_HISTORY_COMMITS` (1000) - i.e. it has not been seeded with - real history yet. Run the bootstrap first. +`resolute-qcom` shares no history with `main` (the CI orchestrator); it holds the +upstream kernel tree the packages are built from. -### Launchpad and GitHub limits (hard-won) +## How it works -- Launchpad's shallow `git fetch --deepen` path is **broken** (it stalls and throws - `error processing shallow info`). It *can* serve a full clone, but spends several - minutes computing the pack server-side, so the seed relaxes the git slow-transfer - abort to ride out that quiet phase. -- GitHub caps a single push at **2 GB**, so the seed is pushed in `< 2 GB` slices. +`resolute-qcom` mirrors the upstream Ubuntu kernel: each upload is fetched and +frozen under an immutable `Ubuntu-qcom-X.Y.Z-A.B` tag. A one-time bootstrap seeds +the branch; thereafter an incremental sync advances it per new upstream upload and +triggers a build. Builds produce `.deb` packages uploaded to a private S3 bucket +(no GitHub artifacts or releases). Downstream patches land on `resolute-qcom-devel`, +never on the mirror. -## Upstream source and version discovery +Upstream source: the Qualcomm Ubuntu kernel on +[Launchpad](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute). -The mirror tracks the carmel-team Qualcomm-Ubuntu kernel on Launchpad: -[https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute). +## Running it -Version discovery is **purely git-tag based - there is no Launchpad REST API call**. -The upstream `Ubuntu-qcom-*` tag is the authoritative version source; the -`check-version` gate compares it against our mirrored tags. To check the newest -upstream upload directly, independent of the mirror: +All three workflows are manual (`Actions → … → Run workflow`, or via `gh`): ```bash -git ls-remote --tags \ - https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute \ - 'refs/tags/Ubuntu-qcom-*' | sort -V | tail -1 -# -> Ubuntu-qcom-7.0.0-1006.8 (example output) -``` - -## Bootstrapping +# Sync the mirror to the latest upstream upload (no inputs; idempotent). +gh workflow run fetch-source-pkg.yml --repo qualcomm-linux/pkg-linux-qcom-canonical -`resolute-qcom` must be **seeded with full history once** before the sync can run -incrementally. This is automated: +# Build .deb packages. Defaults to resolute-qcom-devel HEAD; set the suite input to +# resolute-qcom for the mirror, or kernel_version for an exact tag. +gh workflow run build-kernel.yml --repo qualcomm-linux/pkg-linux-qcom-canonical -```bash -gh workflow run bootstrap-history.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical - # defaults: suite=resolute-qcom (seeds into the resolute-qcom-seed branch) +# One-time only, before the first sync: seed resolute-qcom with history (into +# resolute-qcom-seed, which a human then promotes to the live branch). +gh workflow run bootstrap-history.yml --repo qualcomm-linux/pkg-linux-qcom-canonical ``` -How it works ([`scripts/seed-history.sh`](../scripts/seed-history.sh)): - -- **Single-shot full clone** of the full history (see - [Launchpad and GitHub limits](#launchpad-and-github-limits-hard-won) for why - shallow `--deepen` cannot be used). -- **Sliced push** in `< 2 GB` slices (the GitHub push cap, same section); the - upstream `Ubuntu-qcom-*` tags are pushed **verbatim**. -- **Pushes to `resolute-qcom-seed`, never the live branch.** A human reviews and - promotes it (the production `resolute-qcom` branch is too valuable to clobber - automatically). The cloned repo is cached, so a retry after a failed publish - skips re-downloading the multi-GB history. - -Once seeded and promoted, -[`fetch-source-pkg.yml`](../.github/workflows/fetch-source-pkg.yml) keeps -`resolute-qcom` current; it **refuses to run against an un-seeded branch**. - -## Running a sync - -The sync takes **no inputs** - it always mirrors the latest carmel-team upload -into `resolute-qcom`: - -```bash -gh workflow run fetch-source-pkg.yml \ - --repo qualcomm-linux/pkg-linux-qcom-canonical -# or: Actions → "Sync: Canonical Kernel Sources to Branch" → Run workflow -``` - -If the newest `Ubuntu-qcom-*` tag is already mirrored, the run exits cleanly -(idempotent). Otherwise it mirrors every un-mirrored upload in version order, then -triggers a build of the newest one. - -## Manual build triggers - -Trigger manually: **Actions → Build: Canonical Kernel .deb Packages → Run -workflow**. It builds the `resolute-qcom-devel` integration branch by default; set -the **Branch to build from** field to `resolute-qcom` to build from the mirror branch instead. The `kernel_version` input -controls what is checked out: - -| `kernel_version` | Source checked out | Use | -|------------------|--------------------|-----| -| *empty* | the selected branch HEAD (default `resolute-qcom-devel`) | build the current branch tip | -| `X.Y.Z-A.B` | the tag `Ubuntu-qcom-X.Y.Z-A.B` (validated first; fails fast if absent) | build an exact mirrored upload | - -The only output is the `.deb` upload to **S3** (`lecore-production` runner). No -GitHub Actions artifacts and no GitHub Releases are produced. - -## Workflows reference - -### `fetch-source-pkg.yml` - Sync - -Mirrors new upstream uploads into `resolute-qcom`, history-preserving, via -[`scripts/sync-mirror.sh`](../scripts/sync-mirror.sh). - -- **Trigger**: manual `workflow_dispatch` only (no cron). **No inputs** - the - upstream URL, branch, and tag prefix are fixed constants in the workflow. -- **Runner**: `ubuntu-24.04-arm` (all jobs). - -| Job | What it does | -|-----|-------------| -| `check-version` | Two `ls-remote` calls (no clone): newest upstream `Ubuntu-qcom-*` tag vs our tags. Sets `should_sync`. | -| `sync` | Bare-clones our mirror; for each un-mirrored upload (ascending): fetches the upstream tag (delta only), advances `resolute-qcom`, and atomically pushes the branch + the verbatim tag (lease-pinned, fail-fast). Never merges/rebases. | -| `trigger-build` | Dispatches `build-kernel.yml` for the newest synced upload of the mirror (`suite=resolute-qcom`, `kernel_version=`). | - -Idempotent: if the newest upload is already mirrored, the run exits without cloning -anything. - -### `build-kernel.yml` - Build - -Checks out the kernel source (a verbatim tag, or the branch HEAD) and builds `.deb` -packages inside the base-suite-matched -`ghcr.io/qualcomm-linux/pkg-builder:` container. - -- **Trigger**: dispatched by the sync, or manually. -- **Output**: **S3 only** (`lecore-production` runner). No artifacts, no releases. - -| Input | Default | Trigger | Description | -|-------|---------|---------|-------------| -| `suite` | `resolute-qcom-devel` | dispatch + call | Branch to build from; base suite (`resolute`) derived for container selection. | -| `kernel_version` | *(empty)* | dispatch + call | Builds the exact `Ubuntu-qcom-` tag (validated first). Empty = branch HEAD. | -| `devel_prs` | *(empty)* | dispatch + call | Space-separated PR numbers against `resolute-qcom-devel` to merge before building (e.g. `42 43`). Conflict aborts with a clear error. | -| `ref` | *(empty)* | call only | Exact git ref to checkout, overrides `suite`/`kernel_version` when set. Used by `premerge-pr.yml` to build the PR merge ref. | -| `skip_s3` | `false` | call only | Skip the S3 upload step. Used by `premerge-pr.yml` so premerge builds are build-only and never upload. | - -**Build steps**: -1. Free disk space. -2. Checkout the tag/branch (or `ref` if set) → `kernel-src/`. -3. If `devel_prs` is set: merge each PR against `resolute-qcom-devel` into `kernel-src/` (conflict aborts). -4. Checkout `qualcomm-linux/docker-pkg-build@main`. -5. Derive `BASE_SUITE` (`resolute-qcom` → `resolute`). -6. Build the image: `docker_deb_build.py --rebuild -d `. -7. In-container: `apt-get build-dep linux`; `fakeroot make -f debian/rules clean`; - `fakeroot debian/rules binary- do_skip_checks=true` - (see [Build container notes](#build-container-notes)). -8. Collect `.deb` files and upload to S3 (skipped when `skip_s3=true`). - -**Self-hosted runner requirements** (`lecore-production`): Ubuntu 24.04 arm64, -Docker (runner user in the `docker` group), ≥ 25 GB free disk. - -### Pre-merge validation - -The `premerge-pr.yml` check gives every PR into `resolute-qcom-devel` a build-only -kernel build as a status check. It builds the PR's **merge ref** (the PR as it would look merged into the -current branch HEAD) and uploads nothing (`skip_s3=true`). - -- **Trigger**: `pull_request` on `resolute-qcom-devel` (opened, synchronize, - reopened), run via `build-kernel.yml`'s `workflow_call` interface. -- **Where it lives**: GitHub resolves `pull_request` workflows from the PR's - **base branch** - here `resolute-qcom-devel`, which is a kernel tree carrying - none of `main`'s workflows. (Base equals the default branch only in the common - case; here they differ, so a workflow on `main` alone would never fire.) So the - workflow lives **on the integration branch itself**, at - `resolute-qcom-devel:.github/workflows/premerge-pr.yml` - a thin caller that - invokes `build-kernel.yml@main`, so no build logic lives on the kernel branch. - It is added and maintained by its own PR into `resolute-qcom-devel` (it is - **not** carried on `main`); the reference copy is below. -- **Security**: it uses `pull_request`, **not** `pull_request_target`, so the run - has a read-only token and no secrets, and it builds the merge ref rather than a - raw attacker-controlled head. Contributor PRs come from **forks**, so the build - runs untrusted PR code (`debian/rules`) on the S3-credentialed - `lecore-production` runner. The repo **must** enable **"Require approval for all - fork pull requests"** (Settings > Actions > Fork pull request workflows) so a - maintainer reviews and approves each run before it executes - without it, this - is a pwn-request. Stronger still, and recommended before going public: run - pre-merge builds on a non-credentialed runner, or scope S3 to short-lived OIDC - so validation builds carry no write capability at all. -- **Concurrency**: one build per PR; a new push cancels the in-progress build. - -Before the check is added: `main` must already carry the `workflow_call` interface -(added here), and the repo must have **"Require approval for all fork pull -requests"** enabled (see Security above) so fork PRs cannot run code on the runner -unreviewed. To make the check blocking, add **`Build check / Build`** as a required -status check on the `resolute-qcom-devel` ruleset. - -
-Reference: resolute-qcom-devel:.github/workflows/premerge-pr.yml - -Copy this onto `resolute-qcom-devel` (via a PR into that branch) to add or restore -the check. - -```yaml -# SPDX-License-Identifier: BSD-3-Clause -# -# premerge-pr.yml - pre-merge validation build for resolute-qcom-devel. -# Lives on resolute-qcom-devel (GitHub resolves pull_request workflows from the PR -# base branch). Thin caller of the reusable build-kernel.yml on main. Contributor -# PRs come from forks, so the repo must enable "Require approval for all fork pull -# requests". Rationale + security note: docs/PIPELINE.md#pre-merge-validation. - -name: "Pre-merge PR build" - -on: - pull_request: - branches: - - resolute-qcom-devel - types: [opened, synchronize, reopened] - -permissions: - contents: read - -concurrency: - group: premerge-pr-${{ github.event.pull_request.number }} - cancel-in-progress: true - -jobs: - build: - name: "Build check" - uses: qualcomm-linux/pkg-linux-qcom-canonical/.github/workflows/build-kernel.yml@main - with: - suite: resolute-qcom-devel - ref: refs/pull/${{ github.event.pull_request.number }}/merge - skip_s3: true -``` -
- -## The CI scripts - -| Script | Purpose | Used by | -|--------|---------|---------| -| `scripts/sync-mirror.sh` | Incremental history-preserving mirror-repoint of new uploads | `fetch-source-pkg.yml` | -| `scripts/seed-history.sh` | One-time full-history bootstrap of `resolute-qcom-seed` | `bootstrap-history.yml` | - -`scripts/check-version.sh`, `scripts/fetch-source-pkg.sh`, and -`scripts/build-kernel-deb.sh` are **legacy** standalone local helpers that predate -the mirror model; they are **not** part of the CI path. - -## Build container notes - -### Build environment setup (`debian/rules clean`) - -Before compilation the build runs `fakeroot make -f debian/rules clean`, the -standard Ubuntu kernel build setup path. The `clean` target: - -- Runs `debian/control` as a dependency, generating **`debian/canonical-certs.pem`** - (the X.509 cert the kernel's `certs/x509_certificate_list` target requires - - without it the build fails immediately) and **`debian/control`**. -- Creates **`debian/changelog`** (the active derivative's changelog, - `debian.qcom/changelog`). `dh_installchangelogs`, run at the end of - `binary-qcom`, fails without it - after 2+ hours of compilation. -- Removes stale build artifacts. - -**Why `fakeroot make -f debian/rules` and not `fakeroot debian/rules`?** -`fakeroot` execs the command via `/bin/sh` (dash), which resolves the -`#!/usr/bin/make -f` shebang at exec time; in the container that resolution fails -silently with `debian/rules: not found (exit 127)`. Passing `make -f debian/rules` -explicitly bypasses the shebang lookup. - -### Skipping the config policy check (`do_skip_checks=true`) - -The Ubuntu kernel build runs a config-policy check (`annotations --check`) that -expects `CONFIG_RUST_IS_AVAILABLE=y`. In the `pkg-builder` container `bindgen` is -unavailable, so `olddefconfig` drops that symbol and the policy diff fails. Passing -`do_skip_checks=true` bypasses the check - the standard approach for non-official -builds where optional toolchains are absent. +PRs into `resolute-qcom-devel` get a build-only pre-merge check (`premerge-pr.yml` +on that branch), which calls `build-kernel.yml` with `skip_s3=true`. ## License -See the [main README](../README.md#license): BSD 3-Clause for the CI scripts and -workflows, GPL-2 for the mirrored kernel source. +BSD 3-Clause for the CI scripts and workflows; GPL-2 for the mirrored kernel +source. See the [main README](../README.md#license). From ed5879eaf06e8be879bb32853663099233a967de Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 13 Jul 2026 14:35:24 -0700 Subject: [PATCH 101/115] scripts: add Qualcomm copyright header Add the Qualcomm Technologies copyright line above the existing SPDX-License-Identifier tag in each scripts/*.sh file, matching the header format used in LICENSE.txt and the .github/workflows/*.yml files. Signed-off-by: Bjordis Collaku --- scripts/build-kernel-deb.sh | 1 + scripts/check-version.sh | 1 + scripts/fetch-source-pkg.sh | 1 + scripts/seed-history.sh | 1 + scripts/sync-mirror.sh | 1 + 5 files changed, 5 insertions(+) diff --git a/scripts/build-kernel-deb.sh b/scripts/build-kernel-deb.sh index 0d292916aaee1..a63dde74ba9d8 100755 --- a/scripts/build-kernel-deb.sh +++ b/scripts/build-kernel-deb.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. # SPDX-License-Identifier: BSD-3-Clause # # build-kernel-deb.sh - Build Ubuntu kernel .deb packages from a Canonical diff --git a/scripts/check-version.sh b/scripts/check-version.sh index b6713d34f6c18..95b0fcebb28b7 100755 --- a/scripts/check-version.sh +++ b/scripts/check-version.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. # SPDX-License-Identifier: BSD-3-Clause # # check-version.sh - Query the latest published kernel version from Launchpad diff --git a/scripts/fetch-source-pkg.sh b/scripts/fetch-source-pkg.sh index 2b8f82ea35ed3..e821ece3452c4 100755 --- a/scripts/fetch-source-pkg.sh +++ b/scripts/fetch-source-pkg.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. # SPDX-License-Identifier: BSD-3-Clause # # fetch-source-pkg.sh - Clone the Canonical Ubuntu kernel source from the diff --git a/scripts/seed-history.sh b/scripts/seed-history.sh index b83c1237a85d6..5806c9c5c9939 100755 --- a/scripts/seed-history.sh +++ b/scripts/seed-history.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. # SPDX-License-Identifier: BSD-3-Clause # # seed-history.sh - One-time automated bootstrap: seed a branch of this repo with diff --git a/scripts/sync-mirror.sh b/scripts/sync-mirror.sh index d9d7a2c352f39..51e948c32086e 100755 --- a/scripts/sync-mirror.sh +++ b/scripts/sync-mirror.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. # SPDX-License-Identifier: BSD-3-Clause # # sync-mirror.sh - Incrementally mirror new Canonical kernel upload tags into a From bcaba7e98662239cb72e2764829e43a95aa70a04 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 13 Jul 2026 14:35:34 -0700 Subject: [PATCH 102/115] docs(security): point issue link at this repository The public-issue reporting link in SECURITY.md pointed at a qualcomm/REPLACE-ME placeholder repo. Point it at qualcomm-linux/pkg-linux-qcom-canonical, matching the repository referenced everywhere else in the docs (README.md, CONTRIBUTING.md). Signed-off-by: Bjordis Collaku --- SECURITY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SECURITY.md b/SECURITY.md index 468eda661763f..0c4a6369b489d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -3,7 +3,7 @@ How to Report a Potential Vulnerability? If you would like to report a public issue (for example, one with a released CVE number), please report it as a -[GitHub issue](https://github.com/qualcomm/REPLACE-ME/issues/new). +[GitHub issue](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/issues/new). If you have a patch ready, submit it following the same procedure as any other patch as described in [CONTRIBUTING.md](CONTRIBUTING.md). From dcb97b00981e9d526d00cfe45b67956025c39782 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 16 Jul 2026 20:36:24 -0700 Subject: [PATCH 103/115] Update project documentation wording Update the repository overview to describe the Canonical kernel mirror, Qualcomm contributions, and the resolute-qcom-devel integration branch. Refresh integration and pipeline documentation to use the same wording, remove README sections that no longer match the repository policy, and scope the BSD-3-Clause license summary to workflows, scripts, and documentation maintained on main in the README. Signed-off-by: Bjordis Collaku --- CONTRIBUTING.md | 89 ++------------------------------------------- README.md | 44 ++++++---------------- SECURITY.md | 31 +--------------- docs/INTEGRATION.md | 13 +++---- docs/PIPELINE.md | 9 +---- 5 files changed, 24 insertions(+), 162 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8082beccd7284..cc503a4a9554e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,90 +1,7 @@ # Contributing to pkg-linux-qcom-canonical -Hi there! -We're thrilled that you'd like to contribute to this project. -Your help is essential for keeping this project great and for making it better. +This project is not accepting contributions. -## Two kinds of contribution +The repository mirrors the Canonical kernel optimized for Qualcomm and includes Qualcomm contributions and workflow automation used by the maintainers. -This repository hosts both a CI orchestrator and a mirror of the Canonical kernel, and -the two have **different** contribution flows: - -- **Kernel patches** (changes to the kernel source) do **not** go through this - document. They belong on the `resolute-qcom-devel` integration branch - never on - the `resolute-qcom` mirror. See **[docs/INTEGRATION.md](docs/INTEGRATION.md)**. -- **CI changes** (the GitHub Actions workflows, the `scripts/` helpers, and - the docs - everything on `main`) follow the pull-request flow below. - -The rest of this guide is about **CI changes on `main`**. - -## Branching Strategy - -For CI changes, develop on branches based off of `main`, and open pull -requests against `main`. - -## Submitting a pull request - -1. Please read our [code of conduct](CODE-OF-CONDUCT.md) and [license](LICENSE.txt). -1. [Fork](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/fork) and clone the repository. - - ```bash - git clone https://github.com//pkg-linux-qcom-canonical.git - ``` - -1. Create a new branch based on `main`: - - ```bash - git checkout -b main - ``` - -1. Create an upstream `remote` to make it easier to keep your branches up-to-date: - - ```bash - git remote add upstream https://github.com/qualcomm-linux/pkg-linux-qcom-canonical.git - ``` - -1. Make your changes, add tests, and make sure the tests still pass. -1. Commit your changes using the [DCO](https://developercertificate.org/). You can attest to the DCO by commiting with the **-s** or **--signoff** options or manually adding the "Signed-off-by": - - ```bash - git commit -s -m "Really useful commit message" - ``` - -1. After committing your changes on the topic branch, sync it with the upstream branch: - - ```bash - git pull --rebase upstream main - ``` - -1. Push to your fork. - - ```bash - git push -u origin - ``` - - The `-u` is shorthand for `--set-upstream`. This will set up the tracking reference so subsequent runs of `git push` or `git pull` can omit the remote and branch. - -1. [Submit a pull request](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/pulls) from your branch to `main`. -1. Pat yourself on the back and wait for your pull request to be reviewed. - -## Security Analysis of Pull Requests - -To maintain the security and integrity of this project, all pull requests from external contributors are automatically scanned using [Semgrep](https://github.com/semgrep/semgrep) to detect insecure coding patterns and potential security flaws. - -**Static Analysis with Semgrep:** We use Semgrep to perform lightweight, fast static analysis on every PR. This helps identify risky code patterns and logic flaws early in the development process. - -**Contributor Responsibility:** If any issues are flagged, contributors are expected to resolve them before the PR can be merged. - -**Continuous Improvement:** Our Semgrep ruleset evolves over time to reflect best practices and emerging security concerns. - -By submitting a PR, you agree to participate in this process and help us keep the project secure for everyone. - - -Here are a few things you can do that will increase the likelihood of your pull request to be accepted: - -- Follow the existing style. Shell scripts should pass [shellcheck](https://www.shellcheck.net/); GitHub Actions workflows should pass [actionlint](https://github.com/rhysd/actionlint). -- Write tests. -- Keep your change as focused as possible. - If you want to make multiple independent changes, please consider submitting them as separate pull requests. -- Write a [good commit message](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). -- It's a good idea to arrange a discussion with other developers to ensure there is consensus on large features, architecture changes, and other core code changes. PR reviews will go much faster when there are no surprises. +External pull requests and issue-based contributions are not accepted in this repository. diff --git a/README.md b/README.md index d00677a43b026..bf8c7871d4ccc 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ # pkg-linux-qcom-canonical -Mirrors the Qualcomm-Ubuntu (Canonical) kernel on `resolute-qcom` and builds it -into `.deb` packages. Qualcomm developers add downstream patches on the -`resolute-qcom-devel` integration branch. +Mirrors the Canonical kernel optimized for Qualcomm on `resolute-qcom`, from +[Launchpad](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute/log/?h=master-next), +with Qualcomm contributions. + +This is not a product or actively supported by Qualcomm. We are not accepting +contributions in this repository. > [!TIP] > Latest upload: see the **[tags page](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/tags)**. @@ -11,50 +14,25 @@ into `.deb` packages. Qualcomm developers add downstream patches on the | | | |---|---| -| **Upstream** | [carmel-team Qualcomm-Ubuntu kernel](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) | +| **Upstream** | [https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute) | | **`resolute-qcom`** | Mirror of that kernel - sync-only, do not commit here | -| **`resolute-qcom-devel`** | Integration branch - Qualcomm's downstream patches (via PR) | +| **`resolute-qcom-devel`** | Integration branch - Qualcomm contributions (via PR) | | **Output** | Kernel `.deb` packages, uploaded to S3 | > [!NOTE] > `main` is the **CI orchestrator** - the workflows, scripts, and docs that drive the sync and build. -## Add a downstream patch - -> [!IMPORTANT] -> Patches reach `resolute-qcom-devel` through a **feature branch + pull request**. Branch off the integration branch, push your branch, then open a PR back into it. - -```bash -git clone https://github.com/qualcomm-linux/pkg-linux-qcom-canonical.git -cd pkg-linux-qcom-canonical -git checkout -b my-feature origin/resolute-qcom-devel # branch off the integration branch -# add your patches, then commit DCO-signed: -git commit -s -git push origin my-feature -``` - -Then open a PR from `my-feature` **into `resolute-qcom-devel`**. See -**[docs/INTEGRATION.md](docs/INTEGRATION.md)** for the full workflow. - -## Golden rules - -> [!WARNING] -> - :no_entry: **Never commit to `resolute-qcom`.** The sync is its only writer and force-advances it on every upload. -> - :no_entry: **Never push directly to `resolute-qcom-devel`.** Patches land through a feature branch + pull request. -> - :white_check_mark: DCO `Signed-off-by` is required on every commit (`git commit -s`). - ## Documentation | Doc | For | |-----|-----| | **[docs/INTEGRATION.md](docs/INTEGRATION.md)** | Qualcomm developers - working on `resolute-qcom-devel` | | **[docs/PIPELINE.md](docs/PIPELINE.md)** | Maintainers - sync, build, and mirror operations | -| **[CONTRIBUTING.md](CONTRIBUTING.md)** | Contributing to the CI orchestrator on `main` | -| **[SECURITY.md](SECURITY.md)** | Reporting security issues | +| **[CONTRIBUTING.md](CONTRIBUTING.md)** | This project is not accepting contributions | +| **[SECURITY.md](SECURITY.md)** | This project is not actively maintained. It is a mirror. | ## License | Scope | License | |-------|---------| -| CI scripts and workflows | BSD 3-Clause - see **[LICENSE.txt](LICENSE.txt)** | -| Mirrored kernel source | GPL-2.0 and the individual licences of its components | +| Workflows, scripts, and documentation on `main` | BSD 3-Clause - see **[LICENSE.txt](LICENSE.txt)** | diff --git a/SECURITY.md b/SECURITY.md index 0c4a6369b489d..32b724fd311d1 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,31 +1,4 @@ -How to Report a Potential Vulnerability? -======================================== +# Security Policy -If you would like to report a public issue (for example, one with a released -CVE number), please report it as a -[GitHub issue](https://github.com/qualcomm-linux/pkg-linux-qcom-canonical/issues/new). -If you have a patch ready, submit it following the same procedure as any -other patch as described in [CONTRIBUTING.md](CONTRIBUTING.md). +This project is not actively maintained. It is a mirror. -If you are dealing with a not-yet released or urgent issue, please contact us -via our [Product Security team](mailto:product-security@qualcomm.com) or -see our -[Report a Bug](https://www.qualcomm.com/company/product-security/report-a-bug) -page. Please include the following details while reporting a vulnerability: -- Description of the vulnerability -- Steps to reproduce -- Affected versions -- Potential impact -- Any relevant logs or screenshots - -## Coordinated Disclosure - -We follow a Coordinated Vulnerability Disclosure (CVD) process: - -- **Initial Response**: We will acknowledge your report within 48 hours. - -- **Investigation**: Our team will investigate the issue and provide updates. - -- **Resolution**: We will work with you to resolve the issue and prepare a fix. - -- **Disclosure**: Once the fix is ready, we will disclose the vulnerability and notify affected users. diff --git a/docs/INTEGRATION.md b/docs/INTEGRATION.md index 8c3fe87b887e7..1f14212b9e9f9 100644 --- a/docs/INTEGRATION.md +++ b/docs/INTEGRATION.md @@ -1,8 +1,8 @@ # Developing on `resolute-qcom-devel` -Qualcomm's downstream kernel patches live on the `resolute-qcom-devel` branch, on +Qualcomm contributions live on the `resolute-qcom-devel` branch, on top of the `resolute-qcom` mirror of the Canonical kernel. This guide is for -developers adding those patches. +developers working on those contributions. For how the mirror and build pipeline work, see [PIPELINE.md](PIPELINE.md). @@ -11,13 +11,13 @@ For how the mirror and build pipeline work, see [PIPELINE.md](PIPELINE.md). | Branch | Role | |--------|------| | `resolute-qcom` | The mirror of the upstream Canonical kernel. Only the automated sync writes to it - **do not commit here**. | -| `resolute-qcom-devel` | The integration branch: Qualcomm's downstream patches on top of `resolute-qcom`. **Push your patches here.** | +| `resolute-qcom-devel` | The integration branch: Qualcomm contributions on top of `resolute-qcom`. **Use pull requests for changes here.** | The `-devel` suffix follows Canonical's own `devel` naming convention. -## Contributing patches +## Working on Qualcomm contributions -Downstream kernel patches land on `resolute-qcom-devel` through a **pull request** - +Qualcomm contributions land on `resolute-qcom-devel` through a **pull request** - work on a feature branch and open a PR into `resolute-qcom-devel`. Never commit to `resolute-qcom`: the sync is its only writer and force-advances it on every new upload. @@ -31,8 +31,7 @@ git push origin my-feature ``` Then open a pull request from your feature branch **into `resolute-qcom-devel`**. -Commits must carry a DCO `Signed-off-by` line (`git commit -s`); see -[CONTRIBUTING.md](../CONTRIBUTING.md) for the sign-off policy. +Commits must carry a DCO `Signed-off-by` line (`git commit -s`) as required for Qualcomm contribution review. ## Building diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 064ceba4484ad..894faeef3da3e 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -34,11 +34,10 @@ upstream kernel tree the packages are built from. frozen under an immutable `Ubuntu-qcom-X.Y.Z-A.B` tag. A one-time bootstrap seeds the branch; thereafter an incremental sync advances it per new upstream upload and triggers a build. Builds produce `.deb` packages uploaded to a private S3 bucket -(no GitHub artifacts or releases). Downstream patches land on `resolute-qcom-devel`, +(no GitHub artifacts or releases). Qualcomm contributions land on `resolute-qcom-devel`, never on the mirror. -Upstream source: the Qualcomm Ubuntu kernel on -[Launchpad](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute). +Upstream source: [https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute](https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+git/resolute). ## Running it @@ -60,7 +59,3 @@ gh workflow run bootstrap-history.yml --repo qualcomm-linux/pkg-linux-qcom-canon PRs into `resolute-qcom-devel` get a build-only pre-merge check (`premerge-pr.yml` on that branch), which calls `build-kernel.yml` with `skip_s3=true`. -## License - -BSD 3-Clause for the CI scripts and workflows; GPL-2 for the mirrored kernel -source. See the [main README](../README.md#license). From 1f42b3f7bf0206b6a3532680c17ae81c03ddc847 Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Tue, 28 Jul 2026 14:51:20 +0800 Subject: [PATCH 104/115] ci(build-kernel): support building the -dbgsym.ddeb via dbgsym input - Add a dbgsym workflow input (default true) mapped to a DBGSYM env var and passed into the build container. - Pass do_dbgsym_package=true/false to debian/rules based on DBGSYM. - Collect and list *.ddeb alongside *.deb/*.changes/*.buildinfo, both inside the container and in the outer "Collect built packages" step. - Add a Dbgsym row to the run summary table. - Document the new dbgsym input in docs/PIPELINE.md. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 27 ++++++++++++++++++++++----- docs/PIPELINE.md | 3 ++- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 4209df50a3b4f..15c9802ccbee5 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -40,6 +40,11 @@ on: required: false default: "" type: string + dbgsym: + description: "Build the -dbgsym.ddeb debug symbol packages alongside the .deb packages." + required: false + default: true + type: boolean workflow_call: inputs: @@ -58,6 +63,11 @@ on: required: false default: "" type: string + dbgsym: + description: "Build the -dbgsym.ddeb debug symbol packages alongside the .deb packages." + required: false + default: true + type: boolean ref: description: "Exact git ref to checkout. Overrides suite/kernel_version when set (used by premerge-pr.yml to build the PR merge ref)." required: false @@ -86,6 +96,7 @@ jobs: SUITE: ${{ inputs.suite || 'resolute-qcom-devel' }} ARCH: arm64 FLAVOR: qcom + DBGSYM: ${{ github.event_name == 'schedule' && 'true' || (inputs.dbgsym && 'true' || 'false') }} steps: # ----------------------------------------------------------------------- @@ -297,6 +308,7 @@ jobs: -e JOBS="${JOBS}" \ -e TARGET="${TARGET}" \ -e WS="${GITHUB_WORKSPACE}" \ + -e DBGSYM="${DBGSYM}" \ "ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}" \ bash -c ' set -euo pipefail @@ -348,15 +360,19 @@ jobs: # CONFIG_RUST_IS_AVAILABLE=y — bindgen-0.65 is not available in # the container so Rust support is unavailable, causing the check # to fail. This is the standard approach for non-official builds. - fakeroot debian/rules ${TARGET} do_skip_checks=true + # do_dbgsym_package=true additionally builds the unstripped + # -dbgsym.ddeb (vmlinux + modules with full debug symbols). + DBGSYM_OPT="do_dbgsym_package=false" + if [ "${DBGSYM}" = "true" ]; then DBGSYM_OPT="do_dbgsym_package=true"; fi + fakeroot debian/rules ${TARGET} do_skip_checks=true "${DBGSYM_OPT}" kill "${HEARTBEAT}" 2>/dev/null || true echo "=== Locating built .deb files ===" - find "${WS}" -name "*.deb" | sort || true + find "${WS}" -name "*.deb" -o -name "*.ddeb" | sort || true mkdir -p "${WS}/output/" - find "${WS}" -name "*.deb" -exec cp {} "${WS}/output/" \; + find "${WS}" \( -name "*.deb" -o -name "*.ddeb" \) -exec cp {} "${WS}/output/" \; ls -lh "${WS}/output/" || echo "No .deb files found" ' @@ -370,7 +386,7 @@ jobs: mkdir -p output/ find "${GITHUB_WORKSPACE}" -maxdepth 1 \ - \( -name "*.deb" -o -name "*.changes" -o -name "*.buildinfo" \) \ + \( -name "*.deb" -o -name "*.ddeb" -o -name "*.changes" -o -name "*.buildinfo" \) \ -exec cp {} output/ \; echo "Built packages:" @@ -404,6 +420,7 @@ jobs: echo "| Branch | \`${SUITE}\` |" echo "| Architecture | \`${ARCH}\` |" echo "| Flavour | \`${FLAVOR}\` |" + echo "| Dbgsym | \`${DBGSYM}\` |" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" echo "| Runner | \`${{ runner.name }}\` |" if [ -n "${DEVEL_PRS}" ]; then @@ -412,6 +429,6 @@ jobs: echo "" echo "### Built packages" echo '```' - ls -lh output/*.deb 2>/dev/null || echo "(none)" + ls -lh output/ 2>/dev/null || echo "(none)" echo '```' } >> "$GITHUB_STEP_SUMMARY" diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 894faeef3da3e..4233db986d06e 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -48,7 +48,8 @@ All three workflows are manual (`Actions → … → Run workflow`, or via `gh`) gh workflow run fetch-source-pkg.yml --repo qualcomm-linux/pkg-linux-qcom-canonical # Build .deb packages. Defaults to resolute-qcom-devel HEAD; set the suite input to -# resolute-qcom for the mirror, or kernel_version for an exact tag. +# resolute-qcom for the mirror, or kernel_version for an exact tag. The dbgsym +# input (default true) also builds the unstripped -dbgsym.ddeb. gh workflow run build-kernel.yml --repo qualcomm-linux/pkg-linux-qcom-canonical # One-time only, before the first sync: seed resolute-qcom with history (into From cdcc350f6c44dd6451ae337463bbf19b5170a1d0 Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Wed, 29 Jul 2026 16:19:11 +0800 Subject: [PATCH 105/115] ci(build-kernel): build all flavours (qcom + qcom-rt) FLAVOR was hardcoded to qcom, so the qcom-rt flavour defined in debian.qcom/rules.d/arm64.mk was never built despite being fully wired up (vars.qcom-rt, PREEMPT_RT policy in config/annotations). Set FLAVOR=all so TARGET resolves to the binary target, which depends on every flavour in debian.qcom's flavours list. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 15c9802ccbee5..a0f18d8170266 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -95,7 +95,7 @@ jobs: env: SUITE: ${{ inputs.suite || 'resolute-qcom-devel' }} ARCH: arm64 - FLAVOR: qcom + FLAVOR: all DBGSYM: ${{ github.event_name == 'schedule' && 'true' || (inputs.dbgsym && 'true' || 'false') }} steps: From 2573a268e1e241d6dcc9e991c21fb6c9fb4b8751 Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Thu, 30 Jul 2026 11:16:36 +0800 Subject: [PATCH 106/115] ci(build-kernel): separate qcom-rt packages into output/rt/ qcom and qcom-rt flavour packages were collected into the same flat output/ directory, making the two indistinguishable downstream. Split qcom-rt packages (matched by their -qcom-rt/-qcom-rt-dbgsym suffix) into output/rt/, while qcom and arch-indep packages stay in output/. Also drop the redundant in-container copy step, since the host-side collection step already re-scans and copies the same files. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 32 ++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index a0f18d8170266..83cbe0b7549f8 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -370,27 +370,34 @@ jobs: echo "=== Locating built .deb files ===" find "${WS}" -name "*.deb" -o -name "*.ddeb" | sort || true - - mkdir -p "${WS}/output/" - find "${WS}" \( -name "*.deb" -o -name "*.ddeb" \) -exec cp {} "${WS}/output/" \; - ls -lh "${WS}/output/" || echo "No .deb files found" ' # ----------------------------------------------------------------------- # 8. Collect output packages # The Ubuntu kernel build drops .deb files one level above the # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). + # qcom-rt flavour packages are split into output/rt/ so consumers can + # tell the two flavours apart; qcom flavour + arch-indep packages + # (e.g. linux-qcom-headers-*_all.deb) stay in output/. # ----------------------------------------------------------------------- - name: Collect built packages run: | - mkdir -p output/ + mkdir -p output/rt/ find "${GITHUB_WORKSPACE}" -maxdepth 1 \ \( -name "*.deb" -o -name "*.ddeb" -o -name "*.changes" -o -name "*.buildinfo" \) \ - -exec cp {} output/ \; + -print0 | while IFS= read -r -d '' f; do + case "$(basename "$f")" in + *-qcom-rt_*|*-qcom-rt-dbgsym_*) dest="output/rt/" ;; + *) dest="output/" ;; + esac + cp "$f" "$dest" + done - echo "Built packages:" - ls -lh output/ || echo "(no files found)" + echo "Built packages (qcom):" + ls -lh output/ 2>/dev/null | grep -v '^total\|^d' || echo "(no files found)" + echo "Built packages (qcom-rt):" + ls -lh output/rt/ || echo "(no files found)" # ----------------------------------------------------------------------- # 9. Upload to S3 (lecore-production runner only) @@ -427,8 +434,13 @@ jobs: echo "| Devel PRs | \`${DEVEL_PRS}\` |" fi echo "" - echo "### Built packages" + echo "### Built packages (qcom)" + echo '```' + ls -lh output/ 2>/dev/null | grep -v '^total\|^d' || echo "(none)" + echo '```' + echo "" + echo "### Built packages (qcom-rt)" echo '```' - ls -lh output/ 2>/dev/null || echo "(none)" + ls -lh output/rt/ 2>/dev/null || echo "(none)" echo '```' } >> "$GITHUB_STEP_SUMMARY" From c50bebf886971b151d7510d2378dd4350f0503d7 Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Fri, 31 Jul 2026 08:53:20 +0800 Subject: [PATCH 107/115] ci(build-kernel): share linux-qcom-headers/-tools into output/rt/ linux-headers-*-qcom-rt depends on linux-qcom-headers-*, which is built as a single "Architecture: all" package per ABI rather than per-flavour, so it only ever lands in output/ and never matches the -qcom-rt filename pattern used to populate output/rt/. This leaves output/rt/ missing a dependency, breaking headers installs for the rt flavour when consumed standalone. Copy (not move, since qcom still needs its own copy) linux-qcom-headers-* and linux-qcom-tools-* into output/rt/ as well, so each flavour's output folder is self-contained. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 83cbe0b7549f8..dd901bed96bf2 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -377,8 +377,16 @@ jobs: # The Ubuntu kernel build drops .deb files one level above the # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). # qcom-rt flavour packages are split into output/rt/ so consumers can - # tell the two flavours apart; qcom flavour + arch-indep packages - # (e.g. linux-qcom-headers-*_all.deb) stay in output/. + # tell the two flavours apart; qcom flavour packages stay in output/. + # + # linux-qcom-headers-* and linux-qcom-tools-* are "Architecture: all" + # at the source level, so there is a single shared .deb per ABI + # rather than one per flavour, and they land in output/ (they don't + # match the -qcom-rt filename pattern above). linux-headers-*-qcom-rt + # depends on linux-qcom-headers-*, and a future linux-tools-*-qcom-rt + # would similarly depend on linux-qcom-tools-*, so both are also + # copied (not moved — qcom needs its own copy in output/ too) into + # output/rt/ to keep that folder self-contained. # ----------------------------------------------------------------------- - name: Collect built packages run: | @@ -394,6 +402,12 @@ jobs: cp "$f" "$dest" done + find output -maxdepth 1 \ + \( -name "linux-qcom-headers-*" -o -name "linux-qcom-tools-*" \) \ + -print0 | while IFS= read -r -d '' f; do + cp "$f" output/rt/ + done + echo "Built packages (qcom):" ls -lh output/ 2>/dev/null | grep -v '^total\|^d' || echo "(no files found)" echo "Built packages (qcom-rt):" From 75955ffd34e94edf8f76b90bbabb54876096bbb8 Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Thu, 6 Aug 2026 01:29:27 +0800 Subject: [PATCH 108/115] ci(build-kernel): flavours as space-separated names, indep always built, explicit s3_prefix Rename the flavor input to flavours and switch from a comma-separated token list to space-separated flavour names, matching how Canonical declares flavours in debian.qcom/rules.d/arm64.mk (e.g. "qcom qcom-rt"). binary-indep is no longer an optional token: it is always built alongside the selected flavours, since linux-headers-*/linux-tools-* packages depend on it and previously could be built into an uninstallable combination if the caller omitted it. Replace the S3 upload destination's github.event_name check with an explicit s3_prefix input. A reusable workflow's own github.event_name reflects the caller's original triggering event, not "workflow_call", so premerge-pr.yml calls were always landing under pkg/temp/ instead of the intended pkg/premerge/. Callers now declare their prefix directly. Updates docs/PIPELINE.md to match. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 37 ++++++++++++++++++++++++++---- docs/PIPELINE.md | 8 +++++-- 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index dd901bed96bf2..490c988237ce4 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -45,6 +45,11 @@ on: required: false default: true type: boolean + flavours: + description: "ALL builds every flavour. Otherwise a space-separated list of flavour names (e.g. 'qcom', 'qcom qcom-rt'), no 'binary-' prefix." + required: false + default: "ALL" + type: string workflow_call: inputs: @@ -78,6 +83,16 @@ on: required: false default: false type: boolean + flavours: + description: "ALL builds every flavour. Otherwise a space-separated list of flavour names (e.g. 'qcom', 'qcom qcom-rt'), no 'binary-' prefix." + required: false + default: "ALL" + type: string + s3_prefix: + description: "S3 path prefix under pkg/ (e.g. 'premerge', 'temp'). Callers must pass this explicitly." + required: false + default: "temp" + type: string permissions: contents: read # checkout + tag ls-remote only; output goes to S3 @@ -95,7 +110,7 @@ jobs: env: SUITE: ${{ inputs.suite || 'resolute-qcom-devel' }} ARCH: arm64 - FLAVOR: all + FLAVOURS: ${{ inputs.flavours || 'ALL' }} DBGSYM: ${{ github.event_name == 'schedule' && 'true' || (inputs.dbgsym && 'true' || 'false') }} steps: @@ -294,7 +309,18 @@ jobs: - name: Build kernel packages (ghcr.io/qualcomm-linux/pkg-builder:${{ inputs.suite || 'resolute-qcom' }}) run: | JOBS=$(nproc) - if [ "${FLAVOR}" = "all" ]; then TARGET="binary"; else TARGET="binary-${FLAVOR} binary-indep"; fi + FLAVOURS_LC=$(echo "${FLAVOURS}" | tr '[:upper:]' '[:lower:]') + # binary-indep is always built alongside the selected flavours — + # linux-headers-* and linux-tools-* packages depend on it, so it + # can't be made optional here. + if [ "${FLAVOURS_LC}" = "all" ]; then + TARGET="binary" + else + TARGET="binary-indep" + for f in ${FLAVOURS_LC}; do + TARGET="${TARGET} binary-${f}" + done + fi echo "Building: suite=${SUITE} base_suite=${BASE_SUITE} target=${TARGET} arch=${ARCH} jobs=${JOBS}" @@ -416,6 +442,9 @@ jobs: # ----------------------------------------------------------------------- # 9. Upload to S3 (lecore-production runner only) # Skipped when skip_s3 is set (premerge-pr.yml build-only checks). + # Destination prefix is the caller-supplied s3_prefix input, not + # inferred from github.event_name: a reusable workflow inherits the + # caller's original event, so it is never actually "workflow_call". # ----------------------------------------------------------------------- - name: Upload kernel .deb packages to S3 if: inputs.skip_s3 != true @@ -423,7 +452,7 @@ jobs: with: s3_bucket: qli-prd-lecore-gh-artifacts path: ${{ github.workspace }}/output - destination: ${{ env.ORG_NAME }}/pkg/temp/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ + destination: ${{ env.ORG_NAME }}/pkg/${{ inputs.s3_prefix || 'temp' }}/${{ env.REPO_NAME }}/${{ github.run_id }}-${{ github.run_attempt }}/ # ----------------------------------------------------------------------- # 10. Summary @@ -440,7 +469,7 @@ jobs: echo "|-------|-------|" echo "| Branch | \`${SUITE}\` |" echo "| Architecture | \`${ARCH}\` |" - echo "| Flavour | \`${FLAVOR}\` |" + echo "| Flavour | \`${FLAVOURS}\` |" echo "| Dbgsym | \`${DBGSYM}\` |" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" echo "| Runner | \`${{ runner.name }}\` |" diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 4233db986d06e..9c0db701f0320 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -57,6 +57,10 @@ gh workflow run build-kernel.yml --repo qualcomm-linux/pkg-linux-qcom-canonical gh workflow run bootstrap-history.yml --repo qualcomm-linux/pkg-linux-qcom-canonical ``` -PRs into `resolute-qcom-devel` get a build-only pre-merge check (`premerge-pr.yml` -on that branch), which calls `build-kernel.yml` with `skip_s3=true`. +PRs into `resolute-qcom-devel` get a pre-merge build check (`premerge-pr.yml` on +that branch), which calls `build-kernel.yml` with `flavours=qcom`, +`dbgsym=false`, and `s3_prefix=premerge` (binary-indep is always built +regardless of `flavours`). Its packages are uploaded to S3 under +`pkg/premerge/`, separate from the `pkg/temp/` prefix used by nightly and +manual `workflow_dispatch` runs. From 8dc11cfe2ca45ebfd68e124df9cbed66f093b29a Mon Sep 17 00:00:00 2001 From: Guanquan Tian Date: Tue, 11 Aug 2026 19:00:49 +0800 Subject: [PATCH 109/115] ci(build-kernel): only create output/rt/ when qcom-rt is actually built Collect built packages unconditionally created output/rt/ and copied the architecture-independent linux-qcom-headers-*/linux-qcom-tools-* packages into it regardless of which flavours were selected, so a qcom-only build (e.g. premerge-pr.yml with flavours: qcom) still produced a populated output/rt/ directory. Create output/rt/ only when a qcom-rt artifact is actually found, and gate the header/tools copy and summary output on that. Signed-off-by: Guanquan Tian --- .github/workflows/build-kernel.yml | 43 ++++++++++++++++++------------ 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 490c988237ce4..81c5590d19d0a 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -404,6 +404,9 @@ jobs: # source tree (i.e. in GITHUB_WORKSPACE, not inside kernel-src/). # qcom-rt flavour packages are split into output/rt/ so consumers can # tell the two flavours apart; qcom flavour packages stay in output/. + # output/rt/ is only created on demand (when a qcom-rt artifact is + # actually found below) so a build that didn't select qcom-rt + # doesn't produce an empty/misleading rt directory. # # linux-qcom-headers-* and linux-qcom-tools-* are "Architecture: all" # at the source level, so there is a single shared .deb per ABI @@ -412,32 +415,36 @@ jobs: # depends on linux-qcom-headers-*, and a future linux-tools-*-qcom-rt # would similarly depend on linux-qcom-tools-*, so both are also # copied (not moved — qcom needs its own copy in output/ too) into - # output/rt/ to keep that folder self-contained. + # output/rt/ to keep that folder self-contained — but again, only + # if output/rt/ already exists (i.e. qcom-rt was actually built). # ----------------------------------------------------------------------- - name: Collect built packages run: | - mkdir -p output/rt/ + mkdir -p output/ find "${GITHUB_WORKSPACE}" -maxdepth 1 \ \( -name "*.deb" -o -name "*.ddeb" -o -name "*.changes" -o -name "*.buildinfo" \) \ -print0 | while IFS= read -r -d '' f; do case "$(basename "$f")" in - *-qcom-rt_*|*-qcom-rt-dbgsym_*) dest="output/rt/" ;; - *) dest="output/" ;; + *-qcom-rt_*|*-qcom-rt-dbgsym_*) mkdir -p output/rt/; cp "$f" output/rt/ ;; + *) cp "$f" output/ ;; esac - cp "$f" "$dest" done - find output -maxdepth 1 \ - \( -name "linux-qcom-headers-*" -o -name "linux-qcom-tools-*" \) \ - -print0 | while IFS= read -r -d '' f; do - cp "$f" output/rt/ - done + if [ -d output/rt ]; then + find output -maxdepth 1 \ + \( -name "linux-qcom-headers-*" -o -name "linux-qcom-tools-*" \) \ + -print0 | while IFS= read -r -d '' f; do + cp "$f" output/rt/ + done + fi echo "Built packages (qcom):" ls -lh output/ 2>/dev/null | grep -v '^total\|^d' || echo "(no files found)" - echo "Built packages (qcom-rt):" - ls -lh output/rt/ || echo "(no files found)" + if [ -d output/rt ]; then + echo "Built packages (qcom-rt):" + ls -lh output/rt/ || echo "(no files found)" + fi # ----------------------------------------------------------------------- # 9. Upload to S3 (lecore-production runner only) @@ -481,9 +488,11 @@ jobs: echo '```' ls -lh output/ 2>/dev/null | grep -v '^total\|^d' || echo "(none)" echo '```' - echo "" - echo "### Built packages (qcom-rt)" - echo '```' - ls -lh output/rt/ 2>/dev/null || echo "(none)" - echo '```' + if [ -d output/rt ]; then + echo "" + echo "### Built packages (qcom-rt)" + echo '```' + ls -lh output/rt/ + echo '```' + fi } >> "$GITHUB_STEP_SUMMARY" From f31b4bf6cb1736a7d771738f5f2eb5ac1e6d789f Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Mon, 17 Aug 2026 16:46:20 -0700 Subject: [PATCH 110/115] ci: gate premerge kernels on distro image validation - Observe completed Pre-merge PR build runs from the trusted default-branch workflow and resolve one tested pull request targeting resolute-qcom-devel. - Bind each validation request to the kernel run, attempt, package directory, pull request number, and head SHA. - Start a request-bound Check Run before dispatch and use its external ID to make retries, supersession, and callback completion deterministic. - Dispatch the verified request to qcom-distro-images without exposing repository secrets to the untrusted pull request workflow. - Revalidate the kernel run, pull request, distro run, source SHA, build ID, S3 location, and completion marker before reporting the result. - Complete only the exact matching Check Run while retaining the existing commit status during the reporting transition. - Pin checkout actions to a verified commit and disable persisted Git credentials in every trusted orchestration job. - Document the cross-repository trust boundary, callback lifecycle, artifact layout, and validation reporting contract. Signed-off-by: Bjordis Collaku --- .github/workflows/bootstrap-history.yml | 15 +- .github/workflows/build-kernel.yml | 6 +- .github/workflows/fetch-source-pkg.yml | 14 +- .../workflows/premerge-distro-validation.yml | 226 ++++++++++++++++++ docs/PIPELINE.md | 25 +- scripts/report-distro-validation-check.sh | 202 ++++++++++++++++ scripts/report-distro-validation-status.sh | 30 +++ scripts/resolve-distro-validation-context.sh | 89 +++++++ scripts/send-distro-validation-request.sh | 53 ++++ .../validate-distro-validation-callback.sh | 116 +++++++++ 10 files changed, 763 insertions(+), 13 deletions(-) create mode 100644 .github/workflows/premerge-distro-validation.yml create mode 100755 scripts/report-distro-validation-check.sh create mode 100755 scripts/report-distro-validation-status.sh create mode 100755 scripts/resolve-distro-validation-context.sh create mode 100755 scripts/send-distro-validation-request.sh create mode 100755 scripts/validate-distro-validation-callback.sh diff --git a/.github/workflows/bootstrap-history.yml b/.github/workflows/bootstrap-history.yml index 57c36f0a08d65..fd1c9a3f2a84d 100644 --- a/.github/workflows/bootstrap-history.yml +++ b/.github/workflows/bootstrap-history.yml @@ -157,8 +157,10 @@ jobs: # ----------------------------------------------------------------------- - name: Fail on seed error if: steps.seed.outputs.status == 'failed' + env: + SEED_RC: ${{ steps.seed.outputs.rc }} run: | - echo "::error::seed-history.sh failed (rc=${{ steps.seed.outputs.rc }})." \ + echo "::error::seed-history.sh failed (rc=${SEED_RC})." \ "The clone is cached; investigate and re-dispatch to retry the publish." exit 1 @@ -167,15 +169,18 @@ jobs: # ----------------------------------------------------------------------- - name: Print summary if: always() + env: + SEED_BRANCH: ${{ steps.cfg.outputs.seed_branch }} + SEED_STATUS: ${{ steps.seed.outputs.status || 'failed' }} run: | { echo "## Bootstrap Summary" echo "" echo "| Field | Value |" echo "|-------|-------|" - echo "| Seed branch | \`${{ steps.cfg.outputs.seed_branch }}\` |" - echo "| Status | \`${{ steps.seed.outputs.status || 'failed' }}\` |" - if [ "${{ steps.seed.outputs.status }}" = "complete" ]; then - echo "| Result | Seed pushed -- review and promote \`${{ steps.cfg.outputs.seed_branch }}\` to the live branch |" + echo "| Seed branch | \`${SEED_BRANCH}\` |" + echo "| Status | \`${SEED_STATUS}\` |" + if [ "${SEED_STATUS}" = "complete" ]; then + echo "| Result | Seed pushed -- review and promote \`${SEED_BRANCH}\` to the live branch |" fi } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 81c5590d19d0a..99ae7bacedbb6 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -171,8 +171,9 @@ jobs: if: inputs.kernel_version != '' env: GH_TOKEN: ${{ github.token }} + KERNEL_VERSION: ${{ inputs.kernel_version }} run: | - TAG="${UPSTREAM_PREFIX}-${{ inputs.kernel_version }}" + TAG="${UPSTREAM_PREFIX}-${KERNEL_VERSION}" echo "Verifying tag '${TAG}' exists in repository..." if git ls-remote --exit-code --tags \ @@ -468,6 +469,7 @@ jobs: if: always() env: DEVEL_PRS: ${{ inputs.devel_prs }} + RUNNER_NAME: ${{ runner.name }} run: | { echo "## Kernel Build Summary" @@ -479,7 +481,7 @@ jobs: echo "| Flavour | \`${FLAVOURS}\` |" echo "| Dbgsym | \`${DBGSYM}\` |" echo "| Container | \`ghcr.io/qualcomm-linux/pkg-builder:${BASE_SUITE}\` |" - echo "| Runner | \`${{ runner.name }}\` |" + echo "| Runner | \`${RUNNER_NAME}\` |" if [ -n "${DEVEL_PRS}" ]; then echo "| Devel PRs | \`${DEVEL_PRS}\` |" fi diff --git a/.github/workflows/fetch-source-pkg.yml b/.github/workflows/fetch-source-pkg.yml index 442dbac66755c..9b5eb4aaa35ef 100644 --- a/.github/workflows/fetch-source-pkg.yml +++ b/.github/workflows/fetch-source-pkg.yml @@ -30,8 +30,7 @@ on: workflow_dispatch: # manual only; always mirrors the latest carmel-team upload permissions: - contents: write - actions: write + contents: read # Only one sync at a time -- a second run's lease-pinned push would be rejected. concurrency: @@ -95,6 +94,8 @@ jobs: timeout-minutes: 60 needs: check-version if: needs.check-version.outputs.should_sync == 'true' + permissions: + contents: write outputs: synced_version: ${{ steps.sync.outputs.synced_version }} @@ -126,6 +127,9 @@ jobs: - name: Print summary if: always() + env: + SYNCED_COUNT: ${{ steps.sync.outputs.synced_count || 0 }} + SYNCED_VERSION: ${{ steps.sync.outputs.synced_version }} run: | { echo "## Sync Summary" @@ -133,8 +137,8 @@ jobs: echo "| Field | Value |" echo "|-------|-------|" echo "| Branch | \`${BRANCH}\` |" - echo "| Uploads mirrored | ${{ steps.sync.outputs.synced_count || 0 }} |" - echo "| Latest version | \`${{ steps.sync.outputs.synced_version }}\` |" + echo "| Uploads mirrored | ${SYNCED_COUNT} |" + echo "| Latest version | \`${SYNCED_VERSION}\` |" } >> "$GITHUB_STEP_SUMMARY" # ========================================================================== @@ -145,6 +149,8 @@ jobs: runs-on: ubuntu-24.04-arm needs: [check-version, sync] if: needs.sync.result == 'success' && needs.sync.outputs.synced_count != '0' + permissions: + actions: write steps: - name: Dispatch build-kernel workflow diff --git a/.github/workflows/premerge-distro-validation.yml b/.github/workflows/premerge-distro-validation.yml new file mode 100644 index 0000000000000..b5afa2f642b23 --- /dev/null +++ b/.github/workflows/premerge-distro-validation.yml @@ -0,0 +1,226 @@ +# SPDX-License-Identifier: BSD-3-Clause + +name: "Pre-merge distro image validation" + +on: # zizmor: ignore[dangerous-triggers] + workflow_run: + workflows: + - "Pre-merge PR build" + types: + - completed + + repository_dispatch: + types: + - canonical-premerge-distro-result + +permissions: + contents: read + +concurrency: + group: >- + premerge-distro-validation-${{ + github.event_name == 'repository_dispatch' && 'callback' || 'request' + }}-${{ + github.event.client_payload.pr_number || + github.event.workflow_run.head_repository.full_name || + github.repository + }}-${{ + github.event.workflow_run.head_branch || + github.run_id + }} + cancel-in-progress: true + +jobs: + workflow-context: + name: "Resolve kernel validation context" + if: ${{ github.event_name == 'workflow_run' }} + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + outputs: + should-validate: ${{ steps.context.outputs.should-validate }} + kernel-conclusion: ${{ steps.context.outputs.kernel-conclusion }} + kernel-run-id: ${{ steps.context.outputs.kernel-run-id }} + kernel-run-attempt: ${{ steps.context.outputs.kernel-run-attempt }} + kernel-build-id: ${{ steps.context.outputs.kernel-build-id }} + kernel-s3-prefix: ${{ steps.context.outputs.kernel-s3-prefix }} + pr-number: ${{ steps.context.outputs.pr-number }} + head-sha: ${{ steps.context.outputs.head-sha }} + request-id: ${{ steps.context.outputs.request-id }} + steps: + - name: Checkout orchestration helpers + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + persist-credentials: false + + - id: context + name: Validate triggering workflow context + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + bash ./scripts/resolve-distro-validation-context.sh + + request-validation: + name: "Request distro image validation" + needs: workflow-context + if: ${{ needs.workflow-context.outputs.should-validate == 'true' }} + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + checks: write + contents: read + statuses: write + steps: + - name: Checkout orchestration helpers + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + persist-credentials: false + + - id: check + name: Start distro validation Check Run + env: + GH_TOKEN: ${{ github.token }} + MODE: start + REPOSITORY: ${{ github.repository }} + COMMIT_SHA: ${{ needs.workflow-context.outputs.head-sha }} + PR_NUMBER: ${{ needs.workflow-context.outputs.pr-number }} + REQUEST_ID: ${{ needs.workflow-context.outputs.request-id }} + KERNEL_BUILD_ID: ${{ needs.workflow-context.outputs.kernel-build-id }} + IMAGE_S3_PREFIX: qualcomm-linux/${{ needs.workflow-context.outputs.kernel-s3-prefix }}/${{ needs.workflow-context.outputs.kernel-build-id }} + DETAILS_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + bash ./scripts/report-distro-validation-check.sh + + - name: Report validation pending + if: ${{ steps.check.outputs.should-dispatch == 'true' }} + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + COMMIT_SHA: ${{ needs.workflow-context.outputs.head-sha }} + STATE: pending + DESCRIPTION: "Building Canonical server and desktop distro images" + TARGET_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + bash ./scripts/report-distro-validation-status.sh + + - name: Dispatch trusted distro validation request + if: ${{ steps.check.outputs.should-dispatch == 'true' }} + env: + GH_TOKEN: ${{ secrets.DEB_PKG_BOT_CI_TOKEN }} + DISTRO_REPOSITORY: qualcomm-linux/qcom-distro-images + KERNEL_REPOSITORY: ${{ github.repository }} + KERNEL_RUN_ID: ${{ needs.workflow-context.outputs.kernel-run-id }} + KERNEL_RUN_ATTEMPT: ${{ needs.workflow-context.outputs.kernel-run-attempt }} + KERNEL_BUILD_ID: ${{ needs.workflow-context.outputs.kernel-build-id }} + KERNEL_S3_PREFIX: ${{ needs.workflow-context.outputs.kernel-s3-prefix }} + PR_NUMBER: ${{ needs.workflow-context.outputs.pr-number }} + HEAD_SHA: ${{ needs.workflow-context.outputs.head-sha }} + REQUEST_ID: ${{ needs.workflow-context.outputs.request-id }} + run: | + set -euo pipefail + bash ./scripts/send-distro-validation-request.sh + + kernel-failure: + name: "Report kernel validation failure" + needs: workflow-context + if: ${{ needs.workflow-context.outputs.kernel-conclusion != 'success' }} + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + contents: read + statuses: write + steps: + - name: Checkout status helper + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + persist-credentials: false + + - name: Publish failure status + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + COMMIT_SHA: ${{ needs.workflow-context.outputs.head-sha }} + STATE: failure + DESCRIPTION: "Canonical kernel build failed before distro validation" + TARGET_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + set -euo pipefail + bash ./scripts/report-distro-validation-status.sh + exit 1 + + callback-context: + name: "Validate distro callback" + if: ${{ github.event_name == 'repository_dispatch' }} + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + outputs: + state: ${{ steps.callback.outputs.state }} + description: ${{ steps.callback.outputs.description }} + check-conclusion: ${{ steps.callback.outputs.check-conclusion }} + pr-number: ${{ steps.callback.outputs.pr-number }} + head-sha: ${{ steps.callback.outputs.head-sha }} + request-id: ${{ steps.callback.outputs.request-id }} + kernel-build-id: ${{ steps.callback.outputs.kernel-build-id }} + distro-build-id: ${{ steps.callback.outputs.distro-build-id }} + image-s3-prefix: ${{ steps.callback.outputs.image-s3-prefix }} + distro-run-url: ${{ steps.callback.outputs.distro-run-url }} + steps: + - name: Checkout callback validator + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + persist-credentials: false + + - id: callback + name: Validate returned distro result + env: + GH_TOKEN: ${{ secrets.DEB_PKG_BOT_CI_TOKEN }} + BUCKET: qli-prd-lecore-gh-artifacts + run: | + set -euo pipefail + bash ./scripts/validate-distro-validation-callback.sh + + callback-status: + name: "Report distro validation result" + needs: callback-context + if: ${{ always() && needs.callback-context.result == 'success' }} + runs-on: [self-hosted, lecore-prd-u2404-arm64-xlrg-od-ephem] + permissions: + checks: write + contents: read + statuses: write + steps: + - name: Checkout status helper + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + persist-credentials: false + + - name: Publish final status and summary + env: + GH_TOKEN: ${{ github.token }} + MODE: complete + REPOSITORY: ${{ github.repository }} + COMMIT_SHA: ${{ needs.callback-context.outputs.head-sha }} + PR_NUMBER: ${{ needs.callback-context.outputs.pr-number }} + REQUEST_ID: ${{ needs.callback-context.outputs.request-id }} + KERNEL_BUILD_ID: ${{ needs.callback-context.outputs.kernel-build-id }} + CONCLUSION: ${{ needs.callback-context.outputs.check-conclusion }} + DETAILS_URL: ${{ needs.callback-context.outputs.distro-run-url }} + DISTRO_BUILD_ID: ${{ needs.callback-context.outputs.distro-build-id }} + IMAGE_S3_PREFIX: ${{ needs.callback-context.outputs.image-s3-prefix }} + STATE: ${{ needs.callback-context.outputs.state }} + DESCRIPTION: ${{ needs.callback-context.outputs.description }} + TARGET_URL: ${{ needs.callback-context.outputs.distro-run-url }} + run: | + set -euo pipefail + { + echo "## Canonical Premerge Distro Validation" + echo + echo "| Field | Value |" + echo "|-------|-------|" + echo "| Pull request | \`#${PR_NUMBER}\` |" + echo "| Result | \`${STATE}\` |" + echo "| Distro build ID | \`${DISTRO_BUILD_ID}\` |" + echo "| Image S3 prefix | \`${IMAGE_S3_PREFIX}\` |" + } >> "$GITHUB_STEP_SUMMARY" + + bash ./scripts/report-distro-validation-check.sh + bash ./scripts/report-distro-validation-status.sh + [[ "$STATE" == "success" ]] diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index 9c0db701f0320..606922a3b80ed 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -12,7 +12,8 @@ pkg-linux-qcom-canonical │ ├── .github/workflows/ │ │ ├── fetch-source-pkg.yml ← manual incremental mirror sync │ │ ├── bootstrap-history.yml ← one-time history seed -│ │ └── build-kernel.yml ← build .deb packages (+ reusable workflow_call) +│ │ ├── build-kernel.yml ← build .deb packages (+ reusable workflow_call) +│ │ └── premerge-distro-validation.yml ← trusted distro image validation orchestrator │ ├── scripts/ ← sync-mirror.sh, seed-history.sh (self-documenting) │ └── README.md │ @@ -41,7 +42,7 @@ Upstream source: [https://git.launchpad.net/~carmel-team/ubuntu/+source/linux/+g ## Running it -All three workflows are manual (`Actions → … → Run workflow`, or via `gh`): +The maintenance workflows below are manual (`Actions → … → Run workflow`, or via `gh`): ```bash # Sync the mirror to the latest upstream upload (no inputs; idempotent). @@ -64,3 +65,23 @@ regardless of `flavours`). Its packages are uploaded to S3 under `pkg/premerge/`, separate from the `pkg/temp/` prefix used by nightly and manual `workflow_dispatch` runs. +After the pre-merge kernel workflow completes, `premerge-distro-validation.yml` +runs from the trusted `main` branch through `workflow_run`. It resolves the PR +from the triggering kernel run and dispatches a validated request to the internal +qcom-distro-images repository. The internal receiver calls its local reusable +workflow and builds the fixed Resolute IoT server and desktop matrix using only +the Canonical kernel packages from that exact premerge build. + +The two image tarballs are uploaded alongside the kernel packages under the same +`pkg/premerge/pkg-linux-qcom-canonical/-/` directory. A +`distro-validation.json` completion marker is written only after both image +uploads are verified. qcom-distro-images returns the distro build ID and result +through a repository dispatch callback. The Canonical callback handler verifies +the kernel run, distro run, PR identity, and S3 completion marker before reporting +the final result on the PR head commit. A Check Run named +`qcom-distro-images/canonical-premerge` starts before the distro request and is +completed only after the callback is validated. The request ID is stored as the +Check Run external ID so retries update the same validation and superseded runs +cannot overwrite a newer result. The existing commit status is published in +parallel during the transition. The untrusted PR workflow receives no repository +secrets, and neither side polls the other workflow. diff --git a/scripts/report-distro-validation-check.sh b/scripts/report-distro-validation-check.sh new file mode 100755 index 0000000000000..6a247ea5af705 --- /dev/null +++ b/scripts/report-distro-validation-check.sh @@ -0,0 +1,202 @@ +#!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +: "${MODE:?MODE is required}" +: "${REPOSITORY:?REPOSITORY is required}" +: "${COMMIT_SHA:?COMMIT_SHA is required}" +: "${PR_NUMBER:?PR_NUMBER is required}" +: "${REQUEST_ID:?REQUEST_ID is required}" +: "${KERNEL_BUILD_ID:?KERNEL_BUILD_ID is required}" +: "${IMAGE_S3_PREFIX:?IMAGE_S3_PREFIX is required}" +: "${DETAILS_URL:?DETAILS_URL is required}" + +CHECK_NAME="qcom-distro-images/canonical-premerge" +CONCLUSION="${CONCLUSION:-}" +DISTRO_BUILD_ID="${DISTRO_BUILD_ID:-pending}" + +emit_output() { + [[ -n "${GITHUB_OUTPUT:-}" ]] || return 0 + echo "$1=$2" >> "$GITHUB_OUTPUT" +} + +[[ "$MODE" =~ ^(start|complete)$ ]] || { echo "::error::Invalid Check Run mode: ${MODE}" >&2; exit 1; } +[[ "$REPOSITORY" == "qualcomm-linux/pkg-linux-qcom-canonical" ]] || { echo "::error::Unexpected repository: ${REPOSITORY}" >&2; exit 1; } +[[ "$COMMIT_SHA" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid commit SHA: ${COMMIT_SHA}" >&2; exit 1; } +[[ "$PR_NUMBER" =~ ^[0-9]+$ ]] || { echo "::error::Invalid pull request number: ${PR_NUMBER}" >&2; exit 1; } +[[ "$KERNEL_BUILD_ID" =~ ^[0-9]+-[0-9]+$ ]] || { echo "::error::Invalid kernel build ID: ${KERNEL_BUILD_ID}" >&2; exit 1; } +[[ "$REQUEST_ID" == "${KERNEL_BUILD_ID}-${COMMIT_SHA}" ]] || { echo "::error::Request ID does not match the kernel build and commit SHA." >&2; exit 1; } +[[ "$IMAGE_S3_PREFIX" == "qualcomm-linux/pkg/premerge/pkg-linux-qcom-canonical/${KERNEL_BUILD_ID}" ]] || { echo "::error::Unexpected image S3 prefix: ${IMAGE_S3_PREFIX}" >&2; exit 1; } +[[ "$DETAILS_URL" =~ ^https://github\.com/qualcomm-linux/(pkg-linux-qcom-canonical|qcom-distro-images)/actions/runs/[0-9]+$ ]] || { echo "::error::Unexpected Check Run details URL: ${DETAILS_URL}" >&2; exit 1; } + +if [[ "$MODE" == "complete" ]]; then + [[ "$CONCLUSION" =~ ^(success|failure|cancelled|skipped)$ ]] || { echo "::error::Invalid Check Run conclusion: ${CONCLUSION}" >&2; exit 1; } + [[ "$DISTRO_BUILD_ID" =~ ^[0-9]+-[0-9]+$ ]] || { echo "::error::Invalid distro build ID: ${DISTRO_BUILD_ID}" >&2; exit 1; } +fi + +checks="$({ + gh api \ + --paginate \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "repos/${REPOSITORY}/commits/${COMMIT_SHA}/check-runs?filter=all&per_page=100" +} | jq -s --arg name "$CHECK_NAME" '[.[].check_runs[] | select(.name == $name)]')" + +matching_checks="$(jq --arg request_id "$REQUEST_ID" '[.[] | select((.external_id // "") == $request_id)]' <<< "$checks")" +matching_count="$(jq 'length' <<< "$matching_checks")" + +if [[ "$matching_count" -gt 1 ]]; then + echo "::error::Multiple Check Runs use request ID ${REQUEST_ID}." >&2 + exit 1 +fi + +check_title="Canonical premerge distro validation" +check_summary="Building Canonical server and desktop images for PR #${PR_NUMBER}." +check_text=$(printf "Kernel build ID: \`%s\`\nImage S3 prefix: \`%s\`" "$KERNEL_BUILD_ID" "$IMAGE_S3_PREFIX") + +if [[ "$MODE" == "start" ]]; then + if [[ "$matching_count" == "1" ]]; then + existing_status="$(jq -r '.[0].status' <<< "$matching_checks")" + existing_url="$(jq -r '.[0].html_url' <<< "$matching_checks")" + if [[ "$existing_status" =~ ^(in_progress|completed)$ ]]; then + emit_output should-dispatch false + echo "[INFO] Reusing Check Run for ${REQUEST_ID}: ${existing_url}" + exit 0 + fi + echo "::error::Check Run for ${REQUEST_ID} has unexpected status ${existing_status}." >&2 + exit 1 + fi + + while IFS=$'\t' read -r check_id external_id status; do + [[ -n "$check_id" ]] || continue + [[ "$status" == "in_progress" ]] || continue + [[ "$external_id" != "$REQUEST_ID" ]] || continue + [[ "$external_id" =~ ^[0-9]+-[0-9]+-${COMMIT_SHA}$ ]] || continue + + superseded_payload="$(jq -n \ + --arg details_url "$DETAILS_URL" \ + --arg title "$check_title" \ + --arg summary "Superseded by a newer Canonical premerge validation request." \ + --arg text "Superseding request ID: \`${REQUEST_ID}\`" \ + '{ + status: "completed", + conclusion: "cancelled", + details_url: $details_url, + output: { + title: $title, + summary: $summary, + text: $text + } + }')" + gh api \ + --method PATCH \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "repos/${REPOSITORY}/check-runs/${check_id}" \ + --input - <<< "$superseded_payload" >/dev/null + done < <(jq -r '.[] | [.id, (.external_id // ""), .status] | @tsv' <<< "$checks") + + create_payload="$(jq -n \ + --arg name "$CHECK_NAME" \ + --arg head_sha "$COMMIT_SHA" \ + --arg external_id "$REQUEST_ID" \ + --arg details_url "$DETAILS_URL" \ + --arg title "$check_title" \ + --arg summary "$check_summary" \ + --arg text "$check_text" \ + '{ + name: $name, + head_sha: $head_sha, + external_id: $external_id, + status: "in_progress", + details_url: $details_url, + output: { + title: $title, + summary: $summary, + text: $text + } + }')" + check_run_url="$(gh api \ + --method POST \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "repos/${REPOSITORY}/check-runs" \ + --input - <<< "$create_payload" \ + --jq '.html_url')" + emit_output should-dispatch true + echo "[INFO] Started Check Run for ${REQUEST_ID}: ${check_run_url}" + exit 0 +fi + +if [[ "$matching_count" != "1" ]]; then + echo "::error::No Check Run exists for request ID ${REQUEST_ID}." >&2 + exit 1 +fi + +check_id="$(jq -r '.[0].id' <<< "$matching_checks")" +existing_status="$(jq -r '.[0].status' <<< "$matching_checks")" +existing_conclusion="$(jq -r '.[0].conclusion // ""' <<< "$matching_checks")" +existing_url="$(jq -r '.[0].html_url' <<< "$matching_checks")" + +case "$CONCLUSION" in + success) + check_summary="Canonical server and desktop distro images passed for PR #${PR_NUMBER}." + ;; + failure) + check_summary="Canonical distro image validation failed for PR #${PR_NUMBER}." + ;; + cancelled) + check_summary="Canonical distro image validation was cancelled for PR #${PR_NUMBER}." + ;; + skipped) + check_summary="Canonical distro image validation was skipped for PR #${PR_NUMBER}." + ;; +esac +check_text=$(printf "Kernel build ID: \`%s\`\nDistro build ID: \`%s\`\nImage S3 prefix: \`%s\`" "$KERNEL_BUILD_ID" "$DISTRO_BUILD_ID" "$IMAGE_S3_PREFIX") + +if [[ "$existing_status" == "completed" ]]; then + existing_details_url="$(jq -r '.[0].details_url // ""' <<< "$matching_checks")" + existing_title="$(jq -r '.[0].output.title // ""' <<< "$matching_checks")" + existing_summary="$(jq -r '.[0].output.summary // ""' <<< "$matching_checks")" + existing_text="$(jq -r '.[0].output.text // ""' <<< "$matching_checks")" + if [[ "$existing_conclusion" == "$CONCLUSION" && + "$existing_details_url" == "$DETAILS_URL" && + "$existing_title" == "$check_title" && + "$existing_summary" == "$check_summary" && + "$existing_text" == "$check_text" ]]; then + echo "[INFO] Check Run already completed for ${REQUEST_ID}: ${existing_url}" + exit 0 + fi + echo "::error::Completed Check Run for ${REQUEST_ID} does not match the validated callback result." >&2 + exit 1 +fi + +[[ "$existing_status" == "in_progress" ]] || { echo "::error::Check Run for ${REQUEST_ID} has unexpected status ${existing_status}." >&2; exit 1; } + +complete_payload="$(jq -n \ + --arg conclusion "$CONCLUSION" \ + --arg details_url "$DETAILS_URL" \ + --arg title "$check_title" \ + --arg summary "$check_summary" \ + --arg text "$check_text" \ + '{ + status: "completed", + conclusion: $conclusion, + details_url: $details_url, + output: { + title: $title, + summary: $summary, + text: $text + } + }')" +check_run_url="$(gh api \ + --method PATCH \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "repos/${REPOSITORY}/check-runs/${check_id}" \ + --input - <<< "$complete_payload" \ + --jq '.html_url')" + +echo "[INFO] Completed Check Run for ${REQUEST_ID}: ${check_run_url}" diff --git a/scripts/report-distro-validation-status.sh b/scripts/report-distro-validation-status.sh new file mode 100755 index 0000000000000..704008d18d3b4 --- /dev/null +++ b/scripts/report-distro-validation-status.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +: "${REPOSITORY:?REPOSITORY is required}" +: "${COMMIT_SHA:?COMMIT_SHA is required}" +: "${STATE:?STATE is required}" +: "${DESCRIPTION:?DESCRIPTION is required}" +: "${TARGET_URL:?TARGET_URL is required}" + +STATUS_CONTEXT="${STATUS_CONTEXT:-qcom-distro-images/canonical-premerge}" + +[[ "$REPOSITORY" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]] || { echo "::error::Invalid repository: ${REPOSITORY}" >&2; exit 1; } +[[ "$COMMIT_SHA" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid commit SHA: ${COMMIT_SHA}" >&2; exit 1; } +[[ "$STATE" =~ ^(error|failure|pending|success)$ ]] || { echo "::error::Invalid status state: ${STATE}" >&2; exit 1; } + +description="${DESCRIPTION:0:140}" + +gh api \ + --method POST \ + "repos/${REPOSITORY}/statuses/${COMMIT_SHA}" \ + -f state="$STATE" \ + -f target_url="$TARGET_URL" \ + -f description="$description" \ + -f context="$STATUS_CONTEXT" \ + >/dev/null + +echo "[INFO] Reported ${STATUS_CONTEXT}=${STATE} for ${COMMIT_SHA}." diff --git a/scripts/resolve-distro-validation-context.sh b/scripts/resolve-distro-validation-context.sh new file mode 100755 index 0000000000000..e34b147ef2775 --- /dev/null +++ b/scripts/resolve-distro-validation-context.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +: "${GITHUB_EVENT_NAME:?GITHUB_EVENT_NAME is required}" + +emit_output() { + local name="$1" + local value="$2" + echo "${name}=${value}" >> "$GITHUB_OUTPUT" +} + +validate_common() { + local run_id="$1" + local run_attempt="$2" + local pr_number="$3" + local head_sha="$4" + + [[ "$run_id" =~ ^[0-9]+$ ]] || { echo "::error::Invalid kernel workflow run ID: ${run_id}" >&2; exit 1; } + [[ "$run_attempt" =~ ^[0-9]+$ ]] || { echo "::error::Invalid kernel workflow run attempt: ${run_attempt}" >&2; exit 1; } + [[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "::error::Invalid pull request number: ${pr_number}" >&2; exit 1; } + [[ "$head_sha" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid pull request head SHA: ${head_sha}" >&2; exit 1; } + + emit_output kernel-build-id "${run_id}-${run_attempt}" + emit_output kernel-run-id "$run_id" + emit_output kernel-run-attempt "$run_attempt" + emit_output kernel-s3-prefix "pkg/premerge/pkg-linux-qcom-canonical" + emit_output pr-number "$pr_number" + emit_output head-sha "$head_sha" + emit_output request-id "${run_id}-${run_attempt}-${head_sha}" +} + +if [[ "$GITHUB_EVENT_NAME" == "workflow_run" ]]; then + : "${GITHUB_EVENT_PATH:?GITHUB_EVENT_PATH is required}" + + action="$(jq -r '.action' "$GITHUB_EVENT_PATH")" + event="$(jq -r '.workflow_run.event' "$GITHUB_EVENT_PATH")" + workflow_path="$(jq -r '.workflow_run.path' "$GITHUB_EVENT_PATH")" + repository="$(jq -r '.workflow_run.repository.full_name' "$GITHUB_EVENT_PATH")" + + [[ "$event" == "pull_request" ]] || { echo "::error::Unexpected triggering event: ${event}" >&2; exit 1; } + [[ "$workflow_path" == ".github/workflows/premerge-pr.yml" ]] || { echo "::error::Unexpected triggering workflow path: ${workflow_path}" >&2; exit 1; } + [[ "$repository" == "$GITHUB_REPOSITORY" ]] || { echo "::error::Unexpected triggering repository: ${repository}" >&2; exit 1; } + + [[ "$action" == "completed" ]] || { echo "::error::Unexpected workflow_run activity: ${action}" >&2; exit 1; } + + run_id="$(jq -r '.workflow_run.id' "$GITHUB_EVENT_PATH")" + run_attempt="$(jq -r '.workflow_run.run_attempt' "$GITHUB_EVENT_PATH")" + conclusion="$(jq -r '.workflow_run.conclusion' "$GITHUB_EVENT_PATH")" + head_sha="$(jq -r '.workflow_run.head_sha' "$GITHUB_EVENT_PATH")" + + pull_requests="$( + gh api "repos/${GITHUB_REPOSITORY}/commits/${head_sha}/pulls" \ + -H "Accept: application/vnd.github+json" \ + --jq '[.[] | select(.base.ref == "resolute-qcom-devel" and .head.sha == "'"$head_sha"'")]' + )" + pr_count="$(jq 'length' <<< "$pull_requests")" + [[ "$pr_count" == "1" ]] || { + echo "::error::Expected exactly one resolute-qcom-devel pull request for ${head_sha}; found ${pr_count}." >&2 + exit 1 + } + pr_number="$(jq -r '.[0].number' <<< "$pull_requests")" + + validate_common "$run_id" "$run_attempt" "$pr_number" "$head_sha" + emit_output kernel-conclusion "$conclusion" + if [[ "$conclusion" == "success" ]]; then + emit_output should-validate "true" + else + emit_output should-validate "false" + fi + exit 0 +fi + +if [[ "$GITHUB_EVENT_NAME" == "workflow_dispatch" ]]; then + : "${KERNEL_RUN_ID:?KERNEL_RUN_ID is required}" + : "${KERNEL_RUN_ATTEMPT:?KERNEL_RUN_ATTEMPT is required}" + : "${PR_NUMBER:?PR_NUMBER is required}" + : "${PR_HEAD_SHA:?PR_HEAD_SHA is required}" + + validate_common "$KERNEL_RUN_ID" "$KERNEL_RUN_ATTEMPT" "$PR_NUMBER" "$PR_HEAD_SHA" + emit_output kernel-conclusion "success" + emit_output should-validate "true" + exit 0 +fi + +echo "::error::Unsupported event: ${GITHUB_EVENT_NAME}" >&2 +exit 1 diff --git a/scripts/send-distro-validation-request.sh b/scripts/send-distro-validation-request.sh new file mode 100755 index 0000000000000..a2dee529c5d16 --- /dev/null +++ b/scripts/send-distro-validation-request.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +: "${DISTRO_REPOSITORY:?DISTRO_REPOSITORY is required}" +: "${KERNEL_REPOSITORY:?KERNEL_REPOSITORY is required}" +: "${KERNEL_RUN_ID:?KERNEL_RUN_ID is required}" +: "${KERNEL_RUN_ATTEMPT:?KERNEL_RUN_ATTEMPT is required}" +: "${KERNEL_BUILD_ID:?KERNEL_BUILD_ID is required}" +: "${KERNEL_S3_PREFIX:?KERNEL_S3_PREFIX is required}" +: "${PR_NUMBER:?PR_NUMBER is required}" +: "${HEAD_SHA:?HEAD_SHA is required}" +: "${REQUEST_ID:?REQUEST_ID is required}" + +[[ "$DISTRO_REPOSITORY" == "qualcomm-linux/qcom-distro-images" ]] || { echo "::error::Unexpected distro repository." >&2; exit 1; } +[[ "$KERNEL_REPOSITORY" == "qualcomm-linux/pkg-linux-qcom-canonical" ]] || { echo "::error::Unexpected kernel repository." >&2; exit 1; } +[[ "$KERNEL_RUN_ID" =~ ^[0-9]+$ ]] || { echo "::error::Invalid kernel run ID." >&2; exit 1; } +[[ "$KERNEL_RUN_ATTEMPT" =~ ^[0-9]+$ ]] || { echo "::error::Invalid kernel run attempt." >&2; exit 1; } +[[ "$KERNEL_BUILD_ID" == "${KERNEL_RUN_ID}-${KERNEL_RUN_ATTEMPT}" ]] || { echo "::error::Kernel build ID does not match its run identity." >&2; exit 1; } +[[ "$KERNEL_S3_PREFIX" == "pkg/premerge/pkg-linux-qcom-canonical" ]] || { echo "::error::Unexpected kernel S3 prefix." >&2; exit 1; } +[[ "$PR_NUMBER" =~ ^[0-9]+$ ]] || { echo "::error::Invalid pull request number." >&2; exit 1; } +[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid pull request head SHA." >&2; exit 1; } +[[ "$REQUEST_ID" == "${KERNEL_BUILD_ID}-${HEAD_SHA}" ]] || { echo "::error::Request ID does not match the build and head SHA." >&2; exit 1; } + +payload="$(jq -n \ + --arg event_type "canonical-premerge-validation" \ + --arg kernel_repository "$KERNEL_REPOSITORY" \ + --arg kernel_run_id "$KERNEL_RUN_ID" \ + --arg kernel_run_attempt "$KERNEL_RUN_ATTEMPT" \ + --arg kernel_build_id "$KERNEL_BUILD_ID" \ + --arg kernel_s3_prefix "$KERNEL_S3_PREFIX" \ + --arg pr_number "$PR_NUMBER" \ + --arg head_sha "$HEAD_SHA" \ + --arg request_id "$REQUEST_ID" \ + '{ + event_type: $event_type, + client_payload: { + kernel_repository: $kernel_repository, + kernel_run_id: $kernel_run_id, + kernel_run_attempt: $kernel_run_attempt, + kernel_build_id: $kernel_build_id, + kernel_s3_prefix: $kernel_s3_prefix, + pr_number: $pr_number, + head_sha: $head_sha, + request_id: $request_id + } + }')" + +gh api --method POST "repos/${DISTRO_REPOSITORY}/dispatches" --input - <<< "$payload" + +echo "[INFO] Dispatched distro validation request ${REQUEST_ID}." diff --git a/scripts/validate-distro-validation-callback.sh b/scripts/validate-distro-validation-callback.sh new file mode 100755 index 0000000000000..5825c0107a512 --- /dev/null +++ b/scripts/validate-distro-validation-callback.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. +# SPDX-License-Identifier: BSD-3-Clause +# +set -euo pipefail + +: "${GITHUB_EVENT_PATH:?GITHUB_EVENT_PATH is required}" +: "${BUCKET:?BUCKET is required}" + +emit_output() { + echo "$1=$2" >> "$GITHUB_OUTPUT" +} + +payload="$(jq '.client_payload' "$GITHUB_EVENT_PATH")" +request_id="$(jq -r '.request_id' <<< "$payload")" +kernel_build_id="$(jq -r '.kernel_build_id' <<< "$payload")" +kernel_s3_prefix="$(jq -r '.kernel_s3_prefix' <<< "$payload")" +pr_number="$(jq -r '.pr_number' <<< "$payload")" +head_sha="$(jq -r '.head_sha' <<< "$payload")" +distro_result="$(jq -r '.distro_result' <<< "$payload")" +distro_run_id="$(jq -r '.distro_run_id' <<< "$payload")" +distro_run_attempt="$(jq -r '.distro_run_attempt' <<< "$payload")" +distro_build_id="$(jq -r '.distro_build_id' <<< "$payload")" +distro_ref="$(jq -r '.distro_ref' <<< "$payload")" +image_s3_prefix="$(jq -r '.image_s3_prefix' <<< "$payload")" + +[[ "$(jq -r '.action' "$GITHUB_EVENT_PATH")" == "canonical-premerge-distro-result" ]] || { echo "::error::Unexpected callback event type." >&2; exit 1; } +[[ "$kernel_build_id" =~ ^[0-9]+-[0-9]+$ ]] || { echo "::error::Invalid kernel build ID." >&2; exit 1; } +[[ "$kernel_s3_prefix" == "pkg/premerge/pkg-linux-qcom-canonical" ]] || { echo "::error::Unexpected kernel S3 prefix." >&2; exit 1; } +[[ "$pr_number" =~ ^[0-9]+$ ]] || { echo "::error::Invalid pull request number." >&2; exit 1; } +[[ "$head_sha" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid pull request head SHA." >&2; exit 1; } +[[ "$distro_result" =~ ^(success|failure|cancelled|skipped)$ ]] || { echo "::error::Invalid distro result." >&2; exit 1; } +[[ "$distro_run_id" =~ ^[0-9]+$ && "$distro_run_attempt" =~ ^[0-9]+$ ]] || { echo "::error::Invalid distro run identity." >&2; exit 1; } +[[ "$distro_build_id" == "${distro_run_id}-${distro_run_attempt}" ]] || { echo "::error::Distro build ID does not match its run identity." >&2; exit 1; } +[[ "$distro_ref" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid distro source SHA." >&2; exit 1; } +[[ "$request_id" == "${kernel_build_id}-${head_sha}" ]] || { echo "::error::Request ID does not match the kernel build and head SHA." >&2; exit 1; } +[[ "$image_s3_prefix" == "qualcomm-linux/${kernel_s3_prefix}/${kernel_build_id}" ]] || { echo "::error::Unexpected image S3 prefix." >&2; exit 1; } + +IFS=- read -r kernel_run_id kernel_run_attempt <<< "$kernel_build_id" +kernel_run="$(gh api "repos/qualcomm-linux/pkg-linux-qcom-canonical/actions/runs/${kernel_run_id}")" +jq -e \ + --argjson attempt "$kernel_run_attempt" \ + --arg head_sha "$head_sha" \ + '.event == "pull_request" and + .path == ".github/workflows/premerge-pr.yml" and + .run_attempt == $attempt and + .conclusion == "success" and + .head_sha == $head_sha' \ + <<< "$kernel_run" >/dev/null || { + echo "::error::Kernel workflow run does not match the callback context." >&2 + exit 1 + } + +pull_requests="$( + gh api "repos/qualcomm-linux/pkg-linux-qcom-canonical/commits/${head_sha}/pulls" \ + -H "Accept: application/vnd.github+json" \ + --jq '[.[] | select(.base.ref == "resolute-qcom-devel" and .head.sha == "'"$head_sha"'")]' +)" +[[ "$(jq 'length' <<< "$pull_requests")" == "1" ]] || { echo "::error::Unable to identify one matching Canonical pull request." >&2; exit 1; } +[[ "$(jq -r '.[0].number' <<< "$pull_requests")" == "$pr_number" ]] || { echo "::error::Callback pull request number does not match the kernel commit." >&2; exit 1; } + +distro_run="$(gh api "repos/qualcomm-linux/qcom-distro-images/actions/runs/${distro_run_id}")" +jq -e \ + --argjson attempt "$distro_run_attempt" \ + --arg distro_ref "$distro_ref" \ + '.event == "repository_dispatch" and + .path == ".github/workflows/canonical-premerge-validation.yml" and + .run_attempt == $attempt and + .head_sha == $distro_ref' \ + <<< "$distro_run" >/dev/null || { + echo "::error::Distro workflow run does not match the callback context." >&2 + exit 1 + } + +distro_run_url="$(jq -r '.html_url' <<< "$distro_run")" + +state=failure +description="Canonical distro image validation failed" +check_conclusion="$distro_result" +if [[ "$distro_result" == "success" ]]; then + marker="$(mktemp)" + trap 'rm -f "$marker"' EXIT + aws s3 cp "s3://${BUCKET}/${image_s3_prefix}/distro-validation.json" "$marker" >/dev/null + jq -e \ + --arg request_id "$request_id" \ + --arg distro_build_id "$distro_build_id" \ + --arg distro_ref "$distro_ref" \ + --arg kernel_build_id "$kernel_build_id" \ + --arg kernel_s3_prefix "$kernel_s3_prefix" \ + '.status == "success" and + .request_id == $request_id and + .distro_build_id == $distro_build_id and + .distro_ref == $distro_ref and + .kernel_build_id == $kernel_build_id and + .kernel_s3_prefix == $kernel_s3_prefix and + (.images | sort) == ([ + "qcom-ubuntu-iot-resolute-desktop-canonical.images.tar.gz", + "qcom-ubuntu-iot-resolute-server-canonical.images.tar.gz" + ] | sort)' "$marker" >/dev/null || { + echo "::error::Distro completion marker does not match the callback context." >&2 + exit 1 + } + state=success + description="Canonical server and desktop distro images passed" +fi + +emit_output state "$state" +emit_output description "$description" +emit_output check-conclusion "$check_conclusion" +emit_output pr-number "$pr_number" +emit_output head-sha "$head_sha" +emit_output request-id "$request_id" +emit_output kernel-build-id "$kernel_build_id" +emit_output distro-build-id "$distro_build_id" +emit_output image-s3-prefix "$image_s3_prefix" +emit_output distro-run-url "$distro_run_url" From 6d53414542ce1ecd0ca1896aabd9b27144e2312a Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 20 Aug 2026 16:51:19 -0700 Subject: [PATCH 111/115] ci: install GitHub CLI for distro validation - Install GitHub CLI from Ubuntu repositories in every trusted job that invokes gh api. - Restore kernel context resolution, validation dispatch, failure reporting, callback validation, and Check Run updates. - Remove reliance on the ephemeral runner image providing the GitHub CLI. Signed-off-by: Bjordis Collaku --- .../workflows/premerge-distro-validation.yml | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/.github/workflows/premerge-distro-validation.yml b/.github/workflows/premerge-distro-validation.yml index b5afa2f642b23..f43cdcde827af 100644 --- a/.github/workflows/premerge-distro-validation.yml +++ b/.github/workflows/premerge-distro-validation.yml @@ -51,6 +51,12 @@ jobs: with: persist-credentials: false + - name: Install GitHub CLI + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends gh + - id: context name: Validate triggering workflow context env: @@ -74,6 +80,12 @@ jobs: with: persist-credentials: false + - name: Install GitHub CLI + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends gh + - id: check name: Start distro validation Check Run env: @@ -134,6 +146,12 @@ jobs: with: persist-credentials: false + - name: Install GitHub CLI + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends gh + - name: Publish failure status env: GH_TOKEN: ${{ github.token }} @@ -168,6 +186,12 @@ jobs: with: persist-credentials: false + - name: Install GitHub CLI + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends gh + - id: callback name: Validate returned distro result env: @@ -192,6 +216,12 @@ jobs: with: persist-credentials: false + - name: Install GitHub CLI + run: | + set -euo pipefail + sudo apt-get update + sudo apt-get install -y --no-install-recommends gh + - name: Publish final status and summary env: GH_TOKEN: ${{ github.token }} From 9d8538dae7506c9244f0af59d6ae8f626ef63355 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Thu, 20 Aug 2026 20:12:22 -0700 Subject: [PATCH 112/115] ci: derive distro callback metadata - Derive the distro build ID from the verified workflow run identity. - Read the immutable distro source SHA from the GitHub Actions run and construct the image prefix from validated kernel identity. - Preserve run, pull request, receipt, and completion marker verification for the reduced callback contract. Signed-off-by: Bjordis Collaku --- scripts/validate-distro-validation-callback.sh | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/scripts/validate-distro-validation-callback.sh b/scripts/validate-distro-validation-callback.sh index 5825c0107a512..224a9b4036508 100755 --- a/scripts/validate-distro-validation-callback.sh +++ b/scripts/validate-distro-validation-callback.sh @@ -20,9 +20,6 @@ head_sha="$(jq -r '.head_sha' <<< "$payload")" distro_result="$(jq -r '.distro_result' <<< "$payload")" distro_run_id="$(jq -r '.distro_run_id' <<< "$payload")" distro_run_attempt="$(jq -r '.distro_run_attempt' <<< "$payload")" -distro_build_id="$(jq -r '.distro_build_id' <<< "$payload")" -distro_ref="$(jq -r '.distro_ref' <<< "$payload")" -image_s3_prefix="$(jq -r '.image_s3_prefix' <<< "$payload")" [[ "$(jq -r '.action' "$GITHUB_EVENT_PATH")" == "canonical-premerge-distro-result" ]] || { echo "::error::Unexpected callback event type." >&2; exit 1; } [[ "$kernel_build_id" =~ ^[0-9]+-[0-9]+$ ]] || { echo "::error::Invalid kernel build ID." >&2; exit 1; } @@ -31,10 +28,7 @@ image_s3_prefix="$(jq -r '.image_s3_prefix' <<< "$payload")" [[ "$head_sha" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid pull request head SHA." >&2; exit 1; } [[ "$distro_result" =~ ^(success|failure|cancelled|skipped)$ ]] || { echo "::error::Invalid distro result." >&2; exit 1; } [[ "$distro_run_id" =~ ^[0-9]+$ && "$distro_run_attempt" =~ ^[0-9]+$ ]] || { echo "::error::Invalid distro run identity." >&2; exit 1; } -[[ "$distro_build_id" == "${distro_run_id}-${distro_run_attempt}" ]] || { echo "::error::Distro build ID does not match its run identity." >&2; exit 1; } -[[ "$distro_ref" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid distro source SHA." >&2; exit 1; } [[ "$request_id" == "${kernel_build_id}-${head_sha}" ]] || { echo "::error::Request ID does not match the kernel build and head SHA." >&2; exit 1; } -[[ "$image_s3_prefix" == "qualcomm-linux/${kernel_s3_prefix}/${kernel_build_id}" ]] || { echo "::error::Unexpected image S3 prefix." >&2; exit 1; } IFS=- read -r kernel_run_id kernel_run_attempt <<< "$kernel_build_id" kernel_run="$(gh api "repos/qualcomm-linux/pkg-linux-qcom-canonical/actions/runs/${kernel_run_id}")" @@ -62,16 +56,19 @@ pull_requests="$( distro_run="$(gh api "repos/qualcomm-linux/qcom-distro-images/actions/runs/${distro_run_id}")" jq -e \ --argjson attempt "$distro_run_attempt" \ - --arg distro_ref "$distro_ref" \ '.event == "repository_dispatch" and .path == ".github/workflows/canonical-premerge-validation.yml" and - .run_attempt == $attempt and - .head_sha == $distro_ref' \ + .run_attempt == $attempt' \ <<< "$distro_run" >/dev/null || { echo "::error::Distro workflow run does not match the callback context." >&2 exit 1 } +distro_build_id="${distro_run_id}-${distro_run_attempt}" +distro_ref="$(jq -r '.head_sha' <<< "$distro_run")" +image_s3_prefix="qualcomm-linux/${kernel_s3_prefix}/${kernel_build_id}" +[[ "$distro_ref" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Invalid distro source SHA." >&2; exit 1; } + distro_run_url="$(jq -r '.html_url' <<< "$distro_run")" state=failure From c5e26621b07b4352f2e92f4ee5c5918d4926edb1 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Fri, 21 Aug 2026 10:08:23 -0700 Subject: [PATCH 113/115] ci: expose image prefix in premerge Check Run - Publish only the raw image S3 prefix in Check Run output text for internal tooling compatibility. - Surface the kernel and distro build identities in the trusted callback workflow summary. - Preserve request validation, Check Run lifecycle handling, and S3 completion metadata. Signed-off-by: Bjordis Collaku --- .github/workflows/premerge-distro-validation.yml | 1 + scripts/report-distro-validation-check.sh | 3 +-- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/premerge-distro-validation.yml b/.github/workflows/premerge-distro-validation.yml index f43cdcde827af..3c530610dfc73 100644 --- a/.github/workflows/premerge-distro-validation.yml +++ b/.github/workflows/premerge-distro-validation.yml @@ -247,6 +247,7 @@ jobs: echo "|-------|-------|" echo "| Pull request | \`#${PR_NUMBER}\` |" echo "| Result | \`${STATE}\` |" + echo "| Kernel build ID | \`${KERNEL_BUILD_ID}\` |" echo "| Distro build ID | \`${DISTRO_BUILD_ID}\` |" echo "| Image S3 prefix | \`${IMAGE_S3_PREFIX}\` |" } >> "$GITHUB_STEP_SUMMARY" diff --git a/scripts/report-distro-validation-check.sh b/scripts/report-distro-validation-check.sh index 6a247ea5af705..ba176cfef2127 100755 --- a/scripts/report-distro-validation-check.sh +++ b/scripts/report-distro-validation-check.sh @@ -54,7 +54,7 @@ fi check_title="Canonical premerge distro validation" check_summary="Building Canonical server and desktop images for PR #${PR_NUMBER}." -check_text=$(printf "Kernel build ID: \`%s\`\nImage S3 prefix: \`%s\`" "$KERNEL_BUILD_ID" "$IMAGE_S3_PREFIX") +check_text="$IMAGE_S3_PREFIX" if [[ "$MODE" == "start" ]]; then if [[ "$matching_count" == "1" ]]; then @@ -154,7 +154,6 @@ case "$CONCLUSION" in check_summary="Canonical distro image validation was skipped for PR #${PR_NUMBER}." ;; esac -check_text=$(printf "Kernel build ID: \`%s\`\nDistro build ID: \`%s\`\nImage S3 prefix: \`%s\`" "$KERNEL_BUILD_ID" "$DISTRO_BUILD_ID" "$IMAGE_S3_PREFIX") if [[ "$existing_status" == "completed" ]]; then existing_details_url="$(jq -r '.[0].details_url // ""' <<< "$matching_checks")" From 92c603dd52313c52fa287e61b2a7934f52230e11 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 25 Aug 2026 09:15:12 -0700 Subject: [PATCH 114/115] test(ci): exercise fork premerge distro validation Create a temporary fork-originated pull request targeting `resolute-qcom-devel` to exercise the complete premerge path. - Run the untrusted kernel premerge build from the pull request merge ref. - Delegate Canonical server and desktop image validation from `main`. - Verify the resulting Check Run is returned to the originating pull request. This marker is not intended to merge. Signed-off-by: Bjordis Collaku --- docs/FORK_PREMERGE_SMOKE_TEST.md | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 docs/FORK_PREMERGE_SMOKE_TEST.md diff --git a/docs/FORK_PREMERGE_SMOKE_TEST.md b/docs/FORK_PREMERGE_SMOKE_TEST.md new file mode 100644 index 0000000000000..1e5b3e1a3549e --- /dev/null +++ b/docs/FORK_PREMERGE_SMOKE_TEST.md @@ -0,0 +1,3 @@ +# Fork Premerge Validation Smoke Test + +This temporary marker exercises the fork-based premerge kernel and distro image validation path. It is not intended for merge. From e2fe998e9adfac169c4f4dc82cbe2b7471a90166 Mon Sep 17 00:00:00 2001 From: Bjordis Collaku Date: Tue, 25 Aug 2026 10:34:11 -0700 Subject: [PATCH 115/115] fix(ci): temporarily disable Canonical FIT image generation Canonical commit 544acfcd732a ("UBUNTU: [SAUCE] fit image generation") enables `do_fitimage` for the Qualcomm arm64 flavor. The new packaging rule invokes `mkimage`, but the current Resolute pkg-builder image does not provide that tool, causing all Canonical package builds to fail. - Pass `do_fitimage=false` through the existing `debian/rules` invocation. - Preserve normal kernel, modules, DTB, headers, tools, and dbgsym builds. - Leave qcom-build-utils `dtb.bin` assembly unchanged. This is a temporary workflow-level compatibility measure. The durable resolution is to add `u-boot-tools` to the Resolute builder before re-enabling Canonical FIT generation and publishing its raw `qcom.itb` payload. Signed-off-by: Bjordis Collaku --- .github/workflows/build-kernel.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-kernel.yml b/.github/workflows/build-kernel.yml index 99ae7bacedbb6..07fddad7bd2f0 100644 --- a/.github/workflows/build-kernel.yml +++ b/.github/workflows/build-kernel.yml @@ -391,7 +391,7 @@ jobs: # -dbgsym.ddeb (vmlinux + modules with full debug symbols). DBGSYM_OPT="do_dbgsym_package=false" if [ "${DBGSYM}" = "true" ]; then DBGSYM_OPT="do_dbgsym_package=true"; fi - fakeroot debian/rules ${TARGET} do_skip_checks=true "${DBGSYM_OPT}" + fakeroot debian/rules ${TARGET} do_skip_checks=true "${DBGSYM_OPT}" do_fitimage=false kill "${HEARTBEAT}" 2>/dev/null || true